Embodied intelligence oriented visual-physical self-supervised learning method, device and medium

Abstract

The application discloses a visual-physical self-supervised learning method and device for embodied intelligence, and a medium, relates to the technical field of embodied intelligence, and comprises the following steps: acquiring time-synchronized image frame sequences, depth frame sequences and IMU reading sequence; extracting visual features, geometric features and motion features by using a multi-branch encoder; taking the visual features and the geometric features at the same time as positive sample pairs, and taking the visual features and the geometric features at different times as negative sample pairs, and calculating a cross-modal contrast loss; based on the motion features, a first motion estimation is regressed, a second motion estimation is estimated based on the visual and geometric features, and a motion consistency loss is calculated according to the difference between the two; the geometric features are randomly masked, and the depth value is predicted based on the visual features, and a depth completion loss is calculated; the three losses are weighted and summed to update the encoder parameters, and pre-training parameters are obtained. The application does not need large-scale spatial labeling, enables the model to learn three-dimensional geometry, motion perception and appearance invariance, and improves the perception robustness.

Description

Technical Field

[0001] This invention relates to the field of embodied intelligence technology, and in particular to a visual-physical self-supervised learning method, device and medium for embodied intelligence. Background Technology

[0002] In recent years, the autonomous perception of embodied intelligent agents and mobile machine vision systems in complex physical environments has relied heavily on real-time joint understanding of the scene's 3D structure, their own motion, and the appearance of objects. However, in embodied scenarios such as robot navigation and augmented reality, obtaining accurate pixel-level depth, object pose, and motion trajectory ground truth annotations requires expensive motion capture equipment or cumbersome manual calibration processes. The high annotation cost directly restricts the ability of perception systems to be rapidly deployed in unfamiliar environments.

[0003] To address the aforementioned technical challenges, current self-supervised pre-training methods in machine vision extract supervisory signals from unlabeled color images through proxy tasks such as rotation prediction and instance comparison. While these methods effectively capture 2D texture and semantic patterns, their training signals originate entirely from a single image domain. The pre-training phase never touches upon physical sensing data such as depth and inertia, making it impossible to establish 3D representations of scene occlusion geometry and viewpoint motion. Furthermore, traditional physical model methods based on multi-view geometry and inertial integration, while capable of explicit estimation of structure and motion, heavily rely on manually set prior parameters, making deep integration with data-driven representations difficult. Their generalization ability on large-scale, dynamically changing unlabeled data is extremely limited. To achieve reliable 3D spatial understanding and motion estimation in unknown environments, systems still require significant manual scene-level spatial annotation; otherwise, perceptual robustness drops drastically. Summary of the Invention

[0004] This invention provides a visual-physical self-supervised learning method, device, and medium for embodied intelligence. The technical problem it aims to solve is: how to ensure the robustness of embodied intelligent agents in three-dimensional environment understanding and self-motion perception in dynamic environments without relying on large-scale spatial annotation.

[0005] In a first aspect, the present invention provides a visual-physical self-supervised learning method for embodied intelligence, comprising: Acquire time-synchronized image frame sequences, depth frame sequences, and IMU reading sequences; The image frame sequence is input into the visual branch of the multi-branch encoder to obtain visual features; the depth frame sequence is input into the geometric branch of the multi-branch encoder to obtain geometric features; and the IMU reading sequence is input into the motion branch of the multi-branch encoder to obtain motion features. Based on the visual features and the geometric features, the cross-modal contrast loss is calculated by using visual features and geometric features at the same time as positive sample pairs and visual features and geometric features at different times as negative sample pairs. A first motion estimate is obtained based on the motion feature regression, a second motion estimate is obtained based on the visual features and the geometric features, and a motion consistency loss is calculated based on the difference between the first motion estimate and the second motion estimate. Random region occlusion is performed on the geometric features, and the depth value of the occluded region is predicted based on the visual features to obtain the predicted depth value. The depth completion loss is calculated based on the difference between the predicted depth value and the true depth value. The cross-modal contrast loss, the motion consistency loss, and the depth completion loss are weighted and summed to obtain the total loss. The parameters of the multi-branch encoder are then updated based on the total loss to obtain the pre-trained parameters.

[0006] Optionally, the step of calculating cross-modal contrast loss based on the visual features and the geometric features, using visual features and geometric features at the same time as positive sample pairs and visual features and geometric features at different times as negative sample pairs, includes: The visual features are spatially segmented to obtain multiple image block features, and the geometric features at the same time are spatially segmented accordingly to obtain multiple depth point cloud block features. Image patch features and depth point cloud patch features belonging to the same time and corresponding spatial location are determined as the positive sample pair, and image patch features and depth point cloud patch features belonging to different time or spatial location and not corresponding are determined as the negative sample pair; The similarity between each positive sample pair and each negative sample pair is calculated, and the difference between the similarity between positive sample pairs and the similarity between negative sample pairs is maximized by the cross-entropy loss function to obtain the cross-modal contrast loss.

[0007] Optionally, the step of obtaining a first motion estimate based on the motion feature regression, obtaining a second motion estimate based on the visual features and the geometric features estimation, and calculating a motion consistency loss based on the difference between the first motion estimate and the second motion estimate includes: The motion features are input into a temporal convolutional network to regress the relative pose change between adjacent frames. The relative pose change is then integrated to obtain the first motion estimate. Simultaneously, the covariance matrix of the first motion estimate is output as the first uncertainty measure. Dense optical flow is calculated from the visual features and the geometric features, and camera motion is estimated based on the dense optical flow using visual odometry to obtain the second motion estimate. At the same time, a second uncertainty metric is generated based on the confidence level of the dense optical flow. Based on the first uncertainty measure and the second uncertainty measure, the weighted distance between the first motion estimate and the second motion estimate is calculated to obtain the motion consistency loss, wherein the component with higher uncertainty has a lower weight in the weighted distance.

[0008] Optionally, the step of randomly occluding the geometric features and predicting the depth value of the occluded region based on the visual features to obtain a predicted depth value, and calculating the depth completion loss based on the difference between the predicted depth value and the true depth value, includes: A binary mask of random shape is generated to mask local regions of the geometric features, resulting in masked geometric features. The occluded geometric features and the visual features are concatenated in the channel dimension and input into a depth completion sub-network containing a cross-modal attention module. The cross-modal attention module uses the visual features as the query and the occluded geometric features as the key and value. It fuses the visual features and the occluded geometric features through an attention mechanism to obtain enhanced features. The enhanced features are then decoded, and the predicted depth value of the occluded region is output. Calculate the pixel-wise L1 loss and multi-scale structural similarity loss between the predicted depth value and the true depth value, and sum the L1 loss and the multi-scale structural similarity loss to obtain the depth completion loss.

[0009] Optionally, the visual branch of the multi-branch encoder is a residual convolutional neural network, the geometric branch is a point cloud feature extraction network, and the motion branch is a bidirectional long short-term memory network; the multi-branch encoder further includes a cross-branch feature interaction layer, which is used for: Multi-scale visual feature maps are extracted from the first intermediate layer of the visual branch, and geometric feature volumes of the corresponding scale are extracted from the second intermediate layer of the geometric branch. The correlation matrix between the multi-scale visual feature map and the corresponding scale geometric feature body is calculated by dot product attention, and the multi-scale visual feature map is weighted by the correlation matrix to obtain the enhanced visual features. The geometric feature body at the corresponding scale is weighted by the transpose of the correlation matrix to obtain the enhanced geometric features. The enhanced visual features are input into the convolutional layer in the visual branch after the first intermediate layer, and the enhanced geometric features are input into the point cloud feature extraction layer in the geometric branch after the second intermediate layer.

[0010] Optionally, the weighted summation of the cross-modal contrast loss, the motion consistency loss, and the depth completion loss to obtain the total loss includes: Learnable uncertainty weight parameters are assigned to the cross-modal contrast loss, the motion consistency loss, and the depth completion loss, respectively, and the initial values ​​of the learnable uncertainty weight parameters are preset values. Construct a maximum likelihood joint loss function based on homoscedastic uncertainty, wherein the maximum likelihood joint loss function includes the normalization term of the learnable uncertainty weight parameter for the corresponding loss term; The cross-modal contrast loss, motion consistency loss, and depth completion loss are substituted into the maximum likelihood joint loss function to calculate the total loss, and the learnable uncertainty weight parameters are automatically updated through gradient descent during backpropagation.

[0011] Optionally, after obtaining the pre-trained parameters, the method further includes: The visual branch parameters in the pre-trained parameters are loaded into the downstream task network to obtain the initialized downstream task network; The image training data is input into the initialized downstream task network to obtain the visual features of the image training data, and the IMU training data is input into the motion branch of the pre-training parameters to obtain the motion features of the IMU training data. A decoupling branch is added to the initialized downstream task network. The visual features of the image training data are input into the decoupling branch to obtain scene structure features and appearance texture features. Mutual information minimization constraints are applied to the scene structure features and appearance texture features to reduce the dependency between the scene structure features and appearance texture features. The network parameters corresponding to the scene structure features output by the decoupled branch are fixed, and the network parameters corresponding to the appearance texture features are fine-tuned in a supervised manner using a labeled downstream task dataset. An auxiliary motion consistency regularization term based on the motion features of the IMU training data is added to the fine-tuning loss to obtain the fine-tuned downstream task network parameters.

[0012] Optionally, acquiring the time-synchronized image frame sequence, depth frame sequence, and IMU reading sequence includes: The raw data streams from the camera, depth sensor, and IMU on the embodied device are acquired, and the raw data streams from the IMU are pre-integrated. Multiple IMU measurements between adjacent image frames are integrated into relative rotation, relative velocity, and relative translation changes to obtain motion constraints between adjacent frames. Using the motion constraints as regularization terms, joint nonlinear optimization alignment is performed on the timestamps of image frames, depth frames, and IMU readings in the original data stream to obtain time-aligned image frame sequences, depth frame sequences, and IMU reading sequences. The time-aligned IMU reading sequence is subjected to deviation correction, and the time-aligned image frame sequence and depth frame sequence are subjected to camera extrinsic parameter calibration. The spatial coordinates of the depth frame sequence are transformed to a coordinate system consistent with the image frame sequence to obtain the time-synchronized image frame sequence, depth frame sequence and IMU reading sequence.

[0013] Secondly, the present invention also provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0014] Thirdly, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the above-described method.

[0015] This invention provides a visual-physical self-supervised learning method, device, and medium for embodied intelligence. By jointly utilizing time-synchronized image frame sequences, depth frame sequences, and IMU reading sequences, three complementary self-supervised tasks are constructed in parallel within a unified multi-branch encoder framework. This eliminates the need for any manual annotation during pre-training, fundamentally removing the embodied agent's dependence on large-scale spatial annotations in unfamiliar scenes. Furthermore, cross-modal contrastive learning uses temporal synchronization as a natural supervisory signal, forcing visual and geometric features at the same moment to align in the embedding space, enabling the model to establish a cross-modal correspondence between two-dimensional appearance information and three-dimensional geometric structure. Additionally, motion consistency learning utilizes regression data from motion features... The first motion estimate is constrained to align with the second motion estimate, which is based on visual and geometric features. This forces the visually perceived motion to align with the physical reality, enabling the model to learn its own motion perception capabilities from visual features. Furthermore, deep completion learning uses random region occlusion of geometric features and predicts the depth of the occluded region based on visual features. This forces the model to learn to infer 3D geometric structures using 2D texture cues, improving the geometric completeness of the representation for occluded and incomplete sensor data. Additionally, the loss functions of the three self-supervised tasks are weighted and summed to uniformly drive the multi-branch encoder parameter updates. The resulting pre-trained parameters internalize a deep joint representation of the scene's 3D geometry, its own motion state, and appearance invariance. When these pre-trained parameters are transferred to downstream tasks, only a small amount of labeled data is needed for fine-tuning, enabling the embodied agent to gain robust 3D environmental understanding and its own motion perception capabilities in dynamic environments. This effectively solves the problems of existing solutions where pure visual self-supervision lacks physical motion priors, physical model methods have insufficient generalization, and are highly dependent on spatial annotations. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram illustrating an application scenario of a visual-physical self-supervised learning method for embodied intelligence provided in an embodiment of the present invention. Figure 2 This is a schematic block diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0020] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0021] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0022] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0023] Please see Figure 1 This invention provides a visual-physical self-supervised learning method for embodied intelligence. Embodied intelligence refers to the research field where intelligent agents interact with the physical world through their bodies and generate intelligent behaviors. One of its core challenges is how to enable intelligent agents to learn a deep understanding of the structure of the 3D environment, their own motion state, and the appearance of objects from data from multiple sensors they carry, without human annotation. The method in this embodiment constructs three complementary self-supervised learning tasks, utilizing the natural correlation between time-synchronized image, depth, and inertial measurement data to achieve joint learning of visual and physical representations, providing a general pre-trained model for spatially cognitive-intensive applications such as robot navigation, augmented reality, and autonomous driving. Specifically, the method includes the following steps: S1, acquire time-synchronized image frame sequence, depth frame sequence and IMU reading sequence.

[0024] In practice, time-synchronized image frame sequences, depth frame sequences, and IMU reading sequences are acquired. The image frame sequences are continuously acquired at a certain frame rate by a camera mounted on the device. Each frame contains a two-dimensional pixel array from the device's current viewpoint, with pixel values ​​reflecting the reflectivity of object surfaces in the scene. Furthermore, the depth frame sequences are synchronously acquired by a depth sensor, which can be a depth camera based on structured light coded patterns or a depth camera based on the time-of-flight measurement principle. Its output is an array of distance values ​​corresponding to the image frame space, where each distance value represents the perpendicular distance from the point on the object's surface corresponding to that pixel to the line connecting the camera's optical center. Further, the IMU reading sequences are synchronously acquired by an inertial measurement unit (IMU), a microelectromechanical system sensor integrating a three-axis accelerometer and a three-axis gyroscope. The three-axis accelerometer measures linear acceleration in three orthogonal directions within the device's own coordinate system, and the three-axis gyroscope measures angular velocity around three orthogonal axes. The IMU's sampling frequency is typically much higher than the camera's frame rate; therefore, there are tens to hundreds of IMU readings between adjacent image frames. Furthermore, the three data streams mentioned above can be precisely aligned at the microsecond level by hardware-triggered synchronization cables, or by interpolation alignment at the software layer using the timestamps of each sensor's independent clock, ensuring that image frames, depth frames, and IMU readings under the same time index correspond to the device's state at exactly the same physical moment.

[0025] In some preferred embodiments, acquiring the time-synchronized image frame sequence, depth frame sequence, and IMU reading sequence includes: acquiring raw data streams from the camera, depth sensor, and IMU on the device; pre-integrating the raw IMU data stream; integrating multiple IMU measurements between adjacent image frames into relative rotation, relative velocity, and relative translation changes to obtain motion constraints between adjacent frames; using the motion constraints as regularization terms to perform joint nonlinear optimization alignment on the timestamps of the image frames, depth frames, and IMU readings in the raw data stream to obtain the time-aligned image frame sequence, depth frame sequence, and IMU reading sequence; performing deviation correction on the time-aligned IMU reading sequence; calibrating the camera extrinsic parameters on the time-aligned image frame sequence and depth frame sequence; transforming the spatial coordinates of the depth frame sequence to a coordinate system consistent with the image frame sequence to obtain the time-synchronized image frame sequence, depth frame sequence, and IMU reading sequence.

[0026] In practice, inter-frame motion constraints are generated through IMU pre-integration, and these motion constraints are used as residual constraints to participate in timestamp joint nonlinear optimization alignment. Bias correction and extrinsic parameter calibration are also performed to provide high-precision spatiotemporally aligned multimodal data for self-supervised pre-training. The specific implementation method is as follows: The system acquires raw data streams from the camera, depth sensor, and IMU on the device, and pre-integrates the raw IMU data stream. Multiple IMU measurements between adjacent image frames are integrated into relative rotation, relative velocity, and relative translation changes to obtain motion constraints between adjacent frames. The camera outputs image frames at a certain frame rate, the depth sensor synchronously outputs depth frames, and the IMU continuously outputs angular velocity and acceleration readings at a frequency much higher than the camera's frame rate. The pre-integration numerically integrates multiple IMU measurements between adjacent image frames in the IMU's own coordinate system to obtain the relative rotation, relative velocity, and relative translation changes from the previous frame to the current frame. These relative rotation, relative velocity, and relative translation changes are used as the core components of the motion constraints. Simultaneously, the covariance matrix of the pre-integration result is calculated to quantify the uncertainty introduced by noise propagation during the integration process. This relative motion constraint depends only on the inter-frame IMU measurements themselves and is independent of the global pose of the frame. Therefore, it does not require re-integration due to pose adjustments in subsequent optimization, offering computational convenience.

[0027] Furthermore, using motion constraints as a residual constraint term, joint nonlinear optimization alignment is performed on the timestamps of image frames, depth frames, and IMU readings in the original data stream to obtain time-aligned image frame sequences, depth frame sequences, and IMU reading sequences. Since each sensor's hardware clock is independent, with fixed time offsets and possible clock drift, simple timestamp interpolation alignment can introduce significant errors in high-dynamic scenes. Joint nonlinear optimization uses the time offsets of each sensor as variables to be optimized, constructing a joint residual function. Further, the residual function includes: visual reprojection residuals between image frames, measuring the pixel reprojection error of feature matching points; geometric consistency residuals between depth frames, measuring the difference in depth values ​​after spatial transformation; and IMU pre-integration residuals, measuring the consistency between the motion estimate obtained from IMU pre-integration and the corresponding inter-frame pose changes in the optimization variables. Further, the above residuals are weighted and summed squared, and a nonlinear least squares optimizer is used to iteratively solve for the optimal time offset, maximizing the consistency of timestamps across all sensors. The IMU pre-integration residual serves as a constraint term, limiting timestamp adjustments from disrupting the physical motion consistency of the IMU data.

[0028] Furthermore, deviation correction is performed on the time-aligned IMU reading sequence, and camera extrinsic parameter calibration is performed on the time-aligned image frame sequence and depth frame sequence. The spatial coordinates of the depth frame sequence are transformed to a coordinate system consistent with the image frame sequence, resulting in a time-synchronized image frame sequence, depth frame sequence, and IMU reading sequence. Further, the IMU's accelerometer and gyroscope exhibit zero bias during manufacturing, meaning they output non-zero values ​​in a static state. Deviation correction is achieved by acquiring IMU data at rest, calculating the average value of each axis output as a zero bias estimate, and subtracting it from the original readings in subsequent processing. Further, camera extrinsic parameter calibration is the process of determining the rigid transformation relationship between the depth sensor coordinate system and the camera coordinate system. By simultaneously acquiring observation data of the calibration board under the camera and depth sensor, the two-dimensional coordinates of corner points and the corresponding three-dimensional point positions are extracted. The rotation matrix and translation vector are jointly solved using the perspective n-point algorithm and the iterative nearest point algorithm to obtain the extrinsic parameter matrix. Further, the extrinsic parameter matrix is ​​used to perform a rigid body transformation on the three-dimensional coordinates of each pixel in the depth frame sequence, transforming them from the depth sensor coordinate system to the camera coordinate system, achieving spatial alignment.

[0029] Furthermore, after the above preprocessing steps, the final output image frame sequence, depth frame sequence, and IMU reading sequence achieve sub-frame level synchronization accuracy in the time dimension and pixel-level alignment in the spatial dimension, providing a reliable spatiotemporal correspondence for cross-modal comparison of positive and negative samples, comparison of visual and IMU motion consistency, and ground value supervision of depth completion in self-supervised learning.

[0030] This embodiment addresses the time synchronization error problem caused by independent clocks and trigger delays in multi-sensor systems through IMU pre-integration constraint-assisted timestamp joint nonlinear optimization. The optimization process fully utilizes the precise inter-frame motion information contained in the high-frequency IMU data as a constraint term, unifying visual, depth, and inertial measurements within a single optimization framework for time alignment. This achieves an order-of-magnitude improvement in alignment accuracy compared to traditional timestamp interpolation methods. Furthermore, bias correction eliminates the cumulative error impact of IMU zero bias on long-sequence motion estimation. Precise extrinsic parameter calibration ensures accurate spatial registration of depth and image data, avoiding cross-modal correspondence errors caused by spatial misalignment. Moreover, high-quality data preprocessing is fundamental to the effectiveness of self-supervised pre-training. Only when time synchronization and spatial alignment are highly accurate can the goals of positive / negative sample pair partitioning, motion consistency supervision, and depth completion be reliable and beneficial, driving the model to learn truly transferable joint representations.

[0031] S2, input the image frame sequence into the visual branch of the multi-branch encoder to obtain visual features, input the depth frame sequence into the geometric branch of the multi-branch encoder to obtain geometric features, and input the IMU reading sequence into the motion branch of the multi-branch encoder to obtain motion features.

[0032] In practice, the image frame sequence is input into the visual branch of the multi-branch encoder to obtain visual features. The multi-branch encoder is a neural network architecture containing multiple input channels and a shared representation layer. The visual branch, as its first input channel, is responsible for extracting hierarchical visual representations from the two-dimensional image frames. Furthermore, the visual branch can employ a deep residual network as its backbone structure. The core idea of ​​a deep residual network is to introduce skip connections in the stacking of standard convolutional layers, directly superimposing shallow features onto deep features, thereby alleviating the gradient vanishing problem and supporting deeper network depths. Further, starting from the original pixels, the network undergoes several stages of convolution, batch normalization, and non-linear activation processing, gradually abstracting multi-level features such as edges, corners, textures, local shapes, and even semantic concepts. The final output is a visual feature map that maintains a certain spatial resolution, where each spatial location in the feature map corresponds to a local receptive field of the original image.

[0033] Furthermore, the depth frame sequence is input into the geometric branch of the multi-branch encoder to obtain geometric features. The geometric branch is the second input channel of the multi-branch encoder, specifically processing 3D geometric data. The depth frame first converts the distance value of each pixel into point coordinates in 3D space based on the camera imaging model and known camera intrinsics, forming a point cloud frame. The camera intrinsics, including focal length and principal point position, are necessary parameters for backprojecting from 2D pixel coordinates to 3D spatial coordinates. The geometric branch employs a neural network structure based on point cloud processing, iteratively selecting representative center points in the point cloud through farthest point sampling. Then, using each center point as an anchor point, it groups and aggregates features from its neighborhood points, gradually constructing a hierarchical geometric representation from local to global. Finally, the output geometric features correspond spatially to the visual features to facilitate subsequent cross-modal information interaction and comparison.

[0034] Furthermore, the IMU reading sequence is input into the motion branch of the multi-branch encoder to obtain motion features. The motion branch is the third input channel of the multi-branch encoder, specifically processing time-series sensor data. Since the IMU readings are discrete-time six-dimensional vector sequences, the motion branch employs a recurrent neural network structure to model their temporal dependencies. Furthermore, a Long Short-Term Memory (LSTM) network is a recurrent neural network unit capable of selectively remembering and forgetting historical information. Internally, it contains three gating mechanisms: an input gate, a forget gate, and an output gate, which control the writing of new information, the clearing of old information, and the output of the current state, respectively. Furthermore, the bidirectional design enables the network to simultaneously utilize past and future sensor readings to infer the current motion state, thereby improving the accuracy and contextual richness of motion feature extraction. The output of the motion branch is a sequence of motion feature vectors aligned with the image frame timestamps.

[0035] In some preferred embodiments, the visual branch of the multi-branch encoder is a residual convolutional neural network, the geometric branch is a point cloud feature extraction network, and the motion branch is a bidirectional long short-term memory network. The multi-branch encoder further includes a cross-branch feature interaction layer, which is used to: extract multi-scale visual feature maps from the first intermediate layer of the visual branch and extract geometric feature volumes of the corresponding scale from the second intermediate layer of the geometric branch; calculate the correlation matrix between the multi-scale visual feature maps and the corresponding scale geometric feature volumes through dot product attention, and use the correlation matrix to weight the multi-scale visual feature maps to obtain enhanced visual features; use the transpose of the correlation matrix to weight the corresponding scale geometric feature volumes to obtain enhanced geometric features; input the enhanced visual features into the convolutional layer located after the first intermediate layer in the visual branch, and input the enhanced geometric features into the point cloud feature extraction layer located after the second intermediate layer in the geometric branch.

[0036] In practice, a cross-branch feature interaction layer is introduced between the visual and geometric branches to break the limitation of independent encoding of different modalities in traditional multi-stream networks. This enables bidirectional information fusion of visual and geometric features during the encoding stage, enhancing the expressive power and cross-modal complementarity of features in each branch. The specific implementation method is as follows: The multi-branch encoder employs a residual convolutional neural network for its visual branch, a point cloud feature extraction network for its geometric branch, and a bidirectional long short-term memory network for its motion branch. The visual branch achieves deep feature extraction by stacking multiple residual blocks, each containing two convolutional layers and a skip connection, effectively mitigating gradient degradation. The geometric branch aggregates global geometric information by sampling, grouping, and extracting local features from the point cloud at different scales using a set of abstract modules. The motion branch captures the temporal dependencies of inertial data through a bidirectional recurrent structure, outputting the motion latent state at each time step.

[0037] Furthermore, the multi-branch encoder also includes a cross-branch feature interaction layer, which is used to exchange information between the first intermediate layer of the visual branch and the second intermediate layer of the geometric branch. The first intermediate layer refers to the output feature map of a certain intermediate stage in the visual branch, such as the output of the third residual block of the residual network. This layer's features retain a certain spatial resolution while possessing a high level of semantic abstraction. The second intermediate layer refers to the output feature volume of a certain intermediate stage in the geometric branch, such as the geometric features obtained after passing through a set abstraction module, whose spatial scale is adjusted to match the visual features. Figure 1 To further elaborate, the specific operations are as follows: Multi-scale visual feature maps are extracted from the first intermediate layer of the visual branch, and geometric feature volumes of the corresponding scale are extracted from the second intermediate layer of the geometric branch. The multi-scale visual feature maps are constructed by extracting features from feature layers with different downsampling factors, then uniformly upsampling them to the same resolution and concatenating them along the channel dimension, containing rich information ranging from fine-grained texture to coarse-grained semantics. The geometric feature volumes of the corresponding scale are mapped to a spatial grid of the same size as the visual feature maps through point cloud rasterization and 3D convolution. The features of each grid are the aggregation of features from all points within that grid.

[0038] Furthermore, a correlation matrix between multi-scale visual feature maps and corresponding-scale geometric feature volumes is calculated using dot-product attention. The spatial dimensions of the visual feature maps are flattened to obtain a sequence of visual query vectors; the spatial dimensions of the geometric feature volumes are flattened to obtain sequences of geometric key vectors and value vectors. The dot product of each pair of visual queries and geometric keys is calculated to obtain a correlation matrix. The element values ​​in the matrix represent the feature association strength between a visual location and a geometric location; a larger value indicates that the two are more likely to originate from the same physical region. Further, the correlation matrix is ​​used to weight the multi-scale visual feature maps, injecting geometric structure information into the visual features to obtain enhanced visual features. The transpose of the correlation matrix is ​​used to weight the corresponding-scale geometric feature volumes, injecting visual texture information into the geometric features to obtain enhanced geometric features. This bidirectional enhancement enables the visual branch to perceive the three-dimensional spatial layout, and the geometric branch to perceive the surface properties of objects.

[0039] Furthermore, the enhanced visual features are input into the convolutional layer following the first intermediate layer in the visual branch, and the enhanced geometric features are input into the point cloud feature extraction layer following the second intermediate layer in the geometric branch. The enhanced features serve as new intermediate features for their respective branches and continue to participate in subsequent encoding. Since the interaction layer is positioned in the middle of the encoding stage, the deep network can further extract higher-level semantic information based on the fused features, achieving deep cross-modal fusion.

[0040] This embodiment overcomes the limitation of traditional multimodal schemes that only perform modality alignment in the final feature layer or loss layer by introducing cross-branch feature interaction midway through the encoding process. Furthermore, the visual branch can perceive the 3D geometry of the scene after interaction, which is crucial for understanding object boundaries, spatial occlusion relationships, and geometric layout. Furthermore, the geometric branch incorporates texture and color information after interaction, helping to better distinguish objects with similar geometric shapes but different appearances. Moreover, this deep bidirectional fusion enables the joint representation output by the multi-branch encoder to perform better in geometric understanding, appearance perception, and motion estimation, providing richer and more discriminative pre-trained features for downstream tasks.

[0041] S3. Based on the visual features and the geometric features, calculate the cross-modal contrast loss by using visual features and geometric features at the same time as positive sample pairs and visual features and geometric features at different times as negative sample pairs.

[0042] In specific implementation, based on visual and geometric features, visual and geometric features at the same time are used as positive sample pairs, and visual and geometric features at different times are used as negative sample pairs, and cross-modal contrastive loss is calculated. Furthermore, the goal of contrastive learning is to construct a feature space in which semantically similar (i.e., corresponding to the same physical scene) cross-modal samples are close to each other, while semantically unrelated samples are far apart. Further, in this embodiment, the construction of positive sample pairs is based on temporal synchronization and spatial alignment; that is, image frames and depth frames at the same time and from the same viewpoint are descriptions of the completely identical physical world from different sensing modalities, therefore their features should be highly consistent. Further, negative sample pairs come from feature combinations at different times. Due to device motion or dynamic scene changes, images and depth at different times correspond to different physical content, and their features should be different. In specific calculations, visual and geometric features are mapped to the same embedding space through a projection head, and then the similarity between each positive sample pair and the similarity between all negative sample pairs are calculated. A noise contrastive estimation loss function is used to shift the positive and negative values. The core of noise contrastive estimation loss is to train a binary classifier to distinguish between positive and negative sample pairs. Maximizing classification accuracy is equivalent to minimizing cross-modal contrastive loss.

[0043] In some preferred embodiments, the step of calculating cross-modal contrast loss based on the visual features and the geometric features, using visual features and geometric features at the same time as positive sample pairs and visual features and geometric features at different times as negative sample pairs, includes: spatially dividing the visual features to obtain multiple image patch features, and correspondingly spatially dividing the geometric features at the same time to obtain multiple depth point cloud patch features; determining the image patch features and depth point cloud patch features belonging to the same time and corresponding in spatial location as the positive sample pairs, and determining the image patch features and depth point cloud patch features belonging to different times or not corresponding in spatial location as the negative sample pairs; calculating the similarity of each positive sample pair and each negative sample pair, and maximizing the difference between the similarity of the positive sample pairs and the similarity of the negative sample pairs through the cross-entropy loss function to obtain the cross-modal contrast loss.

[0044] In practice, spatial segmentation expands the comparison at the global feature level to the comparison at the local region level, enabling the model to not only focus on the modal consistency of the entire frame scene, but also delve into the alignment of local details, thereby learning more refined cross-modal representations. The specific implementation method is as follows: Visual features are spatially partitioned to obtain multiple image patch features. The spatial dimensions of the visual features are height and width, determined by the downsampling factor of the residual network. The entire feature map is uniformly divided into several non-overlapping rectangular regions according to a preset grid size. Average pooling is performed on all pixel feature vectors within each region to obtain a fixed-length feature vector as the image patch feature for that region. Furthermore, the average pooling operation aggregates information from all locations within the region and possesses a certain degree of translation invariance, making the patch features insensitive to slight spatial shifts.

[0045] Furthermore, the geometric features at the same time are spatially segmented to obtain multiple depth point cloud block features. To ensure the spatial comparability of the two modalities, the spatial representation of the geometric features needs to be aligned with the visual features. Further, when the geometric features exist in point cloud form, each 3D point is mapped to the corresponding grid of the visual feature map according to the camera projection model. The geometric features of all points falling into the same grid are averaged and aggregated to generate depth point cloud block features. Further, if the geometric features have already been processed into feature maps of the same size as the visual features, the same grid partitioning method as the visual features is directly adopted.

[0046] Furthermore, image patch features and depth point cloud patch features belonging to the same time and corresponding spatial location are identified as positive sample pairs. Since the two types of data have been spatiotemporally aligned, image patches and depth point cloud patches within the same grid region capture the appearance and geometric information of the same local area in the physical world, thus naturally constituting positive samples. Further, image patch features and depth point cloud patch features belonging to different time periods or not corresponding spatial locations are identified as negative sample pairs. Features from different time periods, even if originating from the same spatial grid, become negative samples due to differences in content caused by device movement or scene changes. Furthermore, features not corresponding in spatial location become negative samples because they describe different parts of the scene. By simultaneously utilizing the differences in the temporal and spatial dimensions, a rich number of negative sample pairs can be constructed, improving the discrimination difficulty and representation quality of the contrastive learning task.

[0047] Furthermore, the similarity between each positive and negative sample pair is calculated, and the difference between the similarities of positive and negative sample pairs is maximized using the cross-entropy loss function to obtain the cross-modal contrastive loss. The similarity is further measured using cosine similarity, which is the product of the inner products of two vectors divided by the product of their magnitudes; the result ranges from -1 to 1, with larger values ​​indicating more consistent directions between the two vectors. The contrastive learning is further modeled as a multi-class classification task. For each image patch feature, its similarity to all geometric patch features is used to construct a similarity vector. After normalization using the softmax function, the matching probability distribution of the image patch feature for each geometric patch feature is obtained, where the probability corresponding to a positive sample should approach 1, and the probability corresponding to a negative sample should approach 0. The cross-entropy loss measures the difference between this predicted distribution and the ideal one-hot distribution; the average of the cross-entropy losses of all image patch features is the cross-modal contrastive loss.

[0048] This embodiment degrades the granularity of contrastive learning from the global image level to the local region level through spatial segmentation. This requires the model to precisely align cross-modal information within each spatial region, rather than simply making a general judgment on the matching relationship of the entire frame. Furthermore, local contrast enables the model to learn richer geometric and appearance correspondences, such as feature differences between the two sides of an object's edge, feature continuity within a planar region, and feature distinguishability of different material surfaces. Moreover, this fine-grained cross-modal correspondence knowledge provides high-quality feature initialization for downstream pixel-level tasks such as semantic segmentation, instance segmentation, and depth estimation, significantly improving fine-tuning performance, especially when labeled data is limited.

[0049] S4. A first motion estimate is obtained based on the motion feature regression, and a second motion estimate is obtained based on the visual features and the geometric features estimation. The motion consistency loss is calculated based on the difference between the first motion estimate and the second motion estimate.

[0050] In practice, a first motion estimate is obtained based on motion feature regression, and a second motion estimate is obtained based on visual and geometric feature estimation. The motion consistency loss is calculated based on the difference between the first and second motion estimates. The basic principle of motion consistency learning is that the motion of the same device can be independently estimated from different signal sources, and these estimation results should be consistent. The first motion estimate uses IMU sensor data as input and belongs to motion inference driven by physical measurement. Motion features pass through a fully connected regression layer, outputting relative pose transformation parameters between adjacent frames, including three-dimensional translation increments and three-dimensional rotation increments. Then, through continuous composite operations on the relative pose in the time dimension, the absolute motion trajectory relative to the starting frame is obtained. Further, the second motion estimate uses visual and geometric features as input and belongs to motion inference driven by perceptual data. First, the correspondence between feature points between adjacent frames is calculated to obtain a dense optical flow field reflecting pixel-level motion. Then, the inter-frame motion of the camera is solved from optical flow and depth using epipolar geometric constraints. It should be noted that epipolar geometry describes the projection relationship between two adjacent image planes. Its core is the epipolar constraint equation, which states that the product of the normalized coordinates of a pair of matching feature points and the essential matrix is ​​zero. The essential matrix can be decomposed into a rotation matrix and a translation vector. Furthermore, after obtaining the first and second motion estimates, the differences are calculated on each component of the six-degree-of-freedom motion parameters, and the summation yields the motion consistency loss. Minimizing this loss aligns the visually perceived motion with the physical reality.

[0051] In some preferred embodiments, obtaining a first motion estimate based on the motion feature regression, obtaining a second motion estimate based on the visual features and the geometric features, and calculating a motion consistency loss based on the difference between the first motion estimate and the second motion estimate includes: inputting the motion features into a temporal convolutional network, regressing the relative pose change between adjacent frames, integrating the relative pose change to obtain the first motion estimate, and outputting the covariance matrix of the first motion estimate as a first uncertainty measure; calculating dense optical flow from the visual features and the geometric features, and estimating camera motion based on the dense optical flow using visual odometry to obtain the second motion estimate, and generating a second uncertainty measure based on the confidence level of the dense optical flow; and calculating the weighted distance between the first motion estimate and the second motion estimate based on the first uncertainty measure and the second uncertainty measure to obtain the motion consistency loss, wherein the component with higher uncertainty has a lower weight in the weighted distance.

[0052] In practice, by simultaneously outputting the uncertainty measures of the first and second motion estimates, and adaptively adjusting the weights of each motion component in the loss calculation based on the uncertainty, motion consistency learning becomes more robust to sensor noise and visual estimation errors. The specific implementation method is as follows: Motion features are input into a temporal convolutional network to regress the relative pose changes between adjacent frames. The relative pose changes are then integrated to obtain the first motion estimate, and the covariance matrix of this first motion estimate is output as the first uncertainty measure. The temporal convolutional network utilizes dilated convolutions to expand the receptive field in the temporal dimension, efficiently capturing long-range temporal dependencies while maintaining computational parallelism. The regressed relative pose changes include translational and rotational increments. Translational increments are represented by a three-dimensional vector, and rotational increments are represented by unit quaternions or rotation vectors. Furthermore, the pose increments of adjacent frames in the time series are accumulated frame by frame. That is, starting from the initial pose of the reference frame, the current relative pose transformation is superimposed onto the current absolute pose estimate through matrix multiplication after each frame to obtain the device motion trajectory. Further, the covariance matrix is ​​a symmetric positive definite matrix corresponding to the dimension of the motion parameters. The diagonal elements represent the variance of each degree of freedom estimate, and the off-diagonal elements represent the correlation between degrees of freedom. Furthermore, this covariance matrix is ​​output by an additional prediction head in the network, and its physical meaning is the model's self-assessment of the reliability of its own regression results.

[0053] Dense optical flow is calculated from visual and geometric features, and camera motion is estimated based on this dense optical flow using visual odometry to obtain a second motion estimate. Simultaneously, a second uncertainty metric is generated based on the confidence level of the dense optical flow. Further, the dense optical flow is obtained through an optical flow estimation network or a traditional variational method. The motion vector of each pixel in the optical flow field reflects the projected displacement of the scene point on the image plane. Further, visual odometry filters high-confidence pixel motions from the optical flow field and solves the inter-frame pose of the camera using epipolar geometry and triangulation principles. The optical flow confidence level integrates factors such as pixel gradient strength, forward / backward consistency error, and local motion consistency; pixels with strong gradients and motion consistent with their neighborhood have high confidence. Based on the confidence distribution of all feature points involved in the pose solution, the overall covariance of the second motion estimate can be estimated.

[0054] Furthermore, based on the first and second uncertainty measures, a weighted distance between the first and second motion estimates is calculated to obtain the motion consistency loss, where components with higher uncertainty have lower weights in the weighted distance. Specifically, for each component of the motion estimate, the square of the difference between the first and second motion estimates is calculated, and then divided by the sum of the variances of the two uncertainty measures for that component, i.e., the form of Mahalanobis distance. Mahalanobis distance is a method that uses the covariance matrix to scale the distance measure, eliminating the influence of differences in the dimensions and correlations of the components. Further, the weighted differences of each component are summed to obtain the motion consistency loss. Since variance is in the denominator, the larger the variance (the higher the uncertainty), the smaller the contribution of this term to the total loss, thus achieving automatic weight reduction of unreliable signals.

[0055] This embodiment addresses the issues of temperature drift and noise in IMU measurements, and the accuracy degradation of visual odometry in low-texture or fast-motion scenarios, leading to erroneous supervision in practical applications, by introducing an uncertainty-aware adaptive weighting mechanism. When the device is stationary for extended periods, IMU drift causes the covariance of the first motion estimate to gradually increase. Since visual odometry maintains high accuracy under these conditions, adaptive weighting automatically reduces reliance on IMU estimation, preventing erroneous motion signals from contaminating the model. Conversely, in situations with blurred images or insufficient lighting, the uncertainty of visual estimation increases, and the model automatically trusts the high-frequency motion information provided by the IMU. This dynamic balance enables motion consistency learning to provide stable and reliable supervision signals in various complex scenarios, resulting in a significantly more robust motion-aware representation in variable environments compared to fixed-weight schemes.

[0056] S5, randomly occlude the geometric features, and predict the depth value of the occluded area based on the visual features to obtain the predicted depth value. Calculate the depth completion loss based on the difference between the predicted depth value and the true depth value.

[0057] In practice, geometric features are randomly masked, and the depth value of the masked area is predicted based on visual features. The predicted depth value is then calculated based on the difference between the predicted and true depth values. In practical applications, depth sensors often experience data loss due to material absorption, specular reflection, occlusion, or exceeding the effective measurement range. The depth completion task aims to infer the 3D structure of the missing area from color image cues. This embodiment simulates real-world missing features by randomly generating masks during training. Then, visual features are used as guiding information, and the depth completion sub-network predicts the depth value of the masked area. Furthermore, the depth completion sub-network typically employs an encoder-decoder architecture. The encoder progressively compresses and fuses the features, while the decoder progressively restores the spatial resolution. Skip connections are established between corresponding layers of the encoder and decoder to preserve fine-grained spatial details. The difference between the predicted and true depth values ​​is measured using pixel-level regression loss, while also constraining the structural fidelity of the depth map, ensuring a natural connection between the completed and preserved areas in terms of object boundaries and planar continuity.

[0058] In some preferred embodiments, the step of randomly occluding the geometric features and predicting the depth value of the occluded region based on the visual features to obtain a predicted depth value, and calculating the depth completion loss based on the difference between the predicted depth value and the true depth value, includes: generating a binary mask of random shape to occlude a local region of the geometric features to obtain occluded geometric features; concatenating the occluded geometric features with the visual features in the channel dimension and inputting them into a depth completion sub-network containing a cross-modal attention module, wherein the cross-modal attention module uses the visual features as the query and the occluded geometric features as the key and value, fuses the visual features and the occluded geometric features through an attention mechanism to obtain an enhanced feature, and decodes the enhanced feature to output the predicted depth value of the occluded region; calculating the pixel-wise L1 loss and multi-scale structural similarity loss between the predicted depth value and the true depth value, and summing the L1 loss and the multi-scale structural similarity loss to obtain the depth completion loss.

[0059] In practice, high-quality depth completion of 3D geometric structure inference from visual texture is achieved through random shape mask generation, cross-modal attention feature fusion, and multi-scale loss combination. The specific implementation method is as follows: A binary mask of random shape is generated to occlude local regions of geometric features, resulting in the occluded geometric features. The binary mask is a tensor with values ​​only 0 and 1, where 1 represents the preserved region and 0 represents the occluded region. Mask generation strategies can include random rectangle erasure, where a rectangle is randomly selected in the feature space, all its internal elements are set to 0, and the rectangle's center coordinates, aspect ratio, and area ratio are randomly sampled from a uniform distribution. Alternatively, random irregular shapes can be used, generating masks that more closely resemble the actual occlusion patterns through random walks or polygon cutting. Different masks are randomly generated in each iteration to ensure that the training covers occlusion scenarios of various positions, sizes, and shapes.

[0060] Furthermore, the occluded geometric features and visual features are concatenated along the channel dimension and input into a deep completion subnetwork containing a cross-modal attention module. Channel-dimensional concatenation refers to chaining two feature tensors along the channel axis to form a joint feature with double the number of channels. The core of the cross-modal attention module is a scaled dot product attention mechanism, using the linearly transformed visual features as the query vector and the linearly transformed occluded geometric features as the key and value vectors. The dot product of the query and the key is calculated, and the result is scaled by the square root of the key vector dimension to prevent gradient vanishing. Then, softmax normalization is applied to obtain the attention weight matrix, which represents the degree of attention each visual feature location gives to each geometric feature location. Multiplying the weight matrix by the value vector yields the geometric features enhanced with visual context. The advantage of the attention mechanism is that it can dynamically calculate the internal relationships of the input, is not limited by spatial distance, and can transmit texture information to the occluded region over long distances. The enhanced features are progressively reconstructed in terms of spatial resolution by the encoder-decoder, and shallow detail information is fused in skip connections, ultimately outputting the predicted depth value of the occluded region.

[0061] Furthermore, pixel-wise L1 loss and multi-scale structural similarity loss are calculated between the predicted and true depth values. The sum of these two losses yields the depth completion loss. The L1 loss calculates the absolute difference between the predicted and true values ​​for each pixel location, then averages the results over all pixels. It is insensitive to small, large errors and exhibits strong robustness. The structural similarity loss measures the perceptual similarity between two depth maps across three dimensions: brightness, contrast, and structure. Brightness similarity compares the average depth of the two images; contrast similarity compares the standard deviation of depth values; and structural similarity compares the correlation coefficient between the normalized depth maps. The multi-scale version calculates and weights the structural similarity across multiple downsampling levels, simultaneously addressing both local details and global contours. The combination of these two losses ensures both numerical accuracy and structural plausibility in the completion result.

[0062] This embodiment employs a cross-modal attention mechanism, enabling the depth completion network to adaptively select visual texture information related to the occluded region. It utilizes appearance cues such as object edges, surface textures, and contextual relationships for geometric inference, resulting in a clear and sharp depth value at object boundaries and a smooth, continuous surface within planar regions. Furthermore, a random masking strategy allows the model to learn general geometric inference capabilities, demonstrating good completion performance for various forms of natural occlusion and sensor blind spots. Moreover, the combined use of L1 loss and multi-scale structural similarity loss ensures that the completed depth map closely approximates the true values ​​while maintaining a reasonable overall geometric structure, providing reliable input for downstream tasks such as depth-based 3D reconstruction, scene understanding, and grasping planning.

[0063] S6. The cross-modal contrast loss, motion consistency loss and depth completion loss are weighted and summed to obtain the total loss. The parameters of the multi-branch encoder are updated based on the total loss to obtain the pre-trained parameters.

[0064] In practice, the cross-modal contrast loss, motion consistency loss, and depth completion loss are weighted and summed to obtain the total loss. The key to multi-task learning is balancing the gradient contributions of each task to the model parameters, avoiding one task dominating training and causing underfitting in other tasks. Furthermore, the weighting coefficients can be pre-set to fixed values ​​based on the convergence speed or uncertainty of each task, or they can be adaptively adjusted during training as learnable parameters. Further, the gradient is calculated layer by layer using the backpropagation algorithm based on the total loss, and all trainable parameters of the multi-branch encoder are updated using a stochastic gradient descent optimizer or its adaptive variant. Further, the optimizer automatically adjusts the learning rate by maintaining estimates of the first and second moments of each parameter, achieving efficient and stable convergence. Further, the training process is iterated for multiple epochs until the total loss no longer decreases on the validation set; the final multi-branch encoder parameters are the pre-trained parameters.

[0065] It should be noted that, during the pre-training phase, the Adam optimizer, which is commonly used in this field, can be used instead of the stochastic gradient descent optimizer to improve training convergence speed and stability. The initial learning rate is set to a preset empirical value, and a learning rate decay strategy is adopted during training to promote smooth convergence of the model in the later stages of training. The training batch size can be set according to actual computing resources and the size of the training samples, and the total number of training epochs is based on the total loss stabilizing on the independent validation set.

[0066] It should be noted that the Adam optimizer adaptively adjusts the learning rate by maintaining the first-order moment estimate and the second-order moment estimate of each parameter. The exponential decay rate of the first-order moment estimate is used to control the weighted averaging of historical gradients, and the exponential decay rate of the second-order moment estimate is used to control the weighted averaging of the squared historical gradients. The above hyperparameters can be set in a conventional way according to the specific task and dataset, and this invention does not specifically limit them.

[0067] This embodiment achieves a deep understanding of the scene's 3D geometry, device motion, and object appearance invariance through joint optimization of three self-supervised tasks. Cross-modal contrastive learning utilizes time synchronization as a free supervision signal, establishing alignment between image and depth features in the embedding space, providing a solid feature foundation for downstream cross-modal retrieval and 3D scene understanding. Furthermore, motion consistency learning forces physical motion estimation to remain consistent with visual motion estimation, enabling the model to perceive its own motion from visual features. This capability can be directly transferred to visual odometry and simultaneous localization and mapping (SMR) tasks, significantly reducing localization drift and initialization difficulty. Further, depth completion learning, through repeated training inferring geometry from texture, allows the model to deeply understand the intrinsic relationship between 2D appearance and 3D structure, providing high-quality dense depth estimation in scenes with sparse or missing depth sensor data. Moreover, the entire pre-training process requires no manual annotation; all supervision signals come from the physical consistency of different modal data, significantly reducing the deployment cost of embodied intelligence systems in new environments. Furthermore, when the pre-trained parameters are transferred to downstream tasks, only a small amount of labeled data is needed for fine-tuning, which can achieve or even surpass the performance of training from scratch on a large amount of labeled data.

[0068] In some preferred embodiments, the weighted summation of the cross-modal contrast loss, the motion consistency loss, and the depth completion loss to obtain the total loss includes: assigning learnable uncertainty weight parameters to the cross-modal contrast loss, the motion consistency loss, and the depth completion loss respectively, wherein the initial value of the learnable uncertainty weight parameters is a preset value; constructing a maximum likelihood joint loss function based on homoscedastic uncertainty, wherein the maximum likelihood joint loss function includes normalization terms of the learnable uncertainty weight parameters for the corresponding loss terms; substituting the cross-modal contrast loss, the motion consistency loss, and the depth completion loss into the maximum likelihood joint loss function to calculate the total loss, and automatically updating the learnable uncertainty weight parameters through gradient descent during backpropagation.

[0069] In practice, the weights of each task's loss are automatically learned and adjusted during training using a maximum likelihood joint loss function based on homoscedasticity uncertainty, eliminating the need for manual trial and error to set the optimal weight combination. The specific implementation method is as follows: Learnable uncertainty weights are assigned to the cross-modal contrastive loss, motion consistency loss, and depth completion loss. These weights are scalar variables maintained in the optimizer state along with the neural network parameters and are updated via gradients. Furthermore, the initial values ​​of the uncertainty weights are preset, for example, set to small positive numbers (e.g., 0.1), and automatically scaled during training based on the changing trends of the losses for each task. The physical meaning of the uncertainty weights is the homoscedastic uncertainty of the prediction error for the corresponding task. Homoscedastic uncertainty is a measure of randomness that is independent of the input data and depends only on the inherent noise level of the task itself.

[0070] Furthermore, a maximum likelihood joint loss function based on homoscedastic uncertainty is constructed. This function includes a normalization term for the corresponding loss term, which incorporates learnable uncertainty weight parameters. Assuming that the true output of each task follows a Gaussian distribution with the network prediction as the mean and the task-related uncertainty as the variance, the joint likelihood probability of multiple task outputs under a given input is the product of the likelihood probabilities of each task. Further, taking the negative logarithm transforms maximizing the likelihood into minimizing the negative log-likelihood. Through mathematical derivation, the total loss can be expressed as the sum of the loss of each task divided by its respective uncertainty weight and the logarithm of each weight. The term where the loss of each task is divided by the weight adaptively scales the magnitude of the loss, while the logarithmic term acts as a normalization constraint to prevent the weights from infinitely approaching zero or diverging. This form of loss function automatically optimizes the allocation of task weights during the learning process.

[0071] Furthermore, the cross-modal contrastive loss, motion consistency loss, and deep completion loss are substituted into the maximum likelihood joint loss function to calculate the total loss. During backpropagation, the learnable uncertainty weight parameters are automatically updated via gradient descent. Furthermore, in the early stages of training, the losses for each task may differ significantly; the motion consistency loss has a large value due to the six-dimensional parameter regression, while the contrastive loss may be relatively small. Furthermore, under fixed weights, tasks with large losses will dominate the gradient update direction. Under adaptive weighting, for tasks with large loss values, gradient descent tends to increase their corresponding uncertainty weight parameters, thereby reducing the relative weight of the task's loss term; simultaneously, the logarithmic normalization term inhibits the weights from increasing indefinitely, causing the weights to converge to an equilibrium point. The entire process requires no manual setting or adjustment of the weight hyperparameters.

[0072] This embodiment achieves fully automatic adaptive balancing of multi-task losses by introducing learnable uncertainty weights, eliminating the pain point of requiring extensive manual parameter tuning in traditional multi-task learning. Different tasks have varying loss dimensions, convergence characteristics, and noise levels, making it difficult to maintain balanced learning across all tasks with fixed weights throughout the entire training phase. Furthermore, the automatic weighting mechanism in this embodiment dynamically adjusts the contribution ratio based on the actual performance of each task during training, ensuring that the three tasks always maintain relative weights that facilitate joint representation learning. This prevents premature stagnation due to training difficulties in one task, and also avoids wasting supervisory signals from other tasks due to excessively rapid convergence in one task. The resulting pre-trained representation consistently performs excellently across all downstream tasks.

[0073] In some preferred embodiments, after obtaining the pre-training parameters, the method further includes: loading the visual branch parameters in the pre-training parameters into the downstream task network to obtain an initialized downstream task network; inputting image training data into the initialized downstream task network to obtain visual features of the image training data, and inputting IMU training data into the motion branch in the pre-training parameters to obtain motion features of the IMU training data; adding a decoupling branch to the initialized downstream task network, inputting the visual features of the image training data into the decoupling branch to obtain scene structure features and appearance texture features, and applying a mutual information minimization constraint to the scene structure features and the appearance texture features to reduce the dependency between the scene structure features and the appearance texture features; fixing the network parameters corresponding to the scene structure features output by the decoupling branch, performing supervised fine-tuning of the network parameters corresponding to the appearance texture features using a labeled downstream task dataset, and adding an auxiliary motion consistency regularization term based on the motion features of the IMU training data to the fine-tuning loss to obtain the fine-tuned downstream task network parameters.

[0074] In practice, by separating scene structural features and appearance texture features during the fine-tuning stage and imposing a mutual information minimization constraint on both, the model can fully retain the geometric and motion priors learned during the pre-training stage while adapting to the appearance distribution of downstream tasks. The specific implementation method is as follows: The visual branch parameters from the pre-trained parameters are loaded into the downstream task network to obtain the initialized downstream task network. The downstream task network is a model designed for a specific application, such as an object pose estimation network, a person re-identification network, or a scene semantic segmentation network. These networks typically have their own backbone structure and task-specific heads. The loading operation assigns the weight matrices and bias vectors of each layer of the pre-trained visual branch to the corresponding layers of the downstream task network layer by layer, including all convolutional kernel parameters, and the scaling and offset parameters of the batch normalization layers. After loading, the initial parameters of the downstream task network contain rich geometric, motion, and appearance priors learned in self-supervised pre-training.

[0075] Furthermore, the image training data is input into the initialized downstream task network to obtain the visual features of the image training data, and the IMU training data is input into the motion branch of the pre-training parameters to obtain the motion features of the IMU training data. The image training data consists of samples provided in the downstream task dataset, which may contain scenes and objects with different distributions than those in the pre-training data. The IMU training data consists of inertial measurement readings synchronized with the image training data in time. The motion branch parameters remain frozen at this stage and are used only to extract motion features as a regularization signal.

[0076] Furthermore, a decoupling branch is added to the initialized downstream task network. Visual features from the image training data are input into the decoupling branch to obtain scene structure features and appearance texture features. The decoupling branch consists of two parallel fully connected sub-networks, each receiving visual features and outputting two fixed-dimensional feature vectors. Scene structure features are designed to encode information related to 3D geometry, spatial layout, and depth order, while appearance texture features are designed to encode surface attributes related to color, material, and lighting. To achieve effective separation, a mutual information minimization constraint is imposed on the scene structure features and appearance texture features. Mutual information is an indicator in information theory that measures the amount of information shared between two random variables; zero mutual information indicates that the two variables are statistically independent. The specific implementation of the mutual information minimization constraint can be achieved by training a mutual information neural estimator that distinguishes between feature pairs sampled from joint and marginal distributions. The upper or lower bound of the mutual information is derived using dual representations as a loss term, and minimizing this loss approximates minimizing the mutual information. By imposing this constraint, scene structure features and appearance texture features are forced to learn as independent representations of each other.

[0077] Furthermore, the network parameters corresponding to the scene structure features output by the fixed decoupling branch are used to perform supervised fine-tuning of the network parameters corresponding to the appearance texture features using a labeled downstream task dataset. An auxiliary motion consistency regularization term based on motion features from the IMU training data is added to the fine-tuning loss to obtain the fine-tuned downstream task network parameters. Furthermore, fixing the parameters corresponding to the scene structure features means that the weights of all convolutional and fully connected layers related to the generation of structure features are not updated during the fine-tuning stage. This ensures that the geometric and motion priors obtained during pre-training are not covered by the limited labeled samples downstream. Furthermore, the fine-tuning only updates parameters related to appearance texture features and task-specific heads. The auxiliary motion consistency regularization term requires the fine-tuned network to still make reasonable estimates of the device's own motion while extracting features. It is calculated by comparing the motion consistency of the features generated during fine-tuning with the motion features of the IMU training data and adding this as an additional loss term to the total loss. This regularization term prevents the motion perception ability learned during the pre-training stage from degrading due to over-adaptation to the labeled data distribution during the fine-tuning process.

[0078] This embodiment achieves effective separation of geometric prior and appearance adaptation by introducing feature decoupling and mutual information minimization during the fine-tuning stage. Fixing the structural feature parameters ensures the complete transfer of spatial cognitive abilities acquired during pre-training to downstream tasks, while adjusting only appearance features allows the model to flexibly adapt to color and texture distributions in new scenes. Furthermore, mutual information minimization ensures thorough separation, preventing structural information from being leaked into appearance features and modified through latent variables. Additionally, an auxiliary motion consistency regularization term further protects the motion perception prior, preventing its degradation during fine-tuning. This decoupled fine-tuning strategy is particularly effective in scenarios with extremely limited labeled data, achieving near-full-scale labeled fine-tuning performance with fewer samples.

[0079] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0080] Please see Figure 2 , Figure 2 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a terminal or a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0081] The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0082] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, it enables the processor 502 to perform a visual-physical self-supervised learning method for embodied intelligence.

[0083] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0084] The internal memory 504 provides an environment for the execution of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a visual-physical self-supervised learning method for embodied intelligence.

[0085] The network interface 505 is used for network communication with other devices. Those skilled in the art will understand that the above structure is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. A specific computer device 500 may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements.

[0086] The processor 502 is used to run a computer program 5032 stored in a memory to implement the steps of a visual-physical self-supervised learning method for embodied intelligence.

[0087] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0088] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program may be stored in a storage medium, which is a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0089] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program. When executed by a processor, the computer program causes the processor to perform steps of a visual-physical self-supervised learning method for embodied intelligence.

[0090] The storage medium is a physical, non-transient storage medium, such as a USB flash drive, external hard drive, read-only memory (ROM), magnetic disk, or optical disk, or any other physical storage medium capable of storing program code. The computer-readable storage medium can be non-volatile or volatile.

[0091] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0092] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0093] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0094] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0095] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0096] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, this invention is also intended to include these modifications and variations as long as they fall within the scope of the claims of this invention and their equivalents.

[0097] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A visual-physical self-supervised learning method for embodied intelligence, characterized in that, include: Acquire time-synchronized image frame sequences, depth frame sequences, and IMU reading sequences; The image frame sequence is input into the visual branch of the multi-branch encoder to obtain visual features; the depth frame sequence is input into the geometric branch of the multi-branch encoder to obtain geometric features; and the IMU reading sequence is input into the motion branch of the multi-branch encoder to obtain motion features. Based on the visual features and the geometric features, the cross-modal contrast loss is calculated by using visual features and geometric features at the same time as positive sample pairs and visual features and geometric features at different times as negative sample pairs. A first motion estimate is obtained based on the motion feature regression, a second motion estimate is obtained based on the visual features and the geometric features, and a motion consistency loss is calculated based on the difference between the first motion estimate and the second motion estimate. Random region occlusion is performed on the geometric features, and the depth value of the occluded region is predicted based on the visual features to obtain the predicted depth value. The depth completion loss is calculated based on the difference between the predicted depth value and the true depth value. The cross-modal contrast loss, the motion consistency loss, and the depth completion loss are weighted and summed to obtain the total loss. The parameters of the multi-branch encoder are then updated based on the total loss to obtain the pre-trained parameters.

2. The visual-physical self-supervised learning method for embodied intelligence according to claim 1, characterized in that, The calculation of cross-modal contrast loss based on the visual features and geometric features, using visual features and geometric features at the same time as positive sample pairs and visual features and geometric features at different times as negative sample pairs, includes: The visual features are spatially segmented to obtain multiple image block features, and the geometric features at the same time are spatially segmented accordingly to obtain multiple depth point cloud block features. Image patch features and depth point cloud patch features belonging to the same time and corresponding spatial location are determined as the positive sample pair, and image patch features and depth point cloud patch features belonging to different time or spatial location and not corresponding are determined as the negative sample pair; The similarity between each positive sample pair and each negative sample pair is calculated, and the difference between the similarity between positive sample pairs and the similarity between negative sample pairs is maximized by the cross-entropy loss function to obtain the cross-modal contrast loss.

3. The visual-physical self-supervised learning method for embodied intelligence according to claim 1, characterized in that, The process involves obtaining a first motion estimate based on the motion feature regression, obtaining a second motion estimate based on the visual features and the geometric features, and calculating a motion consistency loss based on the difference between the first and second motion estimates, including: The motion features are input into a temporal convolutional network to regress the relative pose change between adjacent frames. The relative pose change is then integrated to obtain the first motion estimate. Simultaneously, the covariance matrix of the first motion estimate is output as the first uncertainty measure. Dense optical flow is calculated from the visual features and the geometric features, and camera motion is estimated based on the dense optical flow using visual odometry to obtain the second motion estimate. At the same time, a second uncertainty metric is generated based on the confidence level of the dense optical flow. Based on the first uncertainty measure and the second uncertainty measure, the weighted distance between the first motion estimate and the second motion estimate is calculated to obtain the motion consistency loss, wherein the component with higher uncertainty has a lower weight in the weighted distance.

4. The visual-physical self-supervised learning method for embodied intelligence according to claim 1, characterized in that, The process involves randomly occluding the geometric features and predicting the depth value of the occluded region based on the visual features, obtaining a predicted depth value, and calculating the depth completion loss based on the difference between the predicted depth value and the true depth value. This includes: A binary mask of random shape is generated to mask local regions of the geometric features, resulting in masked geometric features. The occluded geometric features and the visual features are concatenated in the channel dimension and input into a depth completion sub-network containing a cross-modal attention module. The cross-modal attention module uses the visual features as the query and the occluded geometric features as the key and value. It fuses the visual features and the occluded geometric features through an attention mechanism to obtain enhanced features. The enhanced features are then decoded, and the predicted depth value of the occluded region is output. Calculate the pixel-wise L1 loss and multi-scale structural similarity loss between the predicted depth value and the true depth value, and sum the L1 loss and the multi-scale structural similarity loss to obtain the depth completion loss.

5. The visual-physical self-supervised learning method for embodied intelligence according to claim 1, characterized in that, The multi-branch encoder's visual branch is a residual convolutional neural network, its geometric branch is a point cloud feature extraction network, and its motion branch is a bidirectional long short-term memory network. The multi-branch encoder also includes a cross-branch feature interaction layer, which is used for: Multi-scale visual feature maps are extracted from the first intermediate layer of the visual branch, and geometric feature volumes of the corresponding scale are extracted from the second intermediate layer of the geometric branch. The correlation matrix between the multi-scale visual feature map and the corresponding scale geometric feature body is calculated by dot product attention, and the multi-scale visual feature map is weighted by the correlation matrix to obtain the enhanced visual features. The geometric feature body at the corresponding scale is weighted by the transpose of the correlation matrix to obtain the enhanced geometric features. The enhanced visual features are input into the convolutional layer in the visual branch after the first intermediate layer, and the enhanced geometric features are input into the point cloud feature extraction layer in the geometric branch after the second intermediate layer.

6. The visual-physical self-supervised learning method for embodied intelligence according to claim 1, characterized in that, The weighted summation of the cross-modal contrast loss, the motion consistency loss, and the depth completion loss yields the total loss, which includes: Learnable uncertainty weight parameters are assigned to the cross-modal contrast loss, the motion consistency loss, and the depth completion loss, respectively, and the initial values ​​of the learnable uncertainty weight parameters are preset values. Construct a maximum likelihood joint loss function based on homoscedastic uncertainty, wherein the maximum likelihood joint loss function includes the normalization term of the learnable uncertainty weight parameter for the corresponding loss term; The cross-modal contrast loss, motion consistency loss, and depth completion loss are substituted into the maximum likelihood joint loss function to calculate the total loss, and the learnable uncertainty weight parameters are automatically updated through gradient descent during backpropagation.

7. The visual-physical self-supervised learning method for embodied intelligence according to claim 1, characterized in that, After obtaining the pre-trained parameters, the method further includes: The visual branch parameters in the pre-trained parameters are loaded into the downstream task network to obtain the initialized downstream task network; The image training data is input into the initialized downstream task network to obtain the visual features of the image training data, and the IMU training data is input into the motion branch of the pre-training parameters to obtain the motion features of the IMU training data. A decoupling branch is added to the initialized downstream task network. The visual features of the image training data are input into the decoupling branch to obtain scene structure features and appearance texture features. Mutual information minimization constraints are applied to the scene structure features and appearance texture features to reduce the dependency between the scene structure features and appearance texture features. The network parameters corresponding to the scene structure features output by the decoupled branch are fixed, and the network parameters corresponding to the appearance texture features are fine-tuned in a supervised manner using a labeled downstream task dataset. An auxiliary motion consistency regularization term based on the motion features of the IMU training data is added to the fine-tuning loss to obtain the fine-tuned downstream task network parameters.

8. The visual-physical self-supervised learning method for embodied intelligence according to claim 1, characterized in that, The acquisition of time-synchronized image frame sequences, depth frame sequences, and IMU reading sequences includes: The raw data streams from the camera, depth sensor, and IMU on the embodied device are acquired, and the raw data streams from the IMU are pre-integrated. Multiple IMU measurements between adjacent image frames are integrated into relative rotation, relative velocity, and relative translation changes to obtain motion constraints between adjacent frames. Using the motion constraints as regularization terms, joint nonlinear optimization alignment is performed on the timestamps of image frames, depth frames, and IMU readings in the original data stream to obtain time-aligned image frame sequences, depth frame sequences, and IMU reading sequences. The time-aligned IMU reading sequence is subjected to deviation correction, and the time-aligned image frame sequence and depth frame sequence are subjected to camera extrinsic parameter calibration. The spatial coordinates of the depth frame sequence are transformed to a coordinate system consistent with the image frame sequence to obtain the time-synchronized image frame sequence, depth frame sequence and IMU reading sequence.

9. A computer device, characterized in that, The computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, can implement the method as described in any one of claims 1-8.

Images