Stereo vision feature extraction method of image, electronic device and program product

Abstract

The present disclosure provides a stereovision feature extraction method for images, an electronic device and a program product. The stereovision feature extraction method for images of the present disclosure comprises: performing weighted summation on original parameter values of each weight parameter in a DINO architecture of a monocular depth estimation model, a three-dimensional vision base model and a DINO model to obtain the DINO architecture with fusion parameter values of each weight parameter as a feature extraction module; inputting an image into the feature extraction module to obtain a fusion feature; inputting the fusion feature after splicing with a camera token into an intra-frame attention module, multiplying the output of the intra-frame attention module with a hyperparameter to obtain a first feature; taking a difference value between a processing result of the fusion feature by a neck module of the monocular depth estimation model and the first feature as a latent feature; and performing dense feature generation on the latent feature by a DPT head to obtain a stereovision feature.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision technology, specifically to a method for extracting stereoscopic visual features from images, an electronic device, and a program product. Background Technology

[0002] Stereo vision is a core branch of 3D vision. In stereo vision, disparity estimation and stereo transformation constitute two core tasks. Disparity estimation (also known as stereo matching) aims to establish pixel correspondences between paired calibration images. Stereo transformation involves converting monocular images or videos into stereo image content. Current stereo transformation methods use disparity priors as guidance to synthesize right-view images. Since disparity is inversely proportional to the depth of an object in the world coordinate system from the imaging plane, it can be estimated from monocular images using predicted depth, camera focal length, and baseline. Inspired by this principle, stereo matching and stereo transformation algorithms typically use monocular depth estimation models to extract depth features as prior information for the task. In resource-constrained scenarios, many applications attempt to calculate disparity by acquiring monocular images and combining the depth output from the monocular depth estimation model with camera parameters to generate stereo vision features. However, because depth estimation often lacks a true scale, the resulting disparity is often only relative and insufficient for high-precision stereo reconstruction. Current monocular depth estimation methods have advantages in edge detection and flying point artifact suppression, but they still have limitations in outputting reliable disparity prior information. Summary of the Invention

[0003] This disclosure provides a method, electronic device, and program product for extracting stereoscopic visual features from an image.

[0004] According to one aspect of this disclosure, a method for extracting stereo vision features from an image is provided, comprising: weighted summing of the original parameter values ​​of each weight parameter in the DINO architecture of a monocular depth estimation model, a 3D vision base model, and a DINO model to obtain a DINO architecture with fusion parameter values ​​of each weight parameter, serving as a feature extraction module; inputting the image into the feature extraction module, wherein the feature extraction module performs feature encoding on the image based on the fusion parameter values ​​to obtain fused features; concatenating the fused features with a camera token to obtain intermediate features; inputting the intermediate features into an intra-frame attention module of the 3D vision base model, and multiplying the output of the intra-frame attention module with a hyperparameter to obtain a first feature, wherein the hyperparameter is used to adjust the influence weight of the intra-frame attention module on the neck module of the monocular depth estimation model; using the difference between a second feature and the first feature as a latent feature, wherein the second feature is the result of multi-scale feature fusion of the fused features by the neck module of the monocular depth estimation model; and generating dense features from the latent features using the DPT head of the monocular depth estimation model to obtain the stereo vision features.

[0005] According to one technical solution, by weighting and summing the original parameter values ​​of each weight parameter in the DINO architecture of the monocular depth estimation model, the 3D vision foundation model, and the DINO model, the feature degradation problem caused by the over-reliance on camera geometric priors in the 3D vision foundation model can be alleviated. Furthermore, by utilizing the intra-frame attention module of the 3D vision foundation model and the neck module of the monocular depth estimation model for bi-branch feature learning, a balance between image feature knowledge and camera knowledge can be further achieved.

[0006] According to at least one embodiment of this disclosure, the original parameter values ​​of each weight parameter in the DINO architecture of the monocular depth estimation model, the 3D vision basic model, and the DINO model are weighted and summed to obtain a DINO architecture with fused parameter values ​​of each weight parameter, which serves as a feature extraction module. This includes: weighting and summing the original parameter values ​​of the same weight parameter in the DINO architecture of the monocular depth estimation model, the 3D vision basic model, and the DINO model to obtain a fused parameter value of the weight parameter; and replacing the original parameter values ​​of the corresponding weight parameters in the DINO architecture with the fused parameter value, using the DINO architecture with the fused parameter value as a feature extraction module.

[0007] According to the technical solution of this embodiment, the scale consistency, geometric accuracy and semantic discrimination ability of stereoscopic vision features can be improved.

[0008] According to at least one embodiment of this disclosure, the monocular depth estimation model has the same weight coefficient for different weight parameters belonging to the same feature layer, the three-dimensional vision basic model has the same weight coefficient for different weight parameters belonging to the same feature layer, and the DINO model has the same weight coefficient for different weight parameters belonging to the same feature layer.

[0009] According to the technical solution of this embodiment, while maintaining the integrity of the model structure, it is possible to achieve an adaptive balance between geometric priors and semantic information at different network depths.

[0010] According to at least one embodiment of this disclosure, the process of determining the weight coefficients of the weight parameters corresponding to each feature layer includes: constructing an objective function, wherein the objective function is the minimum information entropy of the probability values ​​of each weight parameter included in the feature layer, and the probability values ​​of the weight parameters are the weighted sum of the original weight values ​​of the corresponding weight parameters in the monocular depth estimation model and the three-dimensional vision basic model; solving the objective function to obtain the weight coefficients of the weight parameters corresponding to the feature layer in the monocular depth estimation model as first weight coefficients, and obtaining the weight coefficients of the weight parameters corresponding to the feature layer in the three-dimensional vision basic model as second weight coefficients; and subtracting the sum of the first weight coefficient and the second weight coefficient from 1 to obtain the difference as the weight coefficient of the weight parameters corresponding to the feature layer in the DINO model.

[0011] According to the technical solution of this embodiment, by constructing a weight coefficient determination method with information entropy minimization as the objective function, the uncertainty and degradation degree of fusion features can be effectively quantified.

[0012] According to at least one embodiment of this disclosure, solving the objective function includes: updating the first weight coefficient and the second weight coefficient by gradient descent until a stopping condition is reached; and outputting the first weight coefficient and the second weight coefficient when the stopping condition is reached.

[0013] According to the technical solution of this embodiment, it is possible to efficiently and stably converge to the optimal weight combination that minimizes the entropy of the fused feature information.

[0014] According to at least one embodiment of this disclosure, the stopping condition is: the difference between the first weight coefficients in two adjacent iterations is less than or equal to the convergence value, and the difference between the second weight coefficients in two adjacent iterations is less than or equal to the convergence value.

[0015] According to the technical solution of this embodiment, redundant calculations can be avoided while ensuring stable convergence of fusion parameter values.

[0016] According to at least one embodiment of this disclosure, the monocular depth estimation model adopts the MoGe-2 model, the three-dimensional vision foundation model adopts the VGGT model, and the DINO model adopts the DINOv2 model.

[0017] According to the technical solution of this embodiment, the accuracy of stereoscopic vision feature extraction can be improved.

[0018] According to another aspect of this disclosure, an electronic device is provided, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, causing the processor to perform a stereoscopic visual feature extraction method for an image according to any embodiment of this disclosure.

[0019] According to another aspect of this disclosure, a readable storage medium is provided, wherein executable instructions are stored therein, which, when executed by a processor, are used to implement a stereoscopic vision feature extraction method for images according to any embodiment of this disclosure.

[0020] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements a method for extracting stereoscopic visual features of an image according to any embodiment of this disclosure. Attached Figure Description

[0021] The accompanying drawings illustrate exemplary embodiments of the present disclosure and, together with the description thereof, serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0022] Figure 1 This is a flowchart illustrating a method for extracting stereoscopic visual features from an image according to one embodiment of the present disclosure.

[0023] Figure 2 This is a flowchart illustrating a method for determining weight coefficients according to one embodiment of the present disclosure.

[0024] Figure 3 This is a flowchart illustrating a method for solving an objective function according to one embodiment of the present disclosure.

[0025] Figure 4 This is a schematic diagram of the structure of each model used in a stereo vision feature extraction method according to one embodiment of the present disclosure.

[0026] Figure 5 This is a schematic block diagram of a stereoscopic vision feature device according to one embodiment of the present disclosure.

[0027] Figure 6 This is a schematic structural block diagram of an electronic device employing a processor-based hardware implementation according to one embodiment of the present disclosure. Detailed Implementation

[0028] The present disclosure will now be described in further detail with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the accompanying drawings.

[0029] It should be noted that, where there is no conflict, the embodiments and features described in this disclosure can be combined with each other. The technical solutions of this disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0030] Existing monocular depth estimation methods often struggle to accurately estimate the relative spatial arrangement of objects. Vision Foundation Models (VFAMs), another widely used backbone network in stereo vision, suffer from similar shortcomings. Therefore, effectively integrating explicit camera pose information during feature extraction to overcome the inaccuracy in relative spatial arrangement caused by the lack of geometric constraints in existing monocular depth estimation models and VFAMs has become a pressing issue.

[0031] To address this issue, the present disclosure proposes the following technical solution. In this solution, by weighting and summing the original parameter values ​​of each weight parameter in the DINO architecture of the monocular depth estimation model, the 3D vision foundation model, and the DINO model, the feature degradation problem caused by the over-reliance on camera geometric priors in the 3D vision foundation model can be mitigated. Furthermore, by utilizing the intra-frame attention module of the 3D vision foundation model and the neck module of the monocular depth estimation model for bi-branch feature learning, a balance between image feature knowledge and camera knowledge can be further achieved.

[0032] To facilitate description and make the technical solutions of this disclosure easier to understand, the terminology of this disclosure will be explained before describing the technical solutions of this disclosure.

[0033] Monocular depth estimation models are deep learning models that predict the distance from each pixel in a scene to the camera based on a single input image, and are used to generate dense depth maps or geometric priors.

[0034] A 3D vision fundamental model refers to a vision fundamental model that is pre-trained on large-scale 3D data and can encode the relationship between camera pose and 3D geometry. Its feature representation explicitly or implicitly contains camera imaging parameter information.

[0035] The DINO model refers to the VisionTransformer architecture's fundamental visual model trained using self-supervised contrastive learning methods (such as DINO or DINOv2).

[0036] The DINO architecture refers to the underlying network structure adopted by the DINO model, specifically a backbone network based on Vision Transformer (ViT), including an image patch embedding layer, positional encoding, multi-layer Transformer encoders, and optional special tokens (such as class tokens).

[0037] The VGGT (Vision Geometry-aware General-purpose Transformer) model is a fundamental 3D vision model based on the DINOv2 pre-trained Vision Transformer architecture. During training, camera intrinsic and extrinsic parameters are explicitly introduced as supervision signals or conditional inputs. By embedding learnable camera tokens in the Transformer encoder, the VGGT model can learn geometrically perceptual feature representations aligned with real-world camera poses.

[0038] The MoGe-2 model (Monocular Geometry-aware Model v2) is a monocular depth estimation model based on the DINOv2 pre-trained Vision Transformer architecture. The MoGe-2 model inherits the powerful general semantic representation capabilities of DINOv2 and effectively recovers pixel-level spatial details through the DPT head structure.

[0039] A camera token is a pre-trained and frozen learnable embedding vector introduced into a 3D vision foundational model (such as VGGT). During the training phase of the 3D vision foundational model, the camera token participates in the self-attention mechanism together with image patch features, implicitly encoding geometric prior information represented by camera intrinsic and extrinsic parameters. During the inference phase, the camera token is concatenated with image features (such as fused features) and input into the intra-frame attention module to guide the network in perceiving the real-world camera pose and metric scale during feature extraction.

[0040] This disclosure is applicable to image processing tasks such as disparity estimation, stereo matching, or stereo transformation, and is particularly suitable for applications in resource-constrained environments.

[0041] Figure 1 A schematic diagram illustrating the overall flow of a stereoscopic vision feature extraction method for images according to one embodiment of this disclosure is shown. Figure 1 The method shown includes steps S110 to S160. This method can be executed by electronic devices such as mobile phones and tablets.

[0042] In step S110, the original parameter values ​​of each weight parameter in the DINO architecture of the monocular depth estimation model, the 3D vision basic model and the DINO model are weighted and summed to obtain the DINO architecture with the fused parameter values ​​of each weight parameter, which serves as the feature extraction module.

[0043] The DINO architecture is based on the Vision Transformer (ViT) network structure and contains multiple feature layers (i.e., Transformer encoding layers). Each feature layer contains multiple learnable weight parameters.

[0044] The monocular depth estimation model, the 3D vision foundation model, and the DINO model all incorporate the DINO architecture; therefore, all three models share the same DINO architecture weight parameters (same tensor shape and parameter positions). However, due to differences in training objectives and data, the same weight parameters in the three models have different original parameter values. The constructed feature extraction module is a DINO architecture with fused parameter values.

[0045] As one possible implementation, the original parameter values ​​of each weight parameter in the DINO architecture of the monocular depth estimation model, the 3D vision base model, and the DINO model are weighted and summed to obtain a DINO architecture with fused parameter values ​​for each weight parameter. This DINO architecture serves as a feature extraction module. The module includes: weighting and summing the original parameter values ​​of the same weight parameters in the DINO architecture of the monocular depth estimation model, the 3D vision base model, and the DINO model to obtain fused parameter values ​​for the weight parameters; replacing the original parameter values ​​of the corresponding weight parameters in the DINO architecture with these fused parameter values; and using the DINO architecture with the fused parameter values ​​as the feature extraction module.

[0046] As one possible implementation, the monocular depth estimation model preferably adopts the MoGe-2 model, the three-dimensional vision basic model preferably adopts the VGGT model, and the DINO model preferably adopts the DINOv2 model.

[0047] This implementation introduces the DINO architecture of the DINO model and monocular depth estimation model to compensate for the DINO architecture of the 3D vision foundation model. This can alleviate the feature degradation problem caused by the over-reliance on camera geometric priors in the 3D vision foundation model. This implementation can significantly recover image semantic details and geometric structure information while preserving the camera's pose awareness capability. Therefore, it can significantly improve the discriminativeness and geometric consistency of stereo vision features without requiring additional training data.

[0048] In step S120, the image is input into the feature extraction module, which encodes the image based on the fusion parameter values ​​to obtain fused features.

[0049] The feature extraction module employs a DINO architecture with fusion parameter values. The feature extraction module encodes features from the input image and uses its output as the fused features.

[0050] In step S130, the fused features are concatenated with the camera token to obtain intermediate features.

[0051] As one possible implementation, the concatenation method between the fused features and the camera token can be the same as the concatenation method between the output features and the camera token in the DINO architecture of the VGGT model. That is, in the sequence dimension, the camera token is concatenated as a pre-trained and frozen embedding vector with the image patch feature sequence of the fused features, forming an input sequence containing N+1 elements (where N is the number of image patches contained in the fused features, and 1 represents the camera token). This ensures that the intra-frame attention module can jointly model the camera geometric prior and image content in a manner consistent with the original VGGT design. In this implementation, the input sequence containing N+1 elements is the intermediate feature.

[0052] In step S140, the intermediate features are input into the intra-frame attention module of the 3D vision base model, and the output of the intra-frame attention module is multiplied by the hyperparameters to obtain the first feature. The hyperparameters are used to adjust the influence weight of the intra-frame attention module on the neck module of the monocular depth estimation model. For example, the hyperparameters can be set to 0.2.

[0053] The intra-frame attention module in the 3D vision foundation model performs self-attention computation on the intermediate features of the input, based on the global interaction between the camera token and the features of each image patch contained in the fused features. This strengthens the spatial correlation consistent with the camera's geometric prior and outputs geometrically perceived features representing the camera pose constraints. The geometrically perceived features output by the intra-frame attention module are multiplied with hyperparameters, and the result of the multiplication is used as the first feature.

[0054] In step S150, the difference between the second feature and the first feature is used as a latent feature. The second feature is the result of multi-scale feature fusion of the fused features by the neck module of the monocular depth estimation model.

[0055] The neck module of the monocular depth estimation model is used to perform cross-scale feature reconstruction and spatial detail enhancement on the input fused features, and outputs dense structure-aware features (i.e., second features) that represent rich geometric priors. The difference between the second feature and the first feature is the latent feature.

[0056] In step S160, the latent features are generated by dense feature generation using the DPT head of the monocular depth estimation model to obtain stereo vision features.

[0057] The Dense Prediction Transformer (DPT) head of the monocular depth estimation model is used to perform pixel-level dense mapping and semantic-geometric joint decoding of the input latent features. Through multi-scale feature upsampling and cross-layer feature fusion, it reconstructs a high-dimensional feature representation aligned with the input image space, thereby enhancing the fidelity of local details and global semantic consistency of the deep structure. The DPT head of the monocular depth estimation model can output dense stereo vision features suitable for stereo vision tasks.

[0058] The stereo vision features obtained in step S160 can be used for downstream stereo vision tasks. For example, the stereo vision features can be input into a stereo matching decoder to perform a stereo matching task to generate a disparity map. Alternatively, the stereo vision features can be used for stereo image synthesis to perform a stereo transformation task to generate a corresponding stereo view from a monocular image.

[0059] This disclosure mitigates the feature degradation problem caused by the over-reliance on camera geometric priors in the 3D vision foundation model by weighted summation of the original parameter values ​​of each weight parameter in the DINO architecture of the monocular depth estimation model, the 3D vision foundation model, and the DINO model. Furthermore, utilizing the intra-frame attention module of the 3D vision foundation model and the neck module of the monocular depth estimation model for bi-branch feature learning further balances image feature knowledge and camera knowledge.

[0060] While weighted summation of the DINO architecture weight parameters of the monocular depth estimation model, the 3D vision foundation model, and the DINO model can mitigate feature degradation in the 3D vision foundation model, excessively reducing the weights of the 3D vision foundation model will weaken the camera pose information encapsulated within it, thus adversely affecting metric scale estimation. To address this technical problem, this disclosure proposes a method for determining weight coefficients.

[0061] As one possible implementation, in the weight coefficient determination method, the monocular depth estimation model has the same weight coefficients for different weight parameters belonging to the same feature layer, the 3D vision basic model has the same weight coefficients for different weight parameters belonging to the same feature layer, and the DINO model has the same weight coefficients for different weight parameters belonging to the same feature layer. The same feature layer can be the Transformer encoding layer at the same level in the DINO architecture, and all weight parameters within the same feature layer share the same weight coefficients.

[0062] By dividing the weight parameters according to the feature layer and using a uniform weight coefficient to sum the corresponding weight parameters of each model within each feature layer, an adaptive balance between geometric priors and semantic information at different network depths can be achieved while maintaining the integrity of the model structure. This improves the consistent representation ability of the fused feature extraction module in both shallow (texture details) and deep (semantic structure) layers, enhancing the hierarchical representation quality of stereo vision features.

[0063] Figure 2 A schematic diagram illustrating the overall flow of a weighting coefficient determination method according to one embodiment of this disclosure is shown. Figure 2 The method shown includes steps S210 to S230.

[0064] In step S210, an objective function is constructed. The objective function is the minimum information entropy of the probability values ​​of each weight parameter contained in the feature layer. The probability value of the weight parameter is the weighted sum of the original weight values ​​of the corresponding weight parameters in the monocular depth estimation model and the 3D vision basic model.

[0065] In step S220, the objective function is solved to obtain the weight coefficients of the weight parameters corresponding to the feature layers in the monocular depth estimation model, which are used as the first weight coefficients, and the weight coefficients of the weight parameters corresponding to the feature layers in the three-dimensional vision basic model are obtained, which are used as the second weight coefficients.

[0066] In step S230, the difference obtained by subtracting the sum of the first weight coefficient and the second weight coefficient from 1 is used as the weight coefficient of the corresponding weight parameter of the feature layer in the DINO model.

[0067] This implementation method effectively quantifies the uncertainty and degradation degree of fused features by constructing a weight coefficient determination method with the minimization of information entropy as the objective function. Since lower information entropy indicates a more concentrated feature distribution, stronger discriminative power, and less feature degradation, the objective function can guide the feature encoding process of the DINO architecture to suppress feature degradation of the 3D vision base model while avoiding scale blurring caused by excessive reduction of the 3D vision base model weights, which leads to a loss of camera pose information.

[0068] As one possible implementation, the objective function is expressed as: in, Indicates the sequence number of the feature layer; Indicates the first The probability value of the j-th weight parameter of each feature layer; The first digit representing the basic model of 3D vision Weight coefficients of each feature layer; The first monocular depth estimation model represents the... Weight coefficients of each feature layer; The first digit representing the basic model of 3D vision The original parameter values ​​of the j-th weight parameter of each feature layer; The first monocular depth estimation model represents the... The original parameter value of the j-th weight parameter of a feature layer; d represents the total number of weight parameters contained in the feature layer; Indicates the index of the weight parameters contained in the feature layer; Indicates the first The j-th weight parameter of each feature layer; H(.) represents the information entropy function; min represents finding the minimum value.

[0069] Based on the above objective function, the solution obtained... For the monocular depth estimation model The weight coefficients of each weight parameter in each feature layer, i.e., the first weight coefficients. These are obtained based on the objective function described above. The first of the three-dimensional vision basic model The weight coefficients of each weight parameter in each feature layer, i.e., the second weight coefficients. (The last part, "1-", appears to be a typo and can be left as is.) - The result is used as the first in the DINO model The weight coefficients of each feature layer.

[0070] After obtaining the weight coefficients of each feature layer in the three models, the fusion parameter value can be obtained by weighted summation using the following formula.

[0071] in, Indicates the first The fusion parameter value of the j-th weight parameter of each feature layer; The first DINO model represents the... The original parameter values ​​of the j-th weight parameter of each feature layer.

[0072] Figure 3 A schematic diagram illustrating the overall flow of a method for solving the objective function according to one embodiment of this disclosure is shown. Figure 3 The method shown includes steps S310 to S320.

[0073] In step S310, the first weight coefficient and the second weight coefficient are updated by gradient descent until the stopping condition is met.

[0074] In step S320, the first weighting coefficient and the second weighting coefficient when the stopping condition is met are output.

[0075] This implementation uses gradient descent to iteratively optimize the first and second weight coefficients until the stopping condition is met, which can efficiently and stably converge to the optimal weight combination that minimizes the entropy of the fused feature information.

[0076] As one possible implementation, the stopping condition is: the difference between the first weight coefficients in two consecutive iterations is less than or equal to the convergence value, and the difference between the second weight coefficients in two consecutive iterations is less than or equal to the convergence value. For example, the convergence value can be set to... This implementation method sets a stopping condition that the change in weight coefficients between adjacent iterations does not exceed a preset convergence value, which can ensure stable convergence of the fusion parameter values ​​while avoiding redundant calculations.

[0077] Figure 4 A schematic diagram of the structure of each model used in a stereo vision feature extraction method according to one embodiment of the present disclosure is shown.

[0078] Figure 4 The DINO parameter merging section in the lower left corner is used to represent the weighted fusion of each weight parameter in the DINO architecture of the three models: VGGT, MoGe-2, and DINOv2.

[0079] Figure 4 The StereoVGGT network architecture section above describes the main modules used in the stereo vision feature extraction method. The DINOv2 module can be the feature extraction module obtained through step S110. The fused features obtained from the image input DINOv2 module are concatenated with the camera token, and the concatenated result (intermediate features) is simultaneously input into the intra-frame attention module of the VGGT model and the neck module of the MoGe-2 model. The output of the intra-frame attention module of the VGGT model is multiplied by the hyperparameters. The first feature is then obtained. The second feature is taken as the input to the neck module of the MoGe-2 model. The difference between the second feature and the first feature is taken as the latent feature. After the latent feature is input into the DPT head of the MoGe-2 model, the stereo vision feature (i.e., depth prior) is obtained.

[0080] Figure 4 The downstream tasks section in the lower right corner describes the downstream tasks that can be applied to the stereo vision features obtained through this embodiment. For example, they can be applied to the stereo transformation task in Task 1, or to the stereo matching task in Task 2.

[0081] According to any of the above embodiments, this disclosure also provides a stereoscopic vision feature extraction device 500. Figure 5 This is a schematic block diagram of a stereoscopic vision feature extraction device 500 according to one embodiment of this disclosure. Figure 5As shown, the stereo vision feature extraction device 500 includes a feature extraction module construction module 510, a fusion feature acquisition module 520, an intermediate feature acquisition module 530, a first feature acquisition module 540, a latent feature acquisition module 550, and a stereo vision feature acquisition module 560.

[0082] The feature extraction module construction module 510 is used to perform weighted summation of the original parameter values ​​of each weight parameter in the DINO architecture of the monocular depth estimation model, the 3D vision basic model and the DINO model, to obtain the DINO architecture with the fused parameter values ​​of each weight parameter, which serves as the feature extraction module.

[0083] The fusion feature acquisition module 520 is used to input the image into the feature extraction module. The feature extraction module performs feature encoding on the image based on the fusion parameter values ​​to obtain the fusion features.

[0084] The intermediate feature acquisition module 530 is used to concatenate the fused features with the camera token to obtain intermediate features.

[0085] The first feature acquisition module 540 is used to input the intermediate features into the intra-frame attention module of the 3D vision base model, and multiply the output of the intra-frame attention module with the hyperparameters to obtain the first feature. The hyperparameters are used to adjust the influence weight of the intra-frame attention module on the neck module of the monocular depth estimation model.

[0086] The latent feature acquisition module 550 is used to take the difference between the second feature and the first feature as the latent feature. The second feature is the result of multi-scale feature fusion of the fusion features by the neck module of the monocular depth estimation model.

[0087] The stereo vision feature acquisition module 560 is used to generate dense features from latent features through the DPT head of the monocular depth estimation model to obtain stereo vision features.

[0088] The stereo vision feature extraction device 500 disclosed herein can be implemented through a computer software architecture.

[0089] According to further embodiments of this disclosure, an electronic device is also provided. Figure 6This diagram illustrates a schematic block diagram of an electronic device employing a processor-based hardware implementation according to an embodiment of the present disclosure. The hardware structure of the electronic device of the present disclosure can be implemented using a bus architecture. The bus architecture can include any number of interconnect buses and bridges, depending on the specific application and overall design constraints of the hardware. Bus 1100 connects various circuits including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400 such as peripheral devices, voltage regulators, power management circuits, external antennas, etc. Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one connecting line is used in this figure, but this does not indicate that there is only one bus or one type of bus.

[0090] This disclosure also provides a readable storage medium storing a computer program that, when executed by a processor, is used to implement the methods described above. A "readable storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples of a readable storage medium include: an electrical connection with one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM), etc.

[0091] This disclosure also provides a computer program product, the methods of which can be implemented wholly or partially through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented wholly or partially as a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed, all or part of the processes or functions of this disclosure are performed.

[0092] Computer programs or instructions can be stored in a readable storage medium or transferred from one readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The readable storage medium can be any available medium capable of access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; an optical medium, such as a digital video optical disc; or a semiconductor medium, such as a solid-state drive. The computer-readable storage medium can be a volatile or non-volatile storage medium, or it can include both volatile and non-volatile types of storage media.

[0093] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0094] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0095] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0096] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0097] In the description of this specification, the references to terms such as "one embodiment / mode," "some embodiments / modes," "example," "specific example," or "some examples," etc., refer to specific features, structures, or characteristics described in connection with that embodiment / mode or example, which are included in at least one embodiment / mode or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment / mode or example. Moreover, the specific features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments / modes or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments / modes or examples described in this specification, as well as the features of different embodiments / modes or examples.

[0098] Those skilled in the art should understand that the above embodiments are merely for illustrating the present disclosure and are not intended to limit the scope of the disclosure. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present disclosure.

Claims

1. A method for extracting stereoscopic visual features from an image, characterized in that, include: The original parameter values ​​of each weight parameter in the DINO architecture of the monocular depth estimation model, the 3D vision basic model and the DINO model are weighted and summed to obtain the DINO architecture with the fused parameter values ​​of each weight parameter, which serves as the feature extraction module. The image is input into the feature extraction module, which performs feature encoding on the image based on the fusion parameter values ​​to obtain fused features; The fused feature is concatenated with the camera token to obtain the intermediate feature; The intermediate features are input into the intra-frame attention module of the 3D vision base model, and the output of the intra-frame attention module is multiplied by the hyperparameters to obtain the first feature. The hyperparameters are used to adjust the influence weight of the intra-frame attention module on the neck module of the monocular depth estimation model. The difference between the second feature and the first feature is used as a latent feature, where the second feature is the result of multi-scale feature fusion of the fused features by the neck module of the monocular depth estimation model. as well as The stereo vision features are obtained by generating dense features from the latent features using the DPT head of the monocular depth estimation model.

2. The method for extracting stereoscopic visual features from an image as described in claim 1, characterized in that, The original parameter values ​​of each weight parameter in the DINO architecture of the monocular depth estimation model, the 3D vision basic model, and the DINO model are weighted and summed to obtain the DINO architecture with the fused parameter values ​​of each weight parameter. This architecture serves as the feature extraction module, including: The original parameter values ​​belonging to the same weight parameter in the DINO architecture of the monocular depth estimation model, the 3D vision basic model, and the DINO model are weighted and summed to obtain the fusion parameter value of the weight parameter; and The original parameter values ​​of the corresponding weight parameters in the DINO architecture are replaced with the fusion parameter values, and the DINO architecture with the fusion parameter values ​​is used as a feature extraction module.

3. The method for extracting stereoscopic visual features of an image as described in claim 2, characterized in that, The monocular depth estimation model has the same weight coefficient for different weight parameters belonging to the same feature layer, the three-dimensional vision basic model has the same weight coefficient for different weight parameters belonging to the same feature layer, and the DINO model has the same weight coefficient for different weight parameters belonging to the same feature layer.

4. The method for extracting stereoscopic visual features from an image as described in claim 3, characterized in that, The process of determining the weight coefficients for the weight parameters corresponding to each feature layer includes: Construct an objective function, which is the minimum information entropy of the probability values ​​of each weight parameter contained in the feature layer, and the probability value of the weight parameter is the weighted sum of the original weight values ​​of the corresponding weight parameters in the monocular depth estimation model and the three-dimensional vision basic model; Solving the objective function yields the weight coefficients of the weight parameters corresponding to the feature layers in the monocular depth estimation model, which are used as the first weight coefficients; and the weight coefficients of the weight parameters corresponding to the feature layers in the 3D vision foundation model are used as the second weight coefficients; and The difference obtained by subtracting the sum of the first weight coefficient and the second weight coefficient from 1 is used as the weight coefficient of the corresponding weight parameter of the feature layer in the DINO model.

5. The method for extracting stereoscopic visual features from an image as described in claim 4, characterized in that, Solving the objective function includes: The first and second weight coefficients are updated using gradient descent until a stopping condition is met; and Output the first weighting coefficient and the second weighting coefficient when the stopping condition is met.

6. The method for extracting stereoscopic visual features from an image as described in claim 5, characterized in that, The stopping condition is: the difference between the first weight coefficients in two adjacent iterations is less than or equal to the convergence value, and the difference between the second weight coefficients in two adjacent iterations is less than or equal to the convergence value.

7. The method for extracting stereoscopic visual features from an image as described in claim 1, characterized in that, The monocular depth estimation model adopts the MoGe-2 model, the three-dimensional vision basic model adopts the VGGT model, and the DINO model adopts the DINOv2 model.

8. An electronic device, characterized in that, include: The memory stores execution instructions; as well as A processor that executes execution instructions stored in the memory, causing the processor to perform the stereoscopic visual feature extraction method for an image according to any one of claims 1 to 7.

9. A readable storage medium, characterized in that, The readable storage medium stores execution instructions, which, when executed by a processor, are used to implement the stereoscopic visual feature extraction method for the image according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the stereoscopic visual feature extraction method for the image according to any one of claims 1 to 7.

Images