Multi-view three-dimensional reconstruction method, device and medium based on cross-domain feature fusion

Abstract

The application belongs to the technical field of computer vision and computer graphics, and discloses a multi-view three-dimensional reconstruction method based on cross-domain feature fusion, equipment and medium, the method comprises the following steps: cross-feature domain coding is carried out on the initial feature token generated by multi-angle images, the coding decomposes the feature into low-frequency and high-frequency components through dual-tree complex wavelet transform, and the amplitude modulation is carried out on the high-frequency component to enhance the details, and the space-frequency fusion feature is generated; the space-frequency fusion features of the multi-angle are weighted and merged through variance embedding, the weight is generated by calculating the variance of each feature token, and adaptive weighted clustering is carried out based on the weight to generate the fused multi-angle feature; three-source attention decoding is carried out on the fused feature, the static embedding is taken as the query, the space-frequency fusion feature is taken as the key, and the fused multi-angle feature is taken as the value, cross-domain attention calculation and up-sampling are carried out, and finally the three-dimensional voxel reconstruction result of the target object is generated.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and computer graphics, and particularly to a multi-view method based on cross-domain feature fusion. Figure Three Reconstruction methods, equipment, and media. Background Technology

[0002] Multiview Figure Three 3D reconstruction is a core task in computer vision, aiming to recover the three-dimensional geometry of an object from multiple two-dimensional images. This technology has broad application prospects in many fields such as autonomous driving, cultural heritage digitization, virtual reality, and biomedical imaging. Existing 3D reconstruction methods can be categorized based on their output representation, including voxel-based, mesh-based, point cloud-based, and implicit representation methods. Among these, voxel-based methods have attracted considerable attention due to their regular structural representation.

[0003] In recent years, with the great success of the Transformer model in natural language processing and 2D vision tasks, its architecture has also been introduced into the field of 3D reconstruction. Multi-view based on Transformer... Figure Three Dimensional reconstruction methods, such as LRGT and VTR, demonstrate superior performance by capturing long-range dependencies between different view features through powerful self-attention mechanisms. However, these methods still have some inherent limitations.

[0004] First, existing methods primarily rely on spatial domain features extracted from images, such as color, texture, and edges, while neglecting frequency domain information. Frequency domain features, especially high-frequency components, encode fine structural details and texture variations in images, which are crucial for high-quality geometric reconstruction. Second, in the multi-view feature fusion stage, existing methods typically employ feature clustering based on fixed metrics such as Euclidean distance. This strategy assigns equal weight to all feature tokens, ignoring the contribution of individual tokens to variance, which may dilute highly discriminative key features during fusion, introducing unnecessary geometric biases. Finally, when the number of input views is limited or sparse, the overlap between cross-view features decreases, making it difficult for the model to establish robust global associations, resulting in a significant drop in reconstruction performance.

[0005] Therefore, effectively fusing frequency domain information to enhance detail representation, designing more reasonable feature fusion weight strategies to retain key features, and improving the model's reconstruction robustness under limited viewpoints are current challenges in multi-view reconstruction. Figure Three Problems that urgently need to be solved in the field of reconstruction. Summary of the Invention

[0006] The purpose of this invention is to overcome the above-mentioned defects of the prior art and provide a multi-view based on cross-domain feature fusion. Figure ThreeThe method, device and medium are used for reconstructing a three-dimensional object, and are used to solve the problems of over-reliance on spatial domain features, geometric deviation of feature fusion strategies and performance decline under limited viewing angles in existing reconstruction methods.

[0007] In a first aspect, the present application provides a multi-view three-dimensional object reconstruction method based on cross-domain feature fusion. Figure Three The method for reconstructing a three-dimensional object comprises the following steps:

[0008] S1: obtaining a plurality of RGB images of a target object, and respectively segmenting and projecting the plurality of RGB images to obtain initial feature tokens;

[0009] S2: performing cross-feature domain encoding processing on the initial feature tokens, the cross-feature domain encoding processing comprising: decomposing the initial feature tokens into low-frequency components and high-frequency components by using a dual-tree complex wavelet transform, performing amplitude-phase decomposition and amplitude modulation processing on the high-frequency components to enhance structural details, and reconstructing the processed low-frequency components and the high-frequency components to generate spatial-frequency fusion features containing spatial domain information and frequency domain information;

[0010] S3: performing variance embedding weighted merging processing on the spatial-frequency fusion features from multiple viewing angles, the variance embedding weighted merging processing comprising: calculating the variance of each feature token in the spatial-frequency fusion features to generate variance embedding weights, and adaptively weighting clustering the spatial-frequency fusion features from different viewing angles based on the variance embedding weights to generate fused multi-view features;

[0011] S4: performing decoding processing on the fused multi-view features by using a three-source attention mechanism, the three-source attention mechanism comprising: taking a group of static learnable embeddings as a query, taking the spatial-frequency fusion features as a key, and taking the fused multi-view features as a value, performing cross-domain attention calculation to obtain decoding features, and performing upsampling on the decoding features to generate and output a three-dimensional voxel reconstruction result of the target object.

[0012] As an optional implementation manner of the first aspect of the present application, in the S2, the specific process of the cross-feature domain encoding processing is as follows: performing first layer normalization processing on the input initial feature tokens to obtain first normalized features; inputting the first normalized features into a cross-feature domain layer for processing to obtain features rich in spatial frequency information; fusing the features rich in spatial frequency information and the first normalized features through a first residual connection to obtain first fusion features; after performing second layer normalization processing on the first fusion features, inputting the first fusion features into a convolution feedforward network to model local spatial dependency to obtain locally enhanced features; fusing the locally enhanced features and the first fusion features through a second residual connection to generate and output the spatial-frequency fusion features.

[0013] As an optional implementation of the first aspect of the application, the first normalized feature is input to a cross-feature domain layer for processing to obtain a feature rich in spatial frequency information, specifically including: using the dual-tree complex wavelet transform to decompose the first normalized feature into the low-frequency component and the high-frequency component; performing adaptive weighted processing on the low-frequency component, the calculation formula of which is: ; wherein, is the processed low-frequency component, is the original low-frequency component, is a learnable weight matrix; performing the amplitude modulation processing on the high-frequency component; using the inverse dual-tree complex wavelet transform to reconstruct the adaptive weighted processed low-frequency component and the amplitude modulated high-frequency component to obtain the feature rich in spatial frequency information.

[0014] As an optional implementation of the first aspect of the application, in S2, the specific process of performing amplitude-phase decomposition and amplitude modulation processing on the high-frequency component is: decomposing the high-frequency component in complex form into its amplitude and phase, the calculation formulas of the amplitude and the phase being respectively: ; wherein, is the real part of the high-frequency component, is the imaginary part of the high-frequency component, is a constant for preventing numerical instability; using two learnable weight matrices to modulate the amplitude to obtain modulated amplitudes, the calculation formulas of which are respectively: ; wherein, is the modulated amplitude obtained by the learnable weight , is the modulated amplitude obtained by the learnable weight , and the dimensions of which correspond to the directional structure and channel structure of the high-frequency component; combining the modulated amplitudes with the original phase to reconstruct the modulated high-frequency component in complex form, the calculation formulas of which are respectively: ; splicing the real part and the imaginary part of the reconstructed complex high-frequency component, and converting them into a floating-point tensor to obtain the amplitude modulated high-frequency component.

[0015] As an optional implementation manner of the first aspect of the present application, in the S3, the specific process of adaptively weighting and clustering the spatial-frequency fusion features from different perspectives based on the variance embedding weight is: calculating the standard deviation of each feature token in the spatial-frequency fusion features in the feature dimension; normalizing the standard deviations of all feature tokens to obtain the relative weight of each feature token; element-level multiplying the relative weight with the corresponding feature token to generate a weighted feature; when performing multi-perspective feature clustering, using the weighted feature to perform adaptive distance calculation instead of Euclidean distance calculation, thereby guiding the clustering process and generating a clustering index.

[0016] As an optional implementation manner of the first aspect of the present application, the variance embedding weight merging processing in the S3 further includes: after the spatial-frequency fusion features of all perspectives are subjected to the second layer normalization processing, inputting them into a scoring multi-layer perceptron to generate an importance score for each feature token; based on the importance score and the clustering index, using the importance score as a weight to perform a weighted average on all feature tokens in each cluster, thereby merging the feature tokens of multiple perspectives into a feature token of a single perspective to form a preliminary fused feature map; inputting the preliminary fused feature map as a query, the spatial-frequency fusion features subjected to the second layer normalization processing as keys and values into a Transformer module to perform cross-view attention calculation, and finally outputting the fused multi-perspective features.

[0017] As an optional implementation manner of the first aspect of the present application, in the S4, the specific process of performing cross-domain attention calculation to obtain a decoding feature is: inputting the query, the key and the value into mutually independent layer normalization modules respectively for preprocessing to eliminate the distribution difference between different feature streams, thereby obtaining preprocessed query, key and value; inputting the preprocessed query, key and value into a scaled dot-product attention module for calculation, so that the attention weight focuses on the geometric detail area provided by the key, thereby obtaining an attention output; normalizing the attention output and inputting it into a feedforward network and a GELU activation function module for nonlinear transformation, so as to integrate the high-frequency detail information from the key and the global semantic information from the value, and finally output the decoding feature.

[0018] As an optional implementation manner of the first aspect of the present application, the specific process of obtaining the initial feature token is: setting the input RGB image resolution to , and dividing each image into non-overlapping image patches, wherein the number of non-overlapping image patches is ; linearly project each non-overlapping image patch into a pre-defined 768-dimensional space to generate a 768-dimensional feature token tensor, and take the feature token tensor as the initial feature token.

[0019] In a second aspect, an embodiment of the present application provides an electronic device, the electronic device comprising a processor, a memory, and a program or instructions stored on the memory and executable on the processor, and the program or instructions are executed by the processor to implement the steps of the method according to the first aspect.

[0020] In a third aspect, an embodiment of the present application provides a readable storage medium, the readable storage medium storing a program or instructions, and the program or instructions are executed by the processor to implement the steps of the method according to the first aspect.

[0021] Compared with the prior art, the present application has the following beneficial effects:

[0022] 1. By performing cross-feature domain processing in the encoding stage, using dual-tree complex wavelet transform and the unique amplitude modulation technology, the frequency domain information, especially the enhanced high-frequency details, is integrated into the feature representation, effectively improving the model's ability to capture fine geometric structures and alleviating the problem of insufficient detail information under limited viewing angles.

[0023] 2. By performing variance embedding weighted merging processing in the merging stage, the variance of the feature token is introduced as a dynamic weight into the clustering process, so that features with high discriminability and stability dominate in the fusion, reducing information dilution and geometric deviation, and improving the quality of feature fusion.

[0024] 3. By performing three-source attention decoding processing in the decoding stage, the sources of keys and values in the attention mechanism are decoupled, the frequency-enhanced features are used as keys to guide the attention to focus on structural details, while the context features are retained as values to maintain global semantics, achieving effective fusion of cross-domain information and improving the fidelity of the final reconstructed voxels. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 is a multi-view Figure Three Overall architecture diagram of the multi-view

[0026] Figure 2 is a cross-feature domain module (CFDB) structure diagram for performing cross-feature domain encoding processing in the present application;

[0027] Figure 3 is a structure diagram of the cross-feature domain layer (CFDL) and its amplitude modulation module in the present application;

[0028] Figure 4Fig. 1 is a schematic diagram of a comparison between a three-source attention mechanism (TSAM) decoding process in the present application and a traditional attention mechanism. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0030] The terms "first", "second", and the like in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein. In addition, "and / or" in the specification and claims indicates at least one of the connected objects, and the character " / ", generally indicates that the front and rear associated objects are in an "or" relationship. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise explicitly specified.

[0031] Embodiment 1

[0032] Please refer to Figure 1 , the multi-view Figure Three reconstruction method (3D-CFDA) based on cross-domain feature fusion proposed in the present application includes three core components: an encoder, a variance embedding weighted merger (VEWM), and a decoder.

[0033] The architecture realizes a complete reconstruction process from multi-view RGB images to three-dimensional (3D) voxels. The workflow is summarized as follows:

[0034] First, the encoder receives multi-view RGB images of the target object (such as each RGB image resolution is ). It processes the input images through a patch embedding layer. This layer divides the multi-view RGB images into non-overlapping image patches, and each non-overlapping image patch is projected into a 768-dimensional space. The number of patches is calculated according to the input size . From this, 196 feature token tensors are generated, each with a dimension of 768. These tokens are then processed through cascaded cross-feature domain modules (CFDB) and standard Transformer modules (TB).

[0035] Next, features from all perspectives are fed into the Variance Embedding Weighted Merger (VEWM). This component performs adaptive multi-view feature fusion, replacing the rigid weighting strategy in traditional methods with a token-based adaptive variance weighting strategy.

[0036] Finally, after receiving the fused multi-view features, the decoder uses the Three-Source Attention (TSAM) algorithm to reduce the dimensionality from... Dimensional reduction Then, the sample is input into the upsampling module to finally generate the target object. Results of three-dimensional voxel reconstruction.

[0037] Mathematically, the multi-view 3D reconstruction process of the 3D-CFDA model is defined by formula (1):

[0038]

[0039] Where I represents the input set consisting of n multi-view RGB images, each with a resolution of . ; , and These represent the encoder, VEWM, and decoder, respectively. This represents the final 3D voxel reconstruction output.

[0040] Specifically, the Cross-Feature Domain Module (CFDB), Variance Embedding Weighted Merger (VEWM), and Three-Source Attention Mechanism (TSAM) involved in this invention will be described in detail below.

[0041] I. Cross-Feature Domain Module (CFDB)

[0042] CFDB is the core enhancement unit of the encoder, fusing frequency domain details with spatial domain semantics through four sequential components, such as... Figure 2 As shown. Input the initial feature token (token) After the first layer of normalization (LayerNorm), the feature distribution across the channel dimension is stabilized, ensuring the consistency of the input range for subsequent frequency domain operations. The Cross-Feature Domain Layer (CFDL) processes the first normalized features: decomposing the features into low-frequency (LF) and high-frequency (HF) components. Element-wise multiplication is then performed on the LF components, while the HF components are input to the amplitude modulation module for amplitude-phase decomposition. Finally, the CFDL reconstructs the features back into the spatial domain, obtaining features with rich spatial frequency information while preserving the original shape. The convolutional feedforward network complements the CFDL by modeling local spatial dependencies: it projects the input features to 3072 dimensions through the first residual connection (Linear 1), employing... Deeply separable convolution (DWConv) captures local patch-level relationships, obtaining locally enhanced features, which are then projected back to 768-dimensional space through a second residual connection (Linear 2). The second residual connection (Linear 2) fuses the cross-feature domain layer (CFDL) with the output of the convolutional feedforward network and its corresponding input, preserving the original information while integrating enhanced features, generating and outputting spatial-frequency fusion features. DropPath regularization (Pdrop = 0.1) is used to prevent overfitting. In addition, in the two cross-feature domain modules (CFDB), the standard Transformer module (TB) captures cross-domain long-range dependencies through multi-head self-attention (MHSA).

[0043] The most critical component in the cross-feature domain layer (CFDL) is the dual-tree complex wavelet transform (DTCWT) and the proposed amplitude modulation module. The dual-tree complex wavelet transform (DTCWT) has the same advantages as the Fourier transform, including smoothness, non-oscillatory characteristics, amplitude response, and near-shift phase invariance, which is achieved by presenting a nearly linear phase response to signal phase shift. It also reduces aliasing effects, enhances the directional selectivity of wavelets in high-dimensional space, and improves the effectiveness of image edge and direction detection. Specifically, the dual-tree complex wavelet transform (DTCWT) processes six directional components, corresponding to and .

[0044] As shown in Figure 3 , the dual-tree complex wavelet transform (DTCWT) uses two sets of parallel real-valued wavelet transforms: one set generates the real part of the transform output, and the other set generates the imaginary part. These two sets of discrete wavelet transforms approximately implement the complete complex wavelet transform through the use of a collaboratively designed filter bank, while meeting the reconstruction criteria. Subsequently, high-frequency (HF) and low-frequency (LF) features are extracted through the dual-tree complex wavelet transform (DTCWT). High-frequency features encode fine information (such as texture, patterns, and subtle details), while low-frequency features capture global information (including overall brightness, contrast, and object edges and contours). To make the low-frequency (LF) and high-frequency (HF) features learnable, the low-frequency features are processed through element-level multiplication with learnable weights , as shown in equation (2):

[0045]

[0046] wherein is the processed low-frequency component, H represents height, W represents width, and C represents channel.

[0047] Further, for high-frequency signals, an amplitude modulation module is designed, which uses amplitude-phase decomposition technology. The specific process is shown in Figure 3 . The weight matrix By and consisting of, is specifically designed to modulate the amplitude of high-frequency components while preserving their phase information. The extracted high-frequency features are represented in complex form, containing real part and imaginary part . The amplitude modulation module explicitly decomposes the complex features into amplitude and phase components. As shown in equation (3), is a constant used to prevent numerical instability. This decomposition separates the intensity information (amplitude) of fine high-frequency features from the location information (phase), enabling the network to focus on adjusting feature intensity while preserving the spatial directional cues encoded in the phase.

[0048]

[0049] where, denotes the amplitude, denotes the phase.

[0050] To balance the recognition rate and noise suppression, the amplitude modulation module employs learnable weights and directly acting on the amplitude component. As shown in equation (4), and the dimensions of and correspond to the spatial structure (6 directions, ) of high-frequency features, as well as the channel structure.

[0051]

[0052] where, is the modulated amplitude obtained by learnable weight , is the modulated amplitude obtained by learnable weight .

[0053] Under the premise of maintaining the integrity of the phase, the amplitude adjustment process is simple and clear, as shown in equation (5). After the amplitude modulation is completed, the modified amplitude information is combined with the original phase information to reconstruct the complex high-frequency features:

[0054]

[0055] where and denote the real part and imaginary part of the reconstructed complex high-frequency components;

[0056] The reconstruction process ensures the preservation of the original spatial orientation information of the high-frequency features by maintaining the phase, and achieves adaptive adjustment of the intensity thereof by modulating the amplitude. After the amplitude and phase processing, the real part and the imaginary part of the complex feature are spliced along the last dimension, so as to restore the complete complex representation. The final result is converted into a floating-point tensor for subsequent network processing.

[0057] II. Variance-embedded weighted merger (VEWM)

[0058] Existing three-dimensional reconstruction multi-view feature fusion strategies can be roughly divided into two categories: convolution pooling-based methods and attention mechanism-based methods. Convolution pooling methods (such as Pix2Vox++ and GARNet) often have the problem of loss of high-frequency details, and have limitations in capturing fine spatial structures. In contrast, attention mechanism-based methods (including LegoFormer, 3D-C2FT, UMIFormar and LRGT) can more effectively preserve local structural features and take into account the diversity of multi-view features. Therefore, we choose the attention mechanism-based fusion method as the basic framework of this design. However, existing attention-based methods still do not fully focus on identifying and prioritizing tokens according to the variance contribution of tokens under different views. For example, UMIFormer and LRGT group tokens based on feature similarity, but lack a weight strategy to further evaluate the intrinsic information value of each token. All tokens are given the same weight through Euclidean distance, which ignores the intrinsic geometric relationship of tokens between different views, thereby introducing geometric bias, resulting in subtle features with distinguishing information between different views being suppressed. This uniform weight method causes information dilution in the aggregation process, thereby affecting the fidelity of the reconstruction.

[0059] The variance-embedded weighted merger (VEWM) of the present application directly incorporates the variance feature of each token into the distance calculation. Its core innovation lies in the variance-embedded weighting stage.

[0060] First, calculate the standard deviation of each feature in the feature dimension: ; where denotes the standard deviation function, denotes the feature, denotes the dimension index of the calculation, and D denotes the feature dimension;

[0061] Next, normalize these variances to obtain relative weights: ) ; where N denotes the number of features, denotes a very small positive number, and its core role is to avoid a zero denominator;

[0062] Then, generate the weighted feature: . This weighted feature representation is then used to guide the adaptive distance computation: where represents the core function in the machine learning library for batch computation of distance between two sets of data points, represents the features that have been multiplied by the weights.

[0063] As shown in Table 1, the Variance Embedding Weighted Merger (VEWM) contains three key stages: first, under the guidance of the scores generated by the Score MLP, the high variability tokens dominate the cluster centers. From the input feature tensor of size (16 batches, 3 views, 196 patches, 768 dimensions), the normalization layer normalizes the features without changing the dimensions. Subsequently, the Score MLP generates importance scores that guide the clustering step to assign all 588 tokens into 196 clusters. This mechanism ensures that each cluster anchors key view-specific information.

[0064] Second, the tokens within each cluster are merged by weighted averaging. Using the scores generated by the Score MLP as weights, the 588 tokens are merged into 196 clusters, generating the fused feature map. This method preserves salient features by emphasizing high-score tokens. Finally, the Transformer module reinforces cross-view dependencies. The fused feature tensor serves as the query, interacting with the original feature tensor as the key-value through the attention mechanism. This mechanism achieves a balance between global consistency and local uniqueness by optimizing the output.

[0065] Table 1. Variance Embedding Weighted Merger (VEWM) processing flow and feature dimension changes data

[0066]

[0067] Three, Tri-Source Attention Mechanism (TSAM)

[0068] After multi-view feature fusion, the task of the decoder is to synthesize the final 3D voxel representation. Common Transformer-based decoders, such as 3D-RETR, 3D-C2FT, LRGT, and VTR, initialize this process with a set of fixed learnable embedding vectors. This usually involves projecting the fused feature sequence 16 (e.g., length 196) to a smaller fixed query set (e.g., length 64) through an embedding layer. However, this approach fails to fully utilize the rich instance-specific information in the fused features, forcing subsequent attention layers to compensate for this barren initialization. Therefore, the decoder lacks flexibility in dynamically adjusting the representation budget according to the complexity of the target object.

[0069] Traditional methods adopt static learnable embeddings as queries while keys and values are derived from context features processed by a combiner. Although this key-value co-design is simple and straightforward, it limits the expressive power of the model because it forces the model to simultaneously derive attention weights and feature content from a single feature distribution. As shown in Figure 4 To break this limitation, the present application proposes a three-source attention mechanism (TSAM) at the decoding stage, the core innovation of which lies in decoupling the sources of keys and values and introducing three independent feature streams: queries still adopt static learnable embeddings to ensure the stability of the decoding process; keys are extracted from CFDL, which contains rich high-frequency structure and fine detail information; and values are still derived from the original context features to maintain global semantic consistency.

[0070] The specific calculation logic of the three-source attention mechanism (TSAM) can be further divided into three stages. First, independent preprocessing of feature streams: exclusive layer normalization (LayerNorm) is applied to queries Q (generated by static embeddings), keys K (CFDL features), and values V (context features). This design avoids interference from distribution differences of different feature streams, ensuring that the detailed information of high-frequency features (keys K) will not be diluted by the global distribution of context features (values V). Second, attention calculation focusing on details: the standard scaled dot-product attention formula However, since keys K are derived from high-frequency features of the CFDL layer, the attention weights will naturally focus on geometric detail areas. Third, cross-domain feature post-fusion: after layer normalization, the attention output is input into the FFN+GELU module. Nonlinear transformation further integrates high-frequency detail information (from keys K) and global semantic information (from values V), finally outputting [64, 768]-dimensional decoding features that provide a feature basis with both fine detail and semantic consistency for voxel reconstruction. The effectiveness of this design is verified through ablation experiments. At the same time, the three-source attention mechanism (TSAM) can be integrated into existing methods (such as LRGT), where the three-source attention mechanism (TSAM) serves as a general enhancement module. This design achieves performance improvement through two core mechanisms: first, frequency-enhanced features are used as key elements to improve the model's ability to capture fine structural patterns. Second, by separating key streams and value streams, more rich feature fusion is promoted, enabling the model to integrate complementary information from different feature domains.

[0071] Optionally, the embodiments of the present application also provide an electronic device, which includes a processor, a memory, a program or instructions stored in the memory and executable on the processor, and the program or instructions are executed by the processor to implement the above-mentioned multi-view volume reconstruction method based on cross-domain feature fusion. Figure ThreeThe various processes of the embodiment of the method for reconstructing the feature vector are implemented, and the same technical effects can be achieved. To avoid repetition, details are not described herein.

[0072] The embodiment of the method for reconstructing the feature vector is implemented, and the same technical effects can be achieved. To avoid repetition, details are not described herein. Figure Three The various processes of the embodiment of the method for reconstructing the feature vector are implemented, and the same technical effects can be achieved. To avoid repetition, details are not described herein.

[0073] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk, or an optical disk.

[0074] It should be noted that in this document, the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusions, so that a process, method, article, or device that includes a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such a process, method, article, or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article, or device that includes the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, but can also include performing functions in a substantially simultaneous manner or in reverse order, for example, the described method can be performed in an order different from that described, and various steps can be added, omitted, or combined. In addition, features described with reference to certain examples can be combined in other examples.

[0075] From the above description of the embodiments, those skilled in the art can clearly understand that the above embodiment method can be implemented by software and a necessary general hardware platform, of course, it can also be implemented by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, or an optical disk), and includes a plurality of instructions for causing a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method described in each embodiment of the present application.

[0076] The embodiments of the present application are described above with reference to the accompanying drawings, but the present application is not limited to the specific embodiments described above, and the specific embodiments described above are merely illustrative, but not restrictive, and a person of ordinary skill in the art can make many forms under the inspiration of the present application without departing from the purpose of the present application and the scope protected by the claims.

Claims

1. A multi-view 3D reconstruction method based on cross-domain feature fusion, characterized in that, Includes the following steps: S1: Acquire multiple RGB images of the target object, and segment and project the multiple RGB images respectively to obtain an initial feature token; S2: Perform cross-feature domain encoding processing on the initial feature token. The cross-feature domain encoding processing includes: using dual-tree complex wavelet transform to decompose the initial feature token into low-frequency components and high-frequency components; performing amplitude-phase decomposition and amplitude modulation processing on the high-frequency components to enhance structural details; and reconstructing the processed low-frequency components and the high-frequency components to generate a spatial-frequency fusion feature containing spatial domain information and frequency domain information. Specifically, the initial feature token is subjected to a first-layer normalization process to obtain a first normalized feature; the first normalized feature is input to a cross-feature domain layer for processing to obtain features enriched with spatial frequency information; the features enriched with spatial frequency information are fused with the first normalized feature through a first residual connection to obtain a first fused feature; the first fused feature is subjected to a second-layer normalization process and then input to a convolutional feedforward network to model local spatial dependencies to obtain local enhanced features; the local enhanced features are fused with the first fused feature through a second residual connection to generate and output the spatial-frequency fusion feature. S3: Perform variance embedding weighted merging processing on the spatial-frequency fusion features from multiple perspectives. This variance embedding weighted merging process includes: calculating the variance of each feature token in the spatial-frequency fusion features to generate variance embedding weights, and performing adaptive weighted clustering on the spatial-frequency fusion features from different perspectives based on the variance embedding weights to generate fused multi-perspective features. Specifically, the process of adaptive weighted clustering on the spatial-frequency fusion features from different perspectives based on the variance embedding weights is as follows: calculating the standard deviation of each feature token in the spatial-frequency fusion features along the feature dimension; standardizing the standard deviations of all feature tokens to obtain the relative weight of each feature token; multiplying the relative weights element-wise with the corresponding feature tokens to generate weighted features; and using the weighted features for adaptive distance calculation instead of Euclidean distance calculation during multi-perspective feature clustering to guide the clustering process and generate a clustering index. S4: The fused multi-view features are decoded using a three-source attention mechanism. The three-source attention mechanism includes: using a set of static learnable embeddings as a query, the spatial-frequency fusion features as a key, and the fused multi-view features as a value, performing cross-domain attention calculation to obtain decoded features, upsampling the decoded features, generating and outputting the three-dimensional voxel reconstruction result of the target object.

2. The method according to claim 1, characterized in that, The first normalized feature is input into a cross-feature domain layer for processing to obtain features rich in spatial frequency information, specifically including: The first normalized feature is decomposed into the low-frequency component and the high-frequency component using the dual-tree complex wavelet transform; The low-frequency components are subjected to adaptive weighting processing, and the calculation formula is as follows: ； in, The processed low-frequency components, The original low-frequency component, The weight matrix is ​​a learnable weight matrix; The high-frequency components are subjected to the amplitude modulation process described above; The low-frequency components after adaptive weighting and the high-frequency components after amplitude modulation are reconstructed by using inverse dual-tree complex wavelet transform to obtain the features that enrich the spatial frequency information.

3. The method according to claim 1, characterized in that, In step S2, the specific process of amplitude-phase decomposition and amplitude modulation of the high-frequency components is as follows: The high-frequency component in complex form is decomposed into its amplitude and phase, wherein the amplitude and phase The calculation formulas are as follows: ； in, Let be the real part of the high-frequency component. Let be the imaginary part of the high-frequency component. A constant used to prevent numerical instability; The amplitude is modulated using two learnable weight matrices to obtain the modulated amplitude, and the calculation formulas are as follows: ； in, Learnable weights The obtained modulated amplitude, Learnable weights The obtained modulated amplitude, and The dimensions correspond to the directional structure and channel structure of the high-frequency components; The modulated amplitude is combined with the original phase to reconstruct the complex form of the modulated high-frequency component, and the calculation formulas are as follows: ； The real part of the reconstructed complex high-frequency components With the imaginary part The components are spliced ​​together and converted into floating-point tensors to obtain the high-frequency components after amplitude modulation processing.

4. The method according to claim 1, characterized in that, The variance embedding weighted merging process in S3 also includes: After performing a second-layer normalization process on the spatial-frequency fusion features of all perspectives, the data is input into a scoring multilayer perceptron to generate an importance score for each feature token. Based on the importance score and the clustering index, within each cluster, the importance score is used as a weight to perform a weighted average of all feature tokens within the cluster, merging feature tokens from multiple perspectives into feature tokens from a single perspective, forming a preliminary fused feature map. The preliminary fused feature map is used as a query, and the spatial-frequency fused features after the second layer of normalization are used as keys and values. These are then input into the Transformer module for cross-view attention calculation, and finally the fused multi-view features are output.

5. The method according to claim 1, characterized in that, In step S4, the specific process of performing cross-domain attention calculation to obtain decoding features is as follows: The query, the key, and the value are respectively input into independent layer normalization modules for preprocessing to eliminate the distribution differences between different feature flows and obtain the preprocessed query, key, and value. The preprocessed query, key, and value are input into the scaled dot product attention module for calculation, so that the attention weights are focused on the geometric detail region provided by the key, and the attention output is obtained. The attention output is normalized and input into the feedforward network and GELU activation function module for nonlinear transformation to integrate high-frequency detail information from the key and global semantic information from the value, and finally outputs the decoded features.

6. The method according to claim 1, characterized in that, The specific process for obtaining the initial feature token is as follows: Set the input RGB image resolution to Segment each image into The number of non-overlapping image patches is . ; Each non-overlapping image patch is linearly projected into a preset 768-dimensional space to generate a feature token tensor with a dimension of 768, and this feature token tensor is used as the initial feature token.

7. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein when the program or instructions are executed by the processor, they implement the steps of a multi-view 3D reconstruction method based on cross-domain feature fusion as described in any one of claims 1-6.

8. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions, which, when executed by a processor, implement the steps of a multi-view 3D reconstruction method based on cross-domain feature fusion as described in any one of claims 1-6.

Images