Dynamic scale feature extraction and Transform combined 3D scene reconstruction device

Abstract

The invention provides a 3D (three-dimensional) scene reconstruction device combining dynamic scale feature extraction and Transform. Comprising an image acquisition module, a preprocessing module, a dynamic scale feature extraction module, a Transform coding module, a feature aggregation module, a decoding module, a post-processing module, a loss calculation module, a model optimization module, an image registration module, a depth estimation module, a geometric information extraction module, a spatio-temporal information fusion module, a feature enhancement module and an output module. Wherein the image acquisition module acquires RGB-D image sequences of a non-rigid dynamic target by using a monocular camera, and the image sequences contain rich scene information and provide basic data for subsequent three-dimensional reconstruction; the preprocessing module carries out denoising, image enhancement, image registration and other preprocessing on the acquired RGB-D image. The dynamic scale feature extraction and Transform combined 3D scene reconstruction device provided by the invention has the advantages of high precision, high efficiency, high coherence and consistency and the like, and can meet the requirements of different application scenes.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and machine learning, and in particular to a 3D scene reconstruction device combining dynamic scale feature extraction with a Transformer. Background Art

[0002] 3D scene reconstruction is an important research direction in the field of computer vision and graphics. It aims to recover the structure and geometric information of 3D scenes from 2D image data. This technology has broad application prospects in virtual reality, augmented reality, autonomous driving, robot navigation and cultural relics protection. However, traditional 3D reconstruction methods often rely on complex algorithms such as multi-view geometry and stereo vision, and have limited reconstruction effects on non-rigid dynamic targets. With the rapid development of deep learning technology, especially the emergence of models such as convolutional neural networks (CNN) and Transformer, 3D reconstruction technology has made significant progress.

[0003] In the deep learning framework, 3D reconstruction usually includes multiple steps such as feature extraction, feature encoding, 3D reconstruction and post-processing. Feature extraction is the key link in 3D reconstruction, which determines the accuracy and robustness of subsequent reconstruction. However, traditional feature extraction methods often use convolution kernels of fixed scales, which are difficult to adapt to features of different scales. In addition, for non-rigid dynamic targets, their shapes and textures change over time, which further increases the difficulty of feature extraction.

[0004] In order to solve this problem, the dynamic scale feature extraction method came into being. Dynamic Scale Convolutional Neural Network (Dynamic Scale CNN) can adaptively adjust the scale of the convolution kernel according to the characteristics of the input image, thereby effectively extracting features of different scales. This method not only improves the accuracy of feature extraction, but also enhances the model's adaptability to complex scenes.

[0005] On the other hand, the Transformer model has achieved great success in the field of natural language processing, and its powerful feature encoding capabilities have attracted widespread attention in the field of computer vision. The Transformer can capture the global relationship between features through the multi-head self-attention mechanism and improve the expressiveness of feature representation. Introducing the Transformer into the field of 3D reconstruction can achieve efficient encoding and decoding of features, further improving the accuracy and efficiency of 3D reconstruction.

[0006] However, relying solely on dynamic scale feature extraction and Transformer encoding is not enough to achieve high-quality 3D reconstruction. In practical applications, it is also necessary to consider multiple aspects such as depth estimation, geometric information extraction, spatiotemporal information fusion and feature enhancement. Depth estimation can provide geometric structure information of the scene and is an important basis for 3D reconstruction. Geometric information extraction can extract geometric information such as surface normals and curvature from the depth map, which helps to restore the surface details of the scene. Spatiotemporal information fusion can capture the dynamic changes of the scene and improve the coherence and consistency of the reconstruction results. Feature enhancement can use additional information (such as lighting conditions, material information, etc.) to enhance features and improve the expressiveness and robustness of features.

[0007] In summary, although the existing three-dimensional reconstruction technology has made significant progress under the deep learning framework, it still faces many challenges. The present invention proposes a 3D scene reconstruction device based on dynamic scale feature extraction and Transformer, which aims to solve the problems of large computational complexity and insufficient reconstruction accuracy in traditional three-dimensional reconstruction methods when dealing with complex scenes, and improve the accuracy, efficiency and consistency of reconstruction. By introducing advanced technologies such as dynamic scale convolutional neural networks, Transformer encoding modules, feature aggregation modules, spatiotemporal information fusion modules and feature enhancement modules, the present invention can achieve high-quality three-dimensional reconstruction of non-rigid dynamic targets to meet the needs of different application scenarios. Summary of the invention

[0008] The present invention aims to solve the problems raised in the background technology and provides a 3D scene reconstruction device combining dynamic scale feature extraction and Transformer, aiming to improve the accuracy and efficiency of 3D scene reconstruction through dynamic scale feature extraction and efficient Transformer structure.

[0009] The specific technical solutions are as follows:

[0010] A 3D scene reconstruction device combining dynamic scale feature extraction and Transformer, comprising:

[0011] Image acquisition module, preprocessing module, dynamic scale feature extraction module, Transformer encoding module, feature aggregation module, decoding module, post-processing module, loss calculation module, model optimization module, image registration module, depth estimation module, geometric information extraction module, spatiotemporal information fusion module, feature enhancement module and output module, among which:

[0012] The image acquisition module is used to acquire an RGB-D image sequence of a non-rigid dynamic target through a single sensor or multiple sensors, is connected to the preprocessing module, and transmits the acquired RGB-D image sequence to the preprocessing module;

[0013] The preprocessing module is connected to the dynamic scale feature extraction module and the image registration module respectively, and performs denoising and image enhancement preprocessing on the received RGB-D image sequence;

[0014] For the RGB-D image to be registered, the preprocessing module first performs denoising and image enhancement preprocessing on the received RGB-D image sequence, and then passes the preprocessed RGB-D image to the image registration module for registration, and the image registration module passes the registered image to the dynamic scale feature extraction module;

[0015] For RGB-D images that do not require registration, the preprocessing module performs denoising and image enhancement preprocessing on the received RGB-D image sequence, and then directly passes the preprocessed RGB-D image to the dynamic scale feature extraction module;

[0016] The dynamic scale feature extraction module is connected to the Transformer encoding module, and uses a dynamic scale convolutional neural network to extract features from the received preprocessed or registered image, and transmits the extracted features to the Transformer encoding module;

[0017] The Transformer encoding module is connected to the feature aggregation module, which uses a multi-head self-attention mechanism to encode the extracted features to capture the global relationship between the features, and uses position encoding technology to perceive the spatial structure of the image, and passes the encoded features to the feature aggregation module;

[0018] The feature aggregation module is connected to the decoding module, and uses the decoder structure of the Transformer to aggregate the encoded features to fuse features of different scales and different perspectives, generate a complete and consistent three-dimensional feature volume, and pass the features of the aggregated three-dimensional feature volume to the decoding module;

[0019] The decoding module is connected to the post-processing module, which generates three-dimensional point cloud data according to the aggregated features and transmits the generated three-dimensional point cloud data to the post-processing module;

[0020] The post-processing module is connected to the loss calculation module and the output module respectively, and performs smoothing, denoising and gridding on the generated three-dimensional point cloud data to obtain a reconstructed three-dimensional scene, and then passes the reconstructed three-dimensional scene to the loss calculation module (for loss calculation in the training process) and the output module (for output of the final result);

[0021] The loss calculation module is connected to the model optimization module, and calculates the difference between the reconstructed three-dimensional scene and the real scene, and calculates the loss value and passes it to the model optimization module;

[0022] The model optimization module is connected to the dynamic scale feature extraction module and the Transformer encoding module respectively, and is used to receive the loss value output by the loss calculation module, optimize the model parameters according to the loss value, and then feed the optimized parameters back to the dynamic scale feature extraction module and the Transformer encoding module (in the training loop);

[0023] The image registration module is connected to the image acquisition module, the preprocessing module and the dynamic scale feature extraction module respectively, and is used to receive the output of the image acquisition module (for the image to be registered) in the preprocessing stage or before feature extraction, and pass the registered image to the preprocessing module or the dynamic scale feature extraction module (depending on whether the registration is performed before or after the preprocessing);

[0024] The depth estimation module is connected to the image acquisition module, the preprocessing module, the geometric information extraction module and the feature aggregation module respectively, and is used to receive the RGB-D image and generate a depth map to pass to the geometric information extraction module or the feature aggregation module (if the depth information is used as an additional feature);

[0025] The geometric information extraction module is connected to the depth estimation module, the feature aggregation module and the decoding module respectively, and is used to receive the output of the depth estimation module, and extract the geometric information and pass it to the feature aggregation module or the decoding module (if the geometric information is used as an additional feature or for the decoding process);

[0026] The spatiotemporal information fusion module is respectively connected to the dynamic scale feature extraction module, the decoding module and the feature aggregation module, and is used to receive the spatiotemporal information from the dynamic scale feature extraction module and the decoding module, and perform fusion processing, and then pass the fused data to the feature aggregation module or the decoding module (depending on whether the fusion is performed before or after the feature aggregation);

[0027] The feature enhancement module is connected to the dynamic scale feature extraction module, the Transformer encoding module, the Transformer coding module and the feature aggregation module respectively, and is used to receive feature input, and enhance the features using additional information (such as lighting conditions, material information, etc.), and then pass the enhanced features to the Transformer encoding module or the feature aggregation module (depending on whether the enhancement is performed before encoding or aggregation);

[0028] The output modules are respectively connected to the post-processing modules, and are used to receive the output of the post-processing modules and output the reconstructed three-dimensional scene data in multiple formats.

[0029] The above-mentioned 3D scene reconstruction device combining dynamic scale feature extraction and Transformer, wherein: the decoding module gradually generates a fine three-dimensional point cloud by decoding layer by layer, thereby realizing scene reconstruction.

[0030] The above-mentioned 3D scene reconstruction device combining dynamic scale feature extraction and Transformer, wherein: the differences calculated by the loss calculation module include geometric loss, color loss and consistency loss. Through loss calculation, feedback can be provided to the model optimization module to guide the update of the model optimization module parameters.

[0031] The above-mentioned 3D scene reconstruction device combining dynamic scale feature extraction and Transformer, wherein: the model optimization module optimizes the model parameters using an optimization algorithm based on the loss calculation result to improve the reconstruction accuracy and efficiency, and the optimization algorithm includes stochastic gradient descent (SGD), Adam, etc.

[0032] The above-mentioned 3D scene reconstruction device combining dynamic scale feature extraction and Transformer, wherein: the registration process of the image registration module includes the steps of feature point detection, feature point matching and image transformation.

[0033] The above-mentioned 3D scene reconstruction device combining dynamic scale feature extraction and Transformer, wherein: the depth estimation module uses deep learning technology, such as convolutional neural network (CNN), to estimate the depth of the image. Depth information is crucial for three-dimensional reconstruction and can provide geometric structure information of the scene.

[0034] The above-mentioned 3D scene reconstruction device combining dynamic scale feature extraction and Transformer, wherein: the geometric information extracted by the geometric information extraction module includes surface normals, curvatures, etc., which are helpful for subsequent three-dimensional reconstruction and restoration of surface details.

[0035] The above-mentioned 3D scene reconstruction device combining dynamic scale feature extraction and Transformer, wherein: the feature enhancement module uses additional information including lighting conditions, material information, etc. to enhance the extracted features. Through feature enhancement, the expressiveness and robustness of the features can be improved, thereby improving the reconstruction accuracy.

[0036] The above-mentioned 3D scene reconstruction device combining dynamic scale feature extraction and Transformer, wherein: the output module outputs the reconstructed three-dimensional scene in the form of a three-dimensional mesh, point cloud file, etc., and the output module also supports conversion and export of multiple formats to meet the needs of different applications.

[0037] The above-mentioned 3D scene reconstruction device combining dynamic scale feature extraction and Transformer, wherein: the dynamic scale feature extraction module uses a dynamic scale convolutional neural network to adaptively adjust the scale of the convolution kernel to adapt to features of different scales, thereby improving the accuracy and robustness of feature extraction.

[0038] The present invention has the following beneficial effects:

[0039] The 3D scene reconstruction device combining dynamic scale feature extraction and Transformer proposed in the present invention has the following technical effects:

[0040] 1. Improve reconstruction accuracy:

[0041] Through dynamic scale feature extraction and Transformer encoding module, features of different scales can be adaptively extracted and encoded, improving the accuracy and robustness of feature extraction, thereby improving the accuracy of 3D reconstruction.

[0042] 2. Enhance feature expression capabilities:

[0043] The feature aggregation module and feature enhancement module can fuse features of different scales and perspectives, and use additional information to enhance the features to improve the expressiveness and robustness of the features.

[0044] 3. Improve reconstruction efficiency:

[0045] The Transformer encoding and decoding modules use parallel computing methods, which can efficiently process large-scale data and improve the efficiency of 3D reconstruction.

[0046] 4. Improve reconstruction coherence and consistency:

[0047] The spatiotemporal information fusion module can capture the dynamic changes of the scene, fuse the information at different time points and different perspectives, and improve the coherence and consistency of the reconstruction results.

[0048] 5.Support multiple output formats:

[0049] The output module supports conversion and export in multiple formats to meet the needs of different applications and facilitate users to perform subsequent processing and analysis.

[0050] 6. Reduce hardware requirements:

[0051] Through the application of deep learning technology and optimization algorithms, the requirements for high-performance hardware can be reduced to a certain extent, making 3D reconstruction technology more popular and practical.

[0052] In summary, the 3D scene reconstruction device combining dynamic scale feature extraction and Transformer provided by the present invention has the advantages of high precision, high efficiency, high coherence and consistency, and can meet the needs of different application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A schematic diagram of the architecture of a 3D scene reconstruction device combining dynamic scale feature extraction and Transformer provided in an embodiment of the present invention;

[0054] Figure 2 This is a workflow diagram of a 3D scene reconstruction device combining dynamic scale feature extraction and Transformer provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The technical solution of the present invention is further described below with reference to the accompanying drawings and through specific implementation methods.

[0056] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on this patent; in order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; for those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0057] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "inner", "outer", etc. appear, the orientation or position relationship indicated is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as a limitation on this patent. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0058] In the description of the present invention, unless otherwise clearly specified and limited, if the term "connection" or the like appears to indicate the connection relationship between components, the term should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two components or the interaction relationship between two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0059] Example

[0060] The 3D scene reconstruction device provided in this embodiment combines dynamic scale feature extraction with Transformer, such as Figure 1-2 As shown, it includes an image acquisition module, a preprocessing module, a dynamic scale feature extraction module, a Transformer encoding module, a feature aggregation module, a decoding module, a post-processing module, a loss calculation module, a model optimization module, an image registration module, a depth estimation module, a geometric information extraction module, a spatiotemporal information fusion module, a feature enhancement module and an output module, wherein:

[0061] The image acquisition module uses a monocular camera to obtain RGB-D image sequences of non-rigid dynamic targets. These image sequences contain rich scene information and provide basic data for subsequent 3D reconstruction.

[0062] The preprocessing module performs denoising, image enhancement, image registration and other preprocessing on the collected RGB-D images. Through preprocessing, the image quality can be improved, noise interference can be reduced, and high-quality input can be provided for subsequent feature extraction and reconstruction;

[0063] The dynamic scale feature extraction module uses the dynamic scale convolutional neural network (Dynamic Scale CNN) to extract features from the preprocessed image. The dynamic scale feature extraction module uses the dynamic scale convolutional neural network to adaptively adjust the scale of the convolution kernel to adapt to features of different scales, thereby improving the accuracy and robustness of feature extraction;

[0064] The Transformer encoding module uses a multi-head self-attention mechanism to encode the extracted features. Through the self-attention mechanism, the Transformer encoding module can capture the global relationship between features and improve the expressiveness of feature representation. At the same time, the position encoding technology is used to perceive the spatial structure of the image.

[0065] The feature aggregation module uses the decoder structure of Transformer to aggregate the encoded features. Through feature aggregation, the feature aggregation module can fuse features of different scales and perspectives to generate a more complete and consistent three-dimensional feature volume;

[0066] The decoding module generates a detailed 3D point cloud layer by layer, and generates 3D point cloud data based on the aggregated features, thereby reconstructing the scene.

[0067] The post-processing module smoothes, denoises and meshes the generated 3D point cloud to improve the visual effect and accuracy of the reconstruction results.

[0068] The loss calculation module calculates the difference between the reconstruction result and the real scene. The difference includes geometric loss, color loss and consistency loss. Through loss calculation, it can provide feedback to the model optimization module and guide the update of the model optimization module parameters.

[0069] The model optimization module optimizes the model parameters based on the loss calculation results using optimization algorithms to improve reconstruction accuracy and efficiency. The optimization algorithms include stochastic gradient descent (SGD) and Adam.

[0070] The image registration module registers images from different perspectives. The registration process includes steps such as feature point detection, feature point matching, and image transformation for subsequent feature extraction and reconstruction.

[0071] The specific working principle of the image registration module is as follows:

[0072] The image registration module plays a vital role in the 3D scene reconstruction device that combines dynamic scale feature extraction with Transformer. Its main task is to accurately align images taken from different perspectives to ensure that subsequent feature extraction and 3D reconstruction can be based on consistent and accurate image data. The following is the specific working principle of the image registration module:

[0073] 1. Feature point detection

[0074] The first step in image registration is to detect points in each image that are distinctive and easy to identify. These points are usually called feature points. The feature point detection algorithm traverses each pixel of the image, calculates its local features (such as gradients, edges, corners, etc.), and determines which pixels are suitable as feature points based on these features. Common feature point detection algorithms include Harris corner detection, SIFT (Scale-Invariant Feature Transform), and SURF (Speeded Up Robust Features).

[0075] In this embodiment, a feature point detection algorithm with scale invariance and rotation invariance, such as SIFT or SURF, may be selected to adapt to complex scenes of non-rigid dynamic targets.

[0076] 2. Feature point matching

[0077] Once the feature points are detected, the next step is to match these feature points between images from different viewpoints. This is usually done by comparing the local descriptors of the feature points, i.e. feature vectors. The feature vectors describe the image texture and shape information around the feature points, making it possible to accurately match the same physical points even when the viewpoint, lighting or scale changes.

[0078] The feature point matching algorithm calculates the similarity between each pair of feature points (such as Euclidean distance, cosine similarity, etc.), and selects matching feature point pairs based on the similarity threshold. In order to improve the accuracy and robustness of the matching, algorithms such as RANSAC (Random Sample Consensus) can also be used to eliminate incorrect matches.

[0079] 3. Image Transformation

[0080] After obtaining the matching feature point pairs, the image registration module needs to calculate a transformation matrix to transform one image into the coordinate system of another image to achieve image alignment. This transformation matrix usually includes parameters such as translation, rotation, scaling, and affine transformation.

[0081] According to the coordinate information of the feature point pairs, the least square method or other optimization algorithms can be used to solve the transformation matrix. Once the transformation matrix is ​​determined, it can be applied to transform the image so that images from different perspectives can be accurately aligned together.

[0082] 4. Fine-tune

[0083] Although the transformation matrix can roughly achieve image alignment, in some cases, fine adjustments may be required to improve the accuracy of the alignment. This can be achieved through iterative optimization algorithms (such as ICP, Iterative Closest Point), which iteratively adjust the transformation matrix until the predetermined alignment accuracy is achieved.

[0084] 5. Output results

[0085] Finally, the image registration module outputs aligned image data, which will serve as the basis for subsequent feature extraction and 3D reconstruction. The aligned images will have higher consistency and accuracy, thereby improving the accuracy and robustness of the entire 3D reconstruction process.

[0086] In summary, the image registration module accurately aligns images from different perspectives through steps such as feature point detection, feature point matching, image transformation and fine adjustment, providing high-quality and consistent input data for subsequent steps such as dynamic scale feature extraction and Transformer encoding.

[0087] The depth estimation module uses deep learning techniques, such as convolutional neural networks (CNNs), to estimate the depth of images. Depth information is crucial for 3D reconstruction and can provide geometric structure information of the scene.

[0088] The geometric information extraction module extracts geometric information from the depth estimation results, including surface normals, curvature, etc. This information is helpful for subsequent 3D reconstruction and restoration of surface details;

[0089] The spatiotemporal information fusion module fuses information from different time points and different perspectives to improve the coherence and consistency of the reconstruction results. By fusing spatiotemporal information, the dynamic changes of the scene can be captured to generate a more realistic three-dimensional scene.

[0090] The feature enhancement module uses additional information to enhance the features, including lighting conditions, material information, etc., to enhance the extracted features. Through feature enhancement, the expressiveness and robustness of the features can be improved, thereby improving the reconstruction accuracy.

[0091] The output module outputs the reconstructed 3D scene in an appropriate form, such as 3D mesh, point cloud file, etc. The output module also supports conversion and export in multiple formats to meet the needs of different applications.

[0092] The depth estimation module uses deep learning techniques, such as convolutional neural networks (CNN), to estimate the depth of images. Depth information is crucial for 3D reconstruction and can provide geometric structure information of the scene.

[0093] Among them, the loss calculation module is used to calculate the difference between the reconstruction result and the real scene, including geometric loss, color loss and consistency loss. These losses can be expressed as:

[0094] L total =λ1L geo +λ2L color +λ3L cons

[0095] Among them, Lgeo is the geometric loss, Lcolor is the color loss, Lcons is the consistency loss, and λ1, λ2, and λ3 are weight coefficients.

[0096] The geometric loss can be calculated based on the distance between point clouds, for example using the Chamfer distance:

[0097] L geo =Σ i∈A min j∈B ||a i -b j || 2 +Σ j∈B min i∈A ||b j -a i || 2

[0098] Among them, A and B are the reconstructed point cloud and the real point cloud respectively.

[0099] Color loss can be calculated based on color differences at the pixel level, for example using mean squared error (MSE):

[0100]

[0101] Among them, C recon and C real are the color values ​​of the reconstructed image and the real image respectively, and N is the number of pixels.

[0102] Consistency loss can be used to ensure information consistency between different perspectives or time points. The specific implementation method can be designed according to the specific application scenario.

[0103] By introducing the above equations and formulas, the working principle and optimization process of the 3D scene reconstruction device combining dynamic scale feature extraction and Transformer in this embodiment can be explained in more detail.

[0104] Workflow

[0105] S1. Image acquisition:

[0106] A monocular camera is used to acquire RGB-D image sequences of non-rigid dynamic targets.

[0107] S2. Preprocessing:

[0108] The collected RGB-D images are preprocessed by denoising, image enhancement, and image registration to improve image quality.

[0109] S3, dynamic scale feature extraction:

[0110] The Dynamic Scale Convolutional Neural Network (Dynamic Scale CNN) is used to adaptively adjust the scale of the convolution kernel to extract features from the image.

[0111] S4. Depth Estimation:

[0112] Convolutional Neural Network (CNN) is used to estimate the depth of the image and obtain the geometric structure information of the scene.

[0113] S5. Geometric information extraction:

[0114] Extract geometric information such as surface normals and curvature from the depth estimation results.

[0115] S6, Transformer encoding:

[0116] A multi-head self-attention mechanism is used to encode the extracted features and capture the global relationship between features.

[0117] S7. Feature aggregation:

[0118] The decoder structure of Transformer is used to aggregate the encoded features to generate a more complete and consistent three-dimensional feature volume.

[0119] S8, decoding:

[0120] By decoding layer by layer, detailed three-dimensional point cloud data is gradually generated.

[0121] S9. Space-time information fusion:

[0122] The information from different time points and different perspectives is integrated to improve the coherence and consistency of the reconstruction results.

[0123] S10, Feature Enhancement:

[0124] Use additional information (such as lighting conditions, material information, etc.) to enhance features and improve their expressiveness and robustness.

[0125] S11, post-processing:

[0126] The generated 3D point cloud is smoothed, denoised and meshed to improve the visual effect and accuracy of the reconstruction results.

[0127] S12. Loss calculation:

[0128] Calculate the difference between the reconstruction result and the real scene, including geometric loss, color loss and consistency loss.

[0129] S13. Model optimization:

[0130] According to the loss calculation results, the model parameters are optimized using optimization algorithms (such as SGD, Adam, etc.).

[0131] S14, output:

[0132] The reconstructed 3D scene is output in an appropriate form, such as 3D mesh, point cloud file, etc., and supports conversion and export in multiple formats.

[0133] Beneficial Effects

[0134] 1. Improve reconstruction accuracy:

[0135] Through dynamic scale feature extraction and Transformer encoding module, features of different scales can be adaptively extracted and encoded, improving the accuracy and robustness of feature extraction, thereby improving the accuracy of 3D reconstruction.

[0136] 2. Enhance feature expression capabilities:

[0137] The feature aggregation module and feature enhancement module can fuse features of different scales and perspectives, and use additional information to enhance the features to improve the expressiveness and robustness of the features.

[0138] 3. Improve reconstruction efficiency:

[0139] The Transformer encoding and decoding modules use parallel computing methods, which can efficiently process large-scale data and improve the efficiency of 3D reconstruction.

[0140] 4. Improve reconstruction coherence and consistency:

[0141] The spatiotemporal information fusion module can capture the dynamic changes of the scene, fuse the information at different time points and different perspectives, and improve the coherence and consistency of the reconstruction results.

[0142] 5.Support multiple output formats:

[0143] The output module supports conversion and export in multiple formats to meet the needs of different applications and facilitate users to perform subsequent processing and analysis.

[0144] 6. Reduce hardware requirements:

[0145] Through the application of deep learning technology and optimization algorithms, the requirements for high-performance hardware can be reduced to a certain extent, making 3D reconstruction technology more popular and practical.

[0146] In summary, the 3D scene reconstruction device combining dynamic scale feature extraction and Transformer provided in this embodiment has the advantages of high precision, high efficiency, high coherence and consistency, and can meet the needs of different application scenarios.

[0147] The above are only preferred embodiments of the present invention, and are not intended to limit the implementation methods and protection scope of the present invention. Those skilled in the art should be aware that all solutions obtained by equivalent substitutions and obvious changes made using the description and illustrations of the present invention should be included in the protection scope of the present invention.

Claims

1. A 3D scene reconstruction device combining dynamic scale feature extraction and Transformer, characterized in that: include: Image acquisition module, preprocessing module, dynamic scale feature extraction module, Transformer encoding module, feature aggregation module, decoding module, post-processing module, loss calculation module, model optimization module, image registration module, depth estimation module, geometric information extraction module, spatiotemporal information fusion module, feature enhancement module and output module, among which: The image acquisition module is used to acquire an RGB-D image sequence of a non-rigid dynamic target through a single sensor or multiple sensors, is connected to the preprocessing module, and transmits the acquired RGB-D image sequence to the preprocessing module; The preprocessing module is connected to the dynamic scale feature extraction module and the image registration module respectively, and the preprocessing module performs denoising and image enhancement preprocessing on the received RGB-D image sequence; For the RGB-D image to be registered, the preprocessing module first performs denoising and image enhancement preprocessing on the received RGB-D image sequence, and then passes the preprocessed RGB-D image to the image registration module for registration, and the image registration module passes the registered image to the dynamic scale feature extraction module; For RGB-D images that do not require registration, the preprocessing module performs denoising and image enhancement preprocessing on the received RGB-D image sequence, and then directly passes the preprocessed RGB-D image to the dynamic scale feature extraction module; The dynamic scale feature extraction module is connected to the Transformer encoding module, and uses a dynamic scale convolutional neural network to extract features from the received preprocessed or registered image, and transmits the extracted features to the Transformer encoding module; The Transformer encoding module is connected to the feature aggregation module, which uses a multi-head self-attention mechanism to encode the extracted features to capture the global relationship between the features, and uses position encoding technology to perceive the spatial structure of the image, and passes the encoded features to the feature aggregation module; The feature aggregation module is connected to the decoding module, and uses the decoder structure of the Transformer to aggregate the encoded features to fuse features of different scales and different perspectives, generate a complete and consistent three-dimensional feature volume, and pass the features of the aggregated three-dimensional feature volume to the decoding module; The decoding module is connected to the post-processing module, which generates three-dimensional point cloud data according to the aggregated features and transmits the generated three-dimensional point cloud data to the post-processing module; The post-processing module is connected to the loss calculation module and the output module respectively, and performs smoothing, denoising and gridding on the generated three-dimensional point cloud data to obtain a reconstructed three-dimensional scene, and then transmits the reconstructed three-dimensional scene to the loss calculation module and the output module; The loss calculation module is connected to the model optimization module, and calculates the difference between the reconstructed three-dimensional scene and the real scene, and calculates the loss value and passes it to the model optimization module; The model optimization module is connected to the dynamic scale feature extraction module and the Transformer encoding module respectively, and is used to receive the loss value output by the loss calculation module, optimize the model parameters according to the loss value, and then feed the optimized parameters back to the dynamic scale feature extraction module and the Transformer encoding module; The image registration module is connected to the image acquisition module, the preprocessing module and the dynamic scale feature extraction module respectively, and is used to receive the output of the image acquisition module in the preprocessing stage or before feature extraction, and transmit the registered image to the preprocessing module or the dynamic scale feature extraction module; The depth estimation module is connected to the image acquisition module, the preprocessing module, the geometric information extraction module and the feature aggregation module respectively, and is used to receive the RGB-D image and generate a depth map to pass to the geometric information extraction module or the feature aggregation module; The geometric information extraction module is connected to the depth estimation module, the feature aggregation module and the decoding module respectively, and is used to receive the output of the depth estimation module, and extract the geometric information and pass it to the feature aggregation module or the decoding module; The spatiotemporal information fusion module is respectively connected to the dynamic scale feature extraction module, the decoding module and the feature aggregation module, and is used to receive the spatiotemporal information from the dynamic scale feature extraction module and the decoding module, perform fusion processing, and then pass the fused data to the feature aggregation module or the decoding module; The feature enhancement module is respectively connected to the dynamic scale feature extraction module, the Transformer encoding module, the Transformer encoding module and the feature aggregation module, and is used to receive feature input, enhance the features using additional information, and then pass the enhanced features to the Transformer encoding module or the feature aggregation module; The output modules are respectively connected to the post-processing modules, and are used to receive the output of the post-processing modules and output the reconstructed three-dimensional scene data in multiple formats.

2. The 3D scene reconstruction device combining dynamic scale feature extraction and Transformer according to claim 1, characterized in that: The decoding module gradually generates a fine three-dimensional point cloud by decoding layer by layer, thereby realizing scene reconstruction.

3. The 3D scene reconstruction device combining dynamic scale feature extraction and Transformer according to claim 2, characterized in that: The differences calculated by the loss calculation module include geometric loss, color loss and consistency loss.

4. The 3D scene reconstruction device combining dynamic scale feature extraction and Transformer according to claim 3 is characterized in that: The model optimization module optimizes the model parameters using an optimization algorithm according to the loss calculation result, and the optimization algorithm includes a stochastic gradient descent algorithm and an Adam algorithm.

5. The 3D scene reconstruction device combining dynamic scale feature extraction and Transformer according to claim 4 is characterized in that: The registration process of the image registration module includes feature point detection, feature point matching and image transformation steps.

6. The 3D scene reconstruction device combining dynamic scale feature extraction and Transformer according to claim 5, characterized in that: The depth estimation module uses deep learning technology and a convolutional neural network to perform depth estimation on the image.

7. The 3D scene reconstruction device combining dynamic scale feature extraction and Transformer according to claim 6, characterized in that: The geometric information extracted by the geometric information extraction module includes surface normals and curvatures.

8. The 3D scene reconstruction device combining dynamic scale feature extraction and Transformer according to claim 7, characterized in that: The feature enhancement module utilizes additional information including lighting conditions and material information to enhance the extracted features.

9. The 3D scene reconstruction device combining dynamic scale feature extraction and Transformer according to claim 8, characterized in that: The output module outputs the reconstructed three-dimensional scene in the form of a three-dimensional grid or a point cloud file.

10. The 3D scene reconstruction device combining dynamic scale feature extraction and Transformer according to claim 9, characterized in that: The dynamic scale feature extraction module uses a dynamic scale convolutional neural network to adaptively adjust the scale of the convolution kernel to adapt to features of different scales.

Images