Single-view-angle three-dimensional reconstruction method based on visual saliency driving
Through the single-view three-dimensional reconstruction method driven by visual significance, combined with human visual significance prior and deep learning technology, the global and local shape information are integrated, and the problems of geometric accuracy and visual authenticity in single-view three-dimensional reconstruction are solved, and the problem of geometric accuracy and visual authenticity is realized. More natural and realistic three-dimensional model reconstruction.
Patent Information
- Application Number
- CN202411187080.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2024-12-03
AI Technical Summary
The existing single-view three-dimensional reconstruction method is not ideal when dealing with complex geometric details and differences in human visual perception, and it is difficult to effectively combine low-level image clues, component structure knowledge and high-level semantic information, resulting in inconsistent with human visual perception.
Using a single-view three-dimensional reconstruction method based on visual significance, a new loss function and deep neural network is designed by constructing a three-dimensional significance perceptron and reconstruction network model, combining human visual significance priors, integrating global and local shape information features. Architecture, optimize geometric accuracy and visual authenticity during reconstruction.
It improves the geometric similarity between the reconstruction model and the real model, enhances the naturalness and authenticity of the reconstruction results, improves the computer vision system's understanding and analysis capabilities of scenes, and solves the shortcomings of traditional methods in handling complex geometric details.
Smart Images

Figure CN119068118A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image-based three-dimensional reconstruction methods, and in particular relates to a single-view three-dimensional reconstruction method driven by visual saliency. Background Art
[0002] Image-based 3D reconstruction is the process of restoring the 3D geometric shape model of an object from an input 2D image. Depending on the number of input images, it can be divided into multi-view Figure 3 Dimensional reconstruction and single view Figure 3 There are two methods for dimensional reconstruction. Figure 3 Due to its low input cost and wider applicability, single-view reconstruction is particularly suitable for scenes where it is difficult to obtain images from multiple angles. However, compared with multi-view methods, single-view reconstruction faces challenges such as limited viewpoints and lack of spatial and semantic information.
[0003] Solving single vision Figure 3 The basic problem of 3D reconstruction includes extracting object information from a single image, discovering visual features and patterns in different dimensions through semantic expression, geometric cognition and statistical learning, and finally restoring the 3D model of the object. The current challenge is how to effectively combine low-level image clues, component structure knowledge and high-level semantic information to infer an accurate 3D model.
[0004] The human visual system can quickly identify the differences between objects by perceiving and analyzing them from different angles. In contrast, computer vision is still limited in this regard and usually relies on physical similarity calculations and projections to explore the differences between objects. Current 3D reconstruction methods often focus on optimizing the geometric error between the reconstructed model and the real model, but the effect may not be ideal when dealing with complex geometric details and differences in human visual perception.
[0005] Therefore, it is necessary to develop more efficient and accurate single-view Figure 3 Dimensional reconstruction methods should be developed to better meet the challenges and requirements of human visual perception. These methods should be able to effectively integrate multi-source information, improve the geometric accuracy and visual realism in the reconstruction process, and thus better simulate and meet the high requirements of the human visual system for object shape and structure. Summary of the invention
[0006] The present invention aims to solve the technical problems existing in the background technology and provides a single-view 3D reconstruction method based on visual saliency drive. By collecting human visual perception information and building an algorithm model based on human visual saliency, it can imitate human perception of 3D shapes and integrate global and local shape information features to provide a new single-view 3D reconstruction method. Figure 3dimensional reconstruction method. Compared with existing methods, the method of the present invention not only improves the geometric similarity between the reconstructed model and the real model, but also pays more attention to the differences in human visual perception. By introducing the human visual saliency prior, the method of the present invention can effectively capture and utilize the important areas and features that humans pay attention to when observing objects. This method is not limited to a single geometric error optimization, but by comprehensively considering the global and local information characteristics of the shape, the reconstruction result is more in line with the human intuitive cognition of the shape and structure of the object, and the consistency between the reconstruction method and subjective perception is improved.
[0007] In order to solve the technical problem, the technical solution of the present invention is:
[0008] A single-view 3D reconstruction method based on visual saliency driving, the method comprising:
[0009] S1: Obtain a training set and a validation set consisting of multiple RGB images and their corresponding 3D models;
[0010] S2: Collect and preprocess the ranking annotations of the salient components of the 3D model by humans to obtain the training set and test set of the 3D saliency sensor;
[0011] S3: Construct and train a 3D saliency perceptron, sample the 3D shapes in the training set and group them into multiple sub-point clouds, use a 3D point cloud feature extractor to extract features from the sub-point clouds, and obtain features corresponding to each sub-point cloud; use the corresponding features as the input of the 3D saliency perceptron, design a regression head to hierarchically connect the extracted sub-point cloud features, and use a multi-layer perceptron and multiple dynamic graph convolutional networks to perform saliency prediction;
[0012] S4: Build and train a 3D reconstruction network model, design an RGB image feature extractor, extract features of RGB images from the training set, design a 3D decoder, input the RGB image features, obtain a reconstructed 3D shape, perform weighted processing on the 3D shape using the saliency prediction results, build a new loss function, and continuously optimize the reconstruction results during the training process.
[0013] Furthermore, the method further comprises:
[0014] S5: Repeat steps S3 and S4 to gradually optimize the performance and generalization ability of the model;
[0015] S6: Using the validation set, evaluate the performance and accuracy of the model.
[0016] Further, the step S2 comprises:
[0017] S201: manually annotating the preset 3D model data, dynamically displaying the 3D model from multiple angles, and sorting the components of the 3D model according to their significance, with the most significant component marked as R=1, and the remaining components sorted according to their significance;
[0018] S202: According to the ranking results of the salient components, normalization processing is performed to obtain a preprocessed training set and a test set of the three-dimensional saliency sensor.
[0019] Further, the step S3 comprises:
[0020] S301: Sampling the three-dimensional shape, processing the point cloud using the farthest point sampling and K nearest neighbor algorithm, grouping to obtain the center point and the sub-point cloud, arranging the sub-point cloud into a continuous sequence according to the center point, and extracting rich geometric features of the sub-point cloud through the three-dimensional point cloud feature extractor;
[0021] S302: Construct a saliency decoder, propagate features to each point in layers, and predict saliency weight scores through multi-layer perceptrons, dynamic graph convolutional networks, and fully connected layers;
[0022] S303: Perform regression loss on the predicted significance weight score result and the true significance weight label to optimize the significance sensor.
[0023] Furthermore, the deep neural network model of the three-dimensional saliency sensor includes a three-dimensional point cloud feature extractor and a saliency decoder;
[0024] The deep neural network model for constructing the salient component perceptron includes:
[0025] The three-dimensional shape is sampled, and the sampled point cloud can be expressed as Where each point p i is the vector of its three-dimensional coordinates; use the farthest point sampling and K nearest neighbor algorithm to process the point cloud and group it to get the center point Hezi Point Cloud The sub-point clouds are arranged into a continuous sequence according to the center points. The coordinates of the center points are encoded into a one-dimensional space using Morton code and then sorted to determine the order of these center points. The specific calculation formula is: Then, the point blocks are arranged in the same order, sorting the center point C s and sorting point patch P s The calculation formula is C s ,
[0026] The three-dimensional point cloud feature extractor is a deep neural network architecture, including multiple convolutional layers, batch normalization layers, activation layers and pooling layers;
[0027] Construct a saliency decoder, propagate features to each point in layers, and predict saliency weight scores through multi-layer perceptrons, dynamic graph convolutional networks, and fully connected layers;
[0028] The saliency decoder is constructed by a multi-layer perceptron, two connected dynamic graph convolutional networks and a fully connected layer. The two dynamic graph convolutional networks update the point cloud features in layers to update the features. The final saliency weight score result is obtained through the multi-layer perceptron and the fully connected layer. The dynamic graph convolutional network can better capture and extract geometric features in the input data at multiple levels.
[0029] The predicted significance weight score result and the true significance weight label are used as regression loss to optimize the significance sensor; the regression loss used to predict the significance weight score result and the true significance weight label uses the mean square error loss function in, is the predicted significance weight score, w i is the true significance weight label, and N is the number of samples.
[0030] Further, the step S4 comprises:
[0031] S401: construct an RGB image extractor, including multiple two-dimensional convolution blocks and a deep residual network, to extract image feature information by inputting an RGB image;
[0032] S402: construct a 3D decoder, input feature information and coordinate information extracted from RGB, to obtain a reconstructed 3D shape;
[0033] S403: Input the three-dimensional shape reconstructed in S402 into the saliency sensor trained in S3 to generate its saliency perception weight score, calculate the weighted reconstruction loss function of the reconstructed shape and the true shape, and optimize the model.
[0034] Furthermore, the loss function is a chamfer distance based on saliency, and the specific formula is:
[0035]
[0036] Among them, P pred and P gt Represent the predicted reconstructed point cloud and the real 3D point cloud data respectively, N gt and N pred represents the number of points between the predicted reconstructed point cloud and the real 3D point cloud model, s(x) represents the significance weight of point x, and α is the adjustment factor, which we set to 10.
[0037] Further, the step S6 comprises:
[0038] S601: After each training phase, a validation set with a different distribution from the training samples is used to test the performance of the model. Evaluation indicators include but are not limited to chamfer distance and bulldozer distance.
[0039] S602: According to the results of the performance evaluation, adjust the hyperparameters of the model, including the learning rate and the regularization coefficient, optimize the model structure, and improve the evaluation performance of the model.
[0040] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the single-view 3D reconstruction method based on visual saliency driving as described above is implemented.
[0041] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements any one of the above-mentioned single-view three-dimensional reconstruction methods based on visual saliency driving.
[0042] Compared with the prior art, the advantages of the present invention are:
[0043] The present invention integrates human visual saliency priors and deep learning technology to predict saliency, thereby better simulating and reflecting human visual characteristics. This technology can generate more natural and realistic three-dimensional models, improving the expression and transmission of visual information;
[0044] The present invention has good interactivity, allowing users to specify the area of interest or dynamically adjust reconstruction parameters according to specific needs to achieve customized 3D reconstruction models. This flexibility enables the technology to effectively adapt to diverse 3D data, greatly improving its applicability and user satisfaction in different application scenarios and industries;
[0045] The present invention uses a saliency-based reconstruction loss function in algorithm design, which significantly improves the robustness of the 3D reconstruction algorithm and the accuracy of the prediction results. This method not only enhances the generalization ability of the model, but also effectively alleviates the overfitting problem that traditional methods are prone to when facing diversified data, thereby further improving the quality and stability of the reconstruction results.
[0046] In summary, compared with the prior art, the present invention has a single-view Figure 3 It has significant advantages in accuracy, user-friendliness and practicality in the field of maintenance and reconstruction, bringing higher value and broad application prospects to market applications. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 Flowchart of the single-view 3D reconstruction method driven by visual saliency;
[0048] Figure 2 Schematic diagram of the three-dimensional saliency sensor model;
[0049] Figure 3 It is a schematic diagram of the structure of the 3D sub-point cloud feature extractor;
[0050] Figure 4 For single vision Figure 3 Schematic diagram of the 3D reconstruction model. DETAILED DESCRIPTION
[0051] The specific implementation mode of the present invention is described below in conjunction with embodiments:
[0052] It should be noted that the structures, proportions, sizes, etc. shown in this specification are only used to match the contents disclosed in the specification so that people familiar with this technology can understand and read them, and are not used to limit the conditions under which the present invention can be implemented. Any structural modification, change in proportional relationship or adjustment of size should still fall within the scope of the technical content disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.
[0053] At the same time, the terms such as "upper", "lower", "left", "right", "middle" and "one" cited in this specification are only for the convenience of description and are not used to limit the scope of implementation of the present invention. Changes or adjustments to their relative relationships should be regarded as the scope of implementation of the present invention without substantially changing the technical content.
[0054] Embodiment 1:
[0055] like Figure 1 As shown, the present invention provides a single-view 3D reconstruction method based on visual saliency drive, and the reconstruction method aims to reconstruct the geometric structure of a 3D object that is more in line with human visual saliency perception based on a single input image. First, by collecting the saliency ranking annotations of the 3D object parts perceived by human vision and preprocessing them, an initial training set is established. Secondly, based on these annotated data, a new 3D reconstruction loss function based on human visual saliency prior is proposed. On this basis, a deep neural network model is designed and constructed, and 3D information is extracted using multi-scale, and global and local geometric relationships are fully integrated, and finally an accurate reconstruction result is obtained through a 3D reconstruction decoder. The present invention aims to automatically simulate human visual perception capabilities through computer technology, and is suitable for fields such as augmented reality, virtual reality and intelligent manufacturing. By incorporating human visual saliency priors into the 3D reconstruction process, not only can the geometric structure of the object be reconstructed more accurately, but also the ability of the computer vision system to understand and analyze the scene can be improved. The innovation of this method lies in the organic combination of deep learning technology and human visual psychology, which realizes the ability of high-quality 3D reconstruction of single-view images and significantly improves the quality and authenticity of the reconstruction results.
[0056] The specific steps include:
[0057] S1: Obtain training samples consisting of several RGB images and their corresponding three-dimensional models.
[0058] Preferably, in order to increase the diversity of images and three-dimensional data and improve the robustness of the reconstruction algorithm, S1 includes: acquiring three-dimensional data and preprocessing it, obtaining multiple representation methods of the three-dimensional model, and forming a training set, a verification set and a test set.
[0059] Among them, the three-dimensional data preprocessing, such as normalizing the three-dimensional data, transforming the angle and converting multiple representations, etc., can help the training model better understand and reconstruct the three-dimensional structure.
[0060] Furthermore, in a specific embodiment, the present invention uses the existing 3D model benchmark dataset ShapeNet for training and testing. ShapeNet is a rich, large-scale 3D shape dataset and is currently the most commonly used single-view Figure 3 We use ShapeNet Core as a subcategory and select five representative categories: airplane, car, chair, table, and lamp, which contain 2689, 1824, 3745, 5263, and 1542 3D shapes, respectively. We randomly assign each 3D model in the dataset to the training set, validation set, and test set, with proportions of 70%, 10%, and 20%, respectively.
[0061] S2: Collect and preprocess the human annotations of the salient components of the 3D model, and use them as the training set and / or test set of the 3D saliency sensor.
[0062] S21: Use the existing 3D model data for manual annotation and display the 3D model dynamically from multiple angles. Suppose there is a 3D model (X) that contains (m) parts, which are marked as (P1, P2, …, P m ). Participants sorted the components of the 3D model according to their significance, with the most significant component marked as R = 1, and the remaining components were sorted according to their significance, with the least significant component marked as R = m. We can use the vector R x Represents the ranking result of the significant components of model X: R x =(r1,r2,…r m ), where r i represents the position of the i-th component in the sorting, r i The value range of is (1,m), and the sorting position of each component is different.
[0063] S22: According to the ranking results of the significant components, normalization processing will be performed for use in the subsequent training stage.
[0064] In a more specific embodiment, for each ranking r i , use min-max normalization to convert it to a normalized value This ensures that all sorted values are in the range of [0,1], giving them a uniform scale.
[0065] S3: Construct and train a 3D saliency perceptron, by sampling and grouping the 3D shapes in the training set and / or test set into multiple sub-point clouds, and using a 3D point cloud feature extractor to extract features from these sub-point clouds to obtain features corresponding to each sub-point cloud. These features are used as the input of the saliency decoder, and a regression head is designed to hierarchically connect the extracted sub-point cloud features, and saliency prediction is performed using a multi-layer perceptron and multiple dynamic graph convolutional networks.
[0066] In a specific embodiment, referring to the attached Figure 2 The deep neural network model of the three-dimensional saliency sensor includes a three-dimensional point cloud feature extractor and a saliency decoder.
[0067] Furthermore, the deep neural network model of the saliency component perceptron includes:
[0068] S31: Sampling the 3D shape, the sampled point cloud can be expressed as Where each point p i is the vector of its three-dimensional coordinates. Use the farthest point sampling and K nearest neighbor algorithm to process the point cloud and group it to get the center point C = FPS (P), Hezi Point Cloud The sub-point clouds are arranged into a continuous sequence according to the center points. Specifically, the coordinates of the center points are encoded into a one-dimensional space using Morton code, and then sorted to determine the order O of these center points. The specific calculation formula is: Then, the point blocks are arranged in the same order. s and sorting point patch P s The calculation formula is C s ,
[0069] The rich geometric features of the sub-point cloud are extracted through the 3D point cloud feature extractor.
[0070] In a specific embodiment, referring to the attached Figure 3 The three-dimensional point cloud feature extractor is a deep neural network architecture, which includes multiple convolutional layers, batch normalization layers, activation layers and pooling layers to effectively extract the global features of the three-dimensional point cloud.
[0071] S32: Design a saliency decoder to propagate features to each point in layers and predict saliency weight scores through multi-layer perceptrons, dynamic graph convolutional networks, and fully connected layers.
[0072] In a specific embodiment, the saliency decoder is constructed by a multi-layer perceptron, two connected dynamic graph convolutional networks and a fully connected layer, and the two dynamic graph convolutional networks perform hierarchical updates on point cloud features to perform feature updates. The final saliency weight score result is obtained through the multi-layer perceptron and the fully connected layer. Through the dynamic graph convolutional network, the geometric features in the input data can be better captured and extracted at multiple levels.
[0073] S33: The predicted significance weight score result and the true significance weight label are regressed to optimize the significance perceptron.
[0074] In a specific embodiment, the regression loss for predicting the significance weight score result and the true significance weight label is calculated using the mean square error loss function in, is the predicted significance weight score, w i is the true significance weight label, and N is the number of samples.
[0075] S4: Build and train a 3D reconstruction network model, design an RGB image feature extractor, and extract features of RGB images from the training set and / or test set. Design a 3D decoder, input the RGB image features, and obtain the reconstructed 3D shape. Use the saliency prediction obtained in S3 to weight the 3D shape, build a new loss function, and continuously optimize the reconstruction results during the training process to improve the accuracy and reliability of the reconstruction method.
[0076] In a specific embodiment, referring to the attached Figure 4 , the reconstruction network model includes an RGB image extractor and a three-dimensional decoder.
[0077] Further, the reconstructed network model includes:
[0078] S41: Design an RGB image extractor, which includes multiple 2D convolutional blocks and a deep residual network, to extract image feature information by inputting an RGB image.
[0079] According to some preferred embodiments of the present invention, the deep residual network uses a ResNet18 network.
[0080] In a specific implementation, an RGB image with a resolution of 128×128 is input, and the first layer of convolutional network uses a 7×7 convolution kernel for convolution operation, and the activation function is Leaky ReLU. Then, after a series of ResNet18 networks, the spatial size of the feature map is gradually reduced and the number of channels is increased. Finally, after the convolution layer uses a 4×4 convolution kernel and Sigmoid as the activation function, a 128-dimensional output vector is generated.
[0081] S42: Design a 3D decoder that inputs feature information and coordinate information extracted from RGB to obtain a reconstructed 3D shape.
[0082] In a specific embodiment, the three-dimensional decoder includes multiple layers of fully connected layers. Specifically, the 128-dimensional image feature vector and the 3-dimensional coordinate vector extracted from the input S41 are first connected to form a 131-dimensional input vector. Then, multiple fully connected layers are applied, each of which takes the output of the previous layer as input and passes through the Leaky ReLU activation function. The output dimension of each fully connected layer gradually decreases from 2048 dimensions to 128 dimensions. At each layer, in order to retain the original input information, the model connects the original 131-dimensional input vector with the output vector of the current layer. Finally, the 128-dimensional vector is mapped to a single output node through the last fully connected layer, and a value between 0 and 1 is output through the Sigmoid activation function to represent the probability of the point being inside or outside the shape. The three-dimensional model is then sampled to obtain a point cloud representation of the model.
[0083] S43: The three-dimensional shape reconstructed by S42 is input into the saliency sensor trained by S3 to generate its saliency perception weight score, and the reconstruction loss function of the reconstructed shape and the true shape is weighted to optimize the model.
[0084] In a specific implementation, the loss function is a chamfer distance based on saliency, and the specific formula is:
[0085]
[0086] Among them, P pred and P gt Represent the predicted reconstructed point cloud and the real 3D point cloud data respectively, N gt and N pred represents the number of points between the predicted reconstructed point cloud and the real 3D point cloud model, s(x) represents the significance weight of point x, and α is the adjustment factor, which we set to 10.
[0087] S5: Repeat the above steps S3 and S4 several times to gradually optimize the performance and generalization ability of the model.
[0088] In some embodiments, the more repetitions there are and the fewer difficult samples there are, the smaller the performance improvement is. When the performance no longer improves, the training is terminated.
[0089] S6: Evaluate the performance and accuracy of the model using an independent validation set or test set, where the validation set or test set contains RGB images and the corresponding 3D ground-truth models.
[0090] After each training phase, a validation set with a different distribution from the training samples is used to test the performance of the model. Evaluation indicators include but are not limited to chamfer distance and bulldozer distance.
[0091] To further illustrate, the chamfer distance can be expressed as: Where P pred and P gt Represent the predicted reconstructed point cloud and the real 3D point cloud data respectively, N gt and N pred It represents the number of points between the predicted reconstructed point cloud and the real 3D point cloud model. The bulldozer distance can be expressed as: Where x is a point in the real point cloud, are the adjacent vertices of x.
[0092] The saliency sensor, RGB image encoder, 3D decoder and saliency-based chamfer distance loss function in the above process together constitute a visual saliency driven single-view 3D reconstruction method.
[0093] In a specific embodiment, according to the above specific implementation methods, the evaluation results of the visual saliency driven single-view 3D reconstruction method of the present invention and some existing SOTA models in five categories of airplanes, cars, chairs, table lamps and tables can be obtained as shown in Appendix 1. The smaller the value of chamfer distance (CD), bulldozer distance (EMD) and saliency-based chamfer distance (SCD), the better the reconstruction effect. The values in the table are all calculated results magnified 100 times. It can be seen from the performance comparison table that the method can achieve good reconstruction results under the samples of the ShapeNet dataset, which proves the effectiveness of the visual saliency driven single-view 3D reconstruction method.
[0094] Table 1 - Comparison of the reconstruction results obtained with those of the existing SOTA model
[0095]
[0096] Embodiment 2:
[0097] This embodiment provides a terminal device, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of a single-view three-dimensional reconstruction method based on visual saliency drive, including the following steps:
[0098] S1: Obtain a training set and a validation set consisting of multiple RGB images and their corresponding 3D models;
[0099] S2: Collect and preprocess the ranking annotations of the salient components of the 3D model by humans to obtain the training set and test set of the 3D saliency sensor;
[0100] S3: Construct and train a 3D saliency perceptron, sample the 3D shapes in the training set and group them into multiple sub-point clouds, use a 3D point cloud feature extractor to extract features from the sub-point clouds, and obtain features corresponding to each sub-point cloud; use the corresponding features as the input of the 3D saliency perceptron, design a regression head to hierarchically connect the extracted sub-point cloud features, and use a multi-layer perceptron and multiple dynamic graph convolutional networks to perform saliency prediction;
[0101] S4: Build and train a 3D reconstruction network model, design an RGB image feature extractor, extract features of RGB images from the training set, design a 3D decoder, input the RGB image features, obtain a reconstructed 3D shape, perform weighted processing on the 3D shape using the saliency prediction results, build a new loss function, and continuously optimize the reconstruction results during the training process.
[0102] Embodiment 3:
[0103] This embodiment provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a terminal device for storing programs and data. It is understandable that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and the extended storage medium supported by the terminal device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory.
[0104] The processor may load and execute one or more instructions stored in a computer-readable storage medium to implement the corresponding steps of the single-view 3D reconstruction method based on visual saliency drive in the above embodiment; the processor may load and execute the following steps:
[0105] S1: Obtain a training set and a validation set consisting of multiple RGB images and their corresponding 3D models;
[0106] S2: Collect and preprocess the ranking annotations of the salient components of the 3D model by humans to obtain the training set and test set of the 3D saliency sensor;
[0107] S3: Construct and train a 3D saliency perceptron, sample the 3D shapes in the training set and group them into multiple sub-point clouds, use a 3D point cloud feature extractor to extract features from the sub-point clouds, and obtain features corresponding to each sub-point cloud; use the corresponding features as the input of the 3D saliency perceptron, design a regression head to hierarchically connect the extracted sub-point cloud features, and use a multi-layer perceptron and multiple dynamic graph convolutional networks to perform saliency prediction;
[0108] S4: Build and train a 3D reconstruction network model, design an RGB image feature extractor, extract features of RGB images from the training set, design a 3D decoder, input the RGB image features, obtain a reconstructed 3D shape, perform weighted processing on the 3D shape using the saliency prediction results, build a new loss function, and continuously optimize the reconstruction results during the training process.
[0109] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0110] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0111] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0112] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0113] The preferred embodiments of the present invention are described in detail above, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.
[0114] Many other changes and modifications may be made without departing from the concept and scope of the present invention.It should be understood that the present invention is not limited to the specific embodiments, and the scope of the present invention is defined by the appended claims.
Claims
1. A single-view 3D reconstruction method driven by visual saliency, characterized in that: The method comprises: S1: Obtain a training set and a validation set consisting of multiple RGB images and their corresponding 3D models; S2: Collect and preprocess the ranking annotations of the salient components of the 3D model by humans to obtain the training set and test set of the 3D saliency sensor; S3: Construct and train a 3D saliency perceptron, sample the 3D shapes in the training set and group them into multiple sub-point clouds, use a 3D point cloud feature extractor to extract features from the sub-point clouds, and obtain features corresponding to each sub-point cloud; use the corresponding features as the input of the 3D saliency perceptron, design a regression head to hierarchically connect the extracted sub-point cloud features, and use a multi-layer perceptron and multiple dynamic graph convolutional networks to perform saliency prediction; S4: Build and train a 3D reconstruction network model, design an RGB image feature extractor, extract features of RGB images from the training set, design a 3D decoder, input the RGB image features, obtain a reconstructed 3D shape, perform weighted processing on the 3D shape using the saliency prediction results, build a new loss function, and continuously optimize the reconstruction results during the training process.
2. The single-view 3D reconstruction method based on visual saliency drive according to claim 1, characterized in that: The method further comprises: S5: Repeat steps S3 and S4 to gradually optimize the performance and generalization ability of the model; S6: Using the validation set, evaluate the performance and accuracy of the model.
3. The single-view 3D reconstruction method based on visual saliency drive according to claim 1, characterized in that: The step S2 comprises: S201: manually annotating the preset 3D model data, dynamically displaying the 3D model from multiple angles, and sorting the components of the 3D model according to their significance, with the most significant component marked as R=1, and the remaining components sorted according to their significance; S202: According to the ranking results of the salient components, normalization processing is performed to obtain a preprocessed training set and a test set of the three-dimensional saliency sensor.
4. The single-view 3D reconstruction method based on visual saliency drive according to claim 1, characterized in that: The step S3 comprises: S301: Sampling the three-dimensional shape, processing the point cloud using the farthest point sampling and K nearest neighbor algorithm, grouping to obtain the center point and the sub-point cloud, arranging the sub-point cloud into a continuous sequence according to the center point, and extracting rich geometric features of the sub-point cloud through the three-dimensional point cloud feature extractor; S302: Construct a saliency decoder, propagate features to each point in layers, and predict saliency weight scores through multi-layer perceptrons, dynamic graph convolutional networks, and fully connected layers; S303: Perform regression loss on the predicted significance weight score result and the true significance weight label to optimize the significance sensor.
5. The method for single-view 3D reconstruction based on visual saliency drive according to claim 4, characterized in that: The deep neural network model of the three-dimensional saliency sensor includes a three-dimensional point cloud feature extractor and a saliency decoder; The deep neural network model for constructing the salient component perceptron includes: The three-dimensional shape is sampled, and the sampled point cloud can be expressed as Where each point p i is the vector of its three-dimensional coordinates; use the farthest point sampling and K nearest neighbor algorithm to process the point cloud and group it to get the center point Hezi Point Cloud The sub-point clouds are arranged into a continuous sequence according to the center points. The coordinates of the center points are encoded into a one-dimensional space using Morton code and then sorted to determine the order of these center points. The specific calculation formula is: Then, the point blocks are arranged in the same order, sorting the center point C s and sorting point patch P s The calculation formula is The three-dimensional point cloud feature extractor is a deep neural network architecture, including multiple convolutional layers, batch normalization layers, activation layers and pooling layers; Construct a saliency decoder, propagate features to each point in layers, and predict saliency weight scores through multi-layer perceptrons, dynamic graph convolutional networks, and fully connected layers; The saliency decoder is constructed by a multi-layer perceptron, two connected dynamic graph convolutional networks and a fully connected layer. The two dynamic graph convolutional networks update the point cloud features in layers to update the features. The final saliency weight score result is obtained through the multi-layer perceptron and the fully connected layer. The dynamic graph convolutional network can better capture and extract geometric features in the input data at multiple levels. The predicted significance weight score result and the true significance weight label are used as regression loss to optimize the significance sensor; the regression loss used to predict the significance weight score result and the true significance weight label uses the mean square error loss function in, is the predicted significance weight score, w i is the true significance weight label, and N is the number of samples.
6. The method for single-view 3D reconstruction based on visual saliency drive according to claim 1, characterized in that: The step S4 comprises: S401: construct an RGB image extractor, including multiple two-dimensional convolution blocks and a deep residual network, to extract image feature information by inputting an RGB image; S402: construct a 3D decoder, input feature information and coordinate information extracted from RGB, to obtain a reconstructed 3D shape; S403: Input the three-dimensional shape reconstructed in S402 into the saliency sensor trained in S3 to generate its saliency perception weight score, calculate the weighted reconstruction loss function of the reconstructed shape and the true shape, and optimize the model.
7. The single-view 3D reconstruction method based on visual saliency drive according to claim 1, characterized in that: The loss function is the chamfer distance based on saliency, and the specific formula is: Among them, P pred and P gt Represent the predicted reconstructed point cloud and the real 3D point cloud data respectively, N gt and N pred represents the number of points between the predicted reconstructed point cloud and the real 3D point cloud model, s(x) represents the significance weight of point x, and α is the adjustment factor, which we set to 10.
8. The method for single-view 3D reconstruction based on visual saliency drive according to claim 2, characterized in that: The step S6 comprises: S601: After each training phase, a validation set with a different distribution from the training samples is used to test the performance of the model. Evaluation indicators include but are not limited to chamfer distance and bulldozer distance. S602: According to the results of the performance evaluation, adjust the hyperparameters of the model, including the learning rate and the regularization coefficient, optimize the model structure, and improve the evaluation performance of the model.
9. A computer device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the single-view 3D reconstruction method based on visual saliency driving as described in any one of claims 1 to 8 is implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the single-view three-dimensional reconstruction method based on visual saliency driving according to any one of claims 1 to 8.
