Single-view three-dimensional reconstruction method and system based on feature optimization
Through data enhancement and feature optimization methods, single-view image information and 3D reconstruction algorithm models are used to generate key local feature information and train the model, which solves the problem of low accuracy of single-view 3D reconstruction and realizes complete 3D reconstruction of objects.
Patent Information
- Application Number
- CN202510668167.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-23
- Publication Date
- 2025-10-17
AI Technical Summary
Existing single-view 3D reconstruction methods lack sufficient local detail information, resulting in low accuracy of the reconstructed 3D model of the object.
By obtaining the single-view image information to be reconstructed and the trained 3D reconstruction algorithm model, key local feature information is generated using technologies such as data enhancement, codecs, discriminative contrastive learning loss function and multi-layer perceptron. The 3D reconstruction algorithm model is trained through voxel fusion and six-view projection processing to improve reconstruction accuracy.
It achieves accurate reconstruction of the complete three-dimensional model of the object under single-view conditions, improving the accuracy and completeness of three-dimensional reconstruction.
Smart Images
Figure CN120807765A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of artificial intelligence, in particular to a single-view three-dimensional reconstruction method based on feature optimization and a system thereof. BACKGROUND
[0002] With the rapid development of virtual reality, augmented reality and intelligent design technologies, there is an increasing demand for accurate and non-interactive three-dimensional reconstruction. Traditional manual three-dimensional modeling is gradually changing to computer-based methods, which can automatically generate 3D models from 2D images.
[0003] At present, single-view input is common in actual scenarios, but due to the lack of sufficient local detail information, existing single-view three-dimensional reconstruction methods (such as triangulation methods and stereo imaging methods) often have difficulty in accurately reconstructing the complete three-dimensional model of an object, and there is a problem of low accuracy, which needs to be further improved. Figure Three The existing single-view three-dimensional reconstruction methods (such as triangulation methods and stereo imaging methods) often have difficulty in accurately reconstructing the complete three-dimensional model of an object, and there is a problem of low accuracy, which needs to be further improved. SUMMARY
[0004] Therefore, the embodiments of the present application provide a single-view three-dimensional reconstruction method based on feature optimization and a system thereof to solve the problem of low accuracy in the prior art.
[0005] In a first aspect, the embodiments of the present application provide a single-view three-dimensional reconstruction method based on feature optimization, which comprises:
[0006] obtaining single-view image information to be reconstructed and a trained three-dimensional reconstruction algorithm model;
[0007] inputting the single-view image information to be reconstructed into the trained three-dimensional reconstruction algorithm model to generate three-dimensional reconstruction result information.
[0008] Compared with the prior art, the single-view three-dimensional reconstruction method based on feature optimization provided by the embodiments of the present application has the beneficial effects that the terminal device can first obtain single-view image information to be reconstructed and a trained three-dimensional reconstruction algorithm model, then input the single-view image information to be reconstructed into the trained three-dimensional reconstruction algorithm model to generate three-dimensional reconstruction result information, thereby accurately and completely reconstructing the complete three-dimensional model of an object, effectively improving the accuracy, and to some extent solving the problem of low accuracy.
[0009] In a second aspect, the embodiments of the present application provide a single-view three-dimensional reconstruction system based on feature optimization, which comprises:
[0010] a single-view image information to be reconstructed obtaining module for obtaining single-view image information to be reconstructed and a trained three-dimensional reconstruction algorithm model;
[0011] The three-dimensional reconstruction result information generation module is configured to input the single-view image information to be reconstructed into a trained three-dimensional reconstruction algorithm model to generate three-dimensional reconstruction result information.
[0012] In a third aspect, an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the method in the first aspect when executing the computer program.
[0013] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executable on a processor to implement the steps of the method in the first aspect.
[0014] It can be understood that the beneficial effects of the second aspect to the fourth aspect can be referred to the related description in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS
[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced.
[0016] Figure 1 is a flowchart of a single-view three-dimensional reconstruction method provided by an embodiment of the present application;
[0017] Figure 2 is a schematic diagram of constructing a three-dimensional reconstruction algorithm model provided by an embodiment of the present application;
[0018] Figure 3 is a flowchart of step S101 in the single-view three-dimensional reconstruction method provided by an embodiment of the present application;
[0019] Figure 4 is a flowchart of step S102 in the single-view three-dimensional reconstruction method provided by an embodiment of the present application;
[0020] Figure 5 is a schematic diagram of original single-view image information provided by an embodiment of the present application;
[0021] Figure 6 is a schematic diagram of enhanced single-view image information provided by an embodiment of the present application;
[0022] Figure 7 is a flowchart of step S103 in the single-view three-dimensional reconstruction method provided by an embodiment of the present application;
[0023] Figure 8 is a flowchart of step S104 in the single-view three-dimensional reconstruction method provided by an embodiment of the present application;
[0024] Figure 9 is a flowchart of step S105 in the single-view three-dimensional reconstruction method provided by an embodiment of the present application;
[0025] Figure 10 is a schematic diagram of three-dimensional voxel result information provided by an embodiment of the present application;
[0026] Figure 11 is a first schematic diagram of six-view projection processing provided by an embodiment of the present application;
[0027] Figure 12 is a second schematic diagram of six-view projection processing provided by an embodiment of the present application;
[0028] Figure 13 is a flowchart of step S107 in the single-view three-dimensional reconstruction method provided by an embodiment of the present application;
[0029] Figure 14 is a module block diagram of the single-view three-dimensional reconstruction system provided by an embodiment of the present application;
[0030] Figure 15 is a schematic diagram of a terminal device provided by an embodiment of the present application. DETAILED DESCRIPTION
[0031] In the following description, for the purposes of explanation and not limitation, specific details are set forth, such as particular system configurations, techniques, etc., in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.
[0032] In the description of the present application and the appended claims, the terms "first", "second", "third", etc. are used only to distinguish descriptions, and cannot be understood as indicating or implying relative importance.
[0033] In the present application, the reference "one embodiment" or "some embodiments" and the like means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the present application. Thus, the appearance of the phrases "in one embodiment", "in some embodiments", "in other embodiments", "in additional embodiments" and the like in various places in the specification is not necessarily all referring to the same embodiment, but means "one or more but not all embodiments", unless otherwise specifically stated. The terms "comprise", "include", "have" and their conjugates mean "including but not limited to", unless otherwise specifically stated.
[0034] To illustrate the technical solutions described in the present application, specific embodiments are described below.
[0035] Traditional three-dimensional reconstruction methods, such as triangulation and stereo imaging, have limitations in reconstructing unknown local details, and are strongly dependent on environmental factors. Although three-dimensional reconstruction methods based on deep learning can learn complex patterns and features from data, they still face many challenges in single-view reconstruction, such as incomplete reconstruction due to limited training data, inaccurate depth estimation, sensitivity to object pose and shape changes, and limited generalization ability. Multi-view three-dimensional reconstruction methods can capture more comprehensive information through multiple angle images, improve the accuracy of depth estimation, better handle complex shapes and occlusions, and enhance the generalization ability and robustness. However, in practical applications, multi-view data is often difficult to obtain, and single-view data is more common. Figure Three
[0036] Please refer to Figure 1 , Figure 1 is a flowchart of a single-view three-dimensional reconstruction method based on feature optimization provided by an embodiment of the present application. In this embodiment, the execution subject of the single-view three-dimensional reconstruction method is a terminal device. It can be understood that the types of terminal devices include but are not limited to mobile phones, tablet computers, notebook computers, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), etc., and the specific type of terminal device is not limited by the embodiments of the present application.
[0037] Please refer to Figure 1 , the single-view three-dimensional reconstruction method provided by the embodiments of the present application includes but is not limited to the following steps:
[0038] In S100, the single-view image information to be reconstructed and the trained three-dimensional reconstruction algorithm model are obtained.
[0039] Specifically, the terminal device can first obtain the single-view image information to be reconstructed and the trained three-dimensional reconstruction algorithm model, wherein the trained three-dimensional reconstruction algorithm model is used to output the reconstructed three-dimensional model, and an example is shown in Figure 2 , step S100 can correspond to the "inference stage" in Figure 2 .
[0040] In some possible implementations, in order to accurately output the reconstructed three-dimensional model, before step S100, the method further includes but is not limited to the following steps:
[0041] In S101, a trained three-dimensional reconstruction algorithm model is constructed.
[0042] Specifically, before the terminal device acquires the trained three-dimensional reconstruction algorithm model, the terminal device can first construct the trained three-dimensional reconstruction algorithm model.
[0043] In some possible implementation manners, in order to construct an effective three-dimensional reconstruction algorithm model, please refer to Figure 3 , step S101 includes but is not limited to the following steps:
[0044] In S102, original monocular image information and enhanced monocular image information are acquired.
[0045] Specifically, the terminal device can first acquire the original monocular image information and the enhanced monocular image information.
[0046] In some possible implementation manners, in order to acquire the original monocular image information and the enhanced monocular image information, please refer to Figure 4 , step S102 includes but is not limited to the following steps:
[0047] In S1021, the original monocular image information is acquired.
[0048] Exemplarily, please refer to Figure 5 , the terminal device can first acquire the original monocular image information.
[0049] In S1022, the original monocular image information is subjected to data enhancement processing to generate enhanced monocular image information.
[0050] Exemplarily, please refer to Figure 6 , after the terminal device acquires the original monocular image information, the terminal device can subject the original monocular image information to data enhancement processing to effectively generate the enhanced monocular image information, thereby improving the robustness of the subsequent three-dimensional reconstruction algorithm model to input image changes, enhancing the perception ability of the three-dimensional reconstruction algorithm model to local details, and providing more abundant samples for subsequent feature generation.
[0051] Specifically, the enhanced single-view image information is used to describe the original single-view image information after data augmentation processing. The data augmentation processing includes random rotation angle processing, random cropping region processing, and random image size scaling processing. The random rotation angle processing is to rotate the image at different angles randomly. The random cropping region processing is to crop different regions of the image randomly. The random image size scaling processing is to scale the image size randomly. The data augmentation processing also includes appearance transformation processing, such as random contrast adjustment processing, random saturation adjustment processing, random hue adjustment processing, or Gaussian blur processing. The random contrast adjustment processing is to adjust the contrast of the image randomly. The random saturation adjustment processing is to adjust the saturation of the image randomly. The random hue adjustment processing is to adjust the hue of the image randomly.
[0052] In S103, based on the preset codec, the original single-view image information is coded to generate first local discriminative feature information, and the enhanced single-view image information is coded to generate second local discriminative feature information.
[0053] Specifically, after the terminal device acquires the original single-view image information and the enhanced single-view image information, the terminal device can code the original single-view image information based on the preset codec to effectively generate the first local discriminative feature information, and code the enhanced single-view image information based on the codec to generate the second local discriminative feature information.
[0054] Without loss of generality, the codec includes a weight-shared encoder and a decoder. The weight-shared encoder is used to encode the original image and the enhanced image to obtain a feature representation. The decoder is used to decode the encoded feature to extract a local discriminative feature. The weight-shared encoder and the decoder can be replaced by different codecs such as ResNet, Vision Transformer, and DenseNet.
[0055] In some possible implementations, to implement the generation of the first local discriminative feature information and the second local discriminative feature information, please refer to Figure 7 , step S103 includes but is not limited to the following steps:
[0056] In S1031, based on the preset weight-shared encoder, the original single-view image information is coded to generate first image feature information, and the enhanced single-view image information is coded to generate second image feature information.
[0057] Specifically, the terminal device can code the original single-view image information based on the preset weight-shared encoder to effectively generate the first image feature information, and code the enhanced single-view image information based on the weight-shared encoder to quickly generate the second image feature information.
[0058] In S1032, the first image feature information is decoded based on a preset decoder to generate first local discriminative feature information, and the second image feature information is decoded to generate second local discriminative feature information.
[0059] Specifically, after the terminal device generates the first image feature information and the second image feature information, the terminal device can decode the first image feature information based on a preset decoder to efficiently generate the first local discriminative feature information, and decode the second image feature information based on the decoder to effectively generate the second local discriminative feature information.
[0060] In S104, the key local feature information is generated based on a preset discriminative contrastive learning loss function according to the first local discriminative feature vector information and the second local discriminative feature vector information.
[0061] Specifically, after the terminal device generates the first local discriminative feature information and the second local discriminative feature information, the terminal device can effectively generate the key local feature information based on a preset discriminative contrastive learning loss function according to the first local discriminative feature vector information and the second local discriminative feature vector information, thereby effectively enhancing the distinguishing ability of the three-dimensional reconstruction algorithm model to the local details of different objects, so that the key local features of the object can be more accurately captured in the subsequent three-dimensional reconstruction process.
[0062] In some possible implementations, to generate the key local feature information, refer to Figure 8 , step S104 includes but is not limited to the following steps:
[0063] In S1041, the preset normalized temperature scaling cross-entropy loss function is determined as the discriminative contrastive learning loss function.
[0064] Specifically, the terminal device can first determine the preset normalized temperature scaling cross-entropy loss function as the discriminative contrastive learning loss function.
[0065] In S1042, the first local discriminative feature vector information and the second local discriminative feature vector information are input into the discriminative contrastive learning loss function to generate the key local feature information.
[0066] Specifically, after the terminal device determines the discriminative contrastive learning loss function, the terminal device can input the first local discriminative feature vector information and the second local discriminative feature vector information into the discriminative contrastive learning loss function to effectively generate the key local feature information, where the discriminative contrastive learning loss function is:
[0067]
[0068] wherein, η i represents the first local discriminative feature vector information, i.e., the original single-view image information after encoding processing; represents the second local discriminative feature vector information, i.e., the enhanced single-view image information after encoding processing; represents the key local feature information; τ represents a preset weight parameter, which can be a variable weight parameter, and the weight parameter is greater than 0.
[0069] wherein, i represents a first index of a sample, j represents a second index of the sample, and k represents a third index of the sample, i.e., i, j, and k all represent the index of the sample; N represents the number of the first local discriminative feature vector information; sim(u, v) represents the similarity information; u represents the first input argument of the discriminative contrastive learning loss function; and v represents the second input argument of the discriminative contrastive learning loss function.
[0070] In S105, based on the key local feature information and the preset threshold distance, the three-dimensional voxel result information is generated.
[0071] Specifically, after the terminal device generates the key local feature information, the terminal device can effectively generate the three-dimensional voxel result information based on the key local feature information and the preset threshold distance, so as to preliminarily realize the three-dimensional reconstruction based on voxels.
[0072] In some possible implementation manners, in order to realize the generation of the three-dimensional voxel result information, please refer to Figure 9 , step S105 includes but is not limited to the following steps:
[0073] In S1051, the first additional object feature information is generated according to the preset multi-layer perception and the first local discriminative feature vector information, and the second additional object feature information is generated according to the multi-layer perception and the second local discriminative feature vector information.
[0074] Specifically, after the terminal device generates the key local feature information, the terminal device can effectively generate the first additional object feature information according to the preset multi-layer perception and the first local discriminative feature vector information, and effectively generate the second additional object feature information according to the multi-layer perception and the second local discriminative feature vector information, so as to realize the multi-layer perception processing of the output features of the encoder to extract additional object features.
[0075] In S1052, the key local feature information and the first additional object feature information are separated based on the preset threshold distance, and the key local feature information and the second additional object feature information are separated, to generate the fusion feature information.
[0076] Without loss of generality, the extracted additional object features can be combined with the local discriminative features obtained by the discriminative contrastive learning method.
[0077] Specifically, after the terminal device generates the first additional object feature information and the second additional object feature information, the terminal device can perform separation processing on the key local feature information and the first additional object feature information based on a preset threshold distance, and perform separation processing on the key local feature information and the second additional object feature information based on the threshold distance, to generate fusion feature information, so that the local discriminative features and the additional features are distinguished by setting the threshold distance, so that they can be effectively separated in the feature space, and effective feature fusion is achieved.
[0078] In S1053, the fusion feature information is input into a preset voxel fusion module to generate three-dimensional voxel result information.
[0079] Illustratively, please refer to Figure 10 After the terminal device generates the fusion feature information, the terminal device can input the fusion feature information into a preset voxel fusion module to generate three-dimensional voxel result information, wherein the voxel fusion module is a component of a three-dimensional reconstruction algorithm model, and the three-dimensional voxel result information is used to describe predicted voxel occupancy values.
[0080] In S106, six-view projection processing is performed on the three-dimensional voxel result information to generate a plurality of two-dimensional projection image information.
[0081] Illustratively, please refer to Figure 11 and Figure 12 After the terminal device generates the three-dimensional voxel result information, the terminal device can perform six-view projection processing on the three-dimensional voxel result information to generate a plurality of two-dimensional projection image information, each of which can capture spatial information of the object under a corresponding view angle, thereby helping to comprehensively describe the geometric shape and structure of the object from different angles, and providing more abundant information for subsequent feature generation and optimization.
[0082] In some possible implementations, to achieve the generation of the plurality of two-dimensional projection image information, step S106 includes but is not limited to the following steps:
[0083] In S1061, six-view projection processing is performed on the three-dimensional voxel result information based on a six-view projection technology to generate six two-dimensional projection image information.
[0084] Specifically, after the terminal device generates the three-dimensional voxel result information, the terminal device can perform six-view projection processing on the three-dimensional voxel result information based on a six-view projection technology to generate six two-dimensional projection image information.
[0085] Exemplarily, refer to Figure 11 and Figure 12 The terminal device can project the three-dimensional voxel result onto six different two-dimensional views based on a six-view projection technology, i.e., project the voxel result along the positive and negative directions of the x, y, and z axes respectively to obtain six two-dimensional projection images.
[0086] In S107, based on the plurality of two-dimensional projection image information, the preset view constraint loss function, and the preset voxel sampling loss function, the untrained three-dimensional reconstruction algorithm model is trained to generate a trained three-dimensional reconstruction algorithm model.
[0087] Specifically, after the terminal device generates the plurality of two-dimensional projection image information, the terminal device can train the untrained three-dimensional reconstruction algorithm model based on the plurality of two-dimensional projection image information, the preset view constraint loss function, and the preset voxel sampling loss function, to effectively generate the trained three-dimensional reconstruction algorithm model.
[0088] In some possible implementations, to implement the generation of the trained three-dimensional reconstruction algorithm model, refer to Figure 13 Step S107 includes but is not limited to the following steps:
[0089] In S1071, for each voxel point: based on the preset view constraint loss function and the plurality of two-dimensional projection image information, pixel value difference information is generated.
[0090] Specifically, the terminal device can perform the following processing for each voxel point: based on the preset view constraint loss function and the plurality of two-dimensional projection image information, pixel value difference information is generated, so as to calculate the pixel value difference between the projection images, calculate the spatial-based view constraint loss function, effectively guide the model to pay more attention to the spatial structure details of the object when generating the three-dimensional voxel, and improve the integrity and accuracy of the three-dimensional reconstruction result, wherein the pixel value difference information is used to describe the difference value between the predicted projection value and the real projection value of the voxel point in the six-view projection.
[0091] Without loss of generality, the view constraint loss function can be:
[0092]
[0093] In the formula, indicates the pixel value difference information, p indicates the predicted projection value of the voxel point in the six-view projection, gt indicates the real projection value of the voxel point in the six-view projection, S(p) i indicates the sum of the predicted projection values of each voxel point in the i-th two-dimensional projection image information, S(gt) iwherein, represents the sum of the real projection values corresponding to each voxel point in the i-th two-dimensional projection image information, and N represents the total number of the two-dimensional projection image information.
[0094] In S1072, the first local discriminative feature vector information, the second local discriminative feature vector information, and the six two-dimensional projection image information are aligned based on a preset voxel sampling loss function to generate alignment result information.
[0095] Specifically, after the terminal device generates the pixel value difference information, the terminal device can align the first local discriminative feature vector information, the second local discriminative feature vector information, and the six two-dimensional projection image information based on a preset voxel sampling loss function to generate alignment result information, so as to align the local discriminative features obtained in the feature enhancement stage with the six-view constraint in the feature generation stage, reduce the gap between the two, and make the feature representations in the feature enhancement stage and the feature generation stage more consistent, thereby improving the reconstruction performance of the entire model.
[0096] It should be noted that the metric-based voxel sampling loss function can utilize binary cross-entropy to minimize classification errors.
[0097] Without loss of generality, the voxel sampling loss function can be:
[0098]
[0099] In a possible implementation, during the training process, the terminal device can also store and sample the samples according to their priorities. Illustratively, when a sample is added to the experience replay buffer, the terminal device can calculate a priority of the sample according to its matching degree with the real value and the number of samples in the category to which the sample belongs. Samples with higher priorities (for example, samples with poor reconstruction effects and samples in categories with small numbers of samples) will be sampled more frequently for training, so that the model can focus on these difficult samples and small-numbered category samples during the training process, improve the reconstruction performance of the model for different categories and different difficulty samples, and alleviate the influence of the category imbalance problem on the model training.
[0100] In S1073, it is determined whether the pixel value difference information is less than a preset difference threshold.
[0101] Specifically, after the terminal device generates the alignment result information, the terminal device can determine whether the pixel value difference information is less than a preset difference threshold, wherein the difference threshold can be a preset fixed value.
[0102] In S1074, if the pixel value difference information is less than the difference threshold value, it is determined that the trained three-dimensional reconstruction algorithm model is determined, otherwise the steps of S1071 to S1072 are executed again, until the pixel value difference information is less than the difference threshold value, and the training process of the three-dimensional reconstruction algorithm model is completed.
[0103] Specifically, if the pixel value difference information is less than the difference threshold value, the trained three-dimensional reconstruction algorithm model is effectively determined, otherwise the steps S1071 to S1072 are executed again until the pixel value difference information is less than the difference threshold value, and the training process of the three-dimensional reconstruction algorithm model is completed.
[0104] In S200, the single-view image information to be reconstructed is input into the trained three-dimensional reconstruction algorithm model to generate three-dimensional reconstruction result information.
[0105] Specifically, after the terminal device obtains the single-view image information to be reconstructed, the terminal device can input the single-view image information to be reconstructed into the trained three-dimensional reconstruction algorithm model to generate three-dimensional reconstruction result information, that is, the single-view image is input into the three-dimensional reconstruction algorithm model, without data enhancement, the image is encoded by the encoder, and then the local discriminative features and additional features are obtained by the decoder and the multi-layer perception, and the two are fused and input into the voxel fusion module to generate the reconstructed voxel.
[0106] The implementation principle of the single-view three-dimensional reconstruction method based on feature optimization in the embodiment of the application is that the terminal device can obtain the single-view image information to be reconstructed and the trained three-dimensional reconstruction algorithm model, and then input the single-view image information to be reconstructed into the trained three-dimensional reconstruction algorithm model to generate three-dimensional reconstruction result information, so as to accurately reconstruct the complete three-dimensional model of the object and effectively improve the accuracy.
[0107] It should be noted that the size of the serial number of each step in the above embodiment does not mean the execution order, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiment of the application.
[0108] The embodiment of the application also provides a single-view three-dimensional reconstruction system based on feature optimization, for the convenience of description, only the part related to the application is shown, as shown in Figure 14 The system 140 includes:
[0109] The single-view image information to be reconstructed is obtained by the single-view image information to be reconstructed and the trained three-dimensional reconstruction algorithm model.
[0110] The three-dimensional reconstruction result information generation module 142 is configured to input the single-view image information to be reconstructed into a trained three-dimensional reconstruction algorithm model to generate three-dimensional reconstruction result information.
[0111] Optionally, the system 140 further comprises:
[0112] The three-dimensional reconstruction algorithm model construction module is configured to construct the trained three-dimensional reconstruction algorithm model.
[0113] Optionally, the three-dimensional reconstruction algorithm model construction module comprises:
[0114] The original single-view image information acquisition sub-module is configured to acquire the original single-view image information and the enhanced single-view image information.
[0115] The local discriminative feature information generation sub-module is configured to perform coding and decoding processing on the original single-view image information based on a preset codec to generate first local discriminative feature information, and perform coding and decoding processing on the enhanced single-view image information to generate second local discriminative feature information.
[0116] The key local feature information generation sub-module is configured to generate key local feature information based on a preset discriminative contrastive learning loss function and according to the first local discriminative feature vector information and the second local discriminative feature vector information.
[0117] The three-dimensional voxel result information generation sub-module is configured to generate three-dimensional voxel result information based on the key local feature information and a preset threshold distance.
[0118] The two-dimensional projection image information generation sub-module is configured to perform six-view projection processing on the three-dimensional voxel result information to generate a plurality of two-dimensional projection image information.
[0119] The three-dimensional reconstruction algorithm model generation sub-module is configured to train an untrained three-dimensional reconstruction algorithm model based on the plurality of two-dimensional projection image information, a preset view constraint loss function, and a preset voxel sampling loss function to generate the trained three-dimensional reconstruction algorithm model.
[0120] Optionally, the original single-view image information acquisition sub-module comprises:
[0121] The original single-view image information acquisition unit is configured to acquire the original single-view image information.
[0122] The enhanced single-view image information generation unit is configured to perform data enhancement processing on the original single-view image information to generate the enhanced single-view image information, wherein the data enhancement processing comprises random rotation angle processing, random cropping region processing, random scaling image size processing, random contrast adjustment processing, random saturation adjustment processing, random hue adjustment processing, or Gaussian blur processing.
[0123] Optionally, the codec comprises a weight sharing encoder and a decoder, and the local discriminative feature information generation submodule comprises:
[0124] The image feature information generation unit is configured to encode the original single-view image information based on the preset weight sharing encoder to generate first image feature information, and encode the enhanced single-view image information to generate second image feature information.
[0125] The local discriminative feature information generation unit is configured to decode the first image feature information based on the preset decoder to generate first local discriminative feature information, and decode the second image feature information to generate second local discriminative feature information.
[0126] Optionally, the key local feature information generation submodule comprises:
[0127] The discriminative contrastive learning loss function determination unit is configured to determine the preset normalized temperature scaling cross-entropy loss function as a discriminative contrastive learning loss function.
[0128] The key local feature information generation unit is configured to input the first local discriminative feature vector information and the second local discriminative feature vector information into the discriminative contrastive learning loss function to generate key local feature information.
[0129] Optionally, the three-dimensional voxel result information generation submodule comprises:
[0130] The additional object feature information generation unit is configured to generate first additional object feature information according to a preset multi-layer perceptron and the first local discriminative feature vector information, and generate second additional object feature information according to the multi-layer perceptron and the second local discriminative feature vector information.
[0131] The fusion feature information generation unit is configured to separate the key local feature information and the first additional object feature information based on a preset threshold distance, and separate the key local feature information and the second additional object feature information to generate fusion feature information.
[0132] The three-dimensional voxel result information generation unit is configured to input the fusion feature information into a preset voxel fusion module to generate three-dimensional voxel result information, wherein the three-dimensional voxel result information is used to describe predicted voxel occupancy values.
[0133] Optionally, the two-dimensional projection image information generation submodule comprises:
[0134] The two-dimensional projection image information generation unit is configured to perform six-view projection processing on the three-dimensional voxel result information based on a six-view projection technology to generate six two-dimensional projection image information.
[0135] Optionally, the three-dimensional reconstruction algorithm model generation submodule includes:
[0136] a pixel value difference information generation unit configured to generate, for each voxel point, pixel value difference information based on the preset view constraint loss function and the plurality of two-dimensional projection image information, wherein the pixel value difference information is used to describe a difference value between a predicted projection value and an actual projection value of the voxel point in the six-view projection;
[0137] an alignment result information generation unit configured to align the first local discriminative feature vector information, the second local discriminative feature vector information, and the six two-dimensional projection image information based on a preset voxel sampling loss function to generate alignment result information;
[0138] a pixel value difference information judgment unit configured to judge whether the pixel value difference information is less than a preset difference threshold value;
[0139] a re-execution unit configured to, if the pixel value difference information is less than the difference threshold value, determine the trained three-dimensional reconstruction algorithm model, or otherwise re-execute, for each voxel point, the generation of the pixel value difference information based on the preset view constraint loss function and the plurality of two-dimensional projection image information to the alignment of the first local discriminative feature vector information, the second local discriminative feature vector information, and the six two-dimensional projection image information based on the preset voxel sampling loss function to generate the alignment result information, until the pixel value difference information is less than the difference threshold value.
[0140] It should be noted that the information interaction between the above modules, the execution process, and the like, are based on the same concept as the method embodiments of the present application, and the specific functions and the resulting technical effects can be referred to the method embodiments part, which will not be described here.
[0141] The present application also provides a terminal device, as shown in Figure 15 The terminal device 150 of this embodiment includes a processor 151, a memory 152, and a computer program 153 stored in the memory 152 and executable on the processor 151. The processor 151 implements the steps in the single-view three-dimensional reconstruction method embodiments when executing the computer program 153, such as Figure 1 Steps S100 to S200 shown in the above; or, the processor 151 implements the functions of the modules in the above apparatus when executing the computer program 153, such as Figure 14 The functions of the modules 141 to 142 shown in the above.
[0142] The terminal device 150 can be a desktop computer, a notebook computer, a palm computer, and a cloud server, etc. The terminal device 150 includes but is not limited to the processor 151 and the memory 152. Those skilled in the art can understand that the terminal device 150 can further include other components, and the components of the terminal device 150 are not limited to the above.Figure 15 The terminal device 150 is merely an example and does not constitute a limitation on the terminal device 150, and can include more or fewer components than shown, or a combination of some components, or different components, for example, the terminal device 150 can also include an input / output device, a network access device, a bus, and the like.
[0143] The processor 151 can be a GPU, an NPU, and the like parallel acceleration device, and can also be other general-purpose processors, and a field-programmable gate array (FPGA) or other programmable logic device, and the like; the general-purpose processor can be a microprocessor or the processor can also be any conventional processor, and the like.
[0144] The memory 152 can be an internal storage unit of the terminal device 150, for example, a hard disk or a memory of the terminal device 150, and the memory 152 can also be an external storage device of the terminal device 150, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, and the like equipped on the terminal device 150; further, the memory 152 can include both the internal storage unit and the external storage device of the terminal device 150, and the memory 152 can also store the computer program 153 and other programs and data required by the terminal device 150, and the memory 152 can also be used to temporarily store data that has been output or will be output.
[0145] An embodiment of the present application also provides a computer readable storage medium, the storage medium stores a computer program, and the computer program can implement the steps of each method embodiment described above when executed by a processor. The computer program includes computer program code, which can be in the form of source code, object code, an executable file, or some intermediate form, etc.; the computer readable medium can include any entity or device capable of carrying computer program code, a recording medium, a U disk, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.
[0146] The above are preferred embodiments of the present application, and do not limit the protection scope of the present application, therefore: equivalent changes made according to the methods, principles, and structures of the present application should be covered within the protection scope of the present application.
Claims
1. A single-view 3D reconstruction method based on feature optimization, characterized in that: The method comprises: Obtaining the single-view image information to be reconstructed and the trained 3D reconstruction algorithm model; The single-view image information to be reconstructed is input into a trained three-dimensional reconstruction algorithm model to generate three-dimensional reconstruction result information.
2. The method according to claim 1, characterized in that Before obtaining the single-view image information to be reconstructed and the trained 3D reconstruction algorithm model, the method further includes: Build a trained 3D reconstruction algorithm model; The step of constructing a trained 3D reconstruction algorithm model includes: Obtaining original single-view image information and enhanced single-view image information; Based on a preset codec, encoding and decoding the original single-view image information to generate first local discriminant feature information, and encoding and decoding the enhanced single-view image information to generate second local discriminant feature information; Based on a preset discriminative contrastive learning loss function, generating key local feature information according to the first local discriminant feature vector information and the second local discriminant feature vector information; Generating three-dimensional voxel result information based on the key local feature information and a preset threshold distance; Performing six-view projection processing on the three-dimensional voxel result information to generate a plurality of two-dimensional projection image information; Based on the plurality of two-dimensional projection image information, a preset view constraint loss function and a preset voxel sampling loss function, an untrained three-dimensional reconstruction algorithm model is trained to generate a trained three-dimensional reconstruction algorithm model.
3. The method according to claim 2, characterized in that The obtaining of original single-view image information and enhanced single-view image information includes: Obtain original single-view image information; Performing data enhancement processing on the original single-view image information to generate enhanced single-view image information, wherein the data enhancement processing includes random rotation angle processing, random cropping area processing, random image size scaling processing, random contrast adjustment processing, random saturation adjustment processing, random hue adjustment processing, or Gaussian blur processing.
4. The method according to claim 2, characterized in that The codec includes a weight-sharing encoder and a decoder. The codec based on a preset codec performs encoding and decoding processing on the original single-view image information to generate first local discriminant feature information, and performs encoding and decoding processing on the enhanced single-view image information to generate second local discriminant feature information, including: Based on a preset weight-sharing encoder, encoding the original single-view image information to generate first image feature information, and encoding the enhanced single-view image information to generate second image feature information; Based on a preset decoder, the first image feature information is decoded to generate first local discriminant feature information, and the second image feature information is decoded to generate second local discriminant feature information.
5. The method according to claim 2, characterized in that The method of generating key local feature information based on the preset discriminant contrastive learning loss function and the first local discriminant feature vector information and the second local discriminant feature vector information includes: Determine the preset normalized temperature scaled cross entropy loss function as the discriminative contrastive learning loss function; The first local discriminant feature vector information and the second local discriminant feature vector information are input into a discriminant contrastive learning loss function to generate key local feature information, wherein the discriminant contrastive learning loss function is: Where η i is the first local discriminant feature vector information, is the second local discriminant eigenvector information, is the key local feature information, τ is a preset weight parameter, the weight parameter is greater than 0, i is the first index of the sample, j is the second index of the sample, k is the third index of the sample, N is the number of the first local discriminant feature vector information, sim(u,v) is the similarity information, u is the first input independent variable of the discriminant contrast learning loss function, and v is the second input independent variable of the discriminant contrast learning loss function.
6. The method according to claim 2, characterized in that The generating of three-dimensional voxel result information based on the key local feature information and a preset threshold distance includes: Generate first additional object feature information based on a preset multilayer perceptron and the first local discriminant feature vector information, and generate second additional object feature information based on the multilayer perceptron and the second local discriminant feature vector information; Based on a preset threshold distance, separating and processing the key local feature information and the first additional object feature information, and separating and processing the key local feature information and the second additional object feature information to generate fused feature information; The fused feature information is input into a preset voxel fusion module to generate three-dimensional voxel result information, wherein the three-dimensional voxel result information is used to describe the predicted voxel occupancy value.
7. The method according to claim 2, characterized in that The performing six-view projection processing on the three-dimensional voxel result information to generate a plurality of two-dimensional projection image information includes: Based on the six-view projection technology, the three-dimensional voxel result information is subjected to six-view projection processing to generate six two-dimensional projection image information; Accordingly, the training of an untrained 3D reconstruction algorithm model based on the plurality of 2D projection image information, a preset view constraint loss function, and a preset voxel sampling loss function to generate a trained 3D reconstruction algorithm model includes: For each voxel point: generating pixel value difference information based on a preset view constraint loss function and the plurality of two-dimensional projection image information, wherein the pixel value difference information is used to describe the difference between the predicted projection value and the actual projection value of the voxel point in the six-view projection; Based on a preset voxel sampling loss function, aligning the first local discriminant feature vector information, the second local discriminant feature vector information, and the six two-dimensional projection image information to generate alignment result information; Determining whether the pixel value difference information is less than a preset difference threshold; If the pixel value difference information is less than the difference threshold, the trained three-dimensional reconstruction algorithm model is determined; otherwise, the following is executed again for each voxel point: based on the preset view constraint loss function and the multiple two-dimensional projection image information, the pixel value difference information is generated to the preset voxel sampling loss function, the first local discriminant feature vector information, the second local discriminant feature vector information and the six two-dimensional projection image information are aligned, and alignment result information is generated until the pixel value difference information is less than the difference threshold.
8. A single-view 3D reconstruction system based on feature optimization, characterized in that: The system comprises: The module for acquiring the information of the single-view image to be reconstructed is used to acquire the information of the single-view image to be reconstructed and the trained 3D reconstruction algorithm model; 3D reconstruction result information generation module: used to input the single-view image information to be reconstructed into the trained 3D reconstruction algorithm model to generate 3D reconstruction result information.
9. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.