A dynamic adaptive 3D model generation method based on deep learning

Abstract

The present invention discloses a dynamic adaptive 3D model generation method based on deep learning, comprising: S1: constructing a data set; S2: constructing a 3D model generation model, importing the preprocessed image in S1 into a feature extraction module, and obtaining a multi-scale feature representation of the image; S3: importing the multi-scale feature representation in S2 into a feature fusion module for feature fusion, and obtaining fusion features; S4: importing the fusion features of S3 into a 3D point cloud generation module, obtaining multi-view image features to generate dense 3D point clouds; S5: a camera module performs feature point matching on the dense 3D point cloud in S4, and obtains a matching 3D point cloud; S6: a mesh generation module performs image generation on the matching 3D point cloud in S5 through a generative adversarial network, and obtains a corresponding 3D mesh, i.e., a final 3D model. The present invention can achieve a good balance between global structure and detail features by introducing an adaptive weighted fusion mechanism, and ensures the generation of high-precision 3D models in various scenarios.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a dynamic adaptive 3D model generation method based on deep learning. Background Art

[0002] Traditional 3D model generation methods usually rely on manual modeling, laser scanning, structured light measurement and other technologies. These methods not only have high hardware requirements, but also lack flexibility when processing different types of data, and often require manual adjustment of parameters. Even 3D reconstruction methods based on computer vision and deep learning still have problems with detail loss and unstable model generation when processing complex scenes.

[0003] Therefore, there is an urgent need for a more automated and highly robust 3D model generation method that can adaptively process different types of data and maintain accuracy and preserve details during the model generation process. Summary of the invention

[0004] In view of the defects existing in the existing 3D model generation scheme, the present invention provides a dynamic adaptive 3D model generation method based on deep learning, comprising the following steps:

[0005] Step S1: construct a data set, the data set includes several images, and preprocess the images;

[0006] Step S2: construct a 3D model generation model, which includes a feature extraction module, a feature fusion module, a 3D point cloud generation module, a camera module and a mesh generation module, and import the preprocessed image in step S1 into the feature extraction module to obtain a multi-scale feature representation of the image;

[0007] Step S3: importing the multi-scale feature representation in step S2 into the feature fusion module to perform feature fusion and obtain fusion features;

[0008] Step S4: import the fused features of step S3 into the 3D point cloud generation module, obtain multi-view image features, process the multi-view image features through a triangulation method, obtain a preliminary 3D point cloud, further introduce geometric constraints and edge preservation mechanisms based on the preliminary 3D point cloud, and fuse depth information to generate a dense 3D point cloud;

[0009] Step S5: the camera module performs feature point matching on the dense 3D point cloud in step S4 to obtain a matching 3D point cloud. In this process, the position and posture of the dynamically adjusted camera module are obtained;

[0010] Step S6: The mesh generation module generates an image of the matching 3D point cloud in step S5 through a generative adversarial network to obtain the corresponding 3D mesh, which is the final 3D model.

[0011] Furthermore, the image is preprocessed in step S1, specifically:

[0012] Step S11: obtaining an optimal parameter combination of the image; the optimal parameter combination includes an optimal brightness adjustment parameter, an optimal noise reduction coefficient, an optimal edge enhancement coefficient and an optimal contrast parameter;

[0013] Step S12: image standardization, using the optimal brightness adjustment parameters to standardize the image to ensure the consistency of image brightness under different lighting conditions;

[0014] Step S13: Adaptive noise reduction, adaptively reducing the noise in the image through the optimal noise reduction coefficient to reduce the interference of noise on 3D reconstruction;

[0015] Step S14: edge enhancement, enhancing the edges in the image according to the optimal edge enhancement coefficient to ensure the clarity of edge features when the 3D model is generated;

[0016] Step S15: Contrast adjustment: using the optimal contrast parameter to adjust the contrast of the image to enhance the detail expression of the image.

[0017] Furthermore, in step S2, the preprocessed image in step S1 is imported into the feature extraction module to obtain a multi-scale feature representation of the image; specifically, as follows:

[0018] The feature extraction module extracts the features of the input image at multiple scales simultaneously by constructing a multi-scale convolutional neural network; it is expressed as:

[0019] ;

[0020] in, Indicates that the image is Features extracted at multiple scales, that is, multi-scale feature representation of images, represents a multi-scale convolutional neural network, Represents the input image; each scale of the convolutional layer of the multi-scale convolutional neural network has a different receptive field, which is used to capture image information at different levels.

[0021] Furthermore, step S3 is specifically as follows:

[0022] Perform weighted combination on the multi-scale feature representation of the image to obtain fusion features , expressed as:

[0023] ;

[0024] in, Represents quantities of different scales, Representative The weight of a scale.

[0025] Furthermore, step S4 is specifically as follows:

[0026] Step S41: Introduce viewing angle parameters to correspond to multiple viewing angles in the actual scene and generate the first The fusion features under each perspective correspond to the perspective fusion features; expressed as:

[0027] ;

[0028] in, represents the feature extraction operation, which is implemented using a convolutional neural network; Indicates Perspective fusion features under different perspectives;

[0029] Perform weighted fusion on the perspective fusion features under each perspective to form a fusion perspective feature representation , expressed as:

[0030] ;

[0031] in, represents the dynamic weight parameter, Indicates the total number of viewing angles;

[0032] Step S42: Represent the fused view feature by triangulation method Map into 3D space to generate a preliminary 3D point cloud;

[0033] The specific triangulation method is:

[0034] Obtaining fused view feature representation The feature point is 2D points from each viewpoint , through the camera projection matrix to transform the 2D point With 3D points Associated together, the projection equation is formed, expressed as:

[0035] ;

[0036] ;

[0037] in, Indicates The camera projection matrix for each view, Represents the homogeneous coordinates of feature points in 3D space;

[0038] Get 3D points using projection equations from multiple viewpoints , the coordinates of the 3D point are solved by minimizing the reprojection error, expressed as:

[0039] ;

[0040] in, represents the minimization function, represents the reprojection error;

[0041] Step S42 is specifically as follows:

[0042] By representing each fusion view feature The corresponding feature points are triangulated to generate multiple 3D points. Preliminary 3D point cloud ; expressed as:

[0043] ;

[0044] in, and , Respectively represent the horizontal, vertical and vertical coordinates of the 3D coordinates of each feature point obtained by the triangulation method, Represents a preliminary 3D point cloud The number of 3D points in ;

[0045] Step S43: Introduce geometric constraints and edge preservation mechanism; specifically:

[0046] Define the optimization objective, expressed as:

[0047] ;

[0048] in, and Respectively represent adjacent 3D points in the preliminary 3D point cloud, Represents connecting 3D points and 3D points The weight of

[0049] Step S44: Fusing the depth information to optimize the accuracy of the preliminary 3D point cloud, further fusing the depth information with the fusion features, and generating a dense 3D point cloud; represented as:

[0050] ;

[0051] in, represents a dense 3D point cloud, Indicates that from Generate a 3D point cloud with depth information from the multi-scale feature representation of the image. represents the total number of images, Represents the weight of the depth information.

[0052] Furthermore, step S5 is specifically as follows:

[0053] Step S51: The camera module extracts features from each frame image corresponding to each feature point in the dense 3D point cloud, and obtains matching points through the FLANN algorithm, which is expressed as:

[0054] ;

[0055] in, represents the matched feature points, represents the FLANN algorithm; Indicated in Frame image data The set of feature points extracted from Indicated in Frame image data The set of feature points extracted from

[0056] Step S52: The camera module uses the KLT optical flow algorithm to identify the feature points. The image is tracked in subsequent frames to ensure the dynamic consistency of feature points, which is expressed as:

[0057] ;

[0058] in, Represents the trajectory of feature points tracked by the KLT optical flow algorithm, that is, matching the 3D point cloud. Represents the KLT optical flow algorithm;

[0059] Step S53: Optimize the parameters of the camera module, expressed as:

[0060] ;

[0061] in, represents the optimal camera parameter set, represents the error loss function, which is used to measure the difference between the matched 3D point cloud and the real scene data; Indicated in Frame image data On, use the current parameters The estimated 3D point cloud result; Indicates that the camera parameters are applied to the image data to generate a 3D point cloud that matches the real scene; Indicates The real scene data of the frame;

[0062] An adaptive learning mechanism is introduced to adjust the weights of each parameter in real time, which can be expressed as:

[0063] ;

[0064] in, represents the adaptive weight based on scene complexity, Indicates the total number of image frames;

[0065] Step S54: Accurately estimate the position and posture of the camera module in each frame, using the optimal camera parameter set , combined with matching 3D point cloud Calculate the camera module posture for each frame to ensure accurate positioning of the camera module in dynamic scenes; expressed as:

[0066] ;

[0067] in, Indicates The rotation matrix of the frame, that is, the position of the camera module in each frame, Indicates The translation vector of the frame, that is, the posture of the camera module in each frame, represents the pose estimation function.

[0068] Further, step S6 is expressed as:

[0069] The generator of the generative adversarial network generates images for the matching 3D point cloud and obtains the 3D mesh, which is expressed as:

[0070] ;

[0071] in, represents a 3D grid, Indicates that the generator is The grid generated at each scale is Indicates The fusion weights at each scale are Represents the input matched 3D point cloud.

[0072] Compared with the prior art, the present invention has the following advantages:

[0073] 1) By introducing an adaptive weighted fusion mechanism, the present invention can achieve a good balance between global structure and detail features, ensuring the generation of high-precision 3D models in various scenarios. In addition, by introducing viewing angle parameters and combining depth information, the accuracy and consistency of 3D generation are further improved.

[0074] 2) By combining the FLANN algorithm and the KLT optical flow tracking algorithm, the present invention can achieve efficient and stable feature point matching and tracking, which is suitable for 3D generation tasks in dynamically changing scenes and shows higher stability and noise resistance when processing rapidly changing scenes.

[0075] 3) The present invention can not only adjust the camera parameters in real time, but also show higher accuracy and robustness when processing complex scenes through an adaptive weight mechanism; the present invention further improves the generation quality of 3D point clouds by combining multi-frame posture data for global optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 This is a flowchart of the steps of a dynamic adaptive 3D model generation method based on deep learning of the present invention. DETAILED DESCRIPTION

[0077] Reference Figure 1 , a dynamic adaptive 3D model generation method based on deep learning, in one example, includes the following steps:

[0078] Step S1: construct a data set, the data set includes several images, and preprocess the images;

[0079] Step S2: construct a 3D model generation model, which includes a feature extraction module, a feature fusion module, a 3D point cloud generation module, a camera module and a mesh generation module, and import the preprocessed image in step S1 into the feature extraction module to obtain a multi-scale feature representation of the image;

[0080] Step S3: importing the multi-scale feature representation of the image in step S2 into the feature fusion module to perform feature fusion and obtain fusion features;

[0081] Step S4: import the fused features of step S3 into the 3D point cloud generation module, obtain multi-view image features, process the multi-view image features through a triangulation method, obtain a preliminary 3D point cloud, further introduce geometric constraints and edge preservation mechanisms based on the preliminary 3D point cloud, and fuse depth information to generate a dense 3D point cloud;

[0082] Step S5: the camera module performs feature point matching on the dense 3D point cloud in step S4 to obtain a matching 3D point cloud. In this process, the position and posture of the dynamically adjusted camera module are obtained;

[0083] Step S6: The mesh generation module generates an image of the matching 3D point cloud in step S5 through a generative adversarial network to obtain the corresponding 3D mesh, which is the final 3D model.

[0084] Furthermore, in one example, the image is preprocessed in step S1, specifically:

[0085] Step S11: obtaining an optimal parameter combination of the image; the optimal parameter combination includes an optimal brightness adjustment parameter, an optimal noise reduction coefficient, an optimal edge enhancement coefficient and an optimal contrast parameter;

[0086] Step S12: image standardization, using the optimal brightness adjustment parameters to standardize the image to ensure the consistency of image brightness under different lighting conditions;

[0087] Step S13: Adaptive noise reduction, adaptively reducing the noise in the image through the optimal noise reduction coefficient to reduce the interference of noise on 3D reconstruction;

[0088] Step S14: edge enhancement, enhancing the edges in the image according to the optimal edge enhancement coefficient to ensure the clarity of edge features when the 3D model is generated;

[0089] Step S15: Contrast adjustment: using the optimal contrast parameter to adjust the contrast of the image to enhance the detail expression of the image.

[0090] Further, in one example, step S2 imports the preprocessed image in step S1 into the feature extraction module to obtain a multi-scale feature representation of the image; specifically:

[0091] The feature extraction module extracts the features of the input image at multiple scales simultaneously by constructing a multi-scale convolutional neural network; it is expressed as:

[0092] ;

[0093] in, Indicates that the image is Features extracted at multiple scales, that is, multi-scale feature representation of images, represents a multi-scale convolutional neural network, Represents the input image; each scale of the convolutional layer of the multi-scale convolutional neural network has a different receptive field, which is used to capture image information at different levels.

[0094] Furthermore, in one example, step S3 is specifically:

[0095] Perform weighted combination on the multi-scale feature representation of the image to obtain fusion features , expressed as:

[0096] ;

[0097] in, Represents quantities of different scales, Representative The weight of a scale.

[0098] Furthermore, in one example, step S4 is specifically as follows:

[0099] Step S41: Introduce viewing angle parameters to correspond to multiple viewing angles in the actual scene and generate the first The fusion features under each perspective correspond to the perspective fusion features; expressed as:

[0100] ;

[0101] in, represents the feature extraction operation, which is implemented using a convolutional neural network; Indicates Perspective fusion features under different perspectives;

[0102] Perform weighted fusion on the perspective fusion features under each perspective to form a fusion perspective feature representation , expressed as:

[0103] ;

[0104] in, represents the dynamic weight parameter, Indicates the total number of viewing angles;

[0105] Step S42: Represent the fused view feature by triangulation method Map into 3D space to generate a preliminary 3D point cloud;

[0106] The specific triangulation method is:

[0107] Obtaining fused view feature representation The feature point is 2D points from each viewpoint , through the camera projection matrix to transform the 2D point With 3D points Associated together, the projection equation is formed, expressed as:

[0108] ;

[0109] ;

[0110] in, Indicates The camera projection matrix for each view, Represents the homogeneous coordinates of feature points in 3D space;

[0111] Get 3D points using projection equations from multiple viewpoints , the coordinates of the 3D point are solved by minimizing the reprojection error, expressed as:

[0112] ;

[0113] in, represents the minimization function, represents the reprojection error;

[0114] Step S42 is specifically as follows:

[0115] By representing each fusion view feature The corresponding feature points are triangulated to generate multiple 3D points. Preliminary 3D point cloud ; expressed as:

[0116] ;

[0117] in, and , Respectively represent the horizontal, vertical and vertical coordinates of the 3D coordinates of each feature point obtained by the triangulation method, Represents a preliminary 3D point cloud The number of 3D points in ;

[0118] Step S43: Introduce geometric constraints and edge preservation mechanism; specifically:

[0119] Define the optimization objective, expressed as:

[0120] ;

[0121] in, and Respectively represent adjacent 3D points in the preliminary 3D point cloud, Represents connecting 3D points and 3D points The weight of

[0122] Step S44: Fusing the depth information to optimize the accuracy of the preliminary 3D point cloud, further fusing the depth information with the fusion features, and generating a dense 3D point cloud; represented as:

[0123] ;

[0124] in, represents a dense 3D point cloud, Indicates that from Generate a 3D point cloud with depth information from the multi-scale feature representation of the image. represents the total number of images, Represents the weight of the depth information.

[0125] Furthermore, in one example, step S5 is specifically as follows:

[0126] Step S51: The camera module extracts features from each frame image corresponding to each feature point in the dense 3D point cloud, and obtains matching points through the FLANN algorithm, which is expressed as:

[0127] ;

[0128] in, represents the matched feature points, represents the FLANN algorithm; Indicated in Frame image data The set of feature points extracted from Indicated in Frame image data The set of feature points extracted from

[0129] Step S52: The camera module uses the KLT optical flow algorithm to identify the feature points. The image is tracked in subsequent frames to ensure the dynamic consistency of feature points, which is expressed as:

[0130] ;

[0131] in, Represents the trajectory of feature points tracked by the KLT optical flow algorithm, that is, matching the 3D point cloud. Represents the KLT optical flow algorithm;

[0132] Step S53: Optimize the parameters of the camera module, expressed as:

[0133] ;

[0134] in, represents the optimal camera parameter set, represents the error loss function, which is used to measure the difference between the matched 3D point cloud and the real scene data; Indicated in Frame image data On, use the current parameters The estimated 3D point cloud result; Indicates that the camera parameters are applied to the image data to generate a 3D point cloud that matches the real scene; Indicates The real scene data of the frame;

[0135] An adaptive learning mechanism is introduced to adjust the weights of each parameter in real time, which can be expressed as:

[0136] ;

[0137] in, represents the adaptive weight based on scene complexity, Indicates the total number of image frames;

[0138] Step S54: Accurately estimate the position and posture of the camera module in each frame, using the optimal camera parameter set , combined with matching 3D point cloud Calculate the camera module posture for each frame to ensure accurate positioning of the camera module in dynamic scenes; expressed as:

[0139] ;

[0140] in, Indicates The rotation matrix of the frame, that is, the position of the camera module in each frame, Indicates The translation vector of the frame, that is, the posture of the camera module in each frame, represents the pose estimation function.

[0141] Further, in one example, step S6 is expressed as:

[0142] The generator of the generative adversarial network generates images for the matching 3D point cloud and obtains the 3D mesh, which is expressed as:

[0143] ;

[0144] in, represents a 3D grid, Indicates that the generator is The grid generated at each scale is Indicates The fusion weights at each scale are Represents the input matched 3D point cloud.

[0145] Those skilled in the art will readily appreciate other embodiments of the present application after considering the description and practicing the contents disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The description and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.

Claims

1. A dynamic adaptive 3D model generation method based on deep learning, characterized in that: The following steps are involved: Step S1: construct a data set, the data set includes a number of images, and preprocess the images; Step S2: construct a 3D model generation model, which includes a feature extraction module, a feature fusion module, a 3D point cloud generation module, a camera module and a mesh generation module, and import the preprocessed image in step S1 into the feature extraction module to obtain a multi-scale feature representation of the image; Step S3: import the multi-scale feature representation in step S2 into the feature fusion module to perform feature fusion and obtain fusion features; Step S4: import the fused features of step S3 into the 3D point cloud generation module, obtain multi-view image features, process the multi-view image features through a triangulation method, obtain a preliminary 3D point cloud, further introduce geometric constraints and edge preservation mechanisms based on the preliminary 3D point cloud, and fuse depth information to generate a dense 3D point cloud; Step S5: the camera module performs feature point matching on the dense 3D point cloud in step S4 to obtain a matching 3D point cloud. In this process, the position and posture of the dynamically adjusted camera module are obtained; Step S6: The mesh generation module generates an image of the matching 3D point cloud in step S5 through a generative adversarial network to obtain a corresponding 3D mesh, i.e., the final 3D model; The image is preprocessed in step S1, specifically: Step S11: obtaining an optimal parameter combination of the image; the optimal parameter combination includes an optimal brightness adjustment parameter, an optimal noise reduction coefficient, an optimal edge enhancement coefficient and an optimal contrast parameter; Step S12: image standardization, using the optimal brightness adjustment parameters to standardize the image; Step S13: Adaptive noise reduction, adaptively reducing the noise in the image by using the optimal noise reduction coefficient; Step S14: edge enhancement, enhancing the edge in the image according to the optimal edge enhancement coefficient; Step S15: contrast adjustment, using the optimal contrast parameter to adjust the contrast of the image; Step S2 imports the preprocessed image in step S1 into the feature extraction module to obtain a multi-scale feature representation of the image; specifically: The feature extraction module extracts the features of the input image at multiple scales simultaneously by constructing a multi-scale convolutional neural network; it is expressed as: ； in, Indicates that the image is Features extracted at multiple scales, that is, multi-scale feature representation of images, represents a multi-scale convolutional neural network, Represents the input image; each scale of the convolutional layer of the multi-scale convolutional neural network has a different receptive field, which is used to capture image information at different levels; Step S3 is specifically as follows: Step S31: Perform weighted combination on the multi-scale feature representation of the image to obtain fusion features , expressed as: ； in, Represents quantities of different scales, Representative The weight of each scale; Step S4 is specifically as follows: Step S41: Introduce viewing angle parameters to correspond to multiple viewing angles in the actual scene and generate the first The fusion features under each perspective correspond to the perspective fusion features; expressed as: ； in, represents the feature extraction operation, which is implemented using a convolutional neural network; Indicates Perspective fusion features under different perspectives; Perform weighted fusion on the perspective fusion features under each perspective to form a fusion perspective feature representation , expressed as: ； in, represents the dynamic weight parameter, Indicates the total number of viewing angles; Step S42: Represent the fused view feature by triangulation method Map into 3D space to generate a preliminary 3D point cloud; The specific triangulation method is: Obtaining fused view feature representation The feature point is 2D points from each viewpoint , through the camera projection matrix to transform the 2D point With 3D points Associated together, the projection equation is formed, expressed as: ； ； in, Indicates The camera projection matrix under the viewing angle is Represents the homogeneous coordinates of feature points in 3D space; Get 3D points using projection equations from multiple perspectives , the coordinates of the 3D point are solved by minimizing the reprojection error, expressed as: ； in, represents the minimization function, represents the reprojection error; Step S42 is specifically as follows: By representing each fusion view feature The corresponding feature points are triangulated to generate multiple 3D points. Preliminary 3D point cloud ; expressed as: ； in, and , Respectively represent the horizontal, vertical and vertical coordinates of the 3D coordinates of each feature point obtained by the triangulation method, Represents a preliminary 3D point cloud The number of 3D points in ; Step S43: Introduce geometric constraints and edge preservation mechanism; specifically: Define the optimization objective, expressed as: ； in, and Respectively represent adjacent 3D points in the preliminary 3D point cloud, Represents connecting 3D points and 3D points The weight of Step S44: Fusing the depth information to optimize the accuracy of the preliminary 3D point cloud, further fusing the depth information with the fusion features, and generating a dense 3D point cloud; represented as: ； in, represents a dense 3D point cloud, Indicates that from Generate a 3D point cloud with depth information from the multi-scale feature representation of the image. represents the total number of images, is the weight of the depth information; Step S5 is specifically as follows: Step S51: The camera module extracts features from each frame image corresponding to each feature point in the dense 3D point cloud, and obtains matching points through the FLANN algorithm, which is expressed as: ； in, Represents the matched feature points. represents the FLANN algorithm; Indicated in Frame image data The set of feature points extracted from Indicated in Frame image data The set of feature points extracted from Step S52: The camera module uses the KLT optical flow algorithm to identify the feature points. The image is tracked in subsequent frames, expressed as: ； in, Represents the feature point trajectory obtained by KLT optical flow algorithm, that is, matching 3D point cloud, Represents the KLT optical flow algorithm; Step S53: Optimizing the parameters of the camera module, specifically: ； in, represents the optimal camera parameter set, represents the error loss function; Indicated in Frame image data On, use the current parameters The estimated 3D point cloud result; Indicates that camera parameters are applied to image data to generate a 3D point cloud that matches the real scene; Indicates The real scene data of the frame; An adaptive learning mechanism is introduced to adjust the weights of each parameter in real time, which can be expressed as: ； in, represents the adaptive weight based on scene complexity, Indicates the total number of video frames; Step S54: Estimate the position and posture of the camera module in each frame, using the optimal camera parameter set , combined with matching 3D point cloud Calculate the camera module pose for each frame; expressed as: ； in, Indicates The rotation matrix of the frame, that is, the position of the camera module in each frame, Indicates The translation vector of the frame, that is, the posture of the camera module in each frame, represents the pose estimation function.

2. The method for generating a dynamic adaptive 3D model based on deep learning according to claim 1, characterized in that: Step S6 is expressed as: The generator of the generative adversarial network generates images for the matching 3D point cloud and obtains the 3D mesh, which is expressed as: ； in, represents a 3D grid, Indicates that the generator is The grid generated at each scale is Indicates The fusion weights at each scale are Represents the input matched 3D point cloud.

Images