A 3D image generation method based on feature recognition

Abstract

The application relates to the technical field of three-dimensional image processing, and discloses a 3D image generation method based on feature recognition, which comprises the following steps: receiving a two-dimensional image to be processed, and pre-processing the image; using a multi-scale convolutional neural network to extract features of multiple scales from the pre-processed image, so as to obtain multiple scale feature maps; weighting and fusing the features under different scales to obtain a fused feature map; based on optimal transport theory, obtaining a mapped three-dimensional feature; optimizing the three-dimensional feature through total variation regularization; performing geometric optimization through a variational method; and outputting a finally generated three-dimensional image. The application adopts a multi-scale convolutional neural network (MS-CNN) to extract features, and fuses feature maps of different scales, so that the system can extract multi-level information from a two-dimensional image, from macro to micro, and achieves all-around capturing of image details and structures.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional image processing technology, specifically a 3D image generation method based on feature recognition. Background Technology

[0002] With the continuous development of computer vision and 3D reconstruction technology, the technology of generating 3D images based on 2D images has been widely used in many fields, especially in virtual reality, augmented reality, medical imaging and industrial design. Most existing technical solutions rely on 3D reconstruction systems based on traditional image processing methods. These systems usually use convolutional neural networks (CNNs) for feature extraction and 2D-to-3D mapping. Although these technologies show good results in some simple scenarios, they still have significant shortcomings when processing complex and detailed 3D images.

[0003] In existing technologies, feature extraction modules typically use single-scale convolutional neural networks. While this method can handle images of a certain complexity, its ability to extract multi-level, multi-scale features is relatively limited. Especially when dealing with objects with complex textures and varying sizes, single-scale feature extraction methods often fail to capture all image details, resulting in generated 3D images lacking fine texture and shape details. Therefore, in scenarios with high image complexity, the 3D reconstruction results of traditional methods are often unsatisfactory, lacking detail and accuracy. Furthermore, existing 2D-to-3D mapping techniques generally employ simple linear mapping or rule-based mapping methods. Although these methods can complete basic mapping tasks, their mapping accuracy is low, especially in cases of complex geometry or uneven feature distribution in 3D space, easily leading to distorted mapping results. The shortcoming of traditional mapping techniques is their inability to dynamically adjust mapping strategies for different image features, resulting in generated 3D models that cannot accurately reflect the geometric features of the original image, thus affecting the realism and accuracy of the 3D image. Therefore, those skilled in the art propose a feature recognition-based 3D image generation method to address these problems. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a 3D image generation method based on feature recognition, which solves problems such as insufficient feature extraction, inaccurate 2D to 3D mapping, lack of global and local considerations in image optimization, and inflexible output format.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a 3D image generation method based on feature recognition, comprising the following steps:

[0006] Receive the two-dimensional image to be processed and preprocess the image;

[0007] Multi-scale convolutional neural networks are used to extract features at multiple scales from the preprocessed image, resulting in multi-scale feature maps.

[0008] Features at different scales are weighted and fused to obtain a fused feature map.

[0009] Based on the optimal transmission theory, two-dimensional image features are mapped to three-dimensional space to obtain the mapped three-dimensional features;

[0010] By using total variation regularization, 3D features are optimized to suppress noise in the generated image while maintaining smoothness;

[0011] Geometric optimization is performed using variational methods, combined with surface curvature constraints to optimize the surface geometry of the 3D image.

[0012] Output the final generated 3D image.

[0013] Preferably, when extracting feature maps at multiple scales, the size of the convolution kernel used is selected according to the scale of the object. The multiple scales are low, medium and high scales, each corresponding to a convolution kernel of a different size.

[0014] Preferably, the step of performing weighted fusion includes:

[0015] Use scale attention mechanisms;

[0016] A fused feature map is generated by weighting features at different scales.

[0017] The weighting coefficients are obtained through learning, and the weighting coefficients satisfy the normalization condition, that is, the sum of all coefficients is 1.

[0018] Preferably, the optimal transmission theory minimizes the mapping relationship from two-dimensional features to three-dimensional features by calculating the mapping cost between two-dimensional image features and three-dimensional model features. The mapping relationship is solved by the optimal transmission algorithm, in which the cost function is the minimization of the distance between two-dimensional image feature points and three-dimensional model feature points.

[0019] Preferably, the cost function is:

[0020] ;

[0021] in: Here, is the cost function, representing the matching cost between feature points in a two-dimensional image and feature points in a three-dimensional model; Let be a feature point of a two-dimensional image, representing the th feature point in the two-dimensional image space. One feature point; For a 3D model feature point, it represents the first feature point in 3D space. 1 feature point.

[0022] Preferably, the total variation regularization is used to ensure the smoothness of the image surface and reduce noise in the image.

[0023] Preferably, the energy function for the geometry optimization is:

[0024] ;

[0025] in: An energy function for geometry optimization, used to optimize the surface morphology of 3D images; Surface parameterization for 3D images; The second derivative of the image represents the rate of change of the image surface at a certain point. and These represent the average curvature and Gaussian curvature of the surface, respectively; and Regularization parameters used to control the effect of curvature; This refers to the spatial region of the image.

[0026] Preferably, the three-dimensional image is in the form of a three-dimensional mesh, which has a polygonal mesh structure composed of vertices and faces.

[0027] Preferably, the image preprocessing further includes image normalization and cropping, mapping the pixel values ​​of the image to a uniform range, and cropping it to the target size for subsequent processing.

[0028] A 3D image generation system based on feature recognition, comprising:

[0029] The image input module is used to receive the two-dimensional image to be processed;

[0030] The feature extraction module is used to extract features at multiple scales from an image using a multi-scale convolutional neural network.

[0031] The feature fusion module is used to perform weighted fusion of extracted features at multiple scales;

[0032] The mapping module is used to map two-dimensional image features to three-dimensional space to generate three-dimensional features;

[0033] An optimization module is used to optimize the generated 3D features based on total variation regularization and geometric optimization;

[0034] The output module is used to generate and output the final 3D image.

[0035] This invention provides a 3D image generation method based on feature recognition. It has the following beneficial effects:

[0036] 1. This invention uses a multi-scale convolutional neural network (MS-CNN) for feature extraction. By fusing feature maps of different scales, the system can extract multi-level information from macroscopic to microscopic levels from two-dimensional images, achieving comprehensive capture of image details and structure. Compared with the single-scale convolutional network scheme in the prior art, this invention can more accurately reconstruct complex three-dimensional shapes, avoid the shortcomings of traditional methods in detail reconstruction, and make the final three-dimensional image more refined and realistic.

[0037] 2. This invention employs optimal transmission theory for two-dimensional to three-dimensional feature mapping, accurately calculating the mapping relationship between two-dimensional image features and three-dimensional space, thereby generating a three-dimensional image that conforms to geometric laws. Compared with the simple linear mapping or artificial rule-based schemes in the prior art, this invention improves the mapping accuracy through optimal transmission, avoiding geometric distortion and loss of detail that may be caused by traditional methods.

[0038] 3. This invention combines total variation regularization and geometric optimization to optimize 3D images, improving image smoothness and geometric accuracy. Compared with independent optimization methods in traditional methods, this invention, through joint optimization, not only removes noise but also preserves image details and edges, resulting in a smoother 3D image that conforms to the actual physical model.

[0039] 4. The system design of this invention is flexible, and the output module supports multiple three-dimensional data formats. It can generate three-dimensional mesh or point cloud data according to different application requirements and perform efficient conversion. Compared with the existing technologies with low format conversion efficiency or lack of compatibility, this invention provides a highly compatible and fast data output method, ensuring the wide application and efficient utilization of three-dimensional images in various fields such as virtual reality and three-dimensional printing. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0041] Figure 2 This is a schematic diagram of the system architecture of the present invention. Detailed Implementation

[0042] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] Please see the appendix Figure 1 This invention provides a 3D image generation method based on feature recognition, comprising the following steps:

[0044] S1. Receive the two-dimensional image to be processed and preprocess the image;

[0045] Specifically, image preprocessing is a crucial step that lays a solid foundation for subsequent feature extraction and 3D reconstruction. The goal of preprocessing is to convert the input 2D image into a format suitable for subsequent processing and to improve the accuracy of feature extraction by removing irrelevant information and standardizing operations. Preprocessing is usually the starting point of the entire process and is closely linked to the subsequent steps of feature extraction and 3D image generation.

[0046] Image preprocessing consists of two main parts: image normalization and cropping. Initially, the received two-dimensional images often have different sizes and pixel value ranges. Directly using them for subsequent processing may lead to increased noise and computational instability in the feature extraction process. Therefore, image normalization is the first step. The purpose of normalization is to map image pixel values ​​to a uniform range, typically by scaling the pixel values ​​to the range [0,1]. This process is achieved using the following formula:

[0047] ;

[0048] in: The pixel values ​​of the input image; The minimum pixel value of the input image; The maximum pixel value of the input image; These are the standardized pixel values.

[0049] After standardization, the brightness range of the image is unified, which helps to improve the stability and accuracy of the subsequent feature extraction process.

[0050] Cropping removes irrelevant parts of an image. Often, the background or irrelevant areas of an image can interfere with feature extraction, so cropping is performed to focus on the target area in the image. The size of the cropping varies depending on the specific application requirements, and it is usually cropped into a square or rectangular area. The purpose of cropping is to reduce the amount of computation and improve the efficiency and accuracy of subsequent processing.

[0051] Image cropping can be done in the following ways:

[0052] First, the target region in the image is identified, and then the region of interest (ROI) is obtained using edge detection algorithms or deep learning algorithms.

[0053] Then, the image is cropped into the smallest rectangular region containing the ROI, and irrelevant parts of the image are removed. This step helps to remove background noise and ensures that feature extraction focuses on the target region.

[0054] Cropping can also be dynamically adjusted according to the shape of the target object. For example, if the target object is non-rectangular or irregular in shape, it can be cropped into a region that matches the target shape through segmentation technology, which can further improve the effect of subsequent feature extraction.

[0055] The ultimate goal of image preprocessing is to ensure that the quality of the input image is suitable for subsequent multi-scale convolutional neural network (MS-CNN) feature extraction. Image normalization and cropping steps can significantly improve the robustness of feature extraction and provide clear and accurate data for the subsequent processing.

[0056] S2. Use a multi-scale convolutional neural network to extract features at multiple scales from the preprocessed image to obtain multi-scale feature maps.

[0057] Specifically, feature extraction is a crucial step. After image preprocessing in step S1, the resulting image is fed into a multi-scale convolutional neural network (MS-CNN) for feature extraction. The goal of this stage is to extract feature information at different scales from the preprocessed image for use in the subsequent weighted fusion process. The quality of feature extraction directly affects the accuracy of the final generated 3D image, so the design of this step is crucial.

[0058] Multi-scale convolutional neural networks (MS-CNN) are used for feature extraction. In practical applications, different regions in an image contain information of different scales, which is crucial for 3D reconstruction. Through multi-scale convolutional neural networks, we can extract image features at multiple scales simultaneously. In this way, the network can more comprehensively understand various structures, textures, and local details in the image. This multi-scale feature extraction method can effectively enhance the robustness of the network and ensure effective recognition of targets of different sizes.

[0059] The size of each convolutional kernel in the network is set according to the scale of the target. The low-scale convolutional kernels are larger and are used to extract macroscopic features in the image, such as the shape of large objects. The medium-scale and high-scale convolutional kernels are used to extract finer features, such as details and textures. These convolutional kernels of different scales help to capture hierarchical information from large to small, thereby improving the accuracy of subsequent 3D reconstruction.

[0060] Each scale of the convolutional layer uses a different kernel size. The choice of these kernel sizes is based on the size of the object in the image and the target features. For example, a 7×7 kernel may be used for low scales, a 5×5 kernel for medium scales, and a 3×3 kernel for high scales. In this way, the convolutional neural network is able to capture global features from large scales to details.

[0061] Convolutional neural networks employ a multi-channel input design to extract image features from multiple angles simultaneously. For example, by using the RGB three color channels as input, the network can extract information from each color channel separately and then perform weighted integration during subsequent fusion. This design can further enhance the network's adaptability to complex images, especially in scenes rich in color and texture information.

[0062] Pooling layers are also used in the feature extraction process. Pooling operations help reduce the resolution and computational cost of the image while maintaining the saliency of important features in the image. In multi-scale convolutional neural networks, pooling layers are usually placed after convolutional layers. Through pooling operations, the network can work in a smaller feature space, avoiding the consumption of computational resources caused by excessively large image sizes. This approach not only effectively improves computational efficiency but also ensures the stability of the feature extraction process.

[0063] The result of feature extraction is feature maps at multiple scales. Typically, each scale outputs a corresponding feature map, which will serve as the input for the next weighted fusion step. These feature maps contain information at different levels of the image, ranging from global shape to local details.

[0064] In multi-scale convolutional neural networks, the output of each convolutional layer is a feature map, which contains all the local features obtained from the convolutional kernel of that layer. For feature maps of different scales, larger convolutional kernels are usually used to extract larger structural features, while smaller convolutional kernels are used to extract details and local features. In this way, the network can capture the diverse information of the image from different levels and provide richer features for subsequent processing.

[0065] S3. Weighted fusion of features at different scales is performed to obtain the fused feature map;

[0066] Specifically, in the aforementioned step S2, after processing by the multi-scale convolutional neural network (MS-CNN), feature maps of multiple scales have been obtained. These feature maps contain information at different levels of the image, from global shape to local details, covering a variety of features of the image. In order to effectively fuse these different scales of information and improve the accuracy and quality of 3D image generation, the next step is to perform weighted fusion.

[0067] The feature weighted fusion step is implemented through a scale attention mechanism. The core idea of ​​this mechanism is to assign different weighting coefficients to each scale feature based on its importance in the generation of the 3D image. Through this weighting method, features of different scales contribute different weights in the fusion process, thereby optimizing the final feature representation and enhancing the overall quality of the 3D image.

[0068] The scale attention mechanism first weights the feature maps at each scale. These weighting coefficients are automatically learned by the network to ensure that each scale's features receive appropriate attention based on their role in the generation process. The weighting coefficients are calculated according to the following formula:

[0069] ;

[0070] in: The fused feature map represents the final result obtained by weighted fusion of feature maps from multiple scales. For the first Feature maps at different scales represent image features extracted at different scales; This represents the total number of scales in the feature map, indicating the number of different scales used. The weighting coefficients for the corresponding scales represent the weighting coefficients for the th scale. The importance of each scale feature in the fusion process, and the weighting coefficients satisfy the normalization condition:

[0071] ;

[0072] That is, the sum of the weighting coefficients is 1, ensuring that the relative importance of features at each scale can be reasonably allocated according to the model's learning during the weighted fusion process.

[0073] To obtain the optimal weighting coefficients, the network employs an adaptive learning strategy, which automatically optimizes these weighting coefficients during the training process to maximize the contribution of each scale feature in a specific task. This means that the network can dynamically adjust the weights of each scale feature so that the final fused feature map best represents the whole picture of the input image.

[0074] Low-scale features are mainly used to provide macroscopic structural information in an image, while high-scale features usually contain fine local information about the image. In this case, the weighting coefficients of low-scale features may be smaller, while the weighting coefficients of high-scale features may be larger, in order to reflect the importance of image details.

[0075] If certain features in an image have high-level information (such as texture or edges), these features may be crucial for generating a 3D image. In such cases, the weighting coefficients of these features will be automatically increased. For example, when dealing with objects with complex textures, the network may automatically enhance the weights of high-scale features in order to retain more detail in the final 3D model.

[0076] Through this weighted fusion method, the network can effectively integrate low-, medium-, and high-scale feature information. Low-scale features provide a general framework for the overall shape, medium-scale features help capture finer structures, and high-scale features further reveal the local details of the image. Through this weighted fusion of multi-level features, the final feature map can make full use of the advantages of features at each scale, ensuring the richness and detail of the three-dimensional image.

[0077] The weighted fused feature maps can be further processed by nonlinear activation functions, such as ReLU (Rectified Linear Unit), to enhance the network's ability to express features. Activation functions can effectively improve the nonlinear expression of the fused feature maps, thereby further improving the quality of the generated 3D images.

[0078] The result of weighted fusion is a feature map that integrates multi-scale information. These features will be passed to subsequent steps for 3D reconstruction or other related tasks. This step plays a crucial role in the 3D image generation process because it enables the network to make full use of the multi-level information in the input image and improves the expressiveness and accuracy of the final result through automatically learned weighting.

[0079] S4. Based on the optimal transmission theory, the two-dimensional image features are mapped to the three-dimensional space to obtain the mapped three-dimensional features;

[0080] Specifically, in step S3, the feature maps of multiple scales are weighted and fused through the scale attention mechanism to obtain a fused feature map containing global and local information. The next step is to map these two-dimensional image features to three-dimensional space to provide rich feature information for the generation of three-dimensional images. The core objective of this step is to accurately map the two-dimensional features to three-dimensional space according to the optimal transfer theory to ensure the accuracy and detail of the generated three-dimensional images.

[0081] The 2D to 3D feature mapping is achieved through optimal transfer theory. The role of optimal transfer theory here is to calculate the best matching relationship between 2D image features and corresponding features in 3D space. By minimizing the cost function, 2D features are accurately mapped to 3D space, so that the 3D image can reflect the same structure and details as the 2D image.

[0082] The goal of the optimal transfer algorithm is to minimize the mapping cost between two-dimensional and three-dimensional features. This mapping cost is calculated based on the Euclidean distance between two-dimensional image feature points and points in three-dimensional space. To obtain the optimal mapping relationship, the optimal transfer algorithm solves a problem of minimizing a cost function, defined as:

[0083] ;

[0084] in: Here, is the cost function, representing the matching cost between feature points in a two-dimensional image and feature points in a three-dimensional model; Let be a feature point of a two-dimensional image, representing the th feature point in the two-dimensional image space. One feature point; For a 3D model feature point, it represents the first feature point in 3D space. 1 feature point.

[0085] Cost function The Euclidean distance between two-dimensional and three-dimensional feature points was calculated. The aim was to find the optimal mapping relationship by minimizing this distance. Through the optimal transfer algorithm, the two-dimensional image features can find the best matching point in three-dimensional space to complete the mapping.

[0086] In practical implementation, the mapping results can be optimized by introducing constraints. For example, the mapping process can be restricted to a certain predetermined three-dimensional space range to ensure that two-dimensional feature points are only mapped to specific three-dimensional regions. This helps to improve the efficiency of the mapping process and ensures that the geometric structure of the three-dimensional image is more in line with actual needs.

[0087] This mapping process can also be combined with other optimization methods, such as incremental optimization or gradient descent algorithms, to make the mapping from two dimensions to three dimensions more accurate by iteratively adjusting the mapping relationship. In addition, the mapping results can be corrected by additional constraints to ensure that the point cloud or mesh generated in three-dimensional space conforms to the required geometry.

[0088] This optimal transfer algorithm not only optimizes geometric accuracy but also effectively processes texture and detail information in images, thus ensuring that the generated 3D images have high detail representation. Through the optimal transfer theory, the mapping from 2D features to 3D features can avoid error accumulation in traditional methods, making the 3D images closer to the real scene.

[0089] In practice, the accuracy of the 2D to 3D mapping is crucial for subsequent 3D generation. If the cost function is not chosen properly during the mapping process, it may lead to distortion or aberration in the 3D image. Therefore, choosing the optimal cost function and its optimization strategy will directly affect the quality of the 3D image. Through reasonable cost function design, optimal transfer theory can provide a stable and efficient mapping process to ensure that the generated 3D image has high accuracy.

[0090] In certain specific application scenarios, the mapping process can be further enhanced by combining deep learning models. For example, deep learning methods such as Generative Adversarial Networks (GANs) or Variational Autoencoders (VAEs) can be used to further optimize the mapping results and improve the quality of the generated 3D images. These methods can improve image quality by enhancing the preservation of details in the mapping process, especially in the 3D reconstruction of complex objects and textures, where the effect is particularly outstanding.

[0091] S5. Optimize 3D features through total variation regularization;

[0092] Specifically, in step S4, the two-dimensional image features are successfully mapped to three-dimensional space using optimal transfer theory, and preliminary three-dimensional feature maps are generated. These three-dimensional feature maps contain rich information extracted from the two-dimensional image. However, there may still be noisy or unsmooth areas in the generated three-dimensional image. In order to further improve the quality of the three-dimensional image, this step uses the total variation regularization method for optimization to ensure that the generated three-dimensional image is smoother and the details are more realistic.

[0093] Total Variation Regularization (TV) is used to optimize 3D features, aiming to reduce noise in the image while preserving edge information. TV is a commonly used optimization method in image processing. Its core idea is to suppress noise by controlling the gradient change of the image and ensure the smoothness of the image while maintaining image details. This method is particularly suitable for avoiding surface distortion caused by noise when processing 3D reconstructed images.

[0094] Total variation regularization optimizes the image by minimizing the square of the gradient in the 3D image, thus smoothing out local variations and avoiding overly drastic changes. The mathematical expression for the total variation regularization term is as follows:

[0095] ;

[0096] in: The total variation regularized energy function represents the smoothness or glossiness of the surface parameterization of a 3D image. Surface parameterization of a 3D image; The gradient of the image represents the rate of change of the image at a certain point. Represents the spatial region of an image.

[0097] By minimizing this regularization term, noise in 3D images can be effectively reduced while maintaining their smooth transitions.

[0098] During the optimization process, the regularization term can also be weighted in conjunction with other image feature constraints. For example, image edge detection information may be introduced to further suppress excessive changes in noisy regions while preserving edge details. In this way, the surface of the 3D image can be kept smooth while maintaining the sharpness of object edges, avoiding the loss of details caused by excessive smoothing.

[0099] Total variation regularization can also be combined with local optimization algorithms, such as gradient descent or Newton's method. These optimization methods can further improve the convergence speed and accelerate the iteration of the total variation regularization process. In each iteration, the optimization algorithm updates the image parameters and ensures the improvement of image smoothness by minimizing the sum of squared gradients.

[0100] Total variation regularization can effectively eliminate random noise in an image while preserving its texture and structural features. Therefore, for 3D image generation tasks, applying total variation regularization will result in more realistic and natural 3D images, especially when dealing with objects with complex shapes, as it can effectively avoid unnatural smoothing effects.

[0101] When total variation regularization is applied, the detailed information of the image can still be effectively preserved, especially at the edges of objects or important structural parts. The regularization method does not introduce obvious distortion. While the global smoothness of the image is improved, important local features such as edges and curves can also be clearly presented, thereby improving the quality of the final 3D image.

[0102] Total variation regularization can also be combined with other deep learning methods to further optimize the quality of 3D images. For example, using convolutional neural networks (CNNs) in conjunction with total variation regularization to process the details of 3D images, CNNs can further extract features and accelerate the optimization process, making the details of the images more refined.

[0103] S6. Perform geometric optimization using variational methods, and combine surface curvature constraints to optimize the surface geometry of the 3D image.

[0104] Specifically, in the aforementioned step S5, the three-dimensional image undergoes total variation regularization optimization to remove noise and improve image smoothness. However, although the image after regularization is improved in terms of local details and surface smoothness, the geometric shape of the three-dimensional image may still have some distortions or not conform to the actual physical laws. Therefore, the surface shape of the three-dimensional image is further corrected by geometric optimization methods to ensure that the generated three-dimensional image is more in line with natural geometric constraints and to avoid unnatural surface distortions.

[0105] In this step, geometric optimization is performed using the variational method, a mathematical approach commonly used to solve optimization problems. It optimizes the shape of an image by minimizing an energy function. In 3D image generation, the energy function is usually combined with the curvature information of the image, such as mean curvature and Gaussian curvature, to adjust the image surface, thereby ensuring that the shape of the 3D image is both natural and conforms to geometric principles.

[0106] The energy function for geometric optimization can be defined as:

[0107] ;

[0108] in: An energy function for geometry optimization, used to optimize the surface morphology of 3D images; Surface parameterization for 3D images; The second derivative of the image represents the rate of change of the image surface at a certain point. and These represent the average curvature and Gaussian curvature of the surface, respectively; and Regularization parameters used to control the effect of curvature; This refers to the spatial region of the image.

[0109] This energy function optimizes the surface morphology of an image by balancing its smoothness and curvature constraints, making it more consistent with natural geometry.

[0110] Mean curvature and Gaussian curvature are two important parameters for describing the shape of a surface. Mean curvature can reflect the degree of curvature of a local surface, while Gaussian curvature describes the product of the two principal curvatures on the surface. It is usually used to determine the geometric properties of points on the surface. By incorporating this curvature information into the energy function, the shape changes of the image can be effectively controlled during the optimization process, avoiding geometrically unnatural or excessively deformed situations.

[0111] In practical applications, the weighting coefficients of curvature may be adjusted during the optimization process based on different object types or application requirements. and Dynamic adjustments can be made. For example, when dealing with smooth objects, Gaussian curvature may be given more attention to maintain the smoothness of the overall surface; while when dealing with complex objects or objects with obvious curved structures, constraints on the average curvature may be increased to maintain the richness and accuracy of details.

[0112] In the geometric optimization process, the optimization algorithm iteratively adjusts the surface parameters of the 3D image using a variational method. Each iteration adjusts the surface based on the current energy function value. The optimization objective is to minimize the energy function value, ultimately obtaining a smooth 3D image that conforms to geometric laws. To accelerate the optimization process, optimization algorithms such as gradient descent and Newton's method may be used. These methods can gradually adjust the surface morphology based on gradient information, improving optimization efficiency.

[0113] To better adapt to different geometric structures, optimization can be selectively performed based on local features of the image. For example, at the edges of a 3D object, local optimization algorithms may be used to increase control over curvature to maintain edge clarity and shape; while in relatively flat areas, curvature constraints can be appropriately reduced to improve optimization efficiency.

[0114] This geometric optimization process is crucial to the quality of 3D image generation. It not only effectively corrects morphological distortions in the image but also preserves the naturalness of the 3D model, making the generated 3D image more realistic and reliable. By combining curvature information and variational optimization methods, the surface of the 3D image is effectively adjusted, making the image more perfect in terms of geometric structure and visual effect.

[0115] In scenarios involving complex objects or requiring high precision, more refined optimization strategies can be introduced. For example, by introducing more geometric constraints, the local morphological representation of 3D images can be further enhanced, making them more consistent with the geometric characteristics of real objects. This optimization method has broad application prospects in fields such as virtual reality, medical imaging, and industrial design.

[0116] S7. Output the final generated 3D image.

[0117] Specifically, after the optimization processes in steps S1 to S6, the accuracy and detail of the 3D image are effectively improved. From image preprocessing to feature extraction, weighted fusion, 2D to 3D mapping, total variation regularization optimization, and finally geometric optimization, all processing steps have ensured the quality of the 3D image. Next, the final 3D image will be provided to subsequent applications through the output step. This process is crucial because it transforms the generated 3D data into a form that can be used in practical applications, such as virtual reality, 3D printing, and medical imaging.

[0118] The final generated 3D image is output in the form of a mesh or a point cloud. The output format of the 3D image depends on the needs of subsequent applications. In virtual reality, 3D meshes are often used to represent the surface structure of objects because they can effectively represent the shape and details of objects. In 3D printing, point cloud data is more commonly used because it can provide discrete 3D spatial points, making it easier for printers to generate actual objects based on the point cloud.

[0119] The output process of a 3D mesh includes the following steps: First, through the aforementioned geometric optimization steps, the generated 3D image already has a relatively smooth surface that conforms to geometric rules; next, the surface of the image will be discretized into a mesh structure composed of vertices and faces. Each vertex corresponds to a coordinate in 3D space, while the face is a polygon formed by connecting vertices. In most applications, the mesh face is usually a triangle or a quadrilateral. These faces together constitute the 3D surface of the object.

[0120] The vertex coordinates of the mesh can be obtained through surface parameterization of the 3D image, and the surface is generated by the combination of these vertices. In order to ensure that the generated mesh surface is as accurate and realistic as possible, various optimization algorithms, such as Laplacian smoothing and edge smoothing, can be used to further adjust the structure of the mesh, remove redundant vertices, and optimize the mesh connection relationship.

[0121] The generated mesh can be further refined and optimized. For example, mesh simplification techniques can be used to reduce the number of unnecessary polygons while maintaining surface details. This simplification technique is widely used in computer graphics and 3D reconstruction, especially when dealing with complex models, as it can significantly reduce computation and improve rendering speed.

[0122] For some special application scenarios, the output 3D data can also be processed in the form of point clouds. A point cloud is a data structure composed of discrete 3D points. Each point has a specific coordinate in 3D space. These point cloud data are usually used in fields such as 3D scanning, 3D printing and 3D modeling. When generating a point cloud, each valid point of the image corresponds to a coordinate in 3D space.

[0123] Point cloud generation typically involves establishing a corresponding point in three-dimensional space for each pixel or feature point of an image. To improve the density and accuracy of the point cloud, interpolation methods, such as B-spline interpolation and Kalman filtering, can be used during the generation process to ensure the uniformity and density of the point cloud data. In addition, point cloud data can be post-processed, such as noise removal and point cloud registration, to ensure that the final point cloud data is more accurate.

[0124] In some high-precision applications, the output of 3D images can also be combined with texture mapping technology to map the texture information of the original image onto the surface of the generated 3D model. This method can enhance the realism of 3D images, especially in situations where it is necessary to accurately represent the details and textures of an object's surface, such as in medical imaging and virtual reality.

[0125] The 3D image generation system based on feature recognition described below can be referred to in correspondence with the 3D image generation method based on feature recognition described above.

[0126] Please see the appendix Figure 2 A 3D image generation system based on feature recognition, comprising:

[0127] The image input module is used to receive the two-dimensional image to be processed;

[0128] The feature extraction module is used to extract features at multiple scales from an image using a multi-scale convolutional neural network.

[0129] The feature fusion module is used to perform weighted fusion of extracted features at multiple scales;

[0130] The mapping module is used to map two-dimensional image features to three-dimensional space to generate three-dimensional features;

[0131] An optimization module is used to optimize the generated 3D features based on total variation regularization and geometric optimization;

[0132] The output module is used to generate and output the final 3D image.

[0133] Specifically, the main function of the image input module is to receive the two-dimensional image to be processed and preprocess it into a data format suitable for subsequent processing. The image input module can support multiple image formats to ensure that the system can be compatible with image data from different sources and standards. In practical applications, the image input module may include an image acquisition device interface or a file upload interface. Through these interfaces, the system can acquire the image to be processed.

[0134] The feature extraction module extracts features at different scales from the input 2D image using a multi-scale convolutional neural network (MS-CNN). This module uses multiple convolutional layers to learn multi-level and multi-angle features in the image, including macroscopic shape information and detailed local texture information. The feature map at each scale can provide different levels of spatial information, thus providing sufficient data support for subsequent feature fusion and 3D reconstruction.

[0135] The main task of the feature fusion module is to perform weighted fusion of features extracted from different scales to generate a fused feature map for 3D mapping. Through the scale attention mechanism, the module dynamically adjusts the weighting coefficients of each scale feature according to the importance of each scale feature. The fused feature map contains valuable information from each scale, providing rich feature data for 3D image generation.

[0136] The mapping module is responsible for mapping two-dimensional image features to three-dimensional space to generate corresponding three-dimensional feature maps. Based on the optimal transfer theory, this module calculates the mapping cost between two-dimensional image features and three-dimensional model features to obtain the optimal two-dimensional to three-dimensional mapping relationship. The mapping process ensures that the structure and details of the three-dimensional image are consistent with the original two-dimensional image, avoiding distortion or deformation.

[0137] The optimization module uses total variation regularization and geometric optimization algorithms to optimize the generated 3D features to ensure the surface smoothness, geometric accuracy and visual effect of the 3D image. Total variation regularization helps to reduce noise in the image and maintain surface smoothness, while geometric optimization ensures that the 3D image conforms to natural geometric laws by combining surface curvature constraints.

[0138] The output module converts the processed and optimized 3D feature data into standard 3D image formats, such as meshes and point clouds, and outputs them for subsequent applications. The output module supports a variety of 3D data formats and can convert 3D images into formats suitable for virtual reality, 3D printing, or other visualization applications as needed.

[0139] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A 3D image generation method based on feature recognition, characterized in that, Includes the following steps: Receive the two-dimensional image to be processed and preprocess the image; Multi-scale convolutional neural networks are used to extract features at multiple scales from the preprocessed image, resulting in multi-scale feature maps. Features at different scales are weighted and fused to obtain a fused feature map. Based on the optimal transmission theory, two-dimensional image features are mapped to three-dimensional space to obtain the mapped three-dimensional features; Optimize 3D features through total variation regularization; Geometric optimization is performed using variational methods, combined with surface curvature constraints to optimize the surface geometry of the 3D image. Output the final generated 3D image.

2. The 3D image generation method based on feature recognition according to claim 1, characterized in that, When extracting feature maps at multiple scales, the size of the convolution kernel used is selected according to the scale of the object. The multiple scales are low, medium and high scales, each corresponding to a convolution kernel of a different size.

3. The 3D image generation method based on feature recognition according to claim 1, characterized in that, The steps for weighted fusion include: Use scale attention mechanisms; A fused feature map is generated by weighting features at different scales. The weighting coefficients are obtained through learning, and the weighting coefficients satisfy the normalization condition, that is, the sum of all coefficients is 1.

4. The 3D image generation method based on feature recognition according to claim 1, characterized in that, The optimal transmission theory minimizes the mapping relationship from two-dimensional features to three-dimensional features by calculating the mapping cost between two-dimensional image features and three-dimensional model features. The mapping relationship is solved by the optimal transmission algorithm, in which the cost function is the minimum distance between two-dimensional image feature points and three-dimensional model feature points.

5. The 3D image generation method based on feature recognition according to claim 4, characterized in that, The cost function is: ； in: Here, is the cost function, representing the matching cost between feature points in a two-dimensional image and feature points in a three-dimensional model; Let be a feature point of a two-dimensional image, representing the th feature point in the two-dimensional image space. One feature point; For a 3D model feature point, it represents the first feature point in 3D space. 1 feature point.

6. The 3D image generation method based on feature recognition according to claim 1, characterized in that, The total variation regularization is used to ensure the smoothness of the image surface and reduce noise in the image.

7. The 3D image generation method based on feature recognition according to claim 1, characterized in that, The energy function for the geometry optimization is: ； in: An energy function for geometry optimization, used to optimize the surface morphology of 3D images; Surface parameterization for 3D images; The second derivative of the image represents the rate of change of the image surface at a certain point. and These represent the average curvature and Gaussian curvature of the surface, respectively; and Regularization parameters used to control the effect of curvature; This refers to the spatial region of the image.

8. The 3D image generation method based on feature recognition according to claim 1, characterized in that, The three-dimensional image is in the form of a three-dimensional mesh, which is a polygonal mesh structure composed of vertices and faces.

9. The 3D image generation method based on feature recognition according to claim 1, characterized in that, The image preprocessing further includes image normalization and cropping, mapping the pixel values ​​of the image to a uniform range and cropping it to the target size for subsequent processing.

10. A 3D image generation system based on feature recognition, applied to the 3D image generation method based on feature recognition as described in any one of claims 1-9, characterized in that, include: The image input module is used to receive the two-dimensional image to be processed; The feature extraction module is used to extract features at multiple scales from an image using a multi-scale convolutional neural network. The feature fusion module is used to perform weighted fusion of extracted features at multiple scales; The mapping module is used to map two-dimensional image features to three-dimensional space to generate three-dimensional features; An optimization module is used to optimize the generated 3D features based on total variation regularization and geometric optimization; The output module is used to generate and output the final 3D image.

Images