Methods, systems, terminal devices, and storage media for three-dimensional reconstruction of two-dimensional images

Abstract

This invention discloses a method, system, terminal device, and storage medium for three-dimensional reconstruction of two-dimensional images. The method includes: acquiring an image to be processed; inputting the image to be processed into a pre-created recognition model to obtain a feature map; generating a three-dimensional image based on the feature map; mapping the annotation information of the image to be processed onto the three-dimensional image to obtain a final image; and associating the image to be processed and its annotation information with the final image and its annotation information to obtain a database. This invention reduces information loss during the conversion of two-dimensional planar images to three-dimensional images, improves the accuracy of the conversion, provides more precise model recognition results, and offers users a more intuitive browsing experience.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, system, terminal device, and storage medium for three-dimensional reconstruction of two-dimensional images. Background Technology

[0002] Artificial intelligence-assisted diagnosis relies on various machine learning algorithms, primarily convolutional neural networks, to convert two-dimensional images into three-dimensional models, identify labeled content within the two-dimensional images, and then convert the labeled content into the three-dimensional model. However, some information is lost during the conversion process. Summary of the Invention

[0003] The main objective of this invention is to provide a method, system, terminal device, and storage medium for three-dimensional reconstruction of two-dimensional images, aiming to reduce information loss when converting two-dimensional planar images into three-dimensional models, improve the accuracy of three-dimensional models, provide more accurate model recognition results, and offer users a more intuitive browsing experience.

[0004] To achieve the above objectives, embodiments of the present invention provide a method for three-dimensional reconstruction of a two-dimensional image, the method comprising:

[0005] Obtain the image to be processed;

[0006] The image to be processed is input into a pre-created recognition model for processing to obtain a feature map;

[0007] A three-dimensional image is generated based on the feature map;

[0008] The annotation information of the image to be processed is mapped onto the three-dimensional image to obtain the final image;

[0009] The image to be processed and its annotation information are associated with the final image and its annotation information to obtain a database.

[0010] Optionally, the step of generating a 3D image based on the feature map includes:

[0011] Extract keyframes from the feature map;

[0012] Based on the keyframes, calculate the estimated depth;

[0013] The 3D image is generated based on the keyframes, the estimated depth, and the image to be processed.

[0014] Optionally, the step of inputting the image to be processed into a pre-created recognition model for processing to obtain a feature map includes:

[0015] Train the recognition model;

[0016] The steps for training the recognition model include:

[0017] Acquire training images;

[0018] Identify the training image to obtain the detection bounding box of the training image;

[0019] The detection boxes of the training images are classified to obtain the category information of the detection boxes of the training images;

[0020] The category information of the detection boxes of the training image is fed back to the recognition model, and the execution step is returned: recognize the training image to obtain the detection boxes of the training image;

[0021] This process is repeated to iterate the parameters until the recognition model converges, at which point training is terminated.

[0022] Optionally, the step of inputting the image to be processed into a pre-created recognition model to obtain a feature map includes:

[0023] The image to be processed is input into the pre-created recognition model for recognition, and the detection box of the image to be processed is obtained;

[0024] The recognition model is used to classify the detection boxes of the image to be processed to obtain the feature map, wherein the feature map carries the category information of the detection boxes of the image to be processed.

[0025] Optionally, the step of calculating the estimated depth based on the keyframe includes:

[0026] Establish a matching relationship based on the feature points of the keyframes;

[0027] A depth map is established based on the keyframes and the matching relationships;

[0028] The 3D image is generated based on the depth map, the estimated depth, and the image to be processed.

[0029] Optionally, after the step of inputting the image to be processed into a pre-created recognition model to obtain a feature map, the following steps are included:

[0030] The coordinates of the feature map are adjusted according to the preset frame.

[0031] Optionally, the step of inputting the image to be processed into a pre-created recognition model for processing to obtain a feature map includes:

[0032] The image to be processed is preprocessed to obtain a standardized image;

[0033] The standardized image is input into the recognition model for processing to obtain the feature map.

[0034] Furthermore, to achieve the above objectives, the present invention also provides a three-dimensional reconstruction system for a two-dimensional image, the system comprising:

[0035] The image acquisition module is used to acquire the image to be processed.

[0036] The image recognition module is used to input the image to be processed into a pre-created recognition model for processing to obtain a feature map;

[0037] An image generation module is used to generate a three-dimensional image based on the feature map;

[0038] An image mapping module is used to map the annotation information of the image to be processed onto the three-dimensional image to obtain the final image;

[0039] The database generation module is used to associate the image to be processed and its annotation information with the final image and its annotation information to obtain a database.

[0040] In addition, to achieve the above objectives, the present invention also provides a terminal device, the terminal device comprising: a memory, a processor, and a three-dimensional reconstruction method for a two-dimensional image stored in the memory and executable on the processor, wherein when the program for three-dimensional reconstruction of the two-dimensional image is executed by the processor, the steps of the three-dimensional reconstruction method for the two-dimensional image as described above are implemented.

[0041] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a program for three-dimensional reconstruction of a two-dimensional image, wherein when the program for three-dimensional reconstruction of a two-dimensional image is executed by a processor, the program implements the steps of the three-dimensional reconstruction method of the two-dimensional image as described above.

[0042] This invention proposes a method, system, terminal device, and storage medium for 3D reconstruction of 2D images. The process involves: acquiring an image to be processed; inputting the image to be processed into a pre-created recognition model to obtain a feature map; generating a 3D image based on the feature map to convert the 2D image into a 3D image, reducing information loss during the conversion; mapping the annotation information of the image to be processed onto the 3D image to obtain a final image, thereby improving the integrity of information during the 3D image conversion and facilitating user identification of the specific components of the 3D image based on the annotation information; and associating the image to be processed and its annotation information with the final image and its annotation information to obtain a database. Through this method, the invention reduces information loss during the conversion of 2D images to 3D images, improves the accuracy of the 2D image to 3D image conversion, provides a more accurate model recognition effect, and offers users a more intuitive browsing experience. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the functional modules of the terminal device to which the two-dimensional image three-dimensional reconstruction device of the present invention belongs;

[0044] Figure 2 This is a flowchart illustrating the first embodiment of the three-dimensional reconstruction method for two-dimensional images according to the present invention.

[0045] Figure 3 This is a flowchart illustrating a second embodiment of the three-dimensional reconstruction method for two-dimensional images according to the present invention.

[0046] Figure 4 This is a flowchart illustrating the third embodiment of the three-dimensional reconstruction method for two-dimensional images according to the present invention.

[0047] Figure 5 This is a schematic diagram of the functional modules of the three-dimensional reconstruction system for two-dimensional images according to the present invention.

[0048] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0049] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0050] The main solution of this invention is as follows: acquiring an image to be processed; inputting the image to be processed into a pre-created recognition model for processing to obtain a feature map; generating a three-dimensional image based on the feature map; mapping the annotation information of the image to be processed onto the three-dimensional image to obtain a final image; and associating the image to be processed and its annotation information with the final image and its annotation information to obtain a database.

[0051] Technical terms involved in the embodiments of this invention:

[0052] CNN (Convolutional Neural Networks) is a type of feedforward neural network that incorporates convolutional computations and has a deep structure. It is one of the representative algorithms of deep learning. Convolutional neural networks have representation learning capabilities and can perform translation-invariant classification of input information according to their hierarchical structure.

[0053] The input layer of a convolutional neural network can process multidimensional data. Typically, the input layer of a one-dimensional convolutional neural network receives a one-dimensional or two-dimensional array, where the one-dimensional array is usually a time or spectral sample; the two-dimensional array may contain multiple channels; the input layer of a two-dimensional convolutional neural network receives a two-dimensional or three-dimensional array; and the input layer of a three-dimensional convolutional neural network receives a four-dimensional array.

[0054] The hidden layers of convolutional neural networks (CNNs) commonly include three types of architectures: convolutional layers, pooling layers, and fully connected layers. More modern algorithms may employ more complex architectures such as Inception modules and residual blocks. Among these common architectures, convolutional and pooling layers are unique to CNNs. The kernels in convolutional layers contain weights, while pooling layers do not; therefore, pooling layers may not be considered independent layers in the literature. Taking LeNet-5 as an example, the order of these three common architectures in the hidden layers is typically: input - convolutional layer - pooling layer - fully connected layer - output.

[0055] In convolutional neural networks (CNNs), the upstream layer of the output layer is typically a fully connected layer; therefore, its structure and operating principle are the same as the output layer in traditional feedforward neural networks. For image classification problems, the output layer uses a logistic function or a normalized exponential function to output classification labels. In object recognition problems, the output layer can be designed to output the center coordinates, size, and classification of the object. In image semantic segmentation, the output layer directly outputs the classification result for each pixel.

[0056] AI (Artificial Intelligence) is a new technical science that studies and develops theories, methods, technologies, and application systems to simulate, extend, and expand human intelligence. Artificial intelligence is a branch of computer science, and research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems.

[0057] DnCNN (Denoising Convolutional Neural Network) is a modified version of VGG. Its network structure is a cascaded structure of (convolution, BN, ReLU). Unlike ResNet, it does not have jump connections inside the model. Instead, it uses residual learning at the network output.

[0058] FFDnet (Fast and Flexible Denoising Net), also known as DnCNN, effectively removes uniform Gaussian noise using Batch Normalization and residual learning, suppressing noise across a certain noise level range. However, real-world noise is not uniform Gaussian noise; it is signal-dependent, correlated across color channels, and non-uniform, potentially varying with spatial location. In this case, FFDNet uses a noise estimation map as input, balancing noise suppression with detail preservation to handle more complex real-world scenarios.

[0059] CBDNet (Convolutional Blind Denoising Net) uses a noise estimation map as input, balancing the suppression of uniformly distributed noise with the preservation of details to cope with more complex real-world scenarios. It also implements the noise level estimation process in a sub-network, enabling the entire network to achieve blind denoising.

[0060] SIFT (Scale-invariant feature transform) is a computer vision algorithm used to detect and describe local features in images. It finds extreme points in space and extracts their position, scale, and rotation invariants.

[0061] GPU (Graphics Processing Unit), also known as a display core, visual processor, or display chip, is a microprocessor specifically designed for performing image and graphics-related calculations on personal computers, workstations, game consoles, and some mobile devices (such as tablets and smartphones).

[0062] SLAM (simultaneous localization and mapping) places a robot in an unknown location within an unknown environment, allowing the robot to gradually create a complete map of that environment as it moves.

[0063] DTAM (Dense Tracking and Mapping) constructs a globally optimized energy equation to estimate the inverse depth mapping of keyframes; this energy equation consists of the sum of a photometric error data term and a robust spatial regularization term.

[0064] Bundler is a Structure from Motion (SFM) system developed in C and C++. It generates sparse point 3D information from multiple images of a scene taken from different angles and estimates the camera parameters (intrinsic and extrinsic) for each image. The earliest version was used in the Photo Tourism project. Bundler's input consists of images, image features, and image matching information; the output is a 3D reconstruction model of the scene based on these images, along with a small amount of identified camera and scene geometry information.

[0065] CMVS (Cowards Internet-scale Multi-view Stereo) clusters sparse points from SFM into different image clusters, and performs MVS on each image cluster separately. Based on the information from the sparse points in SFM, the massive image is divided into multiple smaller blocks, and MVS is performed on each block separately, reducing the resource spikes of the MVS algorithm.

[0066] PMVS (patch-based MVS algorithm, a 3D reconstruction algorithm) is a technique for 3D modeling using multiple images of the same object from different angles. It can only perform 3D modeling on portions appearing in two or more images.

[0067] YOLO (You Only Look Once) treats object detection as a regression problem. It uses a single end-to-end network to process the input from the original image and output the object's location and category.

[0068] FPN (Feature Pyramid Networks, Gaussian Pyramids) obtains downsampled images through Gaussian smoothing and subsampling. In other words, the Kth layer of the Gaussian pyramid can obtain the K+1th layer of Gaussian image through smoothing and subsampling operations. The Gaussian pyramid contains a series of low-pass filters, and its cutoff frequency gradually increases by a factor of 2 from the previous layer to the next layer. Therefore, the Gaussian pyramid can span a very large frequency range.

[0069] SS (Selective Search, object detection) is the process of finding and classifying a variable number of objects in an image. The detection includes: object type and quantity, object scale, external environmental interference, bounding box classification, and confidence level (with threshold limitations).

[0070] Artificial intelligence-assisted diagnosis relies on various machine learning algorithms, primarily convolutional neural networks, to convert two-dimensional images into three-dimensional models and identify labeled content within the two-dimensional images. However, some information is lost during the conversion process.

[0071] This invention provides a solution aimed at reducing information loss when converting two-dimensional planar images into three-dimensional models and improving the accuracy of three-dimensional models.

[0072] Specifically, refer to Figure 1 , Figure 1This is a schematic diagram of the functional modules of the terminal device to which the two-dimensional image three-dimensional reconstruction device of this invention belongs. The two-dimensional image three-dimensional reconstruction device can be an independent device capable of image processing and network model training, and can be implemented on the terminal device in hardware or software form. The terminal device can be a smart mobile terminal with data processing capabilities, such as a mobile phone or tablet computer, or a fixed terminal device or server with data processing capabilities.

[0073] In this embodiment, the terminal device to which the three-dimensional reconstruction device of the two-dimensional image belongs includes at least an output module 110, a processor 120, a memory 130, and a communication module 140.

[0074] The memory 130 stores the operation method and the 3D reconstruction program of the 2D image; the output module 110 may be a display screen, etc. The communication module 140 may include a WIFI module, a mobile communication module, and a Bluetooth module, etc., and communicates with external devices or servers through the communication module 140.

[0075] When the processor executes the 3D reconstruction program of the 2D image in memory 130, it performs the following steps:

[0076] Obtain the image to be processed;

[0077] The image to be processed is input into a pre-created recognition model for processing to obtain a feature map;

[0078] A three-dimensional image is generated based on the feature map;

[0079] The annotation information of the image to be processed is mapped onto the three-dimensional image to obtain the final image;

[0080] The image to be processed and its annotation information are associated with the final image and its annotation information to obtain a database.

[0081] Furthermore, when the processor executes the 3D reconstruction program of the 2D image in memory 130, it also performs the following steps:

[0082] Extract keyframes from the feature map;

[0083] Based on the keyframes, calculate the estimated depth;

[0084] The 3D image is generated based on the keyframes, the estimated depth, and the image to be processed.

[0085] Furthermore, when the processor executes the 3D reconstruction program of the 2D image in memory 130, it also performs the following steps:

[0086] Train the recognition model;

[0087] The steps for training the recognition model include:

[0088] Acquire training images;

[0089] Identify the training image to obtain the detection bounding box of the training image;

[0090] The detection boxes of the training images are classified to obtain the category information of the detection boxes of the training images;

[0091] The category information of the detection boxes of the training image is fed back to the recognition model, and the execution step is returned: recognize the training image to obtain the detection boxes of the training image;

[0092] This process is repeated to iterate the parameters until the recognition model converges, at which point training is terminated.

[0093] Furthermore, when the processor executes the 3D reconstruction program of the 2D image in memory 130, it also performs the following steps:

[0094] The image to be processed is input into the pre-created recognition model for recognition, and the detection box of the image to be processed is obtained;

[0095] The recognition model is used to classify the detection boxes of the image to be processed to obtain the feature map, wherein the feature map carries the category information of the detection boxes of the image to be processed.

[0096] Furthermore, when the processor executes the 3D reconstruction program of the 2D image in memory 130, it also performs the following steps:

[0097] Establish a matching relationship based on the feature points of the keyframes;

[0098] A depth map is established based on the keyframes and the matching relationships;

[0099] The 3D image is generated based on the depth map, the estimated depth, and the image to be processed.

[0100] Furthermore, when the processor executes the 3D reconstruction program of the 2D image in memory 130, it also performs the following steps:

[0101] The coordinates of the feature map are adjusted according to the preset frame.

[0102] Furthermore, when the processor executes the 3D reconstruction program of the 2D image in memory 130, it also performs the following steps:

[0103] The image to be processed is preprocessed to obtain a standardized image;

[0104] The standardized image is input into the recognition model for processing to obtain the feature map.

[0105] This embodiment, through the above-described scheme, acquires an image to be processed; inputs the image to be processed into a pre-created recognition model to obtain a feature map; generates a 3D image based on the feature map; and maps the annotation information of the image to be processed onto the 3D image to obtain the final image. This invention reduces information loss when converting a 2D planar image to a 3D image, improves the accuracy of 2D image to 3D image conversion, provides more precise model recognition results, and offers users a more intuitive browsing experience.

[0106] Based on, but not limited to, the terminal device architecture described above, embodiments of the method of the present invention are proposed.

[0107] Reference Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the three-dimensional reconstruction method for a two-dimensional image according to the present invention. The three-dimensional reconstruction method for a two-dimensional image includes:

[0108] Step S101: Obtain the image to be processed.

[0109] Step S102: Input the image to be processed into a pre-created recognition model for processing to obtain a feature map.

[0110] Step S103: Generate a three-dimensional image based on the feature map.

[0111] The execution subject of the method in this embodiment can be a three-dimensional reconstruction device for a two-dimensional image, or a three-dimensional reconstruction terminal device or server for a two-dimensional image. This embodiment takes a three-dimensional reconstruction device for a two-dimensional image as an example. The three-dimensional reconstruction device for a two-dimensional image can be integrated into a terminal device such as a smartphone or tablet computer with data processing function.

[0112] To reduce information loss during the conversion of 2D planar images to 3D images, improve the accuracy of 2D image to 3D image conversion, provide more accurate model recognition results, and offer users a more intuitive browsing experience, the process begins by acquiring the image to be processed, which carries annotation information.

[0113] In one implementation method, in this embodiment, the doctor annotates the patient's endoscopic examination images, including lesions and anatomical locations. For example, the doctor annotates the patient's gastroscopy images, including marking the cardia, pylorus, greater curvature, lesser curvature, fundus, and duodenum. The endoscopic examination image containing the annotated information is used as the image to be processed.

[0114] As another implementation method, endoscopic images are acquired, and anatomical sites in the endoscopic images are identified using AI image recognition technology, with corresponding names assigned to each site. The endoscopic images containing the labeled information are then used as the images to be processed.

[0115] Therefore, by annotating two-dimensional images, the accuracy of recognizing three-dimensional images when converting two-dimensional images into three-dimensional images can be improved.

[0116] Secondly, the image to be processed is preprocessed to obtain a standardized image.

[0117] Specifically, noise reduction, enhancement, and balancing are performed on the image to be processed to address color differences, brightness differences, and occlusion. This process is then used to preprocess and standardize the image, resulting in a standardized image. Noise reduction of the image to be processed includes at least one of the following methods:

[0118] The first method involves denoising the image using filters. For example, a non-local means algorithm or a block-matching-based 3D filtering algorithm can be used.

[0119] The second approach is to denoise the image using a model. Examples include: using a non-local self-similarity model; using a sparse model; using a gradient model; and using a Markov model.

[0120] The third method is to denoise the image through learning. For example, using a convolutional neural network (CNN) to denoise the image; convolutional neural networks include CNN, DnCNN, FFDnet, and CBDnet.

[0121] Therefore, by preprocessing the image to be processed, a standardized image is obtained, thereby improving the image's clarity and making the resulting 3D image clearer and more recognizable.

[0122] Furthermore, the image to be processed is input into a pre-created recognition model for processing to obtain a three-dimensional image.

[0123] Specifically, the image to be processed is input into a pre-created recognition model for recognition to obtain the detection boxes of the image to be processed; the recognition model classifies the detection boxes of the image to be processed to obtain feature maps, wherein the feature maps carry the category information of the detection boxes of the image to be processed, so as to map the labeling information of the two-dimensional image to the three-dimensional image in the future, thereby improving the efficiency and accuracy of the labeling information mapping.

[0124] Furthermore, the coordinates of the feature map are adjusted according to the preset bounding boxes, where the preset bounding boxes are anchor boxes that are set in advance by the user.

[0125] Therefore, when the positions of certain features in an image are predefined by the user, the positions of the generated feature maps will also be adjusted according to the predefined position information, thereby improving the accuracy of image conversion and reducing information loss during image conversion.

[0126] Furthermore, a 3D image is generated based on the feature map. Thus, based on the recognition model, the feature map is obtained and converted into a 3D image, making the information of the 2D image more intuitive to the user and facilitating the user's analysis of the 3D image.

[0127] Step S104: Map the annotation information of the image to be processed onto the three-dimensional image to obtain the final image.

[0128] Step S105: Associate the image to be processed and its annotation information with the final image and its annotation information to obtain a database.

[0129] In this embodiment, the annotation information of the image to be processed is mapped to a three-dimensional image to obtain the final image.

[0130] Specifically, after obtaining the 3D image, the manually annotated 2D annotation information is projected onto the 3D image to establish a 3D annotation database, and the 3D image with the 2D annotation information is stored in the 3D annotation database. The image to be processed and its annotation information are then correlated with the final image and its annotation information to obtain the database.

[0131] By employing a graph-convolutional neural network and a Transformer model, 2D annotation information is mapped to corresponding locations in 3D images, enabling the localization and classification of 3D images with 2D annotation information at preset locations. These preset locations include lesion sites and can be configured according to specific circumstances. The images to be processed and their annotation information are then correlated with the final images and their annotation information to create a database. This allows for the direct 3D reconstruction and localization of the gastric lining and lesion sites based on 2D gastroscopy images, with the annotation information for both 2D and 3D images searchable within the database.

[0132] This embodiment, through the above-described scheme, acquires an image to be processed; inputs the image to be processed into a pre-created recognition model to obtain a feature map; generates a 3D image based on the feature map; maps the annotation information of the image to be processed onto the 3D image to obtain a final image; and associates the image to be processed and its annotation information with the final image and its annotation information to obtain a database. This invention improves the accuracy of 3D image recognition when converting 2D images to 3D images by annotating them, reduces information loss during the conversion, and provides a more accurate model recognition effect, offering users a more intuitive browsing experience.

[0133] Reference Figure 3 , Figure 3 This is a flowchart illustrating a second embodiment of the three-dimensional reconstruction method for two-dimensional images according to the present invention. Based on the above... Figure 2 In the embodiment shown, step S103: generating a three-dimensional image based on the feature map includes:

[0134] Step S1031: Extract the keyframes of the feature map.

[0135] Step S1032: Calculate the estimated depth based on the keyframe.

[0136] Step S1033: Generate the 3D image based on the keyframe, the estimated depth, and the image to be processed.

[0137] The key to achieving 3D reconstruction from 2D images lies in depth map estimation. In this embodiment, keyframes of the feature map are extracted, and the depth is estimated based on the keyframes. Based on the keyframes, the estimated depth, and the image to be processed, a 3D image is generated.

[0138] Specifically, key feature points are extracted from the feature map using the SIFT algorithm, and feature points from two frames are matched to obtain key frames of the feature map. For similar frames with small changes in shooting angle and displacement, similarity matrix is ​​used to remove them, retaining key frames with independent information and continuous intersection. Here, key frames are the parts of the image that need to be processed again. In this embodiment, the image frames of the lesion part are used as key frames.

[0139] SIFT, or Scale Invariant Feature Transform, is a technique used in image processing. This technique is scale invariant and can detect key points in images. The SIFT algorithm has the following characteristics:

[0140] (1) The SIFT algorithm has good stability and invariance, and can adapt to changes in rotation, scale, and brightness. To a certain extent, it is not affected by changes in viewpoint, affine transformation, or noise.

[0141] (2) The SIFT algorithm has good discriminative power and can quickly and accurately distinguish information for matching in a massive feature database.

[0142] (3) The SIFT algorithm has a large quantity, and can generate a large number of feature vectors even if there is only a single object.

[0143] (4) The SIFT algorithm is high-speed and can quickly perform feature vector matching.

[0144] (5) The SIFT algorithm is scalable and can be combined with other forms of feature vectors.

[0145] More specifically, firstly, keypoints are extracted from the feature map using the SIFT algorithm. Keypoints are prominent points that do not disappear due to factors such as lighting, scale, or rotation, such as corner points, edge points, bright spots in dark areas, and dark spots in bright areas. This step searches for image locations across all scale spaces. Potential scale- and rotation-invariant points of interest are then identified using the Gaussian differential function.

[0146] Secondly, keypoints are located and feature orientations are determined using the SIFT algorithm: at each candidate location, the position and scale are determined by a finely fitted model. Keypoints are selected based on their stability. Then, one or more orientations are assigned to each keypoint location based on the local gradient orientation of the image. All subsequent operations on the image data are transformed relative to the orientation, scale, and position of the keypoints, thus providing invariance to these transformations.

[0147] Finally, the SIFT algorithm is used to compare the feature vectors of each key point pairwise to find several pairs of matching feature points and establish the correspondence between the scenes. Several pairs of feature points form key frames, and the correspondence between feature points and scenes is used as the matching relationship.

[0148] Therefore, by acquiring keyframes, image information that needs to be processed again in the two-dimensional image can be obtained, providing a basis for the subsequent conversion of the two-dimensional image into a three-dimensional image, thereby improving the accuracy of the three-dimensional image conversion.

[0149] Furthermore, a depth map is created based on keyframes and matching relationships, so that 3D information can be restored and reconstructed subsequently based on the depth map.

[0150] Furthermore, the depth is estimated based on the keyframes.

[0151] Specifically, depth is estimated using DTAM based on keyframes. DTAM does not rely on feature point extraction but instead employs so-called whole image registration (WAR) matching. That is, given two keyframes, DTAM estimates the 3D spatial depth of each pixel, while also estimating the camera displacement and rotation between the two frames. Each pixel in any frame can be projected onto a point in 3D space, and then onto a pixel in another frame. The mirror image of a frame after 3D transformation projected into another frame should be very similar to the image of the first frame. This is the basic principle of DTAM. Although DTAM densely matches each pixel, the algorithm is highly parallelizable, and even real-time performance can be achieved using current commercial GPU hardware. The aforementioned 3D spatial depth is used as the estimated depth.

[0152] Furthermore, based on the depth map, estimated depth, and the image to be processed, a 3D image is generated.

[0153] Specifically, after establishing the depth map, SLAM 3D reconstruction is performed by combining the image to be processed (2D image frame) and the estimated depth, and then fused into a 3D image. DTAM and Bundler+CMVS techniques are used to reconstruct the 3D image. Bundler and PMVS (CMVS) are a toolkit for 3D reconstruction. Bundler uses a series of unordered images to generate a sparse point cloud of the scene. This sparse point cloud is then converted into a dense point cloud using the PMVS toolkit, thus enabling the reconstruction of the 3D image of the scene.

[0154] Therefore, based on the depth map, estimated depth, and the image to be processed, a 3D image is generated, so that the features of the 3D image retain their attribute characteristics after conversion, reducing information loss during image conversion and making it easier for users to obtain information more intuitively from the 3D image.

[0155] This embodiment employs the above-described scheme to: acquire an image to be processed; input the image to be processed into a pre-created recognition model to obtain a feature map; generate a 3D image based on the feature map; map the annotation information of the image to be processed onto the 3D image to obtain a final image; and associate the image to be processed and its annotation information with the final image and its annotation information to obtain a database. This invention reduces information loss during the conversion of a 2D planar image to a 3D image. When the positions of certain features in the image are pre-defined, the positions of the generated feature map will also be adjusted according to the pre-set position information, thereby improving the accuracy of image conversion and reducing information loss during image conversion.

[0156] Reference Figure 4 , Figure 4 This is a flowchart illustrating the third embodiment of the three-dimensional reconstruction method for two-dimensional images according to the present invention. Based on the above... Figure 2 In the embodiment shown, step S102, before inputting the image to be processed into a pre-created recognition model to obtain a feature map, includes:

[0157] Step S106: Obtain training images.

[0158] Step S107: Identify the training image and obtain the detection box of the training image.

[0159] Step S108: Classify the detection boxes of the training image to obtain the category information of the detection boxes of the training image.

[0160] As one implementation method, in this embodiment, firstly, a training image is acquired, and the detection box of the object to be detected in the training image is identified to obtain the detection box of the training image, that is, the background and the object to be detected are classified into two categories.

[0161] Specifically, the R-CNN series of object detection algorithms identifies the detection boxes of objects to be detected in the training image, and then classifies the detection boxes to obtain the category information of the detection boxes. In this embodiment, Faster R-CNN is used to process the training image using a neural network to generate detection boxes, and the objects to be detected within the detection boxes are classified to obtain the category information of the detection boxes. This replaces the method of generating candidate boxes through rules in other R-CNN algorithms, thereby realizing end-to-end training and significantly speeding up the process.

[0162] The entire Faster R-CNN can be divided into three parts:

[0163] 1. Backbone: Shares a base convolutional layer used to extract features from the entire image. For example, VGG16 or ResNet101, removing the fully connected layers and leaving only the convolutional layers, outputting downsampled feature maps.

[0164] 2. RPN: Region Proposal Networks. Classic detection methods are very time-consuming to generate detection boxes. For example, OpenCV AdaBoost uses a sliding window and image pyramid to generate detection boxes; or R-CNN uses the SS method. Faster R-CNN, however, abandons the traditional sliding window and SS methods and directly uses RPN to generate detection boxes. This is a huge advantage of Faster R-CNN, greatly improving the speed of detection box generation.

[0165] 3. Roi pooling and classification network: Classify the candidate detection boxes and fine-tune the coordinates of the candidate boxes again (in RPN, the network will adjust the coordinates based on the previously manually set anchor boxes, so this is the second adjustment).

[0166] The first part, the backbone, is a regular convolutional network that outputs feature maps for use in the subsequent two stages. The third part, the classification network, uses two fully connected layers, followed by two sister fully connected layers (two fully connected layers of the same size that do not share weights), to output coordinate fine-tuning regression information and the category information of the detection boxes, respectively.

[0167] As another implementation, when performing image segmentation, the YOLO algorithm is used to identify the detection boxes of the objects to be detected in the training images. This algorithm is an end-to-end object detection algorithm based on deep learning.

[0168] Unlike most object detection and recognition methods (such as Fast R-CNN), which involve multiple processes such as object region prediction and category prediction, YOLO integrates object region prediction and object category prediction into a single neural network model. This enables real-time and fast object detection and recognition with high accuracy. Its enhanced version can run at 45fps on a GPU, and the simplified version at 155fps, making it suitable for detection and recognition in real-time gastroscopy imaging scenarios. On 320×320 images, YOLOv3 achieved a runtime of 22.2 milliseconds and an mAP of 28.2.

[0169] The main improvements in YOLOv3 include:

[0170] (1) Changes in network structure: The network structure changed from Darknet-19 to Darknet-53, and the phenomenon of skipping layers became more and more common.

[0171] (2) Multi-scale prediction: Output 3 layers, each with S×S grids, namely 13×13, 26×26, and 52×52.

[0172] Small scale: (13×13 feature map) The network receives a (416×416) image, performs downsampling through 5 convolutions with a stride of 2 (416 / 2^5=13), outputs (13×13×512), and then performs 7 convolutions to obtain the first feature map, and makes the first prediction on this feature map.

[0173] Mid-scale (26×26 feature map): Obtain the output of the third-to-last convolutional layer from the small-scale layer, perform one convolution and one x2 upsampling, and connect the upsampled feature (26×26×256) with the 43rd convolutional feature (26×26×512) to output (26×26×728). After 7 convolutions, obtain the second feature map (26×26×255), and make a second prediction on this feature map.

[0174] Large scale (52×52 feature map): The operation is the same as the medium scale. Obtain the output of the third-to-last convolutional layer from the back to the front, perform one convolution and one x2 upsampling, concatenate the upsampled features with the 26th convolutional features, and after 7 convolutions, obtain the third feature map. Make the third prediction on this feature map. Output (52×52×255).

[0175] The advantage of this change in network characteristics is that it allows the network to learn both deep and shallow features simultaneously. By superimposing shallow feature maps onto adjacent channels, it achieves similarity to umsample+concat (non-sample + connection) in FPN. This method superimposes a 13x13x256 feature map onto a 26x26x512 feature map, giving the model fine-grained features and increasing its ability to recognize small targets.

[0176] Anchor boxes: YOLOv3 has a total of 9 anchor boxes, obtained by k-means clustering. On the COCO dataset, the 9 clusters are: (10×13); (16×30); (33×23); (30×61); (62×45); (59×119); (116×90); (156×198); (373×326). Different sized feature maps correspond to different sized prior boxes. The finer the grid cell, the finer the object it can detect. The larger the scale, the smaller the receptive field, and the more sensitive it is to small objects, so a smaller anchor box is chosen.

[0177] Step S109: The category information of the detection boxes of the training image is sent back to the recognition model, and the process returns to step S106: Recognize the training image to obtain the detection boxes of the training image.

[0178] Step S1010: Repeat this process to iterate the parameters until the recognition model converges, at which point the training is terminated.

[0179] In this embodiment, the category information of the detection boxes of the training images is fed back to the recognition model, and the process returns to step S107: recognizing the training images to obtain the detection boxes of the training images. This process is repeated to iterate the parameters until the recognition model converges, at which point training terminates.

[0180] Therefore, based on the category information of the detection box, it is determined whether the recognition model has converged, confirming that the recognition model can effectively recognize the image information of the two-dimensional image and classify it.

[0181] This embodiment, through the above-described scheme, involves: acquiring an image to be processed; inputting the image to be processed into a pre-created recognition model to obtain a feature map; generating a 3D image based on the feature map; mapping the annotation information of the image to be processed onto the 3D image to obtain a final image; and associating the image to be processed and its annotation information with the final image and its annotation information to obtain a database. This invention reduces information loss when converting 2D planar images to 3D images, improves the accuracy of 2D image to 3D image conversion, provides more accurate model recognition results, and offers users a more intuitive browsing experience.

[0182] Reference Figure 5 , Figure 5 This is a schematic diagram of the functional modules of the three-dimensional reconstruction system for two-dimensional images according to the present invention. The three-dimensional reconstruction system for two-dimensional images includes:

[0183] Image acquisition module 10 is used to acquire the image to be processed.

[0184] Image recognition module 20 is used to input the image to be processed into a pre-created recognition model for processing to obtain a feature map;

[0185] Image generation module 30 is used to generate a three-dimensional image based on the feature map.

[0186] Image mapping module 40 is used to map the annotation information of the image to be processed to the three-dimensional image to obtain the final image;

[0187] The database generation module 50 is used to associate the image to be processed and its annotation information with the final image and its annotation information to obtain a database.

[0188] The principle and implementation process of realizing the three-dimensional reconstruction of two-dimensional images in this embodiment are described in the above embodiments and will not be repeated here.

[0189] Furthermore, this embodiment of the invention also proposes a terminal device, which includes a memory, a processor, and a three-dimensional reconstruction program for a two-dimensional image stored in the memory and executable on the processor. When the three-dimensional reconstruction program for the two-dimensional image is executed by the processor, it implements the steps of the three-dimensional reconstruction method for the two-dimensional image as described above.

[0190] Since the three-dimensional reconstruction program of this two-dimensional image adopts all the technical solutions of all the aforementioned embodiments when it is executed by the processor, it has at least all the beneficial effects brought about by all the technical solutions of all the aforementioned embodiments, which will not be repeated here.

[0191] Furthermore, embodiments of the present invention also propose a computer-readable storage medium storing a three-dimensional reconstruction program for a two-dimensional image, wherein when the three-dimensional reconstruction program for the two-dimensional image is executed by a processor, the steps of the three-dimensional reconstruction method for the two-dimensional image as described above are implemented.

[0192] Since the three-dimensional reconstruction program of this two-dimensional image adopts all the technical solutions of all the aforementioned embodiments when it is executed by the processor, it has at least all the beneficial effects brought about by all the technical solutions of all the aforementioned embodiments, which will not be repeated here.

[0193] Compared to existing technologies, this invention provides a method, system, terminal device, and storage medium for three-dimensional reconstruction of two-dimensional images. The process involves: acquiring an image to be processed; inputting the image to be processed into a pre-created recognition model to obtain a feature map; generating a three-dimensional image based on the feature map; mapping the annotation information of the image to be processed onto the three-dimensional image to obtain a final image; and associating the image to be processed and its annotation information with the final image and its annotation information to obtain a database. This invention reduces information loss during the conversion of two-dimensional planar images to three-dimensional images, improves the accuracy of the conversion, provides more precise model recognition results, and offers users a more intuitive browsing experience.

[0194] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or approach that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or approach. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or approach that includes that element.

[0195] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0196] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, controlled terminal, or network device, etc.) to execute the methods of each embodiment of the present invention.

[0197] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method of three-dimensional reconstruction of a two-dimensional image, characterized in that, The method comprises the following steps: acquire a to-be-processed image, wherein the to-be-processed image carries label information; input the to-be-processed image into a pre-created recognition model for processing to obtain a feature map, wherein the feature map carries class information of a detection frame of the to-be-processed image, wherein the training process of the recognition model comprises: acquiring a training image, processing the training image using a neural network through Faster R-CNN to generate a detection frame, and classifying a to-be-detected object in the detection frame to obtain class information of the detection frame, or using a YOLO algorithm to recognize the detection frame of the to-be-detected object in the training image; returning the class information of the detection frame of the training image to the recognition model and returning to the step of recognizing the training image to obtain the detection frame of the training image; iteratively performing parameters in this cycle until the recognition model converges, and terminating the training; generate a three-dimensional image based on the feature map; map the label information of the to-be-processed image to the three-dimensional image through a Graph-Convolution graph convolutional neural network and a Transformer model to obtain a final image; correspondingly associate the to-be-processed image and its label information with the final image and its label information to obtain a database; the step of generating a three-dimensional image based on the feature map comprises: extract key frames of the feature map, wherein key feature points of the feature map are extracted through a scale-invariant feature transform algorithm, key feature points of any two frames in the feature map are matched, a plurality of pairs of key feature points that are mutually matched are found out for each key feature point to establish a corresponding relationship between scenes, and the plurality of pairs of key feature points are combined to form key frames; wherein, for similar frames with small changes in shooting angle and displacement, key frames with independent information and continuous intersection are retained; estimate depth based on the key frames through a full-pixel matching method in a DTAM algorithm; generate the three-dimensional image based on the key frames, the estimated depth and the to-be-processed image using a Bundler technology and a CMVS / PMVS technology.

2. The method of three-dimensional reconstruction of two-dimensional images according to claim 1, characterized in that, the step of inputting the to-be-processed image into a pre-created recognition model for processing to obtain a feature map comprises: input the to-be-processed image into the pre-created recognition model for recognition to obtain a detection frame of the to-be-processed image; classify the detection frame of the to-be-processed image through the recognition model to obtain the feature map, wherein the feature map carries class information of the detection frame of the to-be-processed image.

3. The method of three-dimensional reconstruction of two-dimensional images according to claim 1, characterized in that, the step of estimating depth based on the key frames comprises: establish a matching relationship according to feature points of the key frames; establish a depth map based on the key frames and the matching relationship; the step of generating the three-dimensional image based on the key frames, the estimated depth and the to-be-processed image comprises: generate the three-dimensional image based on the depth map, the estimated depth and the to-be-processed image.

4. The method of three-dimensional reconstruction of two-dimensional images according to claim 1, characterized in that, the step of inputting the to-be-processed image into a pre-created recognition model for processing to obtain a feature map comprises: The coordinates of the feature map are adjusted according to a preset frame.

5. The method of three-dimensional reconstruction of two-dimensional images according to claim 1, characterized in that, The step of inputting the to-be-processed image into the pre-created recognition model for processing to obtain a feature map comprises: The to-be-processed image is pre-processed to obtain a standardized image. The step of inputting the to-be-processed image into the pre-created recognition model for processing to obtain a feature map comprises: The standardized image is input into the recognition model for processing to obtain the feature map.

6. A system for three-dimensional reconstruction of two-dimensional images, characterized in that Comprise: An image acquisition module is configured to acquire a to-be-processed image, wherein the to-be-processed image carries label information; An image recognition module is configured to input the to-be-processed image into a pre-created recognition model for processing to obtain a feature map, wherein the feature map carries class information of a detection frame of the to-be-processed image, and wherein a training process of the recognition model comprises: acquiring a training image, processing the training image using a neural network through a Faster R-CNN to generate a detection frame, classifying a to-be-detected object in the detection frame to obtain the class information of the detection frame, or identifying the detection frame of the to-be-detected object in the training image using a YOLO algorithm; returning the class information of the detection frame of the training image to the recognition model and returning to the step of identifying the training image to obtain the detection frame of the training image; iteratively performing the above steps until the recognition model converges and the training is terminated; An image generation module is configured to generate a three-dimensional image based on the feature map; extract key frames of the feature map, wherein key feature points of the feature map are extracted through a scale-invariant feature transform algorithm, key feature points of any two frames in the feature map are matched, a plurality of pairs of key feature points that match each other are found for each key feature point to establish a corresponding relationship between scenes, and the plurality of pairs of key feature points are combined to form key frames; wherein for similar frames with small changes in shooting angle and displacement, key frames with independent information and continuous intersection are retained; based on the key frames, an estimated depth is calculated through a full-pixel matching method in a DTAM algorithm; based on the key frames, the estimated depth, and the to-be-processed image, a three-dimensional image is generated using a Bundler technology and a CMVS / PMVS technology; An image mapping module is configured to map label information of the to-be-processed image to the three-dimensional image through a Graph-Convolution graph convolutional neural network and a Transformer model to obtain a final image; A database generation module is configured to correspondingly associate the to-be-processed image and its label information with the final image and its label information to obtain a database.

7. A terminal device, characterized by, The terminal device comprises a memory, a processor, and a three-dimensional reconstruction method of a two-dimensional image stored on the memory and executable on the processor, and the three-dimensional reconstruction method of the two-dimensional image comprises the steps of the three-dimensional reconstruction method of the two-dimensional image as claimed in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores thereon a program of three-dimensional reconstruction of two-dimensional images, which, when executed by a processor, implements the steps of the method of three-dimensional reconstruction of two-dimensional images according to any one of claims 1-5.

Images