Method for converting two-dimensional data into three-dimensional data, terminal equipment and storage medium

Abstract

The invention relates to a method for converting two-dimensional data into three-dimensional data, terminal equipment and a storage medium, and the method comprises the steps: segmenting a single two-dimensional image, separating an object from a background in the two-dimensional image, extracting the depth information of the object and the background to generate a three-dimensional model of the object, and outputting a multi-angle two-dimensional image of the three-dimensional model of the object, supplementing the missing part in the background, combining the multi-angle two-dimensional image with the background to generate two-dimensional images under a plurality of continuous visual angles, and finally synthesizing the two-dimensional images under the plurality of continuous visual angles into a grating three-dimensional image. According to the invention, grating three-dimensional image data or grating three-dimensional video data can be generated by using single two-dimensional image data or single-visual-angle two-dimensional video data, and the generated grating three-dimensional image data can be stored by a storage device. And high-definition printing equipment can be used for printing, and an adaptive grating is added for displaying or three-dimensional display equipment is used for displaying, so that the manufacturing efficiency is improved, and the manufacturing cost is reduced.

Description

Technical Field

[0001] The present invention relates to the field of holographic imaging, and in particular to a method, a terminal device and a storage medium for converting two-dimensional data into three-dimensional data. Background Art

[0002] With the continuous development of computer theory, technology and application, the computing power of computer for image processing has been greatly improved. Three-dimensional images or videos serve as the basis for further comprehensive understanding of image and video content technology. At present, although three-dimensional reconstruction technology has achieved many results, no matter what method is used, it requires manual three-dimensional modeling, or synthesizing raster three-dimensional images or videos after multi-angle shooting of objects, which consumes a lot of manpower and cost. How to quickly and efficiently convert two-dimensional images or videos into three-dimensional images or videos while controlling costs has become an urgent problem to be solved. Summary of the invention

[0003] In view of the above problems, a method for converting two-dimensional data into three-dimensional data, a terminal device and a storage medium are provided. The method comprises the following steps:

[0004] S1: semantic segmentation of a single 2D image;

[0005] S2: separating the object from the background in the two-dimensional image, and extracting depth information of the object in the two-dimensional image;

[0006] S3: Generate a three-dimensional model of the object;

[0007] S4: Outputting the three-dimensional model as a multi-angle image;

[0008] S5: Synthesize the multi-angle images into a grating three-dimensional image.

[0009] Furthermore, step S2 uses a combined depth estimation algorithm and a segmentation algorithm to segment the background and objects in the two-dimensional image.

[0010] Furthermore, in step S3, the NeRF algorithm is used to generate a three-dimensional model of the object according to the depth information of the object and the background in the two-dimensional image.

[0011] Furthermore, step S4 further includes the following steps:

[0012] S41: performing viewpoint calibration on the multi-angle images of the three-dimensional model;

[0013] S42: using a projection method to generate two-dimensional images of the three-dimensional model at different viewing angles;

[0014] S43: The image filling algorithm based on the diffusers artificial intelligence model fills in the missing parts of the image background due to changes in visual angle.

[0015] Further, step S5 includes the following steps:

[0016] S51: performing raster image processing on the multi-angle image to form a raster three-dimensional image;

[0017] S52: Rendering the raster three-dimensional image;

[0018] S53: Outputting a raster three-dimensional image.

[0019] Furthermore, the present invention also includes printing the grating three-dimensional image data and adding an adapted grating for display or displaying it by a three-dimensional display device.

[0020] The present invention further provides a terminal device, which includes a memory, a processor, and a program for converting two-dimensional data into three-dimensional data that is stored on and can be run on the processor. When the program for converting two-dimensional data into three-dimensional data is executed by the processor, the steps of the above-mentioned method of converting an image into a three-dimensional image are implemented.

[0021] The present invention further provides a computer-readable storage medium storing a program for converting two-dimensional data into three-dimensional data. When the program for converting two-dimensional data into three-dimensional data is executed by the processor, the steps of the above-mentioned method for converting an image into a three-dimensional image are implemented.

[0022] The method, terminal device and storage medium for converting two-dimensional data into three-dimensional data provided by the present invention first segment a single two-dimensional image to separate an object from a background in the two-dimensional image, extract depth information of the object and the background in the two-dimensional image and generate a three-dimensional model of the object, output a multi-angle two-dimensional image of the three-dimensional model of the object, then supplement the missing part of the background, merge the multi-angle image with the background to generate a two-dimensional image at a series of continuous visual angles, and finally synthesize the two-dimensional images at a series of continuous visual angles into a grating three-dimensional image. The present invention can generate grating three-dimensional picture data or grating three-dimensional video data using a single two-dimensional image data or a single-viewing angle two-dimensional video data. The generated grating three-dimensional image data can be stored by a storage device, or can be printed by a high-definition printing device and displayed with an adapted grating, or displayed by a three-dimensional display device, thereby improving production efficiency and reducing production costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A flow chart of a method for converting two-dimensional data into three-dimensional data according to the present invention;

[0024] Figure 2 A schematic diagram of relative shifting of pixels when projecting three-dimensional information onto a two-dimensional image in a method for converting two-dimensional data into three-dimensional data according to the present invention;

[0025] Figure 3 A schematic diagram of a method for converting two-dimensional data into three-dimensional data according to the present invention, in which pixels are relatively offset at different visual angles when three-dimensional information is projected onto a two-dimensional image;

[0026] Figure 4 A schematic diagram of the principle of combining image depth estimation with image segmentation in a method for converting two-dimensional data into three-dimensional data according to the present invention;

[0027] Figure 5 This is an integration effect diagram when the separated background is not processed in a method for converting two-dimensional data into three-dimensional data of the present invention;

[0028] Figure 6 This is an integrated effect diagram when background supplement processing is performed on a separated background in a method for converting two-dimensional data into three-dimensional data in the present invention.

[0029] Description of Reference Numerals

[0030] 1Object center. DETAILED DESCRIPTION

[0031] In order to provide a further understanding of the purpose, structure, features, and functions of the present invention, the following detailed description is given in conjunction with the embodiments.

[0032] The present invention provides a method for converting two-dimensional data into three-dimensional data. Figure 1 , Figure 1 This is a flow chart of a method for converting two-dimensional data into three-dimensional data according to the present invention, the method comprising the following steps:

[0033] S1: semantic segmentation of a single 2D image;

[0034] S2: separating the object from the background in the two-dimensional image, and extracting depth information of the object in the two-dimensional image;

[0035] S3: Generate a three-dimensional model of the object;

[0036] S4: Outputting the three-dimensional model as a multi-angle image;

[0037] S5: Synthesize the multi-angle images into a grating three-dimensional image.

[0038] The present invention provides a method for converting two-dimensional data into three-dimensional data, which can convert two-dimensional picture data of a single perspective into raster three-dimensional picture data, and can also convert two-dimensional video data of a single perspective into three-dimensional video data. Taking picture data as an example, specifically, a convolutional neural network is used to perform semantic segmentation on a single two-dimensional image data. This process segments different objects in the image data to mark the pixels of different objects. Through the image segmentation results, the AI ​​algorithms of DensePrediction Transformers and segment anything model are used to separate the pixels of the object from the pixels of the background. Among them, Dense Prediction Transformers is a depth estimation algorithm, and segment anythingmodel is an algorithm for segmenting everything, which helps to clearly locate the object and separate it from the background, and use the deep learning model to estimate the depth information of the object from the segmented image. Based on the obtained depth information, a 3D reconstruction algorithm is used to generate a 3D model of the object. The 3D reconstruction algorithms used include structured light scanning, stereo vision or point cloud reconstruction technologies. The main purpose of generating a 3D model is to map the 2D image information to 3D space and construct the geometric shape of the object. Multi-angle images can be obtained by rotating the 3D model and rendering the corresponding angles. These multi-angle images can be used to display and analyze different perspectives of the object. Finally, a raster image processing algorithm is used to synthesize the grating 3D image.

[0039] The above-mentioned convolutional neural network includes an input layer, a convolutional layer, a pooling layer, a fully connected layer and an output. The output layer uses a logical function or a normalized exponential function to output a classification label. In image semantic segmentation, the output layer directly outputs the classification result of each pixel. The above-mentioned deep learning model can be a deep neural network or a convolutional neural network, and the present invention does not limit it.

[0040] The AI ​​algorithm of Dense Prediction Transformers and segment anything model is used to separate the pixels of the object from the pixels of the background. The above-mentioned Dense Prediction Transformers is a depth estimation algorithm, which can also be simply recorded as DPT. In the present invention, the pytouch instruction is used in conjunction with the DPT instruction to calculate the depth information in the image, wherein PyTorch is an open source Python machine learning library based on Torch and used for applications such as natural language processing. In specific operations, the algorithm model is first imported, including the automatic image processing algorithm model and the DPT depth estimation algorithm model, a preprocessing function is established based on the automatic image processing algorithm model, and then an image depth information calculation function is established based on the DPT depth estimation algorithm model. After importing the image to be processed, the image is preprocessed and the image depth information is calculated, and then the depth map is adjusted and the depth map calculation result is output.

[0041] In actual operation, the pseudo code process is as follows:

[0042]

[0043]

[0044] The above-mentioned segment anything model is a segmentation algorithm, simply recorded as SAM, which is an artificial intelligence model developed by Meta for image segmentation. In the present invention, the pytouch command is used in conjunction with the SAM command to calculate the depth information in the picture. First, the SAM registry and the SAM automatic mask algorithm model are imported, and then the annotation tool is imported. The type of the model SAM is set to VIT-H through the SAM registry. SAM provides three models with the most different parameters, which are VIT-B, VIT-L, and VIT-H from small to large. Among them, VIT-H has a better segmentation effect, and the instances in the image are also segmented very completely. Then an image segmentation function is established, and the image to be processed is imported. The SAM segmentation function is used to segment the image. Finally, the annotation tool is used to annotate the segmented area and output the segmented area information of the image.

[0045] In actual operation, the pseudo code process is as follows:

[0046]

[0047]

[0048] See also Figure 4 , Figure 4The present invention is a schematic diagram of the principle of combining image depth estimation with image segmentation in a method for converting two-dimensional data into three-dimensional data. Image depth estimation is DPT. At this time, depth estimation is performed on information in the image, and the depth information of the image is output. The image segmentation algorithm, that is, SAM, divides the image into N areas, calculates the average depth information of each of the N areas in the N segmented areas, and finally combines the N segmented areas with the average depth information of the area, and finally marks the corresponding area as background or object according to the depth information of the N areas, thereby separating the pixels of the object from the pixels of the background. For example, segmented area 1 is combined with the average depth information of segmented area 1, and segmented area 1 is marked as background or object according to the depth information. By analogy, segmented area 2 is judged to be background or object, until all N segmented areas are judged to be background or object.

[0049] Preferably, step S3 uses the NeRF algorithm to generate a three-dimensional model of the object according to the depth information of the object and the background in the image. The NeRF algorithm is an algorithm based on a neural network. It uses a neural network such as a fully connected layer plus an activation layer to implicitly learn a static three-dimensional scene, realize arbitrary new perspective synthesis or rendering of complex scenes, and is used to reconstruct realistic three-dimensional scenes from images. The NeRF algorithm only uses two-dimensional images as supervision to represent complex three-dimensional scenes. When using the NeRF algorithm for three-dimensional image reconstruction, the corresponding two-dimensional image and the depth information extracted from the image are input into the neural network to generate a set of pixel position, direction and color outputs in three-dimensional space. By using classic volume rendering technology, the three-dimensional information is projected into a two-dimensional image of any visual angle.

[0050] Preferably, the present invention improves the existing NeRF algorithm by adding appearance embedding, learning posture optimization and controllable exposure on the basis of the NeRF algorithm, extending the NeRF algorithm through appearance embedding and learning posture optimization to solve environmental changes and posture errors in collected data, and adding controllable exposure to provide and modify exposure capabilities.

[0051] Preferably, step S4 further includes the following steps:

[0052] S41: performing viewpoint calibration on the multi-angle images of the three-dimensional model;

[0053] S42: using a projection method to generate two-dimensional images of the three-dimensional model at different viewing angles;

[0054] S43: The image filling algorithm based on the diffusers artificial intelligence model fills in the missing parts of the image background due to changes in visual angle.

[0055] In order to synthesize a three-dimensional image, step S41 ensures that all multi-angle images are calibrated with the same coordinate system and viewpoint. Each image needs to be calibrated so that they are aligned in the same space, and then the depth information of the object and background extracted previously is integrated with the multi-angle image. The depth information can be used to determine the position and distance of each pixel in order to correctly render the three-dimensional image.

[0056] Step S42 generates two-dimensional images of the three-dimensional model at different visual angles by projection. Figure 2 , Figure 2 This is a schematic diagram of relative pixel offset when projecting three-dimensional information onto a two-dimensional image in a method for converting two-dimensional data into three-dimensional data in the present invention. The three-dimensional model generated in step S3 is projected to a specific viewing angle to render two-dimensional images at different visual angles using a projection method. Figure 2 In the figure, the middle row is the initial pixels of the background information. The numbers in the pixels are the depth information obtained from the 3D model, such as pink pixels with a depth of 2, green pixels with a depth of 4, and blue pixels with a depth of 6. After the initial pixels of the background information are combined with the depth information, the pixels are depth-marked and offset. After the offset, the 45° right view shows Figure 2 The first row of pixel information, 45° left view sees Figure 2 The third row of pixel information in the figure shows the visual effect produced when the human eye views the image after pixel shifting. Figure 3 , Figure 3This is a schematic diagram of a method for converting two-dimensional data into three-dimensional data of the present invention, in which pixels are relatively offset at different visual angles when three-dimensional information is projected onto a two-dimensional image. After the pixel is offset, the pixel color observed by the human eye is not offset if the human eye looks straight ahead. The effect of the pixel offset can be seen at 45° left and 45° right viewing. When looking left at 45°, if there is only one pixel of one color in the direction of sight, such as a pink pixel, the human eye can only see pink. If there are two pixels of two colors in the direction of sight, such as a pink pixel and a green pixel, the human eye can see the superimposed colors of pink and green. Similarly, when looking right at 45°, if there is only one pixel of one color in the direction of sight, such as a blue pixel, the human eye can only see blue. If there are two pixels of two colors in the direction of sight, such as a blue pixel, the human eye can see the superimposed colors of pink and green. If there are three colors of pixels in the direction of the human eye's sight, such as blue pixels, green pixels and pink pixels, the human eye can see the superimposed colors of blue and green. The object center 1 is the rotation center of the viewing angle, and the rotation center of the viewing angle is the intersection of 45° left viewing, 45° right viewing and the center of the sight line when looking straight. Determining the rotation center of the viewing angle can effectively prevent the image seen by the human eye in different directions from being distorted. It should be noted that the above 45° is only used as an example. In actual applications, views of multiple continuous angles will be rendered. For example, in the range of 45° right viewing to 45° left viewing, a two-dimensional image is rendered every 5°. A total of 19 two-dimensional images need to be rendered, thereby generating a smooth two-dimensional image under a continuous visual angle, producing a similar three-dimensional image effect.

[0057] Preferably, the diffusers artificial intelligence model in step S43 includes an image filling algorithm of Stable Diffusion, Stable Diffusion XL (SDXL) or Kandinsky2.2 artificial intelligence model, so as to supplement the missing part of the background after the object is subtracted due to the change of visual angle. Among them, Stable Diffusion and Stable DiffusionXL (SDXL) are the latest AI image generation models, which can generate realistic faces, readable text in images, and have better image composition capabilities. Kandinsky2.2 is an image generation model that can provide better image generation and text-guided images. For details, see Figure 5 and Figure 6 , Figure 5 This is an integration effect diagram when the separated background is not processed in a method for converting two-dimensional data into three-dimensional data of the present invention. Figure 6 This is an integrated effect diagram of a method for converting two-dimensional data into three-dimensional data in the present invention when a separated background is supplemented. Figure 5In the final synthesized left-eye image and right-eye image, some background is missing. Figure 6 In the process, the image filling algorithm based on Stable Diffusion, Stable Diffusion XL (SDXL) or Kandinsky2.2 artificial intelligence model is used to supplement the missing part of the background after subtracting the object. There is no obvious missing background in the synthesized left-eye image and right-eye image. During the operation, the image filling process model is first imported into the diffusers tool, and then the original image is imported and the background image is retained. Then, the image filling instruction is generated to drive the image filling process model to fill the background, and finally the image depth information is output. Among them, the diffusers tool is the preferred tool for generating the most advanced diffusion models of images, audio and even molecular 3D structures. The pseudo code involved in this step is as follows:

[0058] pseudocode:

[0059]

[0060]

[0061] Preferably, step S5 comprises the following steps:

[0062] S51: performing raster image processing on the multi-angle image to form a raster three-dimensional image;

[0063] S52: Rendering the raster three-dimensional image;

[0064] S53: Outputting a raster three-dimensional image.

[0065] First, ensure that rendered multi-angle images of the two-dimensional image have been obtained, and these images capture different perspectives of the object or scene. Then, select the corresponding pixels for each multi-angle image according to the grating plate parameters to generate a rasterized multi-angle image. Use the rasterized multi-angle image to synthesize a three-dimensional image. At each pixel position, the corresponding multi-angle image pixel is synthesized into the three-dimensional image to produce a rasterized three-dimensional image. Further, rendering techniques such as shadows, lighting and texture mapping can be applied to enhance the appearance and realism of the three-dimensional image. These effects can be achieved through shading algorithms to give the image more depth and texture. The final output rasterized three-dimensional image can be output in various formats, such as JPEG, PNG or other image formats, which are not limited by the present invention.

[0066] After obtaining the output of multi-angle image data or multi-angle video data, the present invention uses a raster image processor RIP to merge the multi-angle images to generate the final raster 3D image data or raster 3D video data. The raster image processor RIP is an indispensable intermediate processing link when outputting the graphic pages arranged by the computer to different media such as laser printouts, color separation printing films or light box films. The raster image processor RIP receives the data transmitted from the computer, processes it into the raster data required by the output device such as a printer or an imagesetter, and then controls the device to output. The present invention generates final grating 3D image data or grating 3D video data. In addition to being stored on an electronic device, the generated 3D grating image data can be directly printed using a high-definition printing device, and combined with an adapted grating for display and collection. Generally, the resolution of a high-definition printing device is 300 DPI or above. DPI is a unit of measurement for the resolution of devices such as printers and mice, and is one of the main parameters for measuring the printing accuracy of a printer. Generally speaking, the higher the DPI value, the higher the printing accuracy of the printer. The final generated grating 3D video data is displayed using a 3D display device, such as a 3D display or a projector.

[0067] In addition, the present invention can convert two-dimensional video data of a single perspective into raster three-dimensional video data, reconstruct three-dimensional data for each frame of two-dimensional data processing of the video, and progressively generate a three-dimensional data model through continuous self-fine-tuning of multiple frames of two-dimensional data, and connect key frames in series to generate a GIF animation or a three-dimensional video. The final generated raster 3D video data is displayed using a 3D display device.

[0068] The present invention further provides a terminal device, which includes a memory, a processor, and a program for converting two-dimensional data into three-dimensional data that is stored on and can be run on the processor. When the program for converting two-dimensional data into three-dimensional data is executed by the processor, the steps of the above-mentioned method of converting an image into a three-dimensional image are implemented.

[0069] The present invention further provides a computer-readable storage medium, on which a program for converting two-dimensional data into three-dimensional data is stored, and when the program for converting two-dimensional data into three-dimensional data is executed by the processor, the steps of the method for converting the above-mentioned image into a three-dimensional image are implemented. The computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. Alternatively, the computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0070] The method, terminal device and storage medium for converting two-dimensional data into three-dimensional data provided by the present invention first segment a single two-dimensional image to separate an object from a background in the two-dimensional image, extract depth information of the object in the two-dimensional image and generate a three-dimensional model of the object, output a multi-angle two-dimensional image of the three-dimensional model of the object, then supplement the missing part of the background, merge the multi-angle image with the background to generate a two-dimensional image at a series of continuous visual angles, and finally synthesize the two-dimensional images at a series of continuous visual angles into a grating three-dimensional image. The present invention can generate grating three-dimensional picture data or grating three-dimensional video data using a single two-dimensional image data or a single-viewing angle two-dimensional video data. The generated grating three-dimensional image data can be stored by a storage device, or can be printed by a high-definition printing device and displayed with an adapted grating or displayed by a three-dimensional display device, thereby improving production efficiency and reducing production costs.

[0071] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0072] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in conjunction with these instruction execution systems, apparatuses or devices.

[0073] The present invention has been described by the above-mentioned relevant embodiments, however, the above-mentioned embodiments are only examples for implementing the present invention. It must be pointed out that the disclosed embodiments do not limit the scope of the present invention. On the contrary, changes and modifications made without departing from the spirit and scope of the present invention are all within the scope of patent protection of the present invention.

Claims

1. A method for converting two-dimensional data into three-dimensional data, characterized in that: The method comprises the following steps: S1: semantic segmentation of a single 2D image; S2: separating the object from the background in the two-dimensional image, and extracting depth information of the object and the background in the two-dimensional image; S3: Generate a three-dimensional model of the object; S4: Outputting the three-dimensional model as a multi-angle image; S5: Synthesize the multi-angle images into a grating three-dimensional image.

2. The method for converting two-dimensional data into three-dimensional data according to claim 1, characterized in that: Step S2 uses a combined depth estimation algorithm and a segmentation algorithm to segment the background and objects in the two-dimensional image.

3. The method for converting two-dimensional data into three-dimensional data according to claim 1, characterized in that: In step S3, the NeRF algorithm is used to generate a three-dimensional model of the object according to the depth information of the object and the background in the two-dimensional image.

4. The method for converting two-dimensional data into three-dimensional data according to claim 1, characterized in that: Step S4 also includes the following steps: S41: performing viewpoint calibration on the multi-angle images of the three-dimensional model; S42: using a projection method to generate two-dimensional images of the three-dimensional model at different viewing angles; S43: The image filling algorithm based on the diffusers artificial intelligence model fills in the missing parts of the image background due to changes in visual angle.

5. The method for converting two-dimensional data into three-dimensional data according to claim 1, characterized in that: Step S5 includes the following steps: S51: performing raster image processing on the multi-angle image to form a raster three-dimensional image; S52: Rendering the raster three-dimensional image; S53: Outputting a raster three-dimensional image.

6. The method for converting two-dimensional data into three-dimensional data according to claim 1, characterized in that: It also includes printing the grating three-dimensional image data and adding an adapted grating for display or displaying it by a three-dimensional display device.

7. A terminal device, characterized in that: The terminal device includes a memory, a processor, and a program for converting two-dimensional data into three-dimensional data that is stored on and can be run on the processor. When the program for converting two-dimensional data into three-dimensional data is executed by the processor, the steps of a method for converting an image into a three-dimensional image as described in any one of claims 1 to 6 are implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program for converting two-dimensional data into three-dimensional data, and when the program for converting two-dimensional data into three-dimensional data is executed by the processor, the steps of the method for converting an image into a three-dimensional image as described in any one of claims 1 to 6 are implemented.

Images