Three-dimensional model output method, apparatus and device, and computer readable storage medium

Abstract

Embodiments of the present application relate to the technical field of three-dimensional reconstruction, and disclosed are a three-dimensional model output method, apparatus and device, and a computer readable storage medium. The method comprises: acquiring a plurality of initial images of a target object or a target area to perform three-dimensional reconstruction; establishing a three-dimensional model for the target object or the target area; then projecting the generated three-dimensional model on a virtual screen; and outputting an image, which is obtained by projection, on the virtual screen. In this way, the generation of the three-dimensional model is faster, there is no need for technicians to manually construct the model, the image output data volume is lower, and the output speed is faster.

Description

Three-dimensional model output method, device, equipment and computer-readable storage medium Technical Field

[0001] The embodiments of the present application relate to the field of three-dimensional reconstruction technology, and specifically to a three-dimensional model output method, device, equipment and computer-readable storage medium. Background Art

[0002] With the development of modern technology, people need to use 3D models in more and more aspects. For example, 3D models are used to print parts in industry and to design and plan buildings. Common 3D models are often manually constructed by technicians using modeling software based on pre-designed drawings. When the constructed 3D model needs to be viewed on other devices, the complete 3D model file needs to be sent to the other device through hardware copying or wireless transmission so that the 3D model can be viewed on other devices.

[0003] However, in some application areas, the existing method of manually constructing 3D models and directly outputting 3D model data when needed may cause inconvenience. For example, when people on foot want to use AR glasses or mobile phones to view the building shapes and road contours of the area they are about to arrive at in order to determine whether the destination is ahead, if technicians manually model all buildings and roads in advance, it will take a lot of time and effort, and the amount of data in the 3D model will also be large. When the 3D model is transmitted to the terminal carried by the user for reading and viewing, the data transmission speed will be limited, resulting in the user having to spend a lot of time waiting for the 3D model data to be transmitted to the user terminal. In addition, if the user walks in an area where the 3D model has not been completed in advance, or when the buildings and roads in the area change, the user will not be able to view or will receive incorrect 3D model information.

[0004] Summary of the Invention

[0005] In view of the above problems, embodiments of the present application provide a three-dimensional model output method, apparatus, device, and computer-readable storage medium to solve the problem of inconvenience in generating and outputting three-dimensional model images.

[0006] According to one aspect of an embodiment of the present application, a three-dimensional model output method is provided, comprising: acquiring a plurality of first initial images captured by cameras at different camera poses;

[0007] Performing three-dimensional reconstruction based on the multiple first initial images to obtain a three-dimensional model;

[0008] Projecting the three-dimensional model to obtain a target image on a virtual screen;

[0009] Output the target image.

[0010] In an optional manner, performing three-dimensional reconstruction based on the plurality of first initial images to obtain a three-dimensional model includes: generating a point cloud set based on the plurality of first initial images; constructing the three-dimensional model based on the point cloud set;

[0011] The three-dimensional model is projected to obtain a target image on a virtual screen, including: calculating a surface normal vector of a point cloud set; obtaining a randomly sampled point cloud using a random algorithm for the point cloud set; generating a target space voxel grid based on the surface normal vector and the randomly sampled point cloud; performing differentiable rendering on the target space voxel grid to obtain a color value of each pixel in the target space voxel grid; and projecting the color value of each pixel in the target space voxel grid onto the virtual screen through a projection matrix to obtain a target image.

[0012] In an optional manner, a target spatial voxel grid is generated based on a surface normal vector and a randomly sampled point cloud, including: combining a point cloud set with a surface normal vector to generate a surface grid; constructing an empty spatial voxel grid; projecting the surface grid into the empty spatial voxel grid to obtain an initial spatial voxel grid; calculating a depth value of each voxel grid in the initial spatial voxel grid; and applying the depth value to the initial spatial voxel grid to obtain a target spatial voxel grid.

[0013] In an alternative approach, according to the formula: Calculate the depth value of each voxel grid in the initial space voxel grid, where d x,y,z is the depth value of the voxel grid with coordinates (x, y, z) in the initial space voxel grid, n is the number of triangles of the surface mesh projected onto the voxel grid, f x,y,z is the default depth value calculated based on the position of the voxel grid in space, d i is the depth value of the triangle projected onto the voxel grid by the surface mesh, T j is the set of coordinates of all voxel grids in the initial space voxel grid.

[0014] In an alternative approach, according to the formula: The color value of each pixel in the target space voxel grid is projected onto the virtual screen, where u, v, and w are the coordinate positions of the pixel with coordinates (X, Y, Z) in the space voxel grid on the virtual screen, and f x With f y is the focal length, c x with c y The center coordinates of the image on the virtual screen.

[0015] In an optional manner, before outputting the target image, the three-dimensional model output method further includes: converting the coordinates of each pixel on the target image into a two-dimensional coordinate system.

[0016] In one optional embodiment, after outputting the target image, the three-dimensional model output method further includes: acquiring a second initial image; superimposing the second initial image with the target image to obtain a new target image; and outputting the new target image. According to another aspect of an embodiment of the present application, a three-dimensional model output device is provided, comprising: a first acquisition module for acquiring multiple first initial images captured by cameras at different camera positions; a construction module for performing three-dimensional reconstruction based on the multiple first initial images to obtain a three-dimensional model; a conversion module for projecting the three-dimensional model to obtain the target image on a virtual screen; and a first output module for outputting the target image.

[0017] According to another aspect of an embodiment of the present application, a three-dimensional model output device is provided, including: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus; the memory is used to store at least one program, and the program enables the processor to execute the operation of any one of the three-dimensional model output methods as above.

[0018] According to another aspect of an embodiment of the present application, a computer-readable storage medium is provided, wherein the storage medium stores executable instructions, wherein the executable instructions cause a three-dimensional model output device to perform the operations of any of the three-dimensional model output methods described above. The above description is merely an overview of the technical solution of the embodiment of the present application. To more clearly understand the technical means of the embodiment of the present application, implementation can be carried out in accordance with the contents of the description. To make the above and other purposes, features, and advantages of the embodiment of the present application more apparent and easy to understand, the following specifically describes the specific implementation methods of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present application. In addition, the same reference symbols are used to represent the same components throughout the drawings. In the drawings:

[0020] FIG1 is a schematic diagram of a flow chart of a three-dimensional model output method provided in an embodiment of the present application;

[0021] FIG2 is a flow chart of the sub-steps included in step 120 and step 130 of the present application;

[0022] FIG3 is a flow chart of the sub-steps included in step 133 of the present application;

[0023] FIG4 is a flow chart of the steps before step 140 of the present application;

[0024] FIG5 is a flow chart of the steps following step 140 of the present application;

[0025] FIG6 is a functional block diagram of a three-dimensional model output device provided in an embodiment of the present application;

[0026] FIG7 is a schematic structural diagram of a three-dimensional model output device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will describe in more detail exemplary embodiments of the present application with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. A three-dimensional model is a polygonal representation of an object, usually displayed by a computer or other video device. The displayed object can be a real-world entity or a fictional object. Anything that exists in the physical world can be represented by a three-dimensional model. Precisely because of this characteristic of three-dimensional models, three-dimensional models can be used in many fields to more conveniently simulate, inspect, and produce. By pre-generating a three-dimensional model, designers and decision makers can accurately understand the appearance and structure of the product in advance before the product goes into production, and make corrections to the product design and structure, etc., to avoid defects after the product goes into production. At the same time, in some construction engineering fields, by analyzing three-dimensional models, it is possible to more accurately estimate the cost of the project in advance by calculating parameters such as volume and mass, which can bring certain conveniences to the operation.

[0028] Now, with the development of technology, 3D models can be applied to all aspects of people's daily lives. People can view pre-built 3D models on various mobile devices and enjoy the convenience brought by 3D models containing rich information anytime and anywhere, which brings convenience to people's daily activities such as map navigation, furniture arrangement and online shopping.

[0029] However, the inventors of this application have noticed that currently common three-dimensional models often require professional technicians to design and model them in advance, and can only be put into use after they are finally completed. The construction speed of three-dimensional models is often slow, and if the three-dimensional model needs to cover things in a larger scene, the complete three-dimensional model data will be larger, and it may take a lot of time to transmit and load the data. This is reflected in the actual application scenario that users cannot view the three-dimensional model of the real scene that has not been built, and when they need to view the three-dimensional model of the real scene of a larger area that has been built, it takes a long time to download the three-dimensional model data. At the same time, due to the performance of the device used by the user, it may be impossible to load the three-dimensional model or there may be a freeze in the process of viewing the three-dimensional model, which is not conducive to the promotion and use of three-dimensional model technology. Therefore, there is an urgent need to develop a three-dimensional model output method with fast model generation and small output data volume.

[0030] Based on this, the inventors have conducted research and designed a three-dimensional model output method. By obtaining multiple initial images of the target object or target area for three-dimensional reconstruction, a three-dimensional model of the target object or target area is established, and the generated three-dimensional model is projected onto a virtual screen. The projected image on the virtual screen is output. This method can make the generation of the three-dimensional model faster, without the need for technicians to manually construct the model, and the image output data volume is smaller and the output speed is faster.

[0031] FIG1 shows a flow chart of a 3D model output method provided in an embodiment of the present application. The method is executed by a 3D model output device, which may be, for example, a mobile phone, a computer, or a display. As shown in FIG1 , the method includes the following steps:

[0032] Step 110: Acquire a plurality of first initial images captured by cameras at different camera positions;

[0033] Step 120: Perform three-dimensional reconstruction based on the multiple first initial images to obtain a three-dimensional model;

[0034] Step 130: Projecting the three-dimensional model to obtain a target image on a virtual screen;

[0035] Step 140: Output the target image.

[0036] In steps 110 to 120, multiple first initial images captured by cameras in different camera poses can be obtained through drone aerial photography or mobile phone camera photography. Since the multiple first initial images will be used for three-dimensional reconstruction, when multiple target images are required to generate a three-dimensional model of the same target, the multiple first initial images should contain at least some of the same target. In some application scenarios, a single image can be used to generate a separate three-dimensional model. In this case, the multiple first initial images can be completely different targets, but with connected or approximately connected boundaries. The multiple three-dimensional models generated in step 120 are then spliced ​​together to obtain a complete three-dimensional model. In order to achieve higher accuracy in the three-dimensional reconstruction of the three-dimensional model, the multiple first initial images should be able to reflect information from multiple angles of the target object or area. When the cameras capturing the first initial images have different camera poses, which means different shooting angles, this can, to a certain extent, ensure that the multiple first initial images can better reflect information from multiple angles of the target object or area. It should be noted that the multiple first initial images taken by cameras with different camera postures may refer to multiple images taken sequentially by the same camera device at different camera postures, or may refer to multiple images taken simultaneously by multiple camera devices with different camera postures deployed simultaneously. Those skilled in the art should be able to adjust the shooting strategy according to actual needs to obtain multiple first initial images that are convenient for three-dimensional reconstruction. The embodiments of the present application do not impose any special restrictions on the order of obtaining the first initial images and the specific shooting method.

[0037] In step 120, 3D reconstruction can be performed based on the multiple first initial images, for example, by generating a point cloud from the multiple initial images, then processing the point cloud to ultimately generate a 3D model. Depending on the actual situation, other similar methods or algorithms may also be used for 3D reconstruction. The goal is to restore the depth information in the multiple first initial images as accurately as possible, so that the ultimately reconstructed 3D model is as consistent as possible with the actual scene or the desired object shape.

[0038] In step 130, projecting the three-dimensional model refers to mapping the three-dimensional model onto a medium to achieve data compression. In the embodiment of the present application, the three-dimensional model is projected onto a virtual screen. For example, this can be used to obtain a two-dimensional planar image that reflects the three-dimensional model as viewed from a certain angle. This can be achieved by vertically mapping all pixels at a certain angle of the three-dimensional model onto the virtual screen. It should be noted that the virtual screen can be a plane, a curved surface, or even a hemispherical surface that can cover a large angle of the three-dimensional model. Depending on the display device required to display the target image, the virtual screen can adopt different sizes and shapes to ensure that the size, dimensions, and viewing angle of the target image obtained by projecting the three-dimensional model onto the virtual screen meet the requirements of the display device.

[0039] It should be noted that the position of the virtual screen relative to the 3D model should be adjustable based on actual output needs. Since typical display devices often only display a single angle of the 3D model, when a user adjusts the display angle of the 3D model through gestures, buttons, or a controller, the position of the virtual screen can change accordingly, so that the target image projected by the 3D model on the virtual screen also changes accordingly.

[0040] In step 140, the target image may be output to, for example, a display screen, a mobile phone, or AR glasses. Generally speaking, when the target image is output to a device capable of viewing large-size, wide-angle images, such as AR glasses, the size of the virtual screen in step 130 may be larger, for example, encompassing a larger angular area surrounding the 3D model, so that the target image projected from the 3D model can reflect more information about the 3D model.

[0041] By acquiring multiple first initial images taken by cameras with different camera postures, and performing three-dimensional reconstruction based on the multiple first initial images, a three-dimensional model is obtained, and then the three-dimensional model is projected to obtain a target image on a virtual screen, and the target image is used for output, the generation of the three-dimensional model in the embodiment of the present application is faster. When the three-dimensional model needs to be output and displayed, the target image obtained by projecting the three-dimensional model is output, so that the amount of output data is reduced, the pressure on bandwidth during data transmission is reduced, and the output speed of the three-dimensional model can be improved.

[0042] Please refer to Figure 2, which is a flowchart of the sub-steps included in step 120 and step 130 of the present application. As shown in Figure 2, in some embodiments of the present application, step 120 includes:

[0043] Step 121: generating a point cloud set based on a plurality of first initial images;

[0044] Step 122: constructing a three-dimensional model based on the point cloud set;

[0045] Step 130 includes:

[0046] Step 131: Calculate the surface normal vector of the point cloud set;

[0047] Step 132: using a random algorithm to obtain a random sampling point cloud for the point cloud set;

[0048] Step 133: Generate a target space voxel grid based on the surface normal vector and the randomly sampled point cloud;

[0049] Step 134: performing differentiable rendering on the target space voxel grid to obtain the color value of each pixel in the target space voxel grid;

[0050] Step 135: Project the color value of each pixel in the target space voxel grid onto the virtual screen through the projection matrix to obtain the target image.

[0051] In steps 121 and 122, a point cloud set refers to a collection of multiple three-dimensional point clouds. Generating a point cloud set from multiple first initial images can, for example, involve calculating the features of each pixel in the first initial images, matching the pixels in the multiple first initial images based on the poses of the multiple cameras when capturing the first initial images, and then constructing the pixels in the multiple first initial images in the same coordinate system to generate multiple three-dimensional point clouds to form the point cloud set.

[0052] In steps 131 to 133, the surface normal vector of the point cloud set is calculated, and a random search is performed around the point cloud set to obtain a randomly sampled point cloud. The randomly sampled point cloud is then combined with the surface normal vector to obtain a target space voxel grid. The purpose of calculating the surface normal vector of the point cloud set is to enable the target space voxel grid generated in step 133 to contain lighting and shadow data, so as to improve the effect of the final generated three-dimensional model. The purpose of calculating the randomly sampled point cloud is to reduce the density of the point cloud while keeping the overall geometric characteristics of the point cloud unchanged, thereby reducing the amount of data to be processed and increasing the calculation speed. In step 134, the target space voxel grid is subjected to differentiable rendering to obtain the color value of each pixel in the target space voxel grid, wherein differentiable rendering refers to a rendering process that can perform differential derivation.

[0053] Through the above steps 121 to 135, the three-dimensional reconstruction effect is better and the calculation speed is faster. In addition, due to the use of differentiable rendering, parameters can be optimized during the three-dimensional reconstruction process, which can achieve a more accurate three-dimensional reconstruction effect.

[0054] Please refer to Figure 3, which is a flow chart of the sub-steps included in step 133 of the present application. As shown in Figure 3, in one embodiment of the present application, step 133 includes:

[0055] Step 1331: combining the point cloud set with the surface normal vector to generate a surface mesh;

[0056] Step 1332: construct an empty space voxel grid;

[0057] Step 1333: Project the surface mesh into an empty spatial voxel grid to obtain an initial spatial voxel grid;

[0058] Step 1334: Calculate the depth value of each voxel grid in the initial spatial voxel grid;

[0059] Step 1335: Apply the depth value to the initial spatial voxel grid to obtain the target spatial voxel grid.

[0060] Specifically, in steps 1131 to 1132, the point cloud set is first combined with the surface normal vector to generate a surface mesh without depth value data, and an empty spatial voxel grid is constructed. In steps 1333 to 1334, each triangular face in the surface mesh is projected into the empty spatial voxel grid, and the covered voxel grid of the obtained initial spatial voxel grid is marked as covered. At this time, for the voxel grid marked as covered, the depth value of its center point is calculated, and the calculated depth value is used as the depth value of all triangular faces in the surface mesh projected onto the marked voxel grid. For the voxel grid that is not marked as covered, its depth value can be calculated based on its position in space.

[0061] In steps 1331 to 1335, a method for generating a target space voxel grid is given, by projecting the surface grid generated by combining the point cloud set with the surface normal vector into an empty space voxel grid to obtain an initial space voxel grid, and calculating the depth value of each voxel grid in the initial space voxel grid, and then applying the depth value to the initial space voxel grid to obtain the target space voxel grid for generating a three-dimensional model.

[0062] In some embodiments of the present application, the calculation method of step 1334 is: according to the formula:

[0063] Calculate the depth value of each voxel grid in the initial space voxel grid, where d x,y,z is the depth value of the voxel grid with coordinates (x, y, z) in the initial space voxel grid, n is the number of triangles projected onto the voxel grid by the surface mesh, and f x,y,zis the default depth value calculated based on the position of the voxel grid in space, d i is the depth value of the triangle projected onto the voxel grid by the surface mesh, T j is the set of coordinates of all voxel grids in the initial space voxel grid.

[0064] The depth value of each voxel grid in the initial space voxel grid is calculated by the above formula, and the depth value can be applied to the initial space voxel grid to obtain the target space voxel grid containing the depth value information of each voxel grid, which is convenient for subsequent reconstruction and rendering of the three-dimensional model. The generated three-dimensional model can reflect the distance between each pixel and the projection perspective, making the three-dimensional model more refined and easier for users to observe and view.

[0065] In some embodiments of the present application, the calculation method of step 135 is: according to the formula:

[0066] The color value of each pixel in the target space voxel grid is projected onto the virtual screen, where u, v, and w are the coordinate positions of the pixel with coordinates (X, Y, Z) in the space voxel grid on the virtual screen, and f x With f y is the focal length, c x with c y The center coordinates of the image on the virtual screen.

[0067] It should be noted that if the virtual screen is a two-dimensional plane, then when projecting the three-dimensional coordinates of the pixels in the target space voxel grid onto the virtual screen, in some cases, the projected coordinates can be converted into two-dimensional coordinates in the screen coordinate system. It is understood that those skilled in the art can adaptively transform the projected coordinates of the target space voxel grid on the virtual screen according to actual needs, including but not limited to conversion from a three-dimensional coordinate system to a two-dimensional coordinate system and conversion of the coordinate system origin, to reduce data volume and improve calculation speed.

[0068] The projection coordinates of the color value of each pixel in the target space voxel grid on the virtual screen are calculated by the above formula, so that the projection position of each pixel in the target space voxel grid on the virtual screen can be unified on the virtual screen, and the problem of coordinate axis confusion is less likely to occur.

[0069] Please refer to Figure 4, which is a flow chart of the steps before step 140 of the present application. As shown in Figure 4, in some embodiments of the present application, before step 140, the following steps are further included:

[0070] Step 1401: Convert the coordinates of each pixel on the target image into a two-dimensional coordinate system.

[0071] In some cases, for example, after projecting the pixels in the target space voxel grid onto the virtual screen, the coordinates of several pixels obtained are: (3, 15, 2) (3, 16, 2)... (5, 3, 2). It is easy to know that the virtual screen is a plane with a Z axis of 2. At this time, the plane with a Z axis of 2 is used as a plane coordinate system, and the coordinates of several pixels obtained after projection are: (3, 15) (3, 16)... (5, 3).

[0072] By converting the coordinates of each pixel on the target image into a two-dimensional coordinate system, the coordinate data of the pixel coordinate values ​​of the target image obtained after projection on the virtual screen is more refined, which facilitates the output of the target image.

[0073] As shown in FIG5 , FIG5 is a flow chart of the steps after step 140 of the present application. In some embodiments of the present application, after step 140, the following steps are further included:

[0074] Step 1410: Acquire a second initial image;

[0075] Step 1420: superimposing the second initial image and the target image to obtain a new target image;

[0076] Step 1430: Output the new target image.

[0077] In step 1410 , the second initial image may be an image captured by a photographic device such as a camera or a mobile phone.

[0078] In steps 1420 to 1430, superimposing the second initial image with the target image to obtain a new target image means superimposing pixels in the second initial image with pixels in the target image. The superposition method can adopt image processing technology in computer vision to perform weighted averaging or transparency blending of the pixels of the two to obtain a new target image, and output the new target image to the display device.

[0079] By acquiring a second initial image and superimposing it with the original target image to obtain a new target image for output, when the target object or area corresponding to the three-dimensional model changes, the output image can reflect the change to a certain extent, allowing the user to more directly distinguish between the generated three-dimensional model and the actual target object or area based on the output new target image, and make corresponding adjustments to the operation or road planning.

[0080] Figure 6 shows a functional block diagram of a 3D model output device 600 according to one embodiment of the present application. As shown in Figure 6, the 3D model output device 600 includes a first acquisition module 601, a construction module 602, a first conversion module 603, and a first output module 604. The first acquisition module 601 is used to acquire multiple first initial images captured by cameras at different camera positions; the construction module 602 is used to perform 3D reconstruction based on the multiple first initial images to obtain a 3D model; the first conversion module 603 is used to project the 3D model to obtain a target image on a virtual screen; and the first output module 604 is used to output the target image.

[0081] In some embodiments, the construction module 602 further includes a first generation unit and a second generation unit. The first generation unit is configured to generate a point cloud set based on the plurality of first initial images, and the second generation unit is configured to construct a three-dimensional model based on the point cloud set.

[0082] In some embodiments, the first conversion module 603 further includes a first calculation unit, a second calculation unit, a third generation unit, a rendering unit, and a projection unit. The first calculation unit is used to calculate the surface normal vector of the point cloud set, the second calculation unit is used to use a random algorithm to obtain a randomly sampled point cloud for the point cloud set, the third generation unit is used to generate a target space voxel grid based on the surface normal vector and the randomly sampled point cloud, the rendering unit is used to perform differentiable rendering on the target space voxel grid to obtain the target space voxel grid, and the projection unit is used to project the color value of each pixel in the target space voxel grid onto a virtual screen using a projection matrix to obtain a target image.

[0083] In some embodiments, the third generation unit further includes a generation element, a construction element, a projection element, a calculation element, and a processing element. The generation element is used to combine the point cloud set with the surface normal vector to generate a surface mesh, the construction element is used to construct an empty spatial voxel mesh, the projection element is used to project the surface mesh into the empty spatial voxel mesh to obtain an initial spatial voxel mesh, the calculation element is used to calculate the depth value of each voxel grid in the initial spatial voxel mesh, and the processing element is used to apply the depth value to the initial spatial voxel mesh to obtain a target spatial voxel mesh.

[0084] In some embodiments, the 3D model output device 600 further includes a second conversion module configured to convert the coordinates of each pixel on the target image into a 2D coordinate system.

[0085] In some embodiments, the 3D model output device 600 further includes a second acquisition module, an overlay module, and a second output module. The second acquisition module is used to acquire a second initial image, the overlay module is used to overlay the second initial image with the target image to obtain a new target image, and the second output module is used to output the new target image.

[0086] According to another aspect of the embodiments of the present application, a 3D model output device is provided. Specifically, please refer to FIG7 , which shows a schematic structural diagram of a 3D model output device according to an embodiment of the present application. The specific embodiments of the present application do not limit the specific implementation of the 3D model output device.

[0087] As shown in FIG. 7 , the three-dimensional model output device may include: a processor 702 , a memory 706 , a communication interface 704 and a communication bus 708 .

[0088] The processor 702, memory 706 and communication interface 704 communicate with each other via a communication bus 708. The memory 706 is used to store at least one program 710, which enables the processor 702 to execute the relevant steps in the above-mentioned embodiment of the three-dimensional model output method.

[0089] Specifically, the program 710 may include program code including computer-executable instructions.

[0090] Processor 702 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the 3D model output device may be of the same type, such as one or more CPUs, or may be of different types, such as one or more CPUs and one or more ASICs.

[0091] The memory 706 is used to store the program 710. The memory 706 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0092] Program 710 can be specifically called by processor 702 to enable the three-dimensional model output device to perform the following operations: obtain multiple first initial images taken by cameras with different camera positions; perform three-dimensional reconstruction based on the multiple first initial images to obtain a three-dimensional model; project the three-dimensional model to obtain a target image on a virtual screen; and output the target image.

[0093] The present application also provides a computer-readable storage medium having executable instructions stored therein. When the executable instructions are executed on a 3D model output device, the 3D model output device executes the 3D model output method of any of the above embodiments.

[0094] The executable instructions can specifically be used to enable the three-dimensional model output device to perform the following operations: obtain multiple first initial images taken by cameras with different camera postures; perform three-dimensional reconstruction based on the multiple first initial images to obtain a three-dimensional model; project the three-dimensional model to obtain a target image on a virtual screen; and output the target image.

[0095] The algorithm or demonstration provided here are not inherently relevant to any particular computer, virtual system or other equipment. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing this type of system. In addition, the present application embodiment is not directed to any specific programming language yet. It should be understood that various programming languages ​​can be utilized to realize the content of the present application described here, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of the present application.

[0096] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0097] Similarly, it should be understood that in order to streamline the present application and assist in understanding one or more of the various aspects of the invention, in the above description of the exemplary embodiments of the present application, the various features of the embodiments of the present application are sometimes grouped together into a single embodiment, figure, or description thereof.

[0098] Those skilled in the art will appreciate that the modules in the devices in the embodiments can be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments can be combined into one module or unit or component, and can be divided into multiple submodules or subunits or subassemblies. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying abstract and drawings) and all processes or units of any method or device disclosed in this manner can be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying abstract and drawings) can be replaced by alternative features providing the same, equivalent or similar purpose.

[0099] It should be noted that the above embodiments are illustrative rather than limiting of the present invention, and that those skilled in the art may design alternative embodiments without departing from the scope of the present invention. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order in which they are to be performed.

Claims

1. A three-dimensional model output method, characterized in that: include: Acquire a plurality of first initial images taken by cameras at different camera positions; Performing three-dimensional reconstruction based on the plurality of first initial images to obtain a three-dimensional model; Projecting the three-dimensional model to obtain a target image on a virtual screen; The target image is output.

2. The three-dimensional model output method according to claim 1, characterized in that: The performing three-dimensional reconstruction according to the plurality of first initial images to obtain a three-dimensional model comprises: generating a point cloud set according to the plurality of first initial images; constructing the three-dimensional model according to the point cloud set; The projecting of the three-dimensional model to obtain a target image on a virtual screen includes: Calculate the surface normal vector of the point cloud set; A random sampling point cloud is obtained by using a random algorithm for the point cloud set; Generating a target space voxel grid according to the surface normal vector and the randomly sampled point cloud; Performing differentiable rendering on the target space voxel grid to obtain a color value of each pixel in the target space voxel grid; The color value of each pixel in the target space voxel grid is projected onto the virtual screen through a projection matrix to obtain the target image.

3. The three-dimensional model output method according to claim 2, characterized in that: The step of generating a target space voxel grid according to the surface normal vector and the randomly sampled point cloud comprises: Combining the point cloud set with the surface normal vector to generate a surface mesh; Construct an empty space voxel grid; Projecting the surface grid into the empty space voxel grid to obtain an initial space voxel grid; Calculating the depth value of each voxel grid in the initial spatial voxel grid; The depth value is applied to the initial spatial voxel grid to obtain a target spatial voxel grid.

4. The three-dimensional model output method according to claim 3, characterized in that: According to the formula: The depth value of each voxel grid in the initial spatial voxel grid is calculated, where d x,y,z is the depth value of the voxel grid with coordinates (x, y, z) in the initial spatial voxel grid, n is the number of triangles projected onto the voxel grid by the surface mesh, and f x,y,z is the default depth value calculated based on the position of the voxel grid in space, d i is the depth value of the triangle projected onto the voxel grid by the surface mesh, T j is the set of coordinates of all voxel grids in the initial space voxel grid.

5. The three-dimensional model output method according to claim 2, characterized in that: According to the formula: The color value of each pixel in the target space voxel grid is projected onto the virtual screen, where u, v and w are the coordinate positions of the pixel with coordinates (X, Y, Z) in the space voxel grid on the virtual screen, and f x With f y is the focal length, c x With c y are the center point coordinates of the image in the virtual screen.

6. The three-dimensional model output method according to claim 5, characterized in that: Before outputting the target image, the three-dimensional model output method further includes: The coordinates of each pixel on the target image are converted into a two-dimensional coordinate system.

7. The three-dimensional model output method according to claim 1, characterized in that: After outputting the target image, the three-dimensional model output method further includes: acquiring a second initial image; Superimposing the second initial image and the target image to obtain a new target image; The new target image is output.

8. A three-dimensional model output device, characterized in that: include: A first acquisition module, used to acquire a plurality of first initial images taken by cameras at different camera positions; A construction module, used for performing three-dimensional reconstruction based on the plurality of first initial images to obtain a three-dimensional model; A first conversion module, used for projecting the three-dimensional model to obtain a target image on a virtual screen; The first output module is used to output the target image.

9. A three-dimensional model output device, characterized in that: include: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one program, and the program enables the processor to execute the operation of the three-dimensional model output method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The storage medium stores at least one program, and when the program is executed on the three-dimensional model output device, the three-dimensional model output device executes the operation of the three-dimensional model output method according to any one of claims 1 to 7.

Images