Free viewpoint video generation method and electronic device
By encoding and decoding multiple input images from different perspectives, obtaining point cloud mapping relationships, performing point cloud conversion and sparse three-dimensional reconstruction, the dependence on camera calibration and layout in the existing technology is solved, and efficient Robust 3D reconstruction effect.
Patent Information
- Application Number
- CN202411060419.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-05
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-08-05
AI Technical Summary
The prior art relies heavily on camera calibration and layout during the three-dimensional reconstruction process, resulting in high deployment costs and high requirements for the system's image processing capabilities.
By obtaining multiple input images collected for the same scene from different perspectives, semantic feature encoding and decoding are performed, point cloud mapping relationship is obtained, point cloud transformation and sparse three-dimensional reconstruction, and finally the target three-dimensional reconstruction result is generated based on the rendering network.
Three-dimensional reconstruction without precalibration is realized, which reduces the number of camera deployment requirements and improves the robustness and efficiency of three-dimensional reconstruction.
Smart Images

Figure CN118587367B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a free viewpoint video generation method and electronic device. Background Art
[0002] In computer vision, multi-view Figure 3 Three-dimensional reconstruction refers to the process of reconstructing three-dimensional information based on images collected from multiple perspectives.
[0003] In the prior art, it is generally necessary to calibrate the camera first, that is, to calculate the relationship between the camera's image coordinate system and the world coordinate system, and then use the information in multiple two-dimensional images to reconstruct three-dimensional information. In addition, a large number of cameras need to be deployed for the reconstruction scene to ensure the quality of the three-dimensional reconstruction results. Therefore, when performing three-dimensional reconstruction tasks, it is highly dependent on pre-tasks such as calibration and camera layout. At the same time, the cost of camera deployment is high, and the large amount of image data that needs to be processed leads to high requirements on the system's image processing capabilities. Summary of the invention
[0004] In order to solve the above technical problems, the present application at least provides a three-dimensional image reconstruction method, a free viewpoint video generation method and an electronic device.
[0005] In a first aspect, the present application provides a method for three-dimensional reconstruction of an image, the method comprising: obtaining multiple input images acquired from the same scene at different viewing angles, performing semantic feature encoding and semantic feature decoding on the multiple input images, and obtaining a point cloud mapping relationship; wherein there is a field of view overlap between at least two viewing angles, and the point cloud mapping relationship is used to store the transformation relationship between the image coordinates corresponding to each input image and the three-dimensional reconstruction coordinates; using the point cloud mapping relationship to perform point cloud conversion on the multiple input images respectively, and obtain point cloud data corresponding to the multiple input images respectively; performing sparse three-dimensional reconstruction on the point cloud data corresponding to the multiple input images respectively, and obtaining a sparse reconstruction result; rendering each two-dimensional face element contained in the sparse reconstruction result based on the multiple input images, and obtaining a target three-dimensional reconstruction result.
[0006] In one embodiment, a three-dimensional reconstruction neural network model is pre-trained, and the three-dimensional reconstruction neural network model contains an image encoder and an image decoder; semantic feature encoding and semantic feature decoding are performed on multiple input images to obtain a point cloud mapping relationship, including: image division is performed on each input image to obtain an image block corresponding to each input image; the image block corresponding to each input image is input into the image encoder for semantic feature encoding to obtain the semantic feature corresponding to each input image output by the image encoder; the semantic feature corresponding to each input image is input into the image decoder for semantic feature decoding to obtain the point cloud mapping relationship output by the image decoder.
[0007] In one embodiment, a three-dimensional reconstruction neural network model is pre-trained, and the three-dimensional reconstruction neural network model contains a face rendering network; each two-dimensional face contained in the sparse reconstruction result is rendered based on multiple input images to obtain a target three-dimensional reconstruction result, including: performing two-dimensional face construction on the sparse reconstruction result to obtain multiple two-dimensional faces; inputting the multiple two-dimensional faces and the multiple input images into the face rendering network to obtain a rendering result for each two-dimensional face output by the face rendering network; and combining the rendering results of each two-dimensional face to obtain a target three-dimensional reconstruction result.
[0008] In one embodiment, the training steps of the facet rendering network include: obtaining multiple sample images acquired from the same scene at different viewing angles, wherein the multiple sample images are marked with actual three-dimensional models; calculating sparse reconstruction results corresponding to the multiple sample images, and constructing two-dimensional facets corresponding to the sparse reconstruction results of the multiple sample images; inputting the two-dimensional facets corresponding to the multiple sample images and the multiple sample images into a pre-trained facet rendering network to obtain a predicted rendering result of each two-dimensional facet output by the pre-trained facet rendering network; calculating the loss between the predicted rendering result of each two-dimensional facet and the actual three-dimensional model; and using the loss to adjust the network parameters of the pre-trained facet rendering network to obtain a trained facet rendering network.
[0009] In one embodiment, the method further includes: calculating the confidence of each two-dimensional surface element based on the transparency and / or depth information of each two-dimensional surface element; and deleting two-dimensional surface elements whose confidence is lower than a preset confidence threshold among the two-dimensional surface elements corresponding to the multiple sample images.
[0010] A second aspect of the present application provides a method for generating a free viewpoint video, the method comprising: obtaining multiple input images acquired from the same scene at different viewing angles, and generating target three-dimensional reconstruction results corresponding to the multiple input images based on the above-mentioned image three-dimensional reconstruction method; constructing a virtual camera and determining the position and posture of the virtual camera relative to the target three-dimensional reconstruction result; based on the position and posture of the virtual camera, projecting the target three-dimensional reconstruction result onto the image plane of the virtual camera to obtain the projection picture corresponding to the virtual camera; generating a free viewpoint video based on the multiple input images and the projection picture corresponding to each virtual camera.
[0011] In one embodiment, constructing a virtual camera includes: calculating the depth variation severity of a preset area of a target three-dimensional reconstruction result; determining a virtual camera density corresponding to the preset area based on the depth variation severity; and constructing a virtual camera in the preset area based on the virtual camera density corresponding to the preset area.
[0012] In one embodiment, multiple input images are acquired by a real camera; based on the multiple input images and the projection screen corresponding to each virtual camera, a free viewpoint video is generated, including: creating an interpolation viewpoint between a virtual camera and / or a real camera, and using a virtual camera and / or a real camera adjacent to the interpolation viewpoint as a reference camera corresponding to the interpolation viewpoint; performing interpolation calculation on the projection screen and / or the input image of the reference camera to obtain an interpolation screen corresponding to the interpolation viewpoint; and combining the multiple input images, the projection screen corresponding to each virtual camera, and the interpolation screen corresponding to each interpolation viewpoint to generate a free viewpoint video.
[0013] In one embodiment, a free viewpoint video is generated based on multiple input images and the projection screen corresponding to each virtual camera, including: detecting a moving foreground contained in the input image; in response to occlusion of the free viewpoint video by the moving foreground, completing the occluded area in the free viewpoint video based on image content corresponding to the occluded area in a target three-dimensional reconstruction result.
[0014] The third aspect of the present application provides an image three-dimensional reconstruction device, which includes: an image acquisition module, which is used to acquire multiple input images obtained from the same scene at different viewing angles, and perform semantic feature encoding and semantic feature decoding on the multiple input images to obtain a point cloud mapping relationship; wherein there is a field of view overlap between at least two viewing angles, and the point cloud mapping relationship is used to store the transformation relationship between the image coordinates corresponding to each input image and the three-dimensional reconstruction coordinates; a point cloud conversion module, which is used to perform point cloud conversion on the multiple input images respectively using the point cloud mapping relationship to obtain point cloud data corresponding to the multiple input images respectively; a sparse reconstruction module, which is used to perform sparse three-dimensional reconstruction in combination with the point cloud data corresponding to the multiple input images respectively to obtain a sparse reconstruction result; and a facet rendering module, which is used to render each two-dimensional facet contained in the sparse reconstruction result based on the multiple input images to obtain a target three-dimensional reconstruction result.
[0015] In a fourth aspect, the present application provides a three-dimensional image reconstruction device, which includes: a three-dimensional reconstruction module, which is used to obtain multiple input images collected from the same scene at different viewing angles, and generate target three-dimensional reconstruction results corresponding to the multiple input images based on the above-mentioned image three-dimensional reconstruction method; a virtual camera construction module, which is used to construct a virtual camera and determine the posture of the virtual camera relative to the target three-dimensional reconstruction result; a picture projection module, which is used to project the target three-dimensional reconstruction result onto the image plane of the virtual camera based on the posture of the virtual camera, and obtain the projection picture corresponding to the virtual camera; a video generation module, which is used to generate a free viewpoint video based on the multiple input images and the projection picture corresponding to each virtual camera.
[0016] In a fifth aspect, the present application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-mentioned image three-dimensional reconstruction method and free viewpoint video generation method.
[0017] In a sixth aspect, the present application provides a computer-readable storage medium having program instructions stored thereon, which implement the above-mentioned three-dimensional image reconstruction method and free viewpoint video generation method when the program instructions are executed by a processor.
[0018] The above scheme obtains multiple input images of the same scene captured from different perspectives, encodes and decodes semantic features of the multiple input images, and obtains a point cloud mapping relationship; uses the point cloud mapping relationship to convert the multiple input images into point clouds respectively, and obtains point cloud data corresponding to the multiple input images; performs sparse 3D reconstruction in combination with the point cloud data corresponding to the multiple input images, and obtains a sparse reconstruction result; renders each two-dimensional face element contained in the sparse reconstruction result based on the multiple input images, and obtains a target 3D reconstruction result. The scheme can fully combine the high-level semantic information and low-level image features of the image, and realize 3D reconstruction from coarse to fine in sequence without pre-calibration, and can achieve highly robust 3D reconstruction even under sparse perspectives, that is, when the number of image acquisition devices deployed is small, the 3D reconstruction effect can be guaranteed.
[0019] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.
[0021] Figure 1 It is a schematic diagram of an implementation environment of a solution provided by an embodiment of the present application;
[0022] Figure 2 is a flow chart of a method for three-dimensional image reconstruction shown in an exemplary embodiment of the present application;
[0023] Figure 3 is a schematic diagram of the deployment of an image acquisition device shown in an exemplary embodiment of the present application;
[0024] Figure 4 is a flowchart of a method for generating a free viewpoint video shown in an exemplary embodiment of the present application;
[0025] Figure 5 is a schematic diagram of constructing an interpolation viewpoint shown in an exemplary embodiment of the present application;
[0026] Figure 6 is a block diagram of an image three-dimensional reconstruction device shown in an exemplary embodiment of the present application;
[0027] Figure 7 is a block diagram of a free viewpoint video generation device shown in an exemplary embodiment of the present application;
[0028] Figure 8 is a schematic structural diagram of an electronic device shown in an exemplary embodiment of the present application;
[0029] Fig. 9 It is a schematic diagram of the structure of a computer-readable storage medium shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION
[0030] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.
[0031] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.
[0032] The term "and / or" in this article is only an association information describing the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the previous and next associated objects are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.
[0033] The following is a description of the three-dimensional image reconstruction method provided in the embodiments of the present application.
[0034] Please refer to Figure 1 , which shows a schematic diagram of a solution implementation environment provided by an embodiment of the present application. The solution implementation environment may include multiple image acquisition devices 110 and servers 120, and the image acquisition devices 110 and servers 120 are connected to each other for communication.
[0035] The multiple image acquisition devices 110 are used to acquire images of the scene to be reconstructed to obtain multiple input images.
[0036] Server 120 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.
[0037] In one example, each image acquisition device 110 performs image acquisition processing to obtain multiple input images. The server 120 can perform sparse reconstruction results and two-dimensional surface rendering in sequence according to the acquired multiple input images to perform three-dimensional reconstruction from coarse to fine according to the multiple input images to obtain a target three-dimensional reconstruction result. Of course, the server 120 can store the target three-dimensional reconstruction result locally, transmit it to other terminals, or further process the target three-dimensional reconstruction result, etc.
[0038] In the image three-dimensional reconstruction method provided in the embodiment of the present application, the execution subject of each step can be the image acquisition device 110, or the server 120, or the image acquisition device 110 and the server 120 can be executed in interaction with each other, that is, part of the steps of the method are executed by the image acquisition device 110 and the other steps are executed by the server 120.
[0039] See also Figure 2 , Figure 2 is a flowchart of a method for three-dimensional image reconstruction shown in an exemplary embodiment of the present application. The method for three-dimensional image reconstruction can be applied to Figure 1 It should be understood that the method can also be applied to other exemplary implementation environments and be specifically executed by devices in other implementation environments, and this embodiment does not limit the implementation environment to which the method is applicable.
[0040] like Figure 2 As shown, the image 3D reconstruction method at least includes steps S210 to S240, which are described in detail as follows:
[0041] Step S210: Acquire multiple input images of the same scene captured from different perspectives, perform semantic feature encoding and semantic feature decoding on the multiple input images, and obtain a point cloud mapping relationship; wherein there is a field of view overlap between at least two perspectives, and the point cloud mapping relationship is used to store the transformation relationship between the image coordinates corresponding to each input image and the three-dimensional reconstruction coordinates.
[0042] By deploying image acquisition devices at different viewing angles, multiple input images of the same scene are acquired through these image acquisition devices.
[0043] For example, see Figure 3 , Figure 3 This is a schematic diagram of the deployment of an image acquisition device shown in an exemplary embodiment of the present application. Figure 3 As shown in FIG. 1 , assuming that the reconstruction task is to perform three-dimensional reconstruction of a specified target, n image acquisition devices can be sparsely set up around the specified target, and the value of n can be 3 to 6, such as Figure 3 The image acquisition devices installed include image acquisition device 110 - 1 , image acquisition device 110 - 2 , and image acquisition device 110 - 3 , and each image acquisition device corresponds to a different viewing angle.
[0044] When setting up the image acquisition device, there is an overlap of fields of view between at least two viewing angles, that is, at least one image acquisition device has an overlap of fields of view with other image acquisition devices.
[0045] In addition, the hue, exposure brightness and other parameters between each image acquisition device are adjusted to be basically consistent, that is, the difference between the hue, exposure brightness and other parameters between each image acquisition device is within the preset difference range; and for any point on the specified target, it should be ensured to be visible in at least one image acquisition device as much as possible. A large number of points that are invisible in the image acquisition device will affect the synthesis quality of nearby virtual viewpoints, that is, the number of specified target points that are invisible in each image acquisition device needs to be less than the preset number threshold.
[0046] After the image acquisition devices are deployed, the image data acquired by each image acquisition device is obtained to obtain multiple input images.
[0047] Then, semantic feature encoding and semantic feature decoding are performed on the multiple input images to obtain a point cloud mapping relationship, wherein the point cloud mapping relationship is used to store the transformation relationship between the image coordinates corresponding to each input image and the three-dimensional reconstruction coordinates.
[0048] By analyzing the semantic information of the input image, the point cloud mapping relationship is obtained without prior calibration. Combined with the semantic information of each input image, highly robust 3D reconstruction can be achieved even in sparse perspectives. That is, the 3D reconstruction effect can be guaranteed even when the number of image acquisition devices deployed is small.
[0049] Step S220: performing point cloud conversion on the multiple input images respectively using the point cloud mapping relationship to obtain point cloud data corresponding to the multiple input images respectively.
[0050] After obtaining the point cloud mapping relationship, point cloud conversion can be performed on each input image to convert the input image into point cloud data.
[0051] Specifically, the pixel coordinates (u, v) of the pixels in the input image are obtained, and the pixel coordinates are converted into 3D reconstruction coordinates (x, y, z) according to the point cloud mapping relationship to obtain the 3D reconstruction points. Combined with the 3D reconstruction points corresponding to each pixel of the input image, the final point cloud data is obtained.
[0052] Optionally, when decoding semantic features, a point cloud confidence map is also obtained. The point cloud confidence map contains the point cloud confidence corresponding to each pixel. The point cloud confidence map indicates the reliability of the three-dimensional reconstructed coordinates obtained by converting the pixels corresponding to the input image. The point cloud confidence map is used to alleviate the problem of positioning imbalance at different distances.
[0053] Step S230: performing sparse 3D reconstruction by combining the point cloud data corresponding to the multiple input images to obtain a sparse reconstruction result.
[0054] After obtaining the point cloud data corresponding to the multiple input images, preliminary three-dimensional reconstruction can be performed based on the point cloud data to obtain a sparse reconstruction result.
[0055] Specifically, based on the geometric relationship and topological structure between each point cloud data, these point cloud data are mapped into a unified three-dimensional space to achieve sparse three-dimensional reconstruction and obtain a sparse reconstruction result.
[0056] It should be noted that the specific method of mapping each point cloud data into a unified three-dimensional space can be flexibly selected according to actual conditions, and this application does not limit this.
[0057] For example, one of the multiple input images is selected as a reference image, and the coordinate system corresponding to the reference image is used as the reference coordinate system. Then, the point cloud data corresponding to each input image is registered, and according to the registration result of each point cloud data, each point cloud data is mapped to the reference coordinate system to achieve sparse 3D reconstruction and obtain a sparse reconstruction result.
[0058] Optionally, the visual field overlap area corresponding to each input image is calculated, and the input image with the largest visual field overlap area is selected as the reference image.
[0059] Of course, in addition to the sparse 3D reconstruction method shown above, sparse 3D reconstruction can also be performed using a pre-trained neural network model, which is not limited in this application.
[0060] Step S240: Rendering each two-dimensional surface element contained in the sparse reconstruction result based on the multi-channel input images to obtain a target three-dimensional reconstruction result.
[0061] After obtaining the sparse reconstruction result, the sparse reconstruction result is used as the initialization position to construct the two-dimensional surface element corresponding to the sparse reconstruction result, so as to render each two-dimensional surface element according to the multi-channel input image. The image features (such as texture features, color features, etc.) in the multi-channel input images can be effectively fused to obtain a high-precision target three-dimensional reconstruction result.
[0062] Exemplarily, a mathematical method may be used to render image features contained in an input image into corresponding two-dimensional surface elements, or a pre-trained neural network model may be used to perform two-dimensional surface element rendering processing, which is not limited in the present application.
[0063] Next, some embodiments of the present application are described in detail.
[0064] In some embodiments, a 3D reconstruction neural network model is pre-trained, and the 3D reconstruction neural network model includes an image encoder and an image decoder; in step S210, semantic feature encoding and semantic feature decoding are performed on multiple input images to obtain a point cloud mapping relationship, including:
[0065] Step S211: performing image division on each input image to obtain an image block corresponding to each input image.
[0066] The input image is divided into image blocks according to a preset size to obtain a plurality of image blocks. For example, the input image is divided into 3*3 blocks to obtain 9 image blocks.
[0067] Step S212: inputting the image blocks corresponding to each input image into the image encoder for semantic feature encoding, and obtaining the semantic features corresponding to each input image output by the image encoder.
[0068] The image encoder is used to extract semantic features of the input image.
[0069] Exemplarily, the image encoder may be implemented based on a VIT (Vision Transformer) neural network, or may be implemented based on a convolutional neural network (CNN), which is not limited in the present application.
[0070] Step S213: inputting the semantic features corresponding to each input image into the image decoder for semantic feature decoding, and obtaining the point cloud mapping relationship output by the image decoder.
[0071] According to the extracted semantic features corresponding to each input image, the point cloud mapping relationship is decoded.
[0072] In some embodiments, semantic features of input images with overlapping fields of view may be determined, and the semantic features of the input images with overlapping fields of view may be input into an image decoder for semantic feature decoding processing to obtain a point cloud mapping relationship output by the image decoder.
[0073] Among them, the image decoder can be implemented based on the cross-attention mechanism, and the network structure of the image decoder can be flexibly determined according to the actual scenario, which is not limited in this application.
[0074] In some embodiments, the image decoder is further configured to output a point cloud confidence map.
[0075] In some embodiments, the training process of the image encoder and the image decoder includes: obtaining multiple sample images captured from different perspectives of the same scene, and the multiple sample images are marked with actual three-dimensional models; inputting the multiple sample images into the image encoder and the image decoder to obtain a point cloud mapping relationship and a point cloud confidence map output by the image decoder; calculating the loss between the point cloud mapping relationship, the point cloud confidence map and the actual three-dimensional model, and the loss calculation function includes but is not limited to the L2 norm and the cross entropy loss function, etc.; using the calculated loss to train the image encoder and the image decoder to obtain a trained image encoder and image decoder.
[0076] In some embodiments, the 3D reconstruction neural network model includes a surface element rendering network; in step S240, each 2D surface element contained in the sparse reconstruction result is rendered based on the multi-channel input images to obtain the target 3D reconstruction result, including:
[0077] Step S241: construct two-dimensional surface elements on the sparse reconstruction result to obtain multiple two-dimensional surface elements.
[0078] The two-dimensional surface element includes surface element features such as normal direction, color, transparency, center position, depth information, etc. Among them, transparency is used to reflect the confidence of the two-dimensional surface element. The lower the transparency, the lower the confidence of the two-dimensional surface element.
[0079] Step S242: inputting a plurality of two-dimensional surfels and a plurality of input images into a surfel rendering network, and obtaining a rendering result of each two-dimensional surfel output by the surfel rendering network.
[0080] The facet rendering network performs facet rendering processing according to the facet features of the input two-dimensional facets and the image features of the input image to obtain the rendering results of each two-dimensional facet.
[0081] Step S243: combining the rendering results of each two-dimensional surface element to obtain the target three-dimensional reconstruction result.
[0082] The sparse reconstruction results are further refined and reconstructed through the surface element rendering network to obtain the target 3D reconstruction results that meet the preset requirements.
[0083] In some embodiments, the training steps of the facet rendering network include: obtaining multiple sample images collected from the same scene at different perspectives, wherein the multiple sample images correspond to actual three-dimensional models; calculating sparse reconstruction results corresponding to the multiple sample images, and constructing two-dimensional facets corresponding to the sparse reconstruction results of the multiple sample images; inputting the two-dimensional facets corresponding to the multiple sample images and the multiple sample images into a pre-trained facet rendering network to obtain a predicted rendering result of each two-dimensional facet output by the pre-trained facet rendering network; calculating the loss between the predicted rendering result of each two-dimensional facet and the actual three-dimensional model; and using the loss to adjust the network parameters of the pre-trained facet rendering network to obtain a trained facet rendering network.
[0084] The loss calculation function includes but is not limited to L2 norm and cross entropy loss function, etc. According to the loss between the predicted rendering result of each two-dimensional face element and the actual three-dimensional model, the face element rendering network is trained to obtain a trained face element rendering network.
[0085] For example, the loss corresponding to each two-dimensional face element is calculated, the total loss is calculated based on the loss corresponding to each two-dimensional face element, and the face element rendering network is trained according to the total loss. The total loss can be obtained by calculating the average value of the loss corresponding to each two-dimensional face element; or the confidence corresponding to each two-dimensional face element can be determined, and the loss corresponding to each two-dimensional face element is weighted averaged according to the confidence corresponding to each two-dimensional face element to obtain the total loss, which is not limited in this application.
[0086] Optionally, in order to improve the training efficiency of the network model and avoid overfitting of the network model, it also includes: calculating the confidence of each two-dimensional surface element based on the transparency and / or depth information of each two-dimensional surface element; deleting the two-dimensional surface elements corresponding to the multiple sample images whose confidence is lower than a preset confidence threshold.
[0087] For example, the confidence of each two-dimensional surface element may be calculated based on the transparency of the two-dimensional surface element; the confidence of each two-dimensional surface element may also be calculated based on the depth information of the two-dimensional surface element; or the confidence of each two-dimensional surface element may be calculated in combination with the transparency and depth information of the two-dimensional surface element.
[0088] Delete the two-dimensional surface elements whose confidence is lower than a preset confidence threshold among the two-dimensional surface elements corresponding to the multiple sample images.
[0089] Among them, the two-dimensional surface elements with confidence lower than the preset confidence threshold can be deleted periodically, such as deleting the two-dimensional surface elements once every 1000 training times; or the convergence speed of the surface element rendering network can be detected. If the convergence speed is within a preset range, the two-dimensional surface elements with confidence lower than the preset confidence threshold are deleted.
[0090] Through the above-mentioned image 3D reconstruction method, the high-level semantic information and low-level image features of the image can be fully combined to realize 3D reconstruction from coarse to fine in sequence without prior calibration, and highly robust 3D reconstruction can be achieved even under sparse viewing angles, that is, the 3D reconstruction effect can be guaranteed even when the number of image acquisition devices deployed is small.
[0091] See also Figure 4 , Figure 4 is a flowchart of a free viewpoint video generation method shown in an exemplary embodiment of the present application. The free viewpoint video generation method can be applied to Figure 1 It should be understood that the method can also be applied to other exemplary implementation environments and be specifically executed by devices in other implementation environments, such as the server is also communicatively connected to a terminal, and the terminal can be used as the execution subject. This embodiment does not limit the implementation environment to which the method is applicable.
[0092] like Figure 4 As shown, the image 3D reconstruction method at least includes steps S410 to S440, which are described in detail as follows:
[0093] Step S410: acquiring multiple input images acquired from the same scene at different viewing angles, and generating target three-dimensional reconstruction results corresponding to the multiple input images based on an image three-dimensional reconstruction method.
[0094] Step S420: construct a virtual camera and determine the position and posture of the virtual camera relative to the target 3D reconstruction result.
[0095] Exemplarily, a virtual camera may be randomly constructed between real image acquisition devices to obtain the position and posture of the virtual camera relative to the target three-dimensional reconstruction result.
[0096] Exemplarily, the depth variation severity of a preset area of the target 3D reconstruction result may also be calculated; the density of virtual cameras corresponding to the preset area may be determined based on the depth variation severity; and a virtual camera may be constructed in the preset area based on the density of virtual cameras corresponding to the preset area.
[0097] Specifically, the density of virtual cameras is proportional to the severity of depth changes to avoid local deformation problems in the generated free viewpoint video.
[0098] By calculating the virtual camera density in the above manner, it is possible to ensure that there are enough virtual cameras in areas where the depth information changes dramatically, thereby improving the quality of the generated free viewpoint video.
[0099] Step S430: Based on the position and posture of the virtual camera, project the target three-dimensional reconstruction result onto the image plane of the virtual camera to obtain the projection screen corresponding to the virtual camera.
[0100] The pre-generated target 3D reconstruction result is used to project the corresponding projection screen of the virtual camera, thereby increasing the image acquisition viewing angle and improving the smoothness of the generated free viewpoint video.
[0101] In some implementations, the projection image corresponding to the virtual camera does not need to be generated in real time, but the mapping relationship between the target 3D reconstruction result and the projection image is determined, and when the projection image corresponding to the virtual camera is needed to generate a free viewpoint video, the projection image is generated according to the mapping relationship. For smooth areas lacking texture, block mapping can be used, that is, each pixel point in the smooth area corresponds to a same mapping relationship, and precise pixel mapping is performed on the corresponding edges and other parts.
[0102] Step S440: Generate a free viewpoint video based on the multiple input images and the projection screen corresponding to each virtual camera.
[0103] Free viewpoint video is generated based on multiple input images and the projection screen corresponding to each virtual camera.
[0104] Optionally, the video playback content of the current viewing angle may be paused at any time, and the free viewpoint video between the current viewing angle and the next switching viewing angle may be displayed when the next switching viewing angle is determined.
[0105] In some embodiments, a free viewpoint video is generated based on multiple input images and the projection screen corresponding to each virtual camera, including: creating an interpolation viewpoint between virtual cameras and / or real cameras, and using the virtual cameras and / or real cameras adjacent to the interpolation viewpoint as reference cameras corresponding to the interpolation viewpoint; performing interpolation calculations on the projection screen and / or input image of the reference camera to obtain an interpolation screen corresponding to the interpolation viewpoint; and combining the multiple input images, the projection screen corresponding to each virtual camera, and the interpolation screen corresponding to each interpolation viewpoint to generate a free viewpoint video.
[0106] For example, see Figure 5 , Figure 5 A schematic diagram of constructing an interpolation viewpoint is shown as an exemplary embodiment of the present application. Figure 5As shown, assuming that the reconstruction task is to perform three-dimensional reconstruction of a specified target, three image acquisition devices are sparsely set up around the specified target, namely real camera 1, real camera 2, and real camera 3. The target three-dimensional reconstruction result is created according to the multi-channel input images collected by real camera 1, real camera 2, and real camera 3.
[0107] Then, virtual camera 1, virtual camera 2, and virtual camera 3 are constructed relative to the target 3D reconstruction result, and virtual camera 1, virtual camera 2, and virtual camera 3 are projected respectively according to the target 3D reconstruction result to obtain the projection images corresponding to virtual camera 1, virtual camera 2, and virtual camera 3 respectively.
[0108] Then, an interpolation viewpoint is created between the virtual camera and / or the real camera, and an interpolation calculation is performed based on the projection picture and / or input image corresponding to the reference camera adjacent to the interpolation viewpoint to obtain an interpolation picture. For example, an interpolation viewpoint is created between virtual camera 1 and real camera 1, and an interpolation calculation is performed on the projection picture of virtual camera 1 and the input image of real camera 1 to obtain an interpolation picture of the interpolation viewpoint.
[0109] In some real-time methods, the mapping relationship between the target 3D reconstruction result and the projection picture is stored, and when performing interpolation calculation, the transformation from the original pixel to the projection picture is directly obtained according to the mapping relationship to reduce the amount of calculation.
[0110] The free viewpoint video is generated by combining multiple input images, the projection picture corresponding to each virtual camera and the interpolation picture corresponding to each interpolation viewpoint.
[0111] Since the amount of calculation required to project the target 3D reconstruction result to obtain the projection image is greater than the amount of calculation required for interpolation calculation, by combining the virtual camera and interpolation viewpoint, more viewing angles can be obtained to improve the quality of the generated free viewpoint video.
[0112] In some embodiments, a free viewpoint video is generated based on multiple input images and the projection screen corresponding to each virtual camera, including: detecting a motion foreground contained in the input image, and in response to occlusion of the free viewpoint video by the motion foreground, completing the occluded area in the free viewpoint video based on image content corresponding to the occluded area in a target three-dimensional reconstruction result.
[0113] The input image is subjected to motion foreground detection and motion foreground segmentation, the motion foreground is decoupled, and the depth information corresponding to the motion foreground is restored to the target 3D reconstruction result. Among them, a depth information restoration model can be pre-built and trained, and the depth information restoration processing can be performed using the depth information restoration model. Different depth information restoration models can be trained for different types of motion foreground (such as rigid foreground and non-rigid foreground) to improve the accuracy of depth information restoration.
[0114] If there is a moving foreground occlusion resulting in an invisible area in the free viewpoint video, the image content corresponding to the occluded area in the target 3D reconstruction result is used to complete the occluded area in the free viewpoint video.
[0115] Through the above embodiments, it is possible to avoid the situation where the local foreground invades and affects the video synthesis effect, and avoid the problem of picture holes.
[0116] In some embodiments, when synthesizing free viewpoint video, a feathering radius can be set to perform local filtering so that the target edge where the depth changes dramatically can be better integrated with the background, thereby improving the visual effect of the free viewpoint video and reducing the aliasing problem in image stitching.
[0117] Compared with the prior art that requires the deployment of a large number of multi-perspective real image acquisition devices, the free viewpoint video generation method provided by the present application only requires the deployment of a small number of image acquisition devices. After generating the target three-dimensional reconstruction result based on the multi-channel input images acquired by these image acquisition devices, the target three-dimensional reconstruction result is used as the prior information for multi-perspective video synthesis, and the picture information corresponding to the virtual camera is generated based on the target three-dimensional reconstruction result. In this way, high-quality free viewpoint video can be directly generated only by relying on images with sparse perspectives, which reduces the equipment cost and the amount of image data processing.
[0118] Figure 6 FIG. 1 is a block diagram of an image 3D reconstruction device shown in an exemplary embodiment of the present application. Figure 6 As shown, the exemplary image 3D reconstruction device 600 includes: an image acquisition module 610, a point cloud conversion module 620, a sparse reconstruction module 630 and a facet rendering module 640. Specifically:
[0119] The image acquisition module 610 is used to acquire multiple input images acquired from the same scene at different viewing angles, and to perform semantic feature encoding and semantic feature decoding on the multiple input images to obtain a point cloud mapping relationship; wherein there is a field of view overlap between at least two viewing angles, and the point cloud mapping relationship is used to store the transformation relationship between the image coordinates corresponding to each input image and the three-dimensional reconstruction coordinates;
[0120] The point cloud conversion module 620 is used to perform point cloud conversion on the multiple input images respectively by using the point cloud mapping relationship to obtain point cloud data corresponding to the multiple input images respectively;
[0121] The sparse reconstruction module 630 is used to perform sparse 3D reconstruction by combining the point cloud data corresponding to the multiple input images to obtain a sparse reconstruction result;
[0122] The facet rendering module 640 is used to render each two-dimensional facet contained in the sparse reconstruction result based on the multi-channel input images to obtain the target three-dimensional reconstruction result.
[0123] It should be noted that the image 3D reconstruction device provided in the above embodiment and the image 3D reconstruction method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment, and will not be repeated here. In practical applications, the image 3D reconstruction device provided in the above embodiment can allocate the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.
[0124] Figure 7 FIG. 1 is a block diagram of a free viewpoint video generation device shown in an exemplary embodiment of the present application. Figure 7 As shown, the exemplary free viewpoint video generation device 700 includes: a 3D reconstruction module 710, a virtual camera construction module 720, a screen projection module 730 and a video generation module 740. Specifically:
[0125] The 3D reconstruction module 710 is used to obtain multiple input images of the same scene acquired from different viewing angles, and generate target 3D reconstruction results corresponding to the multiple input images based on the above-mentioned image 3D reconstruction method;
[0126] A virtual camera construction module 720 is used to construct a virtual camera and determine the position and posture of the virtual camera relative to the target 3D reconstruction result;
[0127] The image projection module 730 is used to project the target 3D reconstruction result onto the image plane of the virtual camera based on the position and posture of the virtual camera to obtain the projection image corresponding to the virtual camera;
[0128] The video generation module 740 is used to generate a free viewpoint video based on multiple input images and the projection screen corresponding to each virtual camera.
[0129] It should be noted that the free viewpoint video generation device provided in the above embodiment and the free viewpoint video generation method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment and will not be repeated here. In practical applications, the free viewpoint video generation device provided in the above embodiment can allocate the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.
[0130] See also Figure 8 , Figure 8 800 is a schematic diagram of the structure of an embodiment of an electronic device of the present application. The electronic device 800 includes a memory 801 and a processor 802, and the processor 802 is used to execute program instructions stored in the memory 801 to implement the steps in any of the above-mentioned three-dimensional image reconstruction methods and free viewpoint video generation method embodiments. In a specific implementation scenario, the electronic device 800 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 800 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.
[0131] Specifically, the processor 802 is used to control itself and the memory 801 to implement the steps in any of the above-mentioned three-dimensional image reconstruction methods and free viewpoint video generation method embodiments. The processor 802 can also be called a central processing unit (CPU). The processor 802 may be an integrated circuit chip with signal processing capabilities. The processor 802 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 802 can be implemented by an integrated circuit chip.
[0132] See also Fig. 9 , Fig. 9 The computer-readable storage medium 900 stores program instructions 910 that can be executed by a processor, and the program instructions 910 are used to implement the steps in any of the above-mentioned three-dimensional image reconstruction method and free viewpoint video generation method embodiments.
[0133] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.
[0134] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.
[0135] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.
[0136] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program code.
Claims
1. A free viewpoint video generation method, characterized in that: include: Acquire multiple input images acquired from the same scene at different viewing angles, perform semantic feature encoding and semantic feature decoding on the multiple input images, and obtain a point cloud mapping relationship; wherein there is a field of view overlap between at least two viewing angles, and the point cloud mapping relationship is used to store the transformation relationship between the image coordinates corresponding to each input image and the three-dimensional reconstruction coordinates; Performing point cloud conversion on the multiple input images respectively using the point cloud mapping relationship to obtain point cloud data corresponding to the multiple input images respectively; Combining the point cloud data corresponding to the multiple input images, respectively, to perform sparse three-dimensional reconstruction to obtain a sparse reconstruction result; Rendering each two-dimensional surface element contained in the sparse reconstruction result based on the multiple input images to obtain a target three-dimensional reconstruction result; Calculating a depth variation severity of the target three-dimensional reconstruction result, and determining a virtual camera density based on the depth variation severity; Based on the virtual camera density, construct a virtual camera; Determining the position and posture of the virtual camera relative to the target three-dimensional reconstruction result; Based on the position and posture of the virtual camera, projecting the target three-dimensional reconstruction result onto the image plane of the virtual camera to obtain a projection picture corresponding to the virtual camera; A free viewpoint video is generated based on the multiple input images and the projection screen corresponding to each virtual camera.
2. The method according to claim 1, characterized in that A 3D reconstruction neural network model is pre-trained, and the 3D reconstruction neural network model includes an image encoder and an image decoder; the semantic feature encoding and semantic feature decoding of the multiple input images are performed to obtain a point cloud mapping relationship, including: Performing image division on each input image to obtain an image block corresponding to each input image; Inputting the image blocks corresponding to each input image into the image encoder for semantic feature encoding, and obtaining the semantic features corresponding to each input image output by the image encoder; The semantic features corresponding to each input image are input into the image decoder for semantic feature decoding, so as to obtain a point cloud mapping relationship output by the image decoder.
3. The method according to claim 1, characterized in that A 3D reconstruction neural network model is pre-trained, and the 3D reconstruction neural network model contains a face rendering network; and rendering each 2D face contained in the sparse reconstruction result based on the multi-channel input images to obtain a target 3D reconstruction result includes: Performing two-dimensional surface element construction on the sparse reconstruction result to obtain a plurality of two-dimensional surface elements; Inputting the plurality of two-dimensional facets and the multi-channel input images into the facet rendering network, and obtaining a rendering result of each two-dimensional facet output by the facet rendering network; The target three-dimensional reconstruction result is obtained by combining the rendering results of each two-dimensional surface element.
4. The method according to claim 3, characterized in that: The training steps of the facet rendering network include: Acquire multiple sample images acquired from the same scene at different viewing angles, wherein the multiple sample images are marked with actual three-dimensional models; Calculating sparse reconstruction results corresponding to the multiple sample images, and constructing two-dimensional bins corresponding to the sparse reconstruction results of the multiple sample images; Inputting the two-dimensional surface elements corresponding to the multi-channel sample images and the multi-channel sample images into a pre-trained surface element rendering network to obtain a predicted rendering result of each two-dimensional surface element output by the pre-trained surface element rendering network; Calculating the loss between the predicted rendering result of each two-dimensional surface element and the actual three-dimensional model; The loss is used to adjust the network parameters of the pre-trained face rendering network to obtain a trained face rendering network.
5. The method according to claim 4, characterized in that The method also includes: Calculating the confidence of each two-dimensional surface element based on the transparency and / or depth information of each two-dimensional surface element; The two-dimensional surface elements corresponding to the multiple sample images and having confidence levels lower than a preset confidence threshold are deleted.
6. The method according to claim 1, characterized in that The multiple input images are acquired by real cameras; The generating a free viewpoint video based on the multiple input images and the projection screen corresponding to each virtual camera includes: Creating an interpolation viewpoint between the virtual camera and / or the real camera, and using the virtual camera and / or the real camera adjacent to the interpolation viewpoint as a reference camera corresponding to the interpolation viewpoint; Performing interpolation calculation on the projection picture of the reference camera and / or the input image to obtain an interpolation picture corresponding to the interpolation viewpoint; The free viewpoint video is generated by combining the multiple input images, the projection picture corresponding to each virtual camera and the interpolation picture corresponding to each interpolation viewpoint.
7. The method according to claim 1, characterized in that The generating a free viewpoint video based on the multiple input images and the projection screen corresponding to each virtual camera includes: Detecting a moving foreground contained in the input image; In response to the free viewpoint video being occluded by the moving foreground, the occluded area in the free viewpoint video is completed based on image content corresponding to the occluded area in the target three-dimensional reconstruction result.
8. An electronic device, characterized in that: The electronic device includes a memory and a processor, and the processor is used to execute program instructions stored in the memory to implement the steps in the free viewpoint video generation method as described in any one of claims 1-7.
Citation Information
Patent Citations
Three-dimensional point cloud reconstruction method based on multi-view weighted aggregation
CN116168162A
Three-dimensional reconstruction method, device and equipment based on neural radiation field and storage medium
CN116310076A
