Image rendering method and apparatus, electronic device, storage medium and program product

By determining the mask, reflected image and camera parameters of the image to be rendered, and directly constructing and rendering the three-dimensional model, the problems of ambient lighting changes and camera position sensitivity in the prior art are solved, and the efficiency and quality of image rendering are improved.

WO2024193609A9PCT designated stage expired Publication Date: 2024-11-07TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2024/082808
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-03-20
Filing Date
2024-03-20
Publication Date
2024-11-07

AI Technical Summary

Technical Problem

The existing three-dimensional reconstruction technology has poor quality when ambient light changes and is sensitive to camera position accuracy, resulting in blurred texture or noise, and it takes a long time to train network parameters.

Method used

By determining the mask, reflected image and camera parameters of the image to be rendered, building a model and mapping and lighting restore, the target image is directly generated, avoiding training network parameters and improving rendering efficiency and quality.

Benefits of technology

Maintain image quality when ambient lighting changes, improve rendering efficiency, adapt to sensitive lighting changes, and reduce rendering time.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024082808_07112024_PF_FP_ABST
    Figure CN2024082808_07112024_PF_FP_ABST
Patent Text Reader

Abstract

Provided by the present application are an image rendering method and apparatus, an electronic device, a computer-readable storage medium and a computer program product. The method comprises: determining a mask, a reflection image and a camera parameter corresponding to an image to be rendered, the image to be rendered comprising a target object; on the basis of the mask and the camera parameter, constructing a model corresponding to the target object; performing texture mapping on the model on the basis of the reflection image, and obtaining a map of the model; and performing illumination restoration on the model on the basis of the map, and obtaining a target image of the target object.
Need to check novelty before this filing date? Find Prior Art

Description

Image rendering method, device, electronic device, storage medium and program product

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] The embodiments of this application are based on and claim the priority of Chinese patent application with application number 202310279295.X and application date March 20, 2023. The entire contents of the Chinese patent application are hereby introduced into the embodiments of this application as a reference. Technical Field

[0003] The present application relates to image technology, and in particular to an image rendering method, device, electronic device, storage medium and program product. Background Art

[0004] 3D reconstruction technologies include active and passive methods. Active methods are represented by depth cameras, laser scanners, and light field camera array reconstruction technologies, while passive methods are represented by traditional image-based photogrammetry and inverse neural rendering. Reconstruction using neural inverse rendering often requires optimizing a set of neural network parameters through supervised training to perform 3D reconstruction based on these parameters. However, training these network parameters is time-consuming, sensitive to changes in ambient lighting, and the resulting 3D model quality is poor. Furthermore, these methods are sensitive to camera pose accuracy, potentially resulting in blurred textures or noise.

[0005] Summary of the Invention

[0006] Embodiments of the present application provide an image rendering method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve image rendering quality.

[0007] The technical solution of the embodiment of the present application is implemented as follows:

[0008] The present invention provides an image rendering method, including:

[0009] Determining a mask, a reflected image, and camera parameters corresponding to an image to be rendered, wherein the image to be rendered includes a target object;

[0010] constructing a model corresponding to the target object based on the mask and the camera parameters;

[0011] Mapping the model based on the reflection image to obtain a map of the model;

[0012] Based on the map, the model is illuminated and restored to obtain a target image of the target object.

[0013] An embodiment of the present application provides an image rendering device, comprising:

[0014] A first processing module is configured to determine a mask, a reflected image, and camera parameters corresponding to an image to be rendered, wherein the image to be rendered includes a target object;

[0015] A second processing module is configured to construct a model corresponding to the target object based on the mask and the camera parameters;

[0016] a third processing module, configured to map the model based on the reflection image to obtain a texture of the model;

[0017] The fourth processing module is configured to perform lighting restoration on the model based on the map to obtain a target image of the target object.

[0018] An embodiment of the present application provides an electronic device, including:

[0019] a memory configured to store computer-executable instructions or computer programs;

[0020] The processor is configured to implement the image rendering method provided in the embodiment of the present application when executing the computer executable instructions or computer program stored in the memory.

[0021] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions or a computer program for implementing the image rendering method provided in the embodiment of the present application when executed by a device.

[0022] An embodiment of the present application provides a computer program product, including a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a device, the image rendering method provided by the embodiment of the present application is implemented.

[0023] The embodiments of the present application have the following beneficial effects:

[0024] By applying the embodiments of the present application, after rendering the image to be rendered, the mask, reflected image and camera parameters corresponding to the image to be processed are determined; based on the mask and camera parameters, a model of the target object in the image to be rendered is constructed to obtain a model of the target object; the model is mapped based on the reflected image to obtain a texture; based on the texture, the model is illuminated and restored to obtain a target image of the target object; in this way, the image rendering efficiency can be improved without training network parameters; in addition, during rendering, mapping is performed based on the reflected image to obtain a texture, and illumination restoration is performed based on the texture to obtain the target image. Since the reflected image reflects the image portion that can remain unchanged under changing lighting conditions in the image to be rendered, the texture obtained based on the reflected image also retains the characteristic that the target object in the image to be rendered remains unchanged under changing lighting conditions. Therefore, the target image obtained by illumination restoration of the texture can adapt to more sensitive environmental lighting changes, thereby improving the rendering quality of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] FIG1 is a schematic diagram of the architecture of an image rendering system 100 provided in an embodiment of the present application;

[0026] FIG2 is a schematic structural diagram of an electronic device 500 for image rendering provided in an embodiment of the present application;

[0027] FIG3A is a schematic diagram of a flow chart of an image rendering method provided in an embodiment of the present application;

[0028] FIG3B is a schematic diagram of a flow chart of an image rendering method provided in an embodiment of the present application;

[0029] FIG4A is a schematic flow chart of a method for determining a contour model provided in an embodiment of the present application;

[0030] FIG4B is a schematic flow chart of a method for determining a contour model provided in an embodiment of the present application;

[0031] FIG4C is a schematic diagram of data flow for constructing a contour model according to an embodiment of the present application;

[0032] FIG5A is a schematic diagram of data flow for texture mapping provided in an embodiment of the present application;

[0033] FIG5B is a schematic diagram of a texture mapping method flow diagram provided in an embodiment of the present application;

[0034] FIG5C is a schematic diagram of a combination of triangular facets provided in an embodiment of the present application;

[0035] FIG6 is a schematic flow chart of a method for determining a target image according to an embodiment of the present application;

[0036] FIG7 is a flow chart of an image rendering method according to an embodiment of the present application;

[0037] FIG8 is a schematic diagram of image segmentation provided in an embodiment of the present application;

[0038] FIG9 is a schematic diagram of a white mold construction provided in an embodiment of the present application;

[0039] FIG10 is a schematic flow chart of a method for constructing a white mold according to an embodiment of the present application;

[0040] FIG11 is a schematic diagram of image intrinsic decomposition provided by an embodiment of the present application;

[0041] FIG12 is a schematic diagram of model material and ambient lighting fitting provided in an embodiment of the present application;

[0042] FIG13 is a schematic diagram of a model material and ambient lighting fitting process according to an embodiment of the present application;

[0043] FIG14 is a schematic diagram of an image rendering result provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0045] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0046] In the following description, the terms "first\second..." are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first\second..." can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0048] 3D reconstruction techniques include active and passive methods. Active methods are represented by depth cameras, laser scanners, and light field camera array reconstruction techniques, while passive methods are represented by traditional image-based photogrammetry and inverse neural rendering. Traditional photogrammetry-based 3D reconstruction techniques generally take a set of 2D color images as input and output a display model and texture material of the target object. The main process includes: first, calculating the camera parameters corresponding to each image in the set of images; then, determining the corresponding 3D spatial position (e.g., a 3D point cloud) for each pixel in each frame; then, based on various constraints, fusing the 3D point clouds corresponding to each frame and removing noise points; finally, extracting the geometric model from the 3D point cloud using a spatial geometric meshing algorithm; then, using texture mapping technology, generating a diffuse texture map for the geometric model. This step is collectively referred to as geometric and texture reconstruction. However, this method is sensitive to changes in ambient lighting, easily affected by interference, and has poor reconstruction capabilities for weak texture areas. The generated 3D model is prone to missing parts, and the reconstructed texture maps mostly only contain diffuse materials, which is not conducive to achieving physically based rendering effects.

[0049] The main process of neural inverse rendering technology includes: first calculating the camera parameters corresponding to each image in the group of images, and then optimizing a set of neural network parameters through supervised training, and then representing the geometric shape and texture material of the three-dimensional model with neural network parameters; when the training is completed, the internal and external parameters of the camera under any new perspective can be input to render the corresponding model image. Although this method assumes that a point in three-dimensional space will be inconsistent in different observations due to changes in lighting, perspective, and occlusion relationships due to the inverse neural rendering technology, it is closer to the objective laws of the real world than the traditional multi-perspective geometry that assumes consistency in observations of a point in the same three-dimensional space under multiple frame perspectives, which improves the integrity and robustness of three-dimensional modeling in certain scenarios (such as weak texture scenes). However, it takes a long time to train the network parameters, is sensitive to changes in ambient lighting, and the corresponding three-dimensional model quality is poor. It is also sensitive to the accuracy of camera pose, which may cause blurred textures and noise.

[0050] To this end, the embodiments of the present application provide an image rendering method, apparatus, device, computer-readable storage medium, and computer program product to at least solve the above-mentioned problems. The following describes exemplary applications of the electronic devices provided by the embodiments of the present application. The electronic devices provided by the embodiments of the present application can be implemented as various types of user terminals such as laptops, tablet computers, desktop computers, set-top boxes, mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), smart phones, smart speakers, smart watches, smart TVs, and vehicle-mounted terminals, and can also be implemented as servers. Below, an exemplary application when the device is implemented as a server will be described.

[0051] Refer to Figure 1, which is a schematic diagram of the architecture of the image rendering system 100 provided in an embodiment of the present application. In order to support an exemplary application, the terminal (terminal 400-1 and terminal 400-2 are shown as an example) is connected to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0052] In some embodiments, the image rendering method provided in the embodiments of the present application can be implemented independently by the terminal or the server 200. When implemented independently by the terminal, it can be installed on the terminal in the form of a client, so that the client on the terminal has a local image rendering function. It can also be used as a plug-in for the relevant client and downloaded to the client for local use as needed. In the above deployment method, the image rendering system can complete all detection processes directly locally without the need to access an external network, thereby ensuring the absolute security of the data.

[0053] In some embodiments, the image rendering method provided by the embodiments of the present application can be implemented collaboratively by the terminal and the server 200. For example, the terminal sends a rendering request for an image to be rendered that includes a target object to the server 200; the server 200 determines the mask, reflection image, and camera parameters corresponding to the image to be rendered based on the rendering request; constructs a model corresponding to the target object based on the mask and camera parameters; maps the model based on the reflection image to obtain a texture; based on the texture, performs lighting restoration on the model to obtain a target image of the target object; and returns the obtained target image to the terminal.

[0054] In some embodiments, the embodiments of the present application can also be implemented with the help of cloud technology. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and network within a wide area network or a local area network to realize the calculation, storage, processing, and sharing of data. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form a resource pool that can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources.

[0055] In some embodiments, the server 200 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDNs), and big data and artificial intelligence platforms. The terminal 400 can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal 400 and the server 200 can be directly or indirectly connected by wired or wireless communication, which is not limited in the embodiments of the present application.

[0056] The structure of an electronic device for image rendering provided by an embodiment of the present application is described below. Referring to FIG2 , FIG2 is a schematic diagram of the structure of an electronic device 500 for image rendering provided by an embodiment of the present application. Taking the electronic device 500 as a server as an example, the electronic device 500 for image rendering shown in FIG2 includes: at least one processor 510, a memory 550, at least one network interface 520 and a user interface 530. The various components in the electronic device 500 are coupled together via a bus system 540. It is understandable that the bus system 540 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, various buses are labeled as bus system 540 in FIG2 .

[0057] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0058] The memory 550 includes a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory. The memory 550 may optionally include one or more storage devices physically remote from the processor 510.

[0059] In some embodiments, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0060] The operating system 551 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic businesses and process hardware-based tasks; the network communication module 552 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 520. Exemplary network interfaces 520 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.

[0061] In some embodiments, the image rendering device provided by the embodiment of the present application can be implemented in software. The image rendering device provided by the embodiment of the present application can be provided as various software embodiments, including various forms including application programs, software, software modules, scripts or codes. FIG2 shows an image rendering device 555 stored in a memory 550, which can be software in the form of programs and plug-ins, and includes a series of modules, including a first processing module 5551, a second processing module 5552, a third processing module 5553 and a fourth processing module 5554. These modules are logical, and therefore can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0062] In other embodiments, the image rendering device provided in the embodiments of the present application can be implemented in hardware. As an example, the image rendering device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the image rendering method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs) or other electronic components.

[0063] Based on the above description of the image rendering system and electronic device provided in the embodiment of the present application, the image rendering method provided in the embodiment of the present application is described below. See Figure 3A, which is a flow chart of the image rendering method provided in the embodiment of the present application. The image rendering method provided in the embodiment of the present application is implemented by the server 200 in Figure 1 as an example.

[0064] In step 101A, the server determines the mask, the reflected image, and the camera parameters of the image to be rendered.

[0065] In some embodiments, referring to FIG. 3B , FIG. 3B is a flow chart of an image rendering method provided in an embodiment of the present application. Step 101A shown in FIG. 3A can be implemented by step 101B shown in FIG. 3B :

[0066] In step 101B, the server determines a foreground mask, a diffuse reflection image, and camera parameters of the image to be rendered.

[0067] The foreground mask is the image region where the target object is located in the image to be rendered. The image rendering method provided in the embodiments of the present application is applicable to rendering scenarios such as video streams or images. For the rendering scenario of a video stream, the image to be rendered can be considered as the image corresponding to a video frame in the video stream. The target object in the image to be rendered can be a variety of physical objects such as a car, a person, a tree, a cup, etc., and the embodiments of the present application do not limit the type of the target object.

[0068] In some embodiments, the server may determine the image to be rendered in the following manner: capturing a video stream, where the video stream includes multiple video frames; extracting key frames from the video stream, and using the key frames as the image to be rendered.

[0069] Among them, the key frame includes at least one of the following video frames: the number of times the target feature point in the point cloud corresponding to the video frame is selected is higher than the number threshold, and the target feature point is a feature point that is repeatedly selected and set as an inner group in the point cloud corresponding to the video frame; the relative motion distance between the video frame and the adjacent previous video frame is lower than the distance threshold; the number of feature points matching between the video frame and the adjacent previous video frame is higher than the number threshold.

[0070] In practical applications, for the rendering scenario of video streams, each video stream includes multiple video frames, but there is usually a lot of redundancy in video frames. In order to improve the rendering processing efficiency and the accuracy of the rendering results, it is necessary to extract key frames from the video stream and use the images corresponding to the extracted key frames as the images to be rendered.

[0071] In some embodiments, when performing key frame extraction, a semantic-based video key frame extraction algorithm can be used. For example, a hierarchical clustering algorithm is first used to perform preliminary extraction of video key frames, and then a semantic correlation algorithm is used to perform histogram comparison on the preliminary extracted key frames to remove redundant frames and determine the key frames of the video; a convolutional autoencoder can also be used to extract deep features of video frames, and K-means clustering is performed on them. In each type of video frames, video frames with a clarity exceeding the target clarity (which can be set according to actual needs) are screened out as the initially extracted key frames, and then the point density method is used to perform secondary optimization on the initially extracted key frames, and the key frames obtained from the secondary optimization are used as the key frames of the video stream.

[0072] In other embodiments, a real-time positioning and mapping algorithm (SLAM, Simultaneous localization and mapping) can also be used to extract key frames. For example, a random sampling algorithm is used for the video frame, and several feature points are randomly selected in the point cloud of the current video frame and set as the inner group, and a model suitable for the inner group is calculated. It is determined whether to eliminate the current video frame based on the number of selections of the repeatedly selected feature points (i.e., the above-mentioned target feature points) in the point cloud corresponding to the current video frame, or it is determined whether to eliminate the current video frame based on the relative motion distance between the current video frame and the adjacent previous video frame, or it is determined whether to eliminate the current video frame based on whether the number of feature points matching between the current video frame and the adjacent previous video frame is higher than a number threshold. Of course, it is also possible to combine two or more of the above conditions to determine whether to eliminate the current video frame. For example, if the number of selections corresponding to the target feature points in the point cloud corresponding to the current video frame is higher than the number threshold, the current video frame is not eliminated (retained) and the current video frame is used as the key frame of the video stream. For example, if the number of selections corresponding to the target feature points in the point cloud corresponding to the current video frame is lower than the number threshold, the current video frame is not eliminated (retained) and the current video frame is used as the key frame of the video stream. For example, if the number of selections corresponding to the target feature points in the point cloud corresponding to the current video frame is higher than the number threshold, it is further determined whether the relative motion distance between the video frame and the adjacent previous video frame is lower than the distance threshold. When the relative motion distance between the video frame and the adjacent previous video frame is not lower than the distance threshold, the current video frame is eliminated, that is, the current video frame is not the key frame of the video stream. When the relative motion distance between the video frame and the adjacent previous video frame is lower than the distance threshold, it is further determined whether the number of feature points matching between the video frame and the adjacent previous video frame is higher than the number threshold. When the number of feature points matching between the video frame and the adjacent previous video frame is higher than the number threshold, the current video frame is not eliminated (retained), that is, the current video frame is determined as the key frame of the video stream. This further improves the screening accuracy of the key frames, reduces redundancy as much as possible, and is conducive to improving the subsequent image rendering efficiency.

[0073] In some embodiments, the server can determine the foreground mask and diffuse reflection image corresponding to the image to be rendered in the following manner: perform background segmentation on the image to be rendered to obtain the foreground mask of the image to be rendered, and perform image intrinsic decomposition on the image to be rendered to obtain the diffuse reflection image of the image to be rendered.

[0074] In practical applications, a neural network segmentation algorithm (or neural network model) for background removal can be used to perform background segmentation on each image to be rendered, removing the background portion and obtaining a foreground mask for the image to be rendered. That is, the image to be rendered can be considered a combination of the foreground mask (foreground portion) and the background portion. The foreground mask is the image obtained by removing the background portion from the image to be rendered. A neural network model for image intrinsic decomposition is used to perform image intrinsic decomposition on the image to be rendered, obtaining a diffuse image and a shadow image. Image intrinsic decomposition decomposes an image into two components: a reflectance map and an illumination map (an image reflecting the lighting conditions of the image to be rendered). The diffuse image here is the reflectance map obtained by performing image intrinsic decomposition on the image to be rendered. Since the reflectance map is the portion of the image that remains unchanged under varying lighting conditions, the diffuse image preserves the properties of the target object in the image to be rendered under varying lighting conditions. This provides initial diffuse texture values ​​for subsequent physically based rendering (PBR) texture material fitting, avoiding the noise that can cause difficult-to-converge when fitting PBR texture materials from randomly initialized parameters.

[0075] [Added by reference (Rule 20.6) 29.04.2024] Camera parameters refer to the relevant parameters of the camera when acquiring the image to be rendered. When determining the camera parameters of the camera when acquiring the image to be rendered, the camera parameters can be solved by sparsely reconstructing the image to be rendered based on the Structure from Motion (SFM) algorithm, such as the camera intrinsic parameter K∈R 3x3 and camera extrinsics [R|t]∈R 3x4 , where K∈R 3x3 It is the intrinsic parameter matrix of the camera, which is the parameter describing the internal properties of the camera, including focal length, principal point (optical center) coordinates, distortion coefficient, etc. The extrinsic parameters of the camera are the parameters describing the position and posture of the camera in the world coordinate system (i.e., posture parameters), which are independent of the parameters of the internal properties of the camera. The posture of the camera is described by the rotation matrix R and the translation vector t. The extrinsic parameters of the camera may change at different camera positions or shooting times.

[0076] In step 102A, a model corresponding to the target object is constructed based on the mask and camera parameters.

[0077] Here, after obtaining the mask of the image to be rendered and the camera parameters of the camera when capturing the image to be rendered, the server can construct a model based on the mask and camera parameters of the image to be rendered to obtain a model corresponding to the target object, which is used to perform three-dimensional construction of the target object.

[0078] In some embodiments, still referring to FIG. 3B , step 102A shown in FIG. 3A may be implemented by step 102B shown in FIG. 3B :

[0079] In step 102B, a contour model corresponding to the target object is constructed based on the foreground mask and camera parameters.

[0080] Here, the foreground mask is the image area where the target object is located, which is extracted from the image to be rendered (two-dimensional image). According to the foreground mask and camera parameters, a contour model of the target object can be constructed to obtain a three-dimensional contour model of the target object (i.e., a three-dimensional white model), that is, a contour model in a three-dimensional scene. This contour model is used to construct the target object in three dimensions.

[0081] In practical applications, visual hull carving technology and inverse neural rendering technology can be used to generate a high-quality contour model of the target object. The processing flow is as follows: the contour of the target object in the foreground mask of the rendered image is extracted by the visual hull carving technology to obtain the initial contour model of the target object (i.e., the initial white model), and then the initial contour model is subjected to surface subdivision processing to obtain an optimized contour model (i.e., the optimized white model). Then, based on the camera parameters when the image to be rendered is acquired, the optimized contour model is subjected to image rendering processing by the neural radiance field (NeRF) to obtain a rendered image, and the neural radiation field is updated based on the residual between the rendered image and the image to be rendered (i.e., the original input image), and the next round of iteration is performed based on the updated neural radiation field, and so on, until a contour model that meets actual needs (the final required white model) is obtained.

[0082] In some embodiments, referring to FIG. 4A , FIG. 4A is a flow chart of a method for determining a contour model provided in an embodiment of the present application. Step 102B in FIG. 3B may be implemented by steps 1021 to 1023 shown in FIG. 4A :

[0083] In step 1021 , a three-dimensional space volume is constructed according to the center of the image to be rendered, the foreground mask, and the camera parameters.

[0084] [Added by reference (Rule 20.6) 29.04.2024] Here, the visual shell carving technology solves the least squares equation system: K(RC+t)=p based on the principle of triangulation according to the center p of the image to be rendered, the foreground mask, the internal and external parameters K, R, and t, and estimates the intersection point C of the rays emitted from the center of the image to be rendered in the world coordinate system. A voxel with a resolution of the target resolution (such as 32*32*32 voxel) is constructed with C as the midpoint (that is, the three-dimensional space includes multiple voxel points), and the length of its unit voxel is determined by the average of the lengths from the center of each camera to the intersection point.

[0085] In step 1022 , each voxel point of the three-dimensional space is projected onto the image to be rendered, and an initial contour model of the target object is extracted from each voxel point according to the projection result.

[0086] Here, each voxel point in the three-dimensional space is projected into the image to be rendered. Whether the corresponding voxel point is occupied is determined based on whether it falls on the foreground mask of the image to be rendered. If the voxel point falls on the foreground mask of the image to be rendered, it can be determined that the voxel point falling on the foreground mask of the image to be rendered is occupied. Finally, the Marching Cube algorithm is used to extract the initial contour model (i.e., the initial white model) from the occupied voxel points.

[0087] For ease of distinction, the contour model extracted from the occupied voxel points (i.e., white model) is called the initial contour model of the target object in the three-dimensional scene, and the contour model subsequently constructed based on the initial contour model is called the final contour model of the target object in the three-dimensional scene (the final contour model, i.e., the final required white model).

[0088] In some embodiments, referring to FIG4B , FIG4B is a flow chart of a method for determining a contour model provided in an embodiment of the present application. Step 1022 in FIG4A can be implemented through steps 10221 to 10223 shown in FIG4B : in step 10221, voxel points falling on the foreground mask are filtered out from each voxel point according to the projection result, and a plurality of voxel elements composed of the filtered voxel points are determined; in step 10222, a plurality of voxels are traversed, and triangular facets are extracted from each voxel; in step 10223, the extracted triangular facets are combined to obtain an initial contour model of the target object.

[0089] Among them, a triangular face is a two-dimensional geometric figure composed of three vertices and three edges. Each edge connects two vertices, eventually forming a closed triangle. Triangular faces are usually used to describe the surface of three-dimensional objects. Complex shapes can be constructed by combining a large number of adjacent triangular faces, such as the initial contour model mentioned above.

[0090] A 3D volume contains multiple voxels, and a voxel is a cube grid consisting of eight adjacent voxels. After projecting each voxel of the 3D volume onto the image to be rendered, the voxels that fall on the foreground mask of the rendered image (i.e., occupied voxels) are filtered out from the voxels, and the number of voxels that can be formed by these filtered voxels is determined.

[0091] Here, we use voxels as units to find the boundary between the content and background of a three-dimensional volume. Triangles are extracted from the voxels to fit this boundary. For convenience, the voxel points containing the volume data are called real points, and the background voxel points other than real points are called virtual points. Such a three-dimensional volume is a dot matrix composed of various real and virtual points. In practical applications, triangles are extracted from boundary voxels (which are voxels that have both real and virtual points. This is because boundary voxels inevitably contain "isosurfaces," which are curved surfaces in space that can represent the boundary between the real and virtual parts of an image). Therefore, when extracting triangles, we traverse all the voxels, filter out the boundary voxels from all the voxels, extract triangles from the boundary voxels, and combine the extracted triangles to form the final three-dimensional mesh surface data, thus obtaining the initial contour model of the target object.

[0092] In step 1023, a contour model is constructed based on the initial contour model and the camera parameters to obtain the final contour model required for the target object.

[0093] In some embodiments, the server can implement contour model construction based on the initial contour model and camera parameters to obtain the contour model of the target object in the following manner: perform the following multiple rounds of iterative processes based on the initial contour model and camera parameters: perform surface subdivision on the contour model to be subdivided to obtain an optimized contour model (wherein, the optimized contour model is the contour model finally obtained by performing surface subdivision on the contour model to be subdivided), where the contour model to be subdivided in the first round of iteration is the initial contour model of the target object; based on the camera parameters, perform image rendering on the optimized contour model through a neural radiation field network to obtain a rendered image of the optimized contour model; obtain the absolute value of the error between the rendered image and the image to be rendered, and enter the next round of iteration when the absolute value of the error is greater than the error threshold, and stop the iteration until the absolute value of the error does not exceed the error threshold, so as to determine the contour model of the target object based on the optimized contour model corresponding to the time of stopping the iteration.

[0094] [Added by reference (rule 20.6) 29.04.2024] In practical applications, after extracting the initial contour model of the target object, a differentiable white model can be constructed based on the inverse neural rendering technology. This process is a multi-stage iterative optimization process. For example, the first round of iterative process is as follows: the initial contour model is subjected to surface subdivision processing to obtain the first optimized contour model of the first round of iteration, and then the first optimized contour model obtained in the first round of iteration is subjected to image rendering processing by a neural radiation field (NeRF, Neural Radiance Field) (such as a fully connected neural network) to obtain a first rendered image of the first optimized contour model, and the absolute value of the error between the first rendered image and the image to be rendered is obtained (which can be the absolute value of the error in at least one dimension of multiple dimensions such as color, brightness, texture, etc.). When the absolute value of the error is greater than the error threshold, the neural radiation field parameters are updated. Since the internal and external parameters of the key frame are fixed, only the geometric shape of the model needs to be considered, so the vertex coordinates V of the optimized model are updated. opti The neural radiation field parameters that implicitly store vertex colors will be optimized and updated to avoid explicitly modeling light, shadow, and color. After the neural radiation field parameters are updated, the next round of iteration is performed based on the updated neural radiation field parameters.

[0095] In the second round of iteration, the first optimized contour model obtained in the first round of iteration is subjected to surface subdivision processing to obtain the second optimized contour model in the second round of iteration, and then the second optimized contour model in the second round of iteration is subjected to image rendering processing through the neural radiation field to obtain a second rendered image of the first optimized contour model, and the absolute value of the error between the second rendered image and the image to be rendered is obtained. When the absolute value of the error is greater than the error threshold, the neural radiation field parameters are updated in the same manner as above. After the neural radiation field parameters are updated, the next round of iteration is performed based on the updated neural radiation field parameters, and so on, until the absolute value of the error between the rendered image obtained by neural radiation field rendering and the image to be rendered is lower than the error threshold, the iteration is stopped, and the optimized contour model corresponding to the time of stopping the iteration is used as the contour model finally required by the target object. For example, assuming that a total of N (a positive integer greater than 2) rounds of iteration are performed, the Nth optimized contour model obtained in the Nth round of iteration is used as the contour model finally required by the target object.

[0096] For example, taking the contour model as a white model, see Figure 4C, which is a data flow diagram for the construction of the white model provided in an embodiment of the present application. First, a three-dimensional space body is constructed according to the image to be rendered, the foreground mask and the camera parameters. Then, each voxel point of the three-dimensional space body is projected onto the image to be rendered, and the initial white model of the target object in the three-dimensional scene is extracted from each voxel point according to the projection result; then, multiple rounds of iterations are performed based on the initial white model and the camera parameters, and the optimized white model corresponding to the last round of iteration is used as the final white model of the target object in the three-dimensional scene. For example, in the first round of iteration, the initial white model (i.e., the white model to be subdivided) is subjected to surface subdivision processing to obtain an optimized white model 1 (i.e., the first optimized white model mentioned above). When the optimized white model 1 is subjected to image rendering processing through NeRF, When the absolute value of the error between the rendered image 1 obtained by NeRF processing (i.e., the first rendered image mentioned above) and the image to be rendered is greater than the error threshold (which can be set according to actual needs), the second round of iteration is entered; in the second round of iteration, the white model to be subdivided is the optimized white model 1 obtained in the first round of iteration, that is, the optimized white model 1 is subjected to surface subdivision processing to obtain the optimized white model 2 (i.e., the second optimized white model mentioned above). If the absolute value of the error between the rendered image 2 (i.e., the second rendered image mentioned above) obtained by NeRF rendering of the optimized white model 2 and the image to be rendered is less than or equal to the error threshold, the third round of iteration is no longer entered. At this time, the optimized white model 2 (i.e., the second optimized white model mentioned above) obtained in the second round of iteration is used as the final white model of the target object in the three-dimensional scene for subsequent processing.

[0097] It should be noted that the initial white model, first optimized white model, second optimized white model, ..., and Nth optimized white model provided in the embodiments of the present application do not refer to any specific white model, but are used to distinguish between the white model that existed before the tessellation process and the updated white model obtained after the tessellation process during the tessellation process. As an example, in the first round of iteration, the white model to be tessellated is tessellated. The white model to be tessellated before the tessellation process can be referred to as the initial white model, and the optimized white model obtained after the tessellation process can be referred to as the first optimized white model. In the second round of iteration, the optimized white model obtained after the tessellation process of the first optimized white model obtained in the first round of iteration is referred to as the second optimized white model.

[0098] Similarly, the first rendered image and the second rendered image provided in the embodiment of the present application do not specifically refer to a certain image, but are used to distinguish between the image to be rendered that already exists before the image rendering process and the rendered image updated after the image rendering process during the image rendering process.

[0099] In some embodiments, the server can perform surface subdivision processing on the subdivided contour model to obtain an optimized contour model in the following manner: performing triangular mesh subdivision on the subdivided contour model to obtain an optimized contour model; or performing quadrilateral mesh subdivision on the subdivided contour model to obtain an optimized contour model; wherein the color error of each triangular facet in the optimized contour model is less than a first error value, or the sum of the color errors of each triangular facet in the optimized contour model is less than a second error value; the color error is the difference between the color value of the vertex of the triangular facet projected onto the image to be rendered and the color value at the corresponding projection position in the image to be rendered.

[0100] Here, the contour model to be subdivided can be subdivided by a surface subdivision algorithm to obtain an optimized contour model, wherein the surface subdivision algorithm includes Catmull-Clark subdivision and Loop subdivision, wherein Catmull-Clark subdivision is a quadrilateral mesh subdivision rule, and the contour model to be subdivided can be regarded as a three-dimensional model composed of multiple triangular facets, and each triangular facet is composed of one or more quadrilateral meshes. During subdivision, a new vertex is calculated for each face of the quadrilateral mesh, and a new vertex is calculated for each edge of the quadrilateral mesh. At the same time, the position of each original vertex of the quadrilateral mesh is updated, and the quadrilateral mesh is divided according to the newly generated vertices and the updated vertices. And so on. Multiple divisions can achieve surface subdivision of the contour model to be subdivided.

[0101] Loop subdivision is a triangular mesh subdivision rule. The contour model to be subdivided can be regarded as a three-dimensional model composed of multiple triangular facets. Each triangular facet is composed of one or more triangular meshes. It is split according to 1-4 triangles. A new vertex is generated for each edge calculation. At the same time, the position of each original vertex is updated. The triangular mesh is divided according to the newly generated and updated vertices. And so on. By performing multiple divisions, the surface subdivision of the contour model to be subdivided can be achieved.

[0102] In practical applications, in order to avoid excessive subdivision causing the contour model to occupy too much space, before each subdivision, it is necessary to obtain the color value of each triangle vertex of the optimized contour model obtained at the end of the previous round of optimization projected onto the image to be rendered, and the color error between the color value at the corresponding projection position in the image to be rendered. For triangles whose color error is less than a certain error threshold (which can be set according to actual needs), no further subdivision will be performed; alternatively, the color errors of each triangle are summed, and when the sum of the color errors is less than the error threshold, the contour model to be subdivided will no longer be subdivided.

[0103] Among them, the corresponding projection position in the image to be rendered refers to the projection position of the vertex of the triangle patch to the image to be rendered. For example, the vertex of triangle patch 1 is projected to position A in the image to be rendered, then position A is the projection position corresponding to triangle patch 1.

[0104] As an example, assume that the contour model to be subdivided can be regarded as a three-dimensional model composed of N (a positive integer greater than 2) triangular facets, and the vertex of triangle facet 1 is projected to position A in the image to be rendered, then the color value of the vertex of triangle facet 1 projected to position A in the image to be rendered is obtained, and the color error 1 between the color value of position A in the image to be rendered and the color value is obtained. For the convenience of description, color error 1 can be called the color error corresponding to triangle facet 1. In this way, the color errors corresponding to all triangle facets are obtained, that is, the color error 1 corresponding to triangle facet 1, the color error 2 corresponding to triangle facet 2,..., the color error N corresponding to triangle facet N are obtained, and the relationship between the color error corresponding to each triangle facet and the first error value is judged. The triangle facet whose color error is less than the first error value is no longer subdivided. For example, if the color error 1 corresponding to triangle facet 1 is greater than or equal to the first error value, triangle facet 1 continues to be subdivided. If the color error 2 corresponding to triangle facet 2 is less than the first error value, triangle facet 2 is no longer subdivided. Alternatively, the color errors corresponding to the vertices of all triangular facets are added up (such as color error 1 corresponding to triangle facet 1, color error 2 corresponding to triangle facet 2, ..., color error N corresponding to triangle facet N) to obtain the sum of the color errors, and the size relationship between the sum of the color errors and the second error value is compared. When the sum of the color errors is less than the error threshold, the contour model is no longer subdivided.

[0105] In some embodiments, the server can determine the contour model of the target object based on the optimized contour model corresponding to when the iteration is stopped in the following manner: perform surface construction on the optimized contour model corresponding to when the iteration is stopped to obtain the constructed optimized contour model, and use the constructed optimized contour model as the contour model finally required by the target object.

[0106] In practical applications, the optimized contour model outputted above can be subjected to surface reconstruction processing, such as using the open source computational geometry algorithm library (CGAL, Computational Geometry Algorithms Library) to optimize the contour model for surface construction processing to obtain the constructed optimized contour model, and the constructed optimized contour model is used as the final contour model of the target object in the three-dimensional scene. In this way, non-popular faces, vertices, holes, etc. that appear in the construction of the differentiable white mold are removed, thereby improving the accuracy of the final contour model. The final contour model is used for subsequent image rendering, which can improve the accuracy of image rendering.

[0107] In step 103A, the model is mapped based on the reflection image to obtain a map of the model.

[0108] Here, after obtaining the model of the target object, mapping processing may be performed on the model of the target object based on the reflected image of the image to be rendered to obtain a texture corresponding to the model of the target object.

[0109] In some embodiments, still referring to FIG. 3B , step 103A shown in FIG. 3A may be implemented by step 103B shown in FIG. 3B : in step 103B, texture mapping is performed on the contour model based on the diffuse reflection image to obtain a texture map of the contour model.

[0110] In practical applications, a contour model contains tens of thousands or even millions of vertices and facets (such as triangles). That is, the contour model includes multiple triangles, each vertex is determined by a set of three-dimensional coordinates in the world coordinate system, and each triangle consists of three vertex index numbers. Multi-View Stereo-Texturing (MVS-Texturing) technology can be used to map the triangles corresponding to the contour model to the diffuse image of the image to be rendered through steps such as view selection, color adjustment, and UV parameterization (UV atlas). Ultimately, a diffuse texture map is formed. UV parameterization involves unfolding adjacent triangles from the same image to be rendered onto a two-dimensional image. The pixel values ​​of the two-dimensional image are filled with sampled textures. The normalized coordinates of the triangle vertices on the image are called UV coordinates (texture coordinates). All triangles in the contour model are processed in a loop to ultimately obtain a diffuse texture map and the UV coordinates of all triangles in the diffuse texture map.

[0111] See Figure 5A, which is a data flow diagram of texture mapping provided by an embodiment of the present application. The input of texture reconstruction: contour model (i.e., white model), image to be rendered, and camera parameters; output: texture map and texture coordinates. In addition to vertices and triangles, a renderable model must also contain at least the texture coordinates (i.e., UV coordinates) of each vertex and the texture map corresponding to the texture coordinates. Texture mapping is the process of generating a texture map based on the contour model, the image to be rendered, and the camera parameters. The most basic unit connecting input and output is the triangle. The core step is to find and intercept an optimal area (i.e., the color area or color map below) for each triangle from the diffuse reflection image corresponding to the input image to be rendered to reflect its color, and paste the intercepted optimal area into the texture map. When the number of triangles is multiple, the optimal areas of multiple triangles are all pasted into the texture map to obtain a texture map (or diffuse reflection texture map).

[0112] In some embodiments, referring to FIG. 5B , FIG. 5B is a schematic flow chart of a texture mapping method provided in an embodiment of the present application. Step 103B in FIG. 3B may be implemented by steps 1031 to 1033 shown in FIG. 5B :

[0113] In step 1031 , the texture label of each triangular facet is determined according to the diffuse reflection image and the camera parameters, and the triangular facets with the same texture label are combined to obtain a plurality of target facets.

[0114] Referring to FIG5C , FIG5C is a schematic diagram of a combination of triangular facets provided in an embodiment of the present application. In practical applications, a contour model (i.e., a white model) contains tens of thousands or even millions of vertices and facets (taking triangular facets as an example). To minimize the seams generated when different triangular facet maps are combined, some continuous triangular facets can be combined into a patch (i.e., a target facet). Before the combination, a perspective selection is required. Here, each input image to be rendered and its corresponding camera parameters are referred to as a perspective. Referring to the camera parameters, the optimal perspective can be found for each triangular facet to determine the color expression of the triangular facet. Here, a texture label is used. Record the index number of the perspective, combine all triangles using the same texture label to form a target patch, and intercept the area where the target patch is located (i.e., the color area) from the diffuse image corresponding to the image to be rendered, so as to paste the intercepted color area in the texture atlas to obtain a diffuse texture map, record the texture coordinates of all vertices contained in each target patch, and then combine triangles using the same texture label (i.e., some continuous triangles) into a target patch (i.e., the patch composed of triangles with the same texture label is called a target patch), so as to paste the color area of ​​the target patch in the texture atlas to obtain a diffuse texture map.

[0115] Among them, the optimal viewing angle involved above should at least meet the following conditions: 1) At this viewing angle, the triangle facet is visible; 2) At this viewing angle, the user can see the front of the triangle facet; 3) At this viewing angle, the normal direction of the triangle facet is close to the direction of looking directly at the triangle facet.

[0116] In step 1032 , the color region of each target patch is extracted from the diffuse reflection image.

[0117] In practical applications, triangular facets with the same texture label (i.e., some continuous triangular facets) can also be combined into a target facet. For each target facet, the color area of ​​the target facet is extracted from the diffuse reflection image, and the extracted color area of ​​the target facet is pasted on the texture map of the target facet. The vertices of the contour model are then mapped to the pixels of the texture map to obtain the diffuse reflection texture map.

[0118] This approach not only reduces computational overhead but also enhances color continuity, making the pattern in the diffuse texture map consistent with the pattern in the image to be rendered, thus avoiding the impact of seams on the final rendering effect.

[0119] In step 1033 , color correction is performed on the color region of each target patch, and the corrected color region of each target patch is mapped to the texture map of the corresponding target patch to obtain a texture map of the contour model.

[0120] Here, since adjacent triangles may be sampled from different images (such as different images to be rendered), and these different images may cause visual color inconsistency due to changes in lighting and occlusion relationships, before the color area of ​​the target patch is pasted on the texture map of the target patch, the color area of ​​the target patch may be subjected to color correction processing such as global uniformity or local uniformity, and the color area after color correction may be pasted on the texture map of the target patch. During color correction, the color continuity of different sampled images may be constrained by minimizing the energy equation, so that the texture color is continuous and consistent, where the energy equation is defined as the sum of the absolute value differences in the colors of all adjacent triangles corresponding to different sampled images.

[0121] That is, after obtaining each target patch, for each target patch, the area containing the target patch is captured from the image to be rendered, indicated by the corresponding view, to form a color map (i.e., the color area described above). This color map is then corrected to obtain a corrected color map. The image coordinates of each vertex in the target patch within the corrected color map are then calculated using the camera parameters for that view. Next, a new texture map is created, the corrected color map corresponding to the target patch is placed into the newly created texture map, and the coordinates of each vertex in the target patch within the texture map are calculated (i.e., the UV parameterization described above). This transfers the image coordinates of each vertex in the target patch from the color map to the texture map. The coordinates in the texture map then become the texture coordinates of each vertex in the target patch. If there are multiple target patches, the color map corresponding to each target patch can be placed into the texture map in the same manner as described above. If a texture map is already full, a new texture map can be created. At this point, the texture map and texture coordinates corresponding to the outline model of the target object in the 3D scene are obtained.

[0122] Through the above method, compared with direct parameterization on the contour model, the UV coordinates and diffuse texture generated by MVS-Texturing technology can give each texture block corresponding to the image to be rendered a greater weight when subsequently fitting the PBR texture material, which can reduce the probability of splicing and color difference to a certain extent.

[0123] In step 104A, the model is illuminated and restored based on the texture to obtain a target image of the target object.

[0124] Here, after obtaining the texture corresponding to the model of the target object, a lighting restoration process may be performed on the model of the target object based on the texture to obtain a target image of the target object in three dimensions.

[0125] In some embodiments, still referring to FIG. 3B , step 104A shown in FIG. 3A may be implemented by step 104B shown in FIG. 3B :

[0126] In step 104B, the contour model is illuminated and restored based on the texture map to obtain a target image of the target object.

[0127] In some embodiments, referring to FIG. 6 , FIG. 6 is a flow chart of a target image acquisition method provided in an embodiment of the present application. Step 104B in FIG. 3B may be implemented by steps 1041 to 1042 shown in FIG. 6 :

[0128] In step 1041, the diffuse reflection material group and the global ambient lighting are initialized, wherein the diffuse reflection material group includes at least a texture map; in step 1042, the material and ambient lighting of the silhouette model are restored according to the diffuse reflection material group, the global ambient lighting and the camera parameters to obtain an optimized texture map of the silhouette model; in step 1043, the target image of the target object is determined based on the optimized texture map of the silhouette model.

[0129] [Added by reference (Article 20.6) 29.04.2024] Here, illumination restoration refers to the restoration of materials and ambient lighting. The main purpose of model materials and ambient lighting is to fit the PBR texture materials suitable for modern graphics engines and the ambient lighting at the time of shooting based on the previous module products (texture maps). Before starting the fitting, load the 3D outline model (i.e. the outline model of the target object in the 3D scene, i.e. the 3D white model) and the diffuse texture map (i.e. the texture map mentioned above), then initialize the PBR material group T init (Diffuse Texture Map A init , normal map N init , Specular Map S init , roughness map R init ) and Global Ambient Lighting GIinit Then, the model material and ambient lighting are optimized. For example, the contour model is rendered using the ambient lighting of the panoramic image and the microsurface model to obtain a rendered image, and the absolute value of the error between the rendered image and the image to be rendered is obtained. When the absolute value of the error is greater than or equal to the error threshold, the gradient of the ambient lighting of the panoramic image and the microsurface model is calculated, and the gradient is used to update the ambient lighting of the panoramic image and the microsurface model parameters for the next round of iteration. During optimization, the diffuse texture map A can be fixed first. init and update the normal map N with a larger learning rate α1 init , Specular Map S init , roughness map R init With Global Environment Illumination (GI) init , get the appropriate normal map N after a certain number of iterations init , Specular Map S init , roughness map R init With Global Environment Illumination (GI) init After that, fix the optimized normal map N init , Specular Map S init , roughness map R init With Global Environment Illumination (GI) init , and use a learning rate α2 smaller than α1 to update the optimized diffuse texture map A init .

[0130] [Added by reference (Article 20.6) 29.04.2024] Here, the final output is the optimized model material group T opti (Diffuse Texture Map A opti , normal map N opti , Specular Map S opti , roughness map R opti ) and Global Ambient Lighting GI opti , and use the diffuse texture map in the optimized model material group as the optimized texture map or the diffuse optimized texture map.

[0131] When determining the target image of the target object based on the optimized texture map of the contour model, the optimized texture map of the contour model can be directly used as the target image of the target object; the optimized texture map can also be re-illuminated to obtain the target image of the target object.

[0132] In some embodiments, the server may relight the optimized texture map to obtain a target image of the target object in the following manner: determine the scene lighting source used for relighting; adjust the pixel values ​​of the optimized texture map according to the scene lighting source to obtain an image that matches the scene lighting source, and use the obtained image that matches the scene lighting source as the target image of the target object.

[0133] Here, after obtaining the optimized model material and global ambient lighting, the model can be edited according to actual needs, such as migrating the ambient lighting or editing the model material, relighting, etc. for the image to be rendered corresponding to each key frame. For example, when relighting, according to the scene lighting source used for relighting, new lighting and material are assigned to the image to be rendered, and then the lighting and material information of the image to be rendered is relighted to obtain the target image of the target object in the image to be rendered under the new lighting and material.

[0134] It is understandable that in the embodiments of the present application, when user information and other related data (such as the user's image to be rendered) are involved, when the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0135] Through the above method, after rendering the image to be rendered, the image rendering efficiency can be improved without training network parameters; in addition, texture mapping is performed based on the diffuse reflection image to obtain a texture map, and lighting restoration is performed based on the texture map to obtain a target image. Since the diffuse reflection image reflects the image part that can remain unchanged under changing lighting conditions in the image to be rendered, the texture map obtained based on the diffuse reflection image also retains the characteristic that the target object in the image to be rendered remains unchanged under changing lighting conditions. Therefore, the target image obtained by lighting restoration of the texture map can adapt to more sensitive environmental lighting changes, thereby improving the rendering quality of the image.

[0136] The following describes an exemplary application of the present invention in a practical application scenario. Referring to Figure 7 , which is a flow chart illustrating the image rendering method provided by the present invention, the entire algorithm process can be divided into four steps: data import, model material reconstruction, model editing, and model display. Each of these steps will be described below.

[0137] 1. Data import

[0138] The video stream to be rendered is collected by a video acquisition device, and the collected video stream is input into the data preprocessing module in the model material reconstruction.

[0139] 2. Model material reconstruction

[0140] Model material reconstruction includes three modules: data preprocessing, model reconstruction, and model material and ambient lighting restoration. Each module will be explained below.

[0141] 2.1 Data Preprocessing

[0142] The data preprocessing module includes three submodules: image preprocessing, camera parameter estimation, and foreground and background separation. Among them, image preprocessing is used to extract key frames from the video stream, such as using the real-time positioning and mapping (SLAM) algorithm to extract N key frames KF from the video stream. i (i∈1,…,N), filter out video frames that may have overexposure or motion blur in the video stream to reduce the input data size of the subsequent algorithm and improve the algorithm data quality.

[0143] [Added by reference (Article 20.6) 29.04.2024] Camera parameter estimation: Sparse reconstruction is performed using the SFM algorithm to solve the KF of each key frame i Corresponding camera parameters, such as camera intrinsic parameters K∈R 3x3 and camera extrinsics [R|t]∈R 3x4 .

[0144] [Added by reference (Article 20.6) 29.04.2024] Foreground and background segmentation: Use a neural network segmentation algorithm to perform image segmentation on each key frame to remove the background part of each key frame and obtain the foreground mask of each key frame. i (i.e., the image area where the target object is located in the image to be rendered), as shown in Figure 8, Figure 8 is a schematic diagram of image segmentation provided by an embodiment of the present application, the original image 801 of the key frame is subjected to image segmentation processing, the background part of the original image 801 is taken out, and the foreground mask 802 of the original image 801 is obtained.

[0145] 2.2 Model Reconstruction

[0146] Model reconstruction includes four sub-modules: differentiable white model construction, white model geometry optimization, image intrinsic decomposition, and model texture mapping. Each module will be explained below.

[0147] 2.2.1 Differentiable White Model Construction

[0148] [Added by reference (Rule 20.6) 29.04.2024] Referring to FIG9, FIG9 is a schematic diagram of the white mold construction provided by an embodiment of the present application. The differentiable white mold construction mainly utilizes the visual shell carving technology and the inverse neural rendering technology to generate a high-quality contour model of the reconstructed object (i.e., the above-mentioned white mold). The processing flow is as follows: the contour of each key frame is extracted by the visual shell carving technology to obtain the initial model M of each key frame. init (ie the above-mentioned initial contour model, or initial white model), and then the initial model M of each key frame init Perform surface subdivision processing to obtain the optimized model M of each key frame opti(i.e. the optimized contour model or optimized white model mentioned above), and then the optimized model M of each key frame is optimized by NeRF (Neural Radiance Field) opti Perform image rendering processing to obtain rendered images of each key frame, and update the neural radiation field parameters based on the residual between the rendered images of each key frame and the original image.

[0149] [Added by reference (rule 20.6) 29.04.2024] The visual shell carving technology is based on the image center p of each key frame, the foreground mask Mask i , internal and external parameters K, R, t, solve the least squares equations according to the principle of triangulation: K(RC+t)=p, estimate the intersection C of the rays emitted from the center of each key frame in the world coordinate system, and construct a voxel with a resolution of 32*32*32 (that is, the three-dimensional space volume mentioned above) with C as the midpoint. The length of its unit voxel is determined by the average length from the center of each camera to the intersection point. Project these voxel centers onto each key frame, and determine whether the voxel is occupied based on whether the center of each voxel falls on the foreground mask of the key frame. Finally, use the Marching Cube algorithm to extract the initial model M from the occupied voxel init .

[0150] [Added by reference (Rule 20.6) 29.04.2024] That is, for each key frame (i.e., the image to be rendered), the voxel points that fall on the foreground mask of the key frame are screened from each voxel point, and a plurality of voxels formed by the screened voxel points are determined. All voxels are traversed, triangular facets are extracted from each voxel, and the extracted triangular facets are combined to obtain the initial model M of the target object indicated by the foreground mask in the key frame in the three-dimensional scene. init .

[0151] The construction of a differentiable white mold based on the inverse neural rendering technology is a multi-stage iterative optimization process. Referring to FIG10 , FIG10 is a flow chart of a method for constructing a white mold provided by an embodiment of the present application. The method includes:

[0152] In step 201, the model to be subdivided is subjected to surface subdivision processing to obtain an optimized model.

[0153] It can be understood that in the first round of iteration, the model to be subdivided is the initial model, and in subsequent iterations, the model to be subdivided is the optimized model obtained in the previous round of iteration. For example, in the first round of iteration, the initial model is subjected to surface subdivision processing to obtain optimized model 1, and in the second round of iteration, the optimized model 1 obtained in the first round of iteration is subjected to surface subdivision processing to obtain optimized model 2, and in the third round of iteration, the optimized model 2 obtained in the second round of iteration is subjected to surface subdivision processing to obtain optimized model 3, and so on. The optimized model obtained by surface subdivision processing in the last round of iteration is used as the final optimized model (that is, the contour model ultimately required for the target object in the above-mentioned image to be rendered).

[0154] [Added by reference (Article 20.6) 29.04.2024] Taking the first round of iteration as an example, the initial model M of each key frame can be tessellated by the surface subdivision algorithm. init Subdivide to get the optimized model M opti (i.e. the optimized contour model or optimized white model mentioned above), the surface subdivision algorithm includes Catmull-Clark subdivision and Loop subdivision, wherein Catmull-Clark subdivision is a quadrilateral mesh subdivision rule, the initial model M init It is composed of multiple triangular facets, each of which can be regarded as composed of one or more quadrilateral meshes. When subdividing, a new vertex is generated for each face of the quadrilateral mesh, and a new vertex is generated for each edge of the quadrilateral mesh. At the same time, the position of each original vertex of the quadrilateral mesh is updated, and the quadrilateral mesh is divided according to the updated vertices to achieve the initial model M. init Surface tessellation.

[0155] [Added by reference (Article 20.6) 29.04.2024] Loop subdivision is a subdivision rule for triangular meshes. The initial model M init It is composed of multiple triangular facets, each of which can be regarded as composed of one or more triangular meshes. It is split according to 1-4 triangles, and each edge is calculated to generate a new vertex. At the same time, each original vertex updates its position, and the quadrilateral mesh is divided according to the updated vertex to achieve the initial model M init Surface tessellation.

[0156] In practical applications, in order to avoid excessive subdivision causing the model to occupy too much space, before each subdivision, it is necessary to obtain the color value of each triangle vertex corresponding to the optimized model at the end of the previous round of optimization projected onto the corresponding key frame, and the color error between the color value at the corresponding projection position in the key frame. For triangles whose color error is less than a certain error threshold (i.e., the first error value mentioned above), no further subdivision will be performed; alternatively, the color errors of each triangle are summed, and when the sum of the color errors is less than the error threshold, the model will no longer be subdivided.

[0157] In step 202, based on the camera parameters, the optimized model is subjected to image rendering processing through the neural radiation field to obtain a rendered image.

[0158] [Added by reference (Article 20.6) 29.04.2024] Here, for each key frame, the camera is set according to the internal and external parameters of the key frame, and the corresponding optimized model M is opti The neural radiation field is rendered to the key frame perspective to obtain a rendered image.

[0159] In step 203, the absolute value of the error between the rendered image and the key frame is obtained.

[0160] In step 204 , it is determined whether the absolute value of the error is less than an error threshold.

[0161] Here, when it is determined that the absolute value of the error between the rendered image and the key frame is greater than or equal to the error threshold, step 205 is executed; when it is determined that the absolute value of the error between the rendered image and the key frame is less than the error threshold, step 207 is executed.

[0162] In step 205 , the gradients of the vertex coordinates of the optimization model and the neural radiation field parameters are calculated.

[0163] In step 206, the vertex coordinates and neural radiation field parameters of the optimization model are updated using the gradient.

[0164] [Added by reference (Article 20.6) 29.04.2024] Here, when it is determined that the absolute value of the error between the rendered image and the key frame is greater than or equal to the error threshold, the next round of iteration is performed. Since the internal and external parameters of the key frame are fixed, only the geometric shape of the model needs to be considered, so the vertex coordinates V of the optimized model are opti Neural radiance field parameters that implicitly store vertex colors are optimized to avoid explicitly modeling lighting, shading, and color.

[0165] In step 207, the optimized model is output.

[0166] Here, when it is determined that the absolute value of the error between the rendered image and the key frame is less than the error threshold, the iteration is stopped, and the optimization model corresponding to the stop of the iteration (that is, the optimization model obtained by the last round of iteration) is determined as the contour model (that is, the white model) under the perspective of the key frame.

[0167] 2.2.2 White Model Geometry

[0168] Here, in order to remove non-popular faces, vertices, holes, etc. that appear in the construction of the differentiable white mold, the optimization model output in step 207 in Section 2.2.1 can be subjected to surface reconstruction processing, such as using CGAL to perform surface reconstruction processing on the optimization model to obtain a reconstructed optimization model, and the reconstructed optimization model is used as the white mold of the key frame in the three-dimensional scene (that is, the above-mentioned contour model).

[0169] 2.2.3 Image Intrinsic Decomposition

[0170] See Figure 11, which is a schematic diagram of the image intrinsic decomposition provided in an embodiment of the present application. A neural network is used to decouple the key frame into a diffuse reflection image and a shadow image. The purpose is to provide an initial value of the diffuse reflection texture for the subsequent fitting of the PBR texture material, and to avoid noise that may be difficult to converge when fitting the PBR texture material from randomly initialized parameters.

[0171] 2.2.4 Model Texture Mapping

[0172] [Added by reference (Article 20.6) 29.04.2024] Using Multi-View Stereo-Texturing (MVS-Texturing) technology, through view selection (View-selection), color correction (Color Adjustment), UV parameterization (UV Atlas) and other steps, the model face is mapped to the diffuse reflection image of the key frame of different view angles, and finally combined into a diffuse reflection texture map. Compared with direct parameterization on the white model, the UV coordinates and diffuse reflection texture A generated by MVS-Texturing technology are more accurate than those generated by direct parameterization on the white model. init When fitting PBR texture materials later, each texture block can be given a greater weight corresponding to the keyframe perspective, which can reduce the probability of seams and color differences to a certain extent.

[0173] The perspective selection is to find the key frame with the largest angle between the normal direction cosine and each triangular facet that constitutes the three-dimensional model (i.e., the outline model or white model mentioned above) from all key frames, and project the triangular facet onto the key frame to sample the corresponding texture sampling area. In practical applications, each key frame and the camera parameters when the key frame was collected can be used as a texture label. The key frames and the white models of each key frame (including multiple vertices and triangular faces), and the camera parameters of each key frame (i.e., the camera parameters when the key frames were collected) are input to output a renderable model. In addition to vertices and triangular faces, the model also needs to include at least the texture coordinates of each vertex and the texture map corresponding to the texture coordinates.

[0174] During rendering, for each triangle that constitutes the white model, the color area corresponding to the triangle is found and intercepted from the diffuse reflection image corresponding to the input image to be rendered (i.e., the key frame) to reflect its color, and the intercepted color area is pasted into the texture map. In actual applications, the color area of ​​each triangle is not directly pasted into the texture map, but triangles with the same texture label (i.e., some continuous triangles) are combined into a target patch, and the color area of ​​the target patch is pasted into the texture map of the target patch to obtain a diffuse reflection texture map.

[0175] In addition, since adjacent triangles may be sampled from different key frames, and these key frames may cause visual color inconsistencies due to changes in lighting and occlusion relationships, before the color area of ​​the target facet is pasted on the texture map, the color area of ​​the target facet may be subjected to color correction processing such as global uniformity or local uniformity, and the color area after color correction may be pasted on the texture map of the target facet. During color correction, the color continuity of different key frames may be constrained by minimizing the energy equation, so that the texture color is continuous and consistent. The energy equation is defined as the sum of the absolute value differences in the colors of all adjacent triangles corresponding to different key frames.

[0176] [Added by reference (Article 20.6) 29.04.2024] UV parameterization is to unfold adjacent triangles from the same keyframe onto a 2D image. The pixel values ​​of the 2D image are filled with the corresponding sampled texture. The normalized coordinates of the vertices of the triangles on the image are called UV coordinates. All triangles in the white model of each keyframe are processed in a loop, and finally the diffuse texture map A is obtained. init And the UV coordinates of all triangles corresponding to the diffuse texture map.

[0177] 2.3 Model material and environmental lighting restoration

[0178] [Added by reference (rule 20.6) 29.04.2024] The main purpose of the model material and ambient lighting restoration module is to restore the model material and ambient lighting to the previous module product (diffuse texture map A init ), the PBR texture material suitable for modern graphics engines and the ambient lighting during shooting are fitted.

[0179] [Added by reference (rule 20.6) 29.04.2024] Referring to FIG12, FIG12 is a schematic diagram of the model material and the ambient light fitting provided in the embodiment of the present application. Before starting the fitting, the three-dimensional white model M obtained in 2.2 is loaded. opti With diffuse texture map A init , then initialize the PBR material group T init (Diffuse Texture Map A init , normal map Ninit , Specular Map S init , roughness map R init ) and Global Ambient Lighting GI init , optimize the model material and ambient lighting to obtain the optimized PBR material group T opti and GI opti .

[0180] Referring to FIG13 , FIG13 is a schematic diagram of a process for fitting model material and ambient lighting according to an embodiment of the present application. The method includes:

[0181] In step 301, the model material and the environment lighting are initialized.

[0182] [Added by reference (rule 20.6) 29.04.2024] In practical applications, initializing the model material is to initialize the model material group T init (Diffuse Texture Map A init , normal map N init , Specular Map S init , roughness map R init ), initializing ambient lighting is to initialize the global ambient lighting GI init .

[0183] In step 302 , the white model is rendered using the ambient lighting of the panoramic image and the microsurface model to obtain a rendered image.

[0184] Here, for each key frame, the camera is set according to the internal and external parameters of the key frame, and the white model is rendered through the ambient lighting of the panoramic image and the microsurface model to obtain the corresponding rendered image.

[0185] [Added by reference (Article 20.6) 29.04.2024] The color of each 3D point can be expressed as the incident radiance L i (ω i ) and the bidirectional scattering distribution function f(ω i ,ω o ), the integral domain is the hemisphere Ω, ω centered on the surface intersection normal n i ,ω o are the incident and outgoing directions of the light, and n is the normal vector of the point.

[0186] [Added by reference (Article 20.6) 29.04.2024] The bidirectional scattering distribution function of the microsurface model is as follows:

[0187] Among them, D, G and F are the normal distribution term, geometric attenuation term and Fresnel term functions respectively. The numerical solutions of the above function terms can be calculated through diffuse texture map, normal map, specular map and roughness map.

[0188] In step 303, the absolute value of the error between the rendered image and the key frame is obtained.

[0189] In step 304 , it is determined whether the absolute value of the error is less than an error threshold.

[0190] Here, when it is determined that the absolute value of the error between the rendered image and the key frame is greater than or equal to the error threshold, step 305 is executed; when it is determined that the absolute value of the error between the rendered image and the key frame is less than the error threshold, step 307 is executed.

[0191] In step 305 , the gradient of the ambient illumination of the panoramic image and the microsurface model is calculated.

[0192] In step 306 , the ambient lighting and microsurface model parameters of the panoramic image are updated using the gradient.

[0193] [Added by reference (rule 20.6) 29.04.2024] When optimizing, you can first fix the diffuse texture map A init and update the normal map N with a larger learning rate α1 init , Specular Map S init , roughness map R init With Global Environment Illumination (GI) init , get the appropriate normal map N after a certain number of iterations init , Specular Map S init , roughness map R init With Global Environment Illumination (GI) init After that, fix the optimized normal map N init , Specular Map S init , roughness map R init With Global Environment Illumination (GI) init , and use a learning rate α2 smaller than α1 to update the optimized diffuse texture map A init .

[0194] [Added by reference (Article 20.6) 29.04.2024] In step 307: output the optimized model material and environment lighting.

[0195] [Added by reference (Article 20.6) 29.04.2024] Here, the final output is the optimized model material group T opti (Diffuse Texture Map A opti , normal map N opti , Specular Map S opti, roughness map R opti ) and Global Ambient Lighting GI opti .

[0196] 2.4 Model Editing

[0197] After optimizing the model material and global environment lighting, you can edit the model according to actual needs, such as migrating the environment lighting or editing the model material for the image to be processed corresponding to each key frame.

[0198] 2.5 Model Display

[0199] After editing the model, the corresponding rendering results can be displayed. For example, the image to be rendered corresponding to each key frame is input into the edited model. After assigning new lighting and materials to the image to be rendered, the lighting and material information of the image to be rendered is re-rendered or re-lighted to obtain the three-dimensional target image (i.e., three-dimensional model) of the image to be rendered under the new lighting and material.

[0200] Refer to Figure 14, which is an image rendering result provided by an embodiment of the present application. After the image to be rendered is re-illuminated, a target image under new illumination is obtained, that is, the illumination of the target image is changed compared with the image to be rendered.

[0201] [Added by reference (rule 20.6) 29.04.2024] Through the above method, after obtaining the optimized model material group T opti (Diffuse Texture Map A opti , normal map N opti , Specular Map S opti , roughness map R opti ) and Global Ambient Lighting GI opti After that, it can be used to reduce costs and speed up the production of game art assets, as well as content creation for user-generated content (UCG) games (which allow players to create their own game content).

[0202] Taking the realistic game production pipeline as an example, the above method can ensure the authenticity of art modeling, and at the same time greatly shorten the art material production cycle. For example, under normal circumstances, it takes a week or even longer for art to produce high-poly models and material mapping information, but the method provided by this application can be completed in one to several hours, and only some post-editing is required to produce a high-quality three-dimensional model. In addition, the method provided in the embodiment of the present application can also provide art designers with a draft of the model for subsequent reference in the stylized modeling task, which also shortens the cycle of game art production. In the creation of UGC game content, players or users can use a smartphone to shoot videos or pictures, upload them to the cloud, and immediately obtain the corresponding three-dimensional model and texture material. Combined with third-party tools or applications (such as ARKit), the produced three-dimensional model can be quickly interacted with real or virtual characters and scenes to create a variety of custom game content.

[0203] Through the above method, the embodiment of the present application provides a low-cost three-dimensional object reconstruction solution, which can automatically reconstruct a three-dimensional model with high-precision geometric shapes and high-quality PBR texture materials from only a video shot by a smartphone, and can support the application of its reconstructed products in various rendering scenarios, reducing the cost of art asset creation and improving development efficiency.

[0204] So far, the image rendering method provided by the embodiment of the present application has been described in combination with the exemplary application and implementation of the electronic device provided by the embodiment of the present application. The following will continue to describe the cooperation of various modules in the image rendering device 555 provided by the embodiment of the present application to implement the image rendering scheme.

[0205] The first processing module 5551 is configured to determine the mask, reflection image and camera parameters of the image to be rendered, wherein the image to be rendered includes a target object; the second processing module 5552 is configured to construct a model corresponding to the target image based on the mask and the camera parameters; the third processing module 5553 is configured to map the model based on the reflection image to obtain a texture of the model; and the fourth processing module 5554 is configured to perform lighting restoration on the model based on the texture to obtain a target image of the target object.

[0206] In some embodiments, the first processing module 5551 is further configured to determine the foreground mask and diffuse reflection image of the image to be rendered; the second processing module 5552 is further configured to construct a contour model corresponding to the target object based on the foreground mask and the camera parameters; the third processing module 5553 is further configured to perform texture mapping on the contour model based on the diffuse reflection image to obtain a texture map of the contour model; the fourth processing module 5554 is further configured to perform lighting restoration on the contour model based on the texture map to obtain a target image of the target object.

[0207] In some embodiments, the device also includes: an image determination module, configured to capture a video stream, the video stream including multiple video frames; extract key frames from the video stream, and use the key frames as the image to be rendered; wherein the key frames include at least one of the following video frames: the number of times the target feature point in the point cloud of the video frame is selected is higher than a number threshold, and the target feature point is a feature point in the point cloud that is repeatedly selected and set as an inner group; the relative motion distance between the video frame and the adjacent previous video frame is lower than a distance threshold; the number of feature points matching between the video frame and the adjacent previous video frame is higher than a number threshold.

[0208] In some embodiments, the first processing module is further configured to perform background segmentation on the image to be rendered, remove the background part from the image to be rendered, obtain the image area where the target object is located in the image to be rendered, and use the image area as the foreground mask of the image to be rendered, and perform image intrinsic decomposition on the image to be rendered to obtain a diffuse reflection image of the image to be rendered.

[0209] In some embodiments, the second processing module is further configured to construct a three-dimensional space body according to the center of the image to be processed, the foreground mask and the camera parameters, wherein the three-dimensional space body includes a plurality of voxel points; project each voxel point of the three-dimensional space body onto the image to be rendered, and extract the initial contour model of the target object from each voxel point according to the projection result; construct a contour model based on the initial contour model and the camera parameters to obtain the contour model of the target object.

[0210] In some embodiments, the second processing module is further configured to filter out voxel points that fall on the foreground mask from each of the voxel points based on the projection results, and determine a plurality of voxels formed by the filtered voxel points; traverse the plurality of voxels and extract triangular facets from the plurality of voxels; and combine the extracted triangular facets to obtain an initial contour model of the target object.

[0211] In some embodiments, the second processing module is further configured to perform the following multiple rounds of iterative processes based on the initial contour model and the camera parameters: performing surface subdivision on the contour model to be subdivided to obtain an optimized contour model, wherein the contour model to be subdivided in the first round of iteration is the initial contour model; performing image rendering through a neural radiation field network based on the optimized contour model and the camera parameters to obtain a rendered image of the optimized contour model; obtaining the absolute value of the error between the rendered image and the image to be rendered, and entering the next round of iteration when the absolute value of the error is greater than an error threshold, and stopping the iteration until the absolute value of the error does not exceed the error threshold, and determining the contour model of the target object based on the optimized contour model corresponding to the time of stopping the iteration.

[0212] In some embodiments, the second processing module is further configured to perform triangular mesh subdivision on the contour model to be subdivided to obtain an optimized contour model; or, perform quadrilateral mesh subdivision on the contour model to be subdivided to obtain an optimized contour model; wherein the color error of each triangular facet in the optimized contour model is less than a first error value, or the sum of the color errors of each triangular facet in the optimized contour model is less than a second error value; the color error is the difference between the color value of the vertex of the triangular facet projected onto the image to be rendered and the color value at the corresponding projection position in the image to be rendered.

[0213] In some embodiments, the second processing module is further configured to perform surface construction on the optimized contour model to obtain a constructed optimized contour model, and use the constructed optimized contour model as the contour model of the target object.

[0214] In some embodiments, the contour model includes multiple triangular facets, and the third processing module is further configured to determine the texture label of each triangular facet based on the diffuse reflection image and the camera parameters, and combine the triangular facets with the same texture label to obtain multiple target facets; extract the color area of ​​each target facet from the diffuse reflection image; perform color correction on the color area of ​​each target facet, and paste the corrected color area of ​​each target facet into the texture map of the corresponding target facet to obtain the texture map of the contour model.

[0215] In some embodiments, the fourth processing module is further configured to initialize a diffuse reflection material group and global ambient lighting, wherein the diffuse reflection material group includes at least the texture map; based on the diffuse reflection material group, the global ambient lighting and the camera parameters, the material and ambient lighting of the contour model are restored to obtain an optimized texture map of the contour model; and based on the optimized texture map of the contour model, the target image of the target object is determined.

[0216] In some embodiments, the fourth processing module is further configured to determine a scene lighting source for re-illumination; adjust the pixel values ​​of the optimized texture map according to the scene lighting source to obtain an image that matches the scene lighting source, and use the obtained image that matches the scene lighting source as the target image of the target object.

[0217] An embodiment of the present application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the image rendering method described in the embodiment of the present application.

[0218] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, which stores computer-executable instructions or computer programs. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the image rendering method provided by an embodiment of the present application, for example, the image rendering method shown in Figure 3A.

[0219] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or various devices including one or any combination of the above memories.

[0220] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0221] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0222] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0223] In summary, the embodiments of the present application provide a low-cost three-dimensional object reconstruction solution, which can automatically reconstruct a three-dimensional model with high-precision geometric shapes and high-quality PBR texture materials from only a video shot by a smartphone, and can support the application of its reconstructed products in various rendering scenarios, reducing the cost of art asset creation and improving development efficiency.

[0224] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. An image rendering method, the method being performed by an electronic device, the method comprising: Determining a mask, a reflected image, and camera parameters of an image to be rendered, wherein the image to be rendered includes a target object; Based on the mask and the camera parameters, construct a model corresponding to the target object; Mapping the model based on the reflection image to obtain a texture of the model; Based on the map, the model is illuminated and restored to obtain a target image of the target object.

2. The method of claim 1, wherein: The step of determining a mask and a reflection image of an image to be rendered includes: Determine the foreground mask and diffuse reflection image of the image to be rendered; The constructing a model corresponding to the target object based on the mask and the camera parameters includes: Based on the foreground mask and the camera parameters, construct a contour model corresponding to the target object; Mapping the model based on the reflection image to obtain a map of the model includes: Performing texture mapping on the contour model based on the diffuse reflection image to obtain a texture map of the contour model; The step of performing illumination restoration on the model based on the map to obtain a target image of the target object includes: Based on the texture map, the contour model is restored by illumination to obtain a target image of the target object.

3. The method of claim 1, wherein: Before determining the mask of the image to be rendered, the method further includes: Capturing a video stream, wherein the video stream includes a plurality of video frames; Extracting key frames from the video stream and using the key frames as the images to be processed; The key frame includes at least one of the following video frames: The number of times the target feature point in the point cloud of the video frame is selected is higher than a number threshold, and the target feature point is a feature point in the point cloud that is repeatedly selected and set as an inner group; The relative motion distance between the video frame and the adjacent previous video frame is lower than a distance threshold; The number of feature points that match between the video frame and an adjacent previous video frame is greater than a number threshold.

4. The method of claim 2, wherein: The step of determining a foreground mask and a diffuse reflection image of the image to be rendered includes: Performing background segmentation on the image to be rendered, removing the background portion from the image to be rendered, obtaining an image region where the target object is located in the image to be rendered, and using the image region as a foreground mask of the image to be rendered; Performing image intrinsic decomposition on the image to be rendered to obtain a diffuse reflection image of the image to be rendered.

5. The method of claim 2, wherein: The constructing a contour model corresponding to the target object based on the foreground mask and the camera parameters includes: Construct a three-dimensional space according to the center of the image to be processed, the foreground mask and the camera parameters. An inter-body, wherein the three-dimensional space body includes a plurality of voxel points; Projecting each voxel point of the three-dimensional space onto the image to be rendered, and extracting an initial contour model of the target object from each voxel point according to the projection result; A contour is constructed based on the initial contour model and the camera parameters to obtain a contour model corresponding to the target object.

6. The method of claim 5, wherein: The step of extracting the initial contour model of the target object from each of the voxel points according to the projection result comprises: According to the projection result, from each of the voxel points, the voxel points projected onto the foreground mask are screened out, and a plurality of voxels formed by the screened voxel points are determined; Traversing the plurality of voxels and extracting triangular facets from the plurality of voxels; The extracted triangular facets are combined to obtain an initial contour model of the target object.

7. The method of claim 5, wherein: The constructing a contour based on the initial contour model and the camera parameters to obtain a contour model corresponding to the target object includes: The following multiple rounds of iterations are performed based on the initial contour model and the camera parameters: Performing surface subdivision on the contour model to be subdivided to obtain an optimized contour model, wherein the contour model to be subdivided in the first round of iteration is the initial contour model; Based on the camera parameters, performing image rendering on the optimized contour model through a neural radiation field network to obtain a rendered image of the optimized contour model; The absolute value of the error between the rendered image and the image to be rendered is obtained, and when the absolute value of the error is greater than an error threshold, the next round of iteration is entered, and the iteration is stopped until the absolute value of the error does not exceed the error threshold, so as to determine the contour model corresponding to the target object based on the optimized contour model corresponding to when the iteration is stopped.

8. The method of claim 7, wherein: The step of performing surface subdivision on the contour model to be subdivided to obtain an optimized contour model comprises: Performing triangular mesh subdivision on the subdivided contour model to obtain an optimized contour model; or performing quadrilateral mesh subdivision on the subdivided contour model to obtain an optimized contour model; Wherein, the color error of each triangular facet in the optimized contour model is less than the first error value, or the sum of the color errors of each triangular facet in the optimized contour model is less than the second error value; The color error is the difference between the color value of the vertex of the triangle patch projected onto the image to be rendered and the color value at the corresponding projection position in the image to be rendered.

9. The method of claim 7, wherein: The determining of the contour model corresponding to the target object based on the optimized contour model corresponding to when the iteration is stopped includes: The optimized contour model corresponding to the time when iteration is stopped is subjected to surface construction to obtain a constructed optimized contour model, and the constructed optimized contour model is used as the contour model of the target object.

10. The method of claim 2, wherein: The contour model includes a plurality of triangular facets, and texture mapping is performed on the contour model based on the diffuse reflection image to obtain a texture map of the contour model, including: Determine the texture label of each of the triangular facets according to the diffuse reflection image and the camera parameters, and combine the triangular facets with the same texture label to obtain a plurality of target facets; Extracting the color area of ​​each target surface patch from the diffuse reflection image; Color correction is performed on the color region of each target face patch, and the color region of each target face patch after correction is pasted onto the texture map of the corresponding target face patch to obtain the texture map of the contour model.

11. The method of claim 2, wherein: The step of performing illumination restoration on the contour model based on the texture map to obtain a target image of the target object includes: Initialize a diffuse reflection material group and global environment lighting, wherein the diffuse reflection material group at least includes the texture map; According to the diffuse reflection material group, the global environment illumination and the camera parameters, the material and environment illumination of the contour model are restored to obtain an optimized texture map of the contour model; A target image of the target object is determined based on the optimized texture map of the contour model.

12. The method of claim 2, wherein: The step of determining a target image of the target object based on the optimized texture map of the contour model comprises: Determine the scene illumination source used for relighting; According to the scene lighting source, the pixel values ​​of the optimized texture map are adjusted to obtain an image matching the scene lighting source, and the obtained image matching the scene lighting source is used as the target image of the target object.

13. An image rendering device, the device comprising: A first processing module is configured to determine a mask, a reflected image, and camera parameters corresponding to an image to be rendered, wherein the image to be rendered includes a target object; A second processing module is configured to construct a model corresponding to the target object based on the mask and the camera parameters; A third processing module is configured to map the model based on the reflection image to obtain a texture of the model; The fourth processing module is configured to perform lighting restoration on the model based on the map to obtain a target image of the target object.

14. An electronic device comprising: a memory configured to store computer executable instructions or computer programs; The processor is configured to implement the image processing method according to any one of claims 1 to 12 when executing the computer executable instructions or computer programs stored in the memory.

15. A computer-readable storage medium storing computer-executable instructions or a computer program, wherein when the computer-executable instructions or the computer program are executed by a processor, the image rendering method according to any one of claims 1 to 12 is implemented.

16. A computer program product, comprising a computer program or a computer executable instruction, wherein when the computer program or the computer executable instruction is executed by a processor, the image rendering method according to any one of claims 1 to 12 is implemented.