Training method of image rendering model and image rendering method

Abstract

The present disclosure relates to a method for training an image rendering model, an image rendering method, an apparatus, an electronic device and a storage medium. The method comprises: obtaining a training data sample for a target object, and pre-training an image rendering model according to the training data sample to obtain an image rendering model satisfying a pre-training completion condition; obtaining a depth image corresponding to the target object according to the image rendering model satisfying the pre-training completion condition, and constructing point cloud data of the target object; performing surface reconstruction on the target object according to the point cloud data of the target object to obtain a surface reconstruction result of the target object, and determining a target sampling point corresponding to the target object based on the surface reconstruction result of the target object; and training the image rendering model satisfying the pre-training completion condition based on the target sampling point to obtain a trained image rendering model. The present method improves the training efficiency of the image rendering model.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to training methods for image rendering models, image rendering methods, apparatus, electronic devices, and storage media. Background Technology

[0002] With the development of image processing technology, image rendering technology has emerged. Current image rendering techniques typically employ the NeRF (Neural Radiance Fields) model to obtain the volume density and color values ​​of points on the surface of a target object in a 3D scene. However, the training process of the NeRF model requires a large amount of data on invalid points (i.e., points not on the surface of the target object), resulting in slow model training speed and consequently, low efficiency in rendering target objects using the NeRF model.

[0003] Current accelerated training methods for NeRF models involve performing a coarse reconstruction of the target object in a 3D space containing the target object, yielding a coarse reconstruction result. This reconstruction result is used to determine the coarse distribution region of the target object. Then, data is sampled within this coarse distribution region to obtain multiple sampling point data points that characterize the target object. The NeRF model is then trained based on this multiple sampling point data points to obtain the trained NeRF model.

[0004] However, in current accelerated training methods for NeRF models, to ensure the integrity of the target object, the coarse distribution region of the target object is often larger than the actual distribution region of the target object. Moreover, the internal space of the target object is also included in the coarse distribution region, and data points in the internal space also participate in data sampling. Therefore, the resulting multiple sampling points still contain a large number of invalid data points, i.e., non-target object surface data. Training on invalid points increases the training time of the NeRF model and reduces the model training efficiency. Summary of the Invention

[0005] This disclosure provides a training method for an image rendering model, an image rendering method, an apparatus, an electronic device, and a storage medium, to at least solve the problem of low training efficiency of image rendering models in related technologies. The technical solution of this disclosure is as follows:

[0006] According to a first aspect of the present disclosure, a method for training an image rendering model is provided, the method comprising:

[0007] Obtain training data samples for the target object, and pre-train the image rendering model based on the training data samples to obtain an image rendering model that meets the pre-training completion conditions.

[0008] Based on the depth image obtained from the image rendering model that meets the pre-training completion conditions, construct the point cloud data of the target object;

[0009] The surface of the target object is reconstructed based on the point cloud data of the target object to obtain the surface reconstruction result of the target object, and the target sampling point corresponding to the target object is determined based on the surface reconstruction result of the target object;

[0010] Based on the target sampling points, the image rendering model that meets the pre-training completion conditions is trained to obtain the trained image rendering model.

[0011] In one exemplary embodiment, obtaining training data samples for the target object includes:

[0012] Acquire a video segment containing the target object within a preset duration; the video segment contains initial image data from multiple preset shooting angles;

[0013] The video segment is subjected to frame extraction processing to obtain an image sequence containing initial image data of multiple target initial video frames;

[0014] The background is removed from each initial image data in the image sequence based on a pre-trained object segmentation model to obtain background-removed image data.

[0015] Determine the camera pose corresponding to each background-removed image data, and obtain training data samples of the target object based on each background-removed image data and the camera pose corresponding to each background-removed image data.

[0016] In an exemplary embodiment, the step of pre-training the image rendering model based on the training data samples to obtain an image rendering model that meets the pre-training completion conditions includes:

[0017] The training data samples are input into the image rendering model, and the image rendering model processes each image data in the training data samples to output the rendered image corresponding to each image data.

[0018] Loss calculations are performed on each preset standard rendered image and each rendered image to obtain the loss result corresponding to the training data sample;

[0019] When the loss result and the number of model training iterations meet the pre-training completion conditions, an image rendering model that meets the pre-training completion conditions is obtained.

[0020] In an exemplary embodiment, the step of inputting the training data samples into an image rendering model, processing each image data in the training data samples through the image rendering model, and outputting a depth image and a rendered image corresponding to each image data includes:

[0021] The training data samples are input into the image rendering model. Based on the camera pose corresponding to each image data in the training data samples, multiple simulated rays corresponding to the image data under the camera pose are generated. Data is sampled in the ray direction of each simulated ray to obtain multiple sampling points corresponding to the simulated ray.

[0022] The color values ​​and volume density values ​​of multiple sampling points on the simulated light are determined, and the depth image and rendered image corresponding to the image data under the camera pose are determined based on the color values, volume density values ​​and position coordinates of the multiple sampling points.

[0023] In an exemplary embodiment, determining the depth image and rendered image corresponding to the image data under the camera pose based on the color values, volume density, and position coordinates of the plurality of sampling points includes:

[0024] Based on the color value, volume density value, and preset integration algorithm of multiple sampling points corresponding to each simulated light ray, the rendered image corresponding to the image data under the camera pose is determined;

[0025] Based on the position coordinates, volume density values, and preset integration algorithm of multiple sampling points corresponding to each simulated ray, the depth image corresponding to the image data under the camera pose is determined.

[0026] In an exemplary embodiment, constructing the point cloud data of the target object based on the depth image obtained from the image rendering model that satisfies the pre-training completion condition includes:

[0027] Based on the position coordinates of the pixels contained in each depth image obtained by the image rendering model that meets the pre-training completion conditions, and the preset conversion algorithm, the pixels contained in each depth image are fused and converted to obtain initial point cloud data.

[0028] According to the preset spatial partitioning strategy, the three-dimensional space corresponding to the initial point cloud data is divided into a preset number of grids;

[0029] The number of initial point cloud data in each grid is counted, and the initial point cloud data in grids with a number less than a preset threshold is eliminated to obtain the denoised point cloud data of the target object.

[0030] According to a second aspect of the present disclosure, an image rendering method includes:

[0031] The target camera pose is input into the trained image rendering model for the target object, and the trained image rendering model is used to process the image and output a rendered image of the target object from the shooting angle corresponding to the target camera pose; wherein, the training process of the trained image rendering model includes performing the training method of the image rendering model as described in any one of the first aspects above.

[0032] According to a third aspect of the present disclosure, a training apparatus for an image rendering model is provided, the apparatus comprising:

[0033] The pre-training unit is configured to acquire training data samples for the target object and pre-train the image rendering model based on the training data samples to obtain an image rendering model that meets the pre-training completion conditions.

[0034] The construction unit is configured to execute the point cloud data of the target object by executing the depth image obtained from the image rendering model that satisfies the pre-training completion condition;

[0035] The reconstruction unit is configured to perform surface reconstruction of the target object based on the point cloud data of the target object, obtain the surface reconstruction result of the target object, and determine the target sampling point corresponding to the target object based on the surface reconstruction result of the target object;

[0036] The training unit is configured to train the image rendering model that meets the pre-training completion conditions based on the target sampling points, so as to obtain the trained image rendering model.

[0037] In one exemplary embodiment, the pre-training unit includes:

[0038] The acquisition subunit is configured to acquire a video segment containing the target object within a preset duration; the video segment contains initial image data from multiple preset shooting angles;

[0039] The frame extraction subunit is configured to perform frame extraction processing on the video segment to obtain an image sequence containing initial image data of multiple target initial video frames;

[0040] The first processing subunit is configured to perform background removal processing on each initial image data contained in the image sequence according to a pre-trained object segmentation model to obtain background-removed image data.

[0041] The construction subunit is configured to determine the camera pose corresponding to each of the background-removed image data, and to obtain training data samples of the target object based on each of the background-removed image data and the camera pose corresponding to each of the background-removed image data.

[0042] In one exemplary embodiment, the pre-training unit includes:

[0043] The second processing subunit is configured to input the training data samples into an image rendering model, process each image data in the training data samples through the image rendering model, and output a rendered image corresponding to each image data.

[0044] The computation subunit is configured to perform loss calculations based on each preset standard rendered image and each of the rendered images to obtain the loss result corresponding to the training data sample;

[0045] A subunit is determined and configured to execute an image rendering model that satisfies the pre-training completion condition when the loss result and the number of model training iterations meet the pre-training completion condition.

[0046] In an exemplary embodiment, the second processing subunit is specifically used to input the training data samples into the image rendering model, generate multiple simulated rays corresponding to the image data under the camera pose based on the camera pose of each image data in the training data samples, and perform data sampling in the ray direction of each simulated ray to obtain multiple sampling points corresponding to the simulated ray.

[0047] The color values ​​and volume density values ​​of multiple sampling points on the simulated light are determined, and the depth image and rendered image corresponding to the image data under the camera pose are determined based on the color values, volume density values ​​and position coordinates of the multiple sampling points.

[0048] In an exemplary embodiment, the second processing subunit is specifically used to determine the rendered image corresponding to the image data under the camera pose based on the color value, volume density value, and preset integration algorithm of multiple sampling points corresponding to each of the simulated light rays;

[0049] Based on the position coordinates, volume density values, and preset integration algorithm of multiple sampling points corresponding to each simulated ray, the depth image corresponding to the image data under the camera pose is determined.

[0050] In one exemplary embodiment, the building unit includes:

[0051] The fusion and transformation unit is configured to execute the position coordinates of the pixels contained in each depth image obtained based on the image rendering model that meets the pre-training completion conditions and a preset transformation algorithm to fuse and transform the pixels contained in each depth image to obtain initial point cloud data.

[0052] The partitioning unit is configured to divide the three-dimensional space corresponding to the initial point cloud data into a preset number of grids according to a preset spatial partitioning strategy;

[0053] The denoising unit is configured to perform statistical analysis on the number of initial point cloud data in each grid and eliminate the initial point cloud data in grids with a number less than a preset threshold, thereby obtaining the denoised point cloud data of the target object.

[0054] According to a fourth aspect of the present disclosure, an image rendering apparatus is provided, the apparatus comprising:

[0055] The rendering unit is configured to input the target camera pose into a trained image rendering model for the target object, and through the processing of the trained image rendering model, output a rendered image of the target object from the shooting angle corresponding to the target camera pose; wherein, the training process of the trained image rendering model includes executing the image rendering model training method as described in any one of the first aspects above.

[0056] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising:

[0057] processor;

[0058] Memory used to store the processor's executable instructions;

[0059] The processor is configured to execute the instructions to implement the training method of the image rendering model as described in any one of the first aspects above, and to implement the image rendering method as described in the second aspect above.

[0060] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided, wherein when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform a training method for an image rendering model as described in any one of the first aspects above, and to implement an image rendering method as described in the second aspect above.

[0061] According to a seventh aspect of the present disclosure, a computer program product is provided, wherein when the instructions are executed by a processor of an electronic device, the electronic device is enabled to perform the training method of the image rendering model described in any one of the first aspects above, and to implement the image rendering method as described in the second aspect above.

[0062] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0063] This method pre-trains the image rendering model to obtain the depth image of the target object output by the image rendering model that meets the pre-training completion conditions. Then, the surface of the target object is reconstructed based on the depth image to obtain the surface reconstruction result. Based on the surface reconstruction result, the target sampling points corresponding to the target object can be determined. This allows the image rendering model to be trained only on the target sampling points during training, reducing the training time for invalid points (i.e., non-target sampling points) and improving the training efficiency of the image rendering model.

[0064] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0065] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0066] Figure 1 This is a flowchart illustrating a training method for an image rendering model according to an exemplary embodiment.

[0067] Figure 2 This is a schematic diagram illustrating a surface reconstruction result of a target object according to an exemplary embodiment.

[0068] Figure 3 This is a schematic diagram illustrating the determination of the intersection point of the surfaces of a target object according to an exemplary embodiment.

[0069] Figure 4 This is a flowchart illustrating a step of obtaining training data samples of a target object according to an exemplary embodiment.

[0070] Figure 5 This is a flowchart illustrating the pre-training steps of an image rendering model according to an exemplary embodiment.

[0071] Figure 6 This is a flowchart illustrating the model processing procedure in an image rendering model according to an exemplary embodiment.

[0072] Figure 7 This is a flowchart illustrating the sampling point data processing process in the image processing of a target object according to an exemplary embodiment.

[0073] Figure 8 This is a flowchart illustrating a step of determining a rendered image and a depth image according to an exemplary embodiment.

[0074] Figure 9 This is a flowchart illustrating an application method of an image rendering model according to an exemplary embodiment.

[0075] Figure 10 This is a schematic diagram illustrating the rendering of images of a target object from various shooting angles according to an exemplary embodiment.

[0076] Figure 11 This is a block diagram illustrating a training apparatus for an image rendering model according to an exemplary embodiment.

[0077] Figure 12 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0078] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0079] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0080] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0081] Figure 1 This is a flowchart illustrating a training method for an image rendering model according to an exemplary embodiment, such as... Figure 1 As shown, the training method for this image rendering model is applied to a computer device, and the method includes the following steps.

[0082] In step S110, training data samples for the target object are obtained, and the image rendering model is pre-trained based on the training data samples to obtain an image rendering model that meets the pre-training completion conditions.

[0083] The target object can be an object such as a chair, table, or teacup, or a living being such as a person, cat, or dog. In other words, the target object is a concrete physical object that can be rendered as an image. This embodiment does not limit the type of the target object.

[0084] In implementation, the computer device pre-stores training data samples of the target object and a preset image rendering model. The training data samples contain image data of the camera device under different camera poses and shooting angles. When image rendering of the target object is required, the computer device obtains the training data samples corresponding to the target object, and pre-trains the preset image rendering model based on the camera poses and the image data corresponding to each camera pose contained in the training data samples. Thus, an image rendering model that meets the pre-training completion conditions is obtained after pre-training.

[0085] In step S120, the depth image corresponding to the target object is obtained based on the image rendering model that meets the pre-training completion conditions, and the point cloud data of the target object is constructed.

[0086] In implementation, after obtaining an image rendering model that meets the pre-training completion conditions, the computer device uses this model to determine the depth image corresponding to each image of the target object in the training data samples. Then, based on the distance data represented by each pixel in each depth image, the computer device constructs point cloud data containing the target object in three-dimensional space. The point cloud data represents the position coordinates of the target object's surface in three-dimensional space.

[0087] In step S120, the surface of the target object is reconstructed based on the point cloud data of the target object to obtain the surface reconstruction result of the target object, and the target sampling point corresponding to the target object is determined based on the surface reconstruction result of the target object.

[0088] The target sampling point is the intersection of the surface reconstruction result of the target object and the preset simulated light ray.

[0089] In practice, computer equipment reconstructs the surface of the target object based on the point cloud data of the target object, i.e. Figure 2 As shown, point cloud data from adjacent points are connected to form patches, resulting in the surface reconstruction of the target object (i.e., a mesh structure). Then, the computer generates simulated rays for the image data at each camera pose in the training data samples. Data is sampled on each simulated ray to obtain multiple sampling points along that ray. Then, as... Figure 3 As shown, based on the multiple sampling points corresponding to the simulated light rays and the surface reconstruction results of the target object, the sampling point at the intersection of the surface reconstruction results of the target object and the simulated light rays is determined as the target sampling point corresponding to the target object.

[0090] In step S140, the image rendering model that meets the pre-training completion conditions is trained based on the target sampling points to obtain the trained image rendering model.

[0091] In implementation, the computer equipment further trains the pre-trained image rendering model, which meets the pre-training completion conditions, based on the target sampling points. During this further training, the volume density value and color of the target sampling points are used to obtain the corresponding rendered image under the camera pose through a preset integral algorithm. The loss result is calculated based on the standard rendered image and the output rendered image, thereby achieving supervised training of the image rendering model and obtaining the final trained image rendering model. Specifically, the training process of the image rendering model is similar to the pre-training process of the image rendering model, and will not be described in detail in this embodiment.

[0092] In the above-mentioned training method for the image rendering model, the depth image of the target object output by the image rendering model that meets the pre-training completion conditions is obtained through pre-training of the image rendering model. Then, the surface of the target object is reconstructed based on the depth image to obtain the surface reconstruction result. Based on the surface reconstruction result, the target sampling points corresponding to the target object can be determined. This allows the image rendering model to be trained only on the target sampling points when it is being trained, reducing the training time for invalid points (i.e., non-target sampling points), improving the training efficiency of the image rendering model, and thus improving the image rendering efficiency.

[0093] In one exemplary embodiment, such as Figure 4 As shown, in step S110, obtaining training data samples for the target object can be achieved through the following steps:

[0094] In step S402, a video segment containing the target object is acquired.

[0095] The video segment contains initial image data from multiple preset shooting angles. Each preset shooting angle corresponds to a camera pose.

[0096] In practice, the camera equipment captures video around the target object. During video capture, the camera equipment can adopt various camera poses to obtain a video segment containing the target object. For example, a video can be captured around the target object, including multiple shooting angles such as front view, top view, and oblique view, with a video length of approximately 1 minute. Then, when image rendering of the target object is required, the computer equipment acquires the video segment containing the target object within a preset duration.

[0097] In step S404, the video segment is subjected to frame extraction processing to obtain an image sequence containing initial image data of multiple target initial video frames.

[0098] In practice, a video segment contains image data of multiple initial video frames. The computer device performs frame extraction processing on the multiple initial video frames in the video segment to obtain an image sequence containing initial image data of multiple target initial video frames after frame extraction processing.

[0099] Optionally, the computer device may perform frame extraction processing on video segments by using a random frame extraction method, a proportional frame extraction method, or a frame extraction method with equal duration. This embodiment of the present disclosure does not limit the frame extraction processing method.

[0100] In step S406, background removal processing is performed on each initial image data contained in the image sequence according to the pre-trained object segmentation model to obtain background-removed image data.

[0101] In implementation, a pre-trained object segmentation model is stored in the computer device. This object segmentation model, after pre-training, can segment image data. Therefore, the computer device performs background removal processing on each initial image data in the image sequence according to the object segmentation model to obtain background-removed image data. The object segmentation model can be a deep learning model, and this embodiment of the disclosure is not limited thereto.

[0102] In step S408, the camera pose corresponding to each background-removed image data is determined, and training data samples of the target object are obtained based on each background-removed image data and the camera pose corresponding to each background-removed image data.

[0103] In practice, for the background-removed image data, the computer equipment uses colmap (a 3D reconstruction software) to determine the camera pose corresponding to each background-removed image data, and obtains the training data sample of the target object based on each background-removed image data corresponding to the target object and the camera pose corresponding to each background-removed image data.

[0104] In this embodiment, the initial image data of the target object from multiple preset shooting angles are processed to determine the camera pose corresponding to each initial image data. Training data samples are then constructed based on each initial image data and the camera pose corresponding to each initial image data, which are then used to train the preset image rendering model.

[0105] In one exemplary embodiment, such as Figure 5 As shown, in step S110, the image rendering model is pre-trained based on the training data samples to obtain an image rendering model that meets the pre-training completion conditions. This can be achieved through the following steps:

[0106] In step S502, the training data samples are input into the image rendering model, and the image rendering model processes each image data in the training data samples and outputs the rendered image corresponding to each image data.

[0107] The image rendering model can be a NeRF (Neural Radiance Fields) model.

[0108] In implementation, the computer equipment inputs each image data point from the training data samples, along with the corresponding camera pose, into the image rendering model. The model then generates multiple simulated light rays for each camera pose, representing the orientation direction of the image data. The specific number of simulated light rays is equal to the size of the image data (w*h rays). Next, sampling is performed on each simulated light ray, calculating the color and volume density values ​​at each sampling point. Using the color and volume density values ​​of multiple simulated light rays, the rendered image corresponding to each image data point is obtained.

[0109] Optionally, in addition to outputting rendered images, the image rendering model can also be used by computer devices to obtain depth images corresponding to each image data based on the intermediate data results obtained during the pre-training process of the image rendering model.

[0110] In step S504, loss calculation is performed based on each preset standard rendered image and each rendered image to obtain the loss result corresponding to the training data sample.

[0111] In implementation, the computer equipment has pre-set standard rendered images. Then, the computer equipment performs loss calculations based on the rendered images obtained by the image rendering model and the pre-set standard rendered images to obtain the loss results corresponding to the training data samples. Specifically, the loss calculation method can be: loss = sum(abs(C-C1)) / n. Where C represents the standard rendered image, C1 represents the rendered image output by the image rendering model, abs() represents taking the absolute value, and n represents the number of image data in the training data samples.

[0112] In step S506, when the loss result and the number of model training iterations meet the pre-training completion conditions, an image rendering model that meets the pre-training completion conditions is obtained.

[0113] In implementation, the computer equipment has a preset threshold for the number of model training iterations (or the number of model iterations threshold). When the number of model training iterations reaches the threshold and the loss result of the image rendering model stabilizes within a preset range, it is determined that the training of the current image rendering model has met the pre-training completion condition, and the computer equipment obtains an image rendering model that meets the pre-training completion condition. This threshold for the number of model training iterations can be 1 / 10 of the total number of model training iterations (i.e., the sum of the pre-training iterations and the final training iterations).

[0114] In this embodiment, the image rendering model is pre-trained using training data samples and preset pre-training completion conditions to obtain an image rendering model that meets the pre-training completion conditions. This allows a depth image to be obtained based on the image rendering model that meets the pre-training completion conditions, thereby completing the surface reconstruction of the target object and improving the accuracy of surface reconstruction so as to accurately fit the surface of the target object.

[0115] In one exemplary embodiment, such as Figure 6 As shown, in step S502, the training data samples are input into the image rendering model. The image rendering model processes each image data in the training data samples and outputs the rendered image corresponding to each image data. This can be achieved through the following steps:

[0116] In step S602, the training data samples are input into the image rendering model. Based on the camera pose corresponding to each image data in the training data samples, multiple simulated rays corresponding to the image data under the camera pose are generated. Data sampling is performed in the ray direction of each simulated ray to obtain multiple sampling points corresponding to the simulated ray.

[0117] In implementation, the computer device inputs training data samples into the image rendering model. Based on the camera pose corresponding to each image data in the training data samples, it generates multiple simulated rays corresponding to the image data under the camera pose. Data sampling is performed on the ray direction of each simulated ray. For example, about 10 points are sampled on a simulated ray, and w*h = 80 simulated rays, thus obtaining multiple sampling points (800 sampling points) corresponding to 80 simulated rays.

[0118] In step S604, the color values ​​and volume density values ​​of multiple sampling points on the simulated light are determined, and the depth image and rendered image corresponding to the image data under the camera pose are determined based on the color values, volume density values ​​and position coordinates of the multiple sampling points.

[0119] In implementation, based on the principle of image rendering, at each sampling point on a simulated ray, the closer the sampling point is to the surface of the target object, the darker the color value and the greater the volume density value; conversely, the farther away the sampling point is from the surface of the target object, the lighter the color value (i.e., the closer to transparency) and the smaller the volume density value. Therefore, as... Figure 7 As shown, the computer device uses a pre-defined standard rendered image to perform supervised learning based on the relative distances between sampling points on each simulated ray and the surface of the target object, determining the color and volume density values ​​of multiple sampling points on each simulated ray. Then, based on the color and volume density values ​​of multiple sampling points and their position coordinates, the depth image and rendered image corresponding to the image data at each camera pose are determined. To make the rendered image output by the image rendering model increasingly closer to the standard rendered image, the image rendering model assigns increasingly larger weights to the sampling points on the target object's surface during this pre-training process. Consequently, after pre-training, the sampling points on the target object's surface can be accurately determined.

[0120] In this embodiment, the image rendering model is trained in a supervised manner based on the preset standard image rendering image and training data samples. During the pre-training process, the image rendering model can gradually increase the weight of the sampling points on the surface of the target object, thereby obtaining the accurate sampling points on the surface of the target object, and thus obtaining the depth image and rendering image corresponding to the image data under each camera pose.

[0121] In one exemplary embodiment, such as Figure 8 As shown, in step S604, determining the depth image and rendered image corresponding to the image data under the camera pose based on the color values, volume density values, and position coordinates of the multiple sampling points can be achieved through the following steps:

[0122] In step S802, the rendered image corresponding to the image data under the camera pose is determined based on the color value, volume density value, and preset integration algorithm of multiple sampling points corresponding to each simulated ray.

[0123] In practice, the computer equipment uses the color values ​​of multiple sampling points corresponding to each simulated ray generated from each image data to perform integral processing on the color values ​​using the volume density value, thereby determining the rendered image corresponding to the image data under the camera pose.

[0124] In step S804, the depth image corresponding to the image data under the camera pose is determined based on the position coordinates, volume density values, and preset integration algorithm of multiple sampling points corresponding to each simulated ray.

[0125] In practice, the computer equipment uses the volume density value to integrate the position coordinates of multiple sampling points corresponding to each simulated ray generated from each image data to determine the depth image corresponding to the image data under the camera pose.

[0126] In this embodiment, based on the preset integration algorithm and the color value, volume density value and position coordinate of each sampling point obtained during the pre-training process, the depth image and rendering image corresponding to the image data are determined, so as to construct point cloud data based on the depth image and complete the surface reconstruction of the target object.

[0127] In one exemplary embodiment, such as Figure 9 As shown, in step S120, the point cloud data of the target object is constructed based on the depth image obtained from the image rendering model that meets the pre-training completion conditions. This can be achieved through the following steps:

[0128] In step S902, based on the position coordinates of the pixels in each depth image obtained by the image rendering model that meets the pre-training completion conditions and the preset conversion algorithm, the pixels in each depth image are fused and converted to obtain the initial point cloud data.

[0129] In implementation, given an image rendering model that meets the pre-training completion conditions, depth images corresponding to each image data can be obtained based on the model. The data value of each pixel in each depth image represents the three-dimensional coordinates (x, y, 12) of the target object surface in the world coordinate system. Then, the computer device transforms the position coordinates of each pixel based on the position coordinates of each pixel and a preset transformation algorithm, and fuses the transformed position coordinates of each pixel in the depth images into three-dimensional space to obtain the initial point cloud data.

[0130] In step S904, the three-dimensional space corresponding to the initial point cloud data is divided into a preset number of grids according to a preset spatial partitioning strategy.

[0131] In implementation, the computer device divides the three-dimensional space corresponding to the initial point cloud data into a preset number of grids according to a preset spatial partitioning strategy. For example, the computer device divides the three-dimensional space into n*n (e.g., 500*500) small grids.

[0132] In step S906, the number of initial point cloud data in each grid is counted, and the initial point cloud data in grids with a number less than a preset threshold is eliminated to obtain the denoised point cloud data of the target object.

[0133] In practice, the computer device counts the number of initial point cloud data in each grid. If the number of initial point cloud data in a grid is greater than or equal to a preset threshold, the initial point cloud data in that grid is retained. If the number of initial point cloud data in a grid is less than the preset threshold, it indicates that the initial point cloud data in that grid is noisy data. The computer device then removes the initial point cloud data in that grid to obtain the denoised point cloud data of the target object.

[0134] In this embodiment, the position coordinates of pixels in each depth image data are fused and transformed to obtain initial point cloud data. Then, the initial point cloud data is denoised to obtain the denoised point cloud data of the target object. Each point in the point cloud data is a point on the surface of the target object after processing, so that the surface of the target object can be reconstructed based on the point cloud data.

[0135] In one exemplary embodiment, an image rendering method is provided, the method comprising:

[0136] The target camera pose is input into the image rendering model that has been trained for the target object. Through the processing of the trained image rendering model, the rendered image of the target object from the shooting perspective corresponding to the target camera pose is output.

[0137] In implementation, the computer device inputs the target camera pose into a trained image rendering model for the target object. The trained image rendering model processes the target camera pose and outputs a rendered image of the target object from the shooting perspective corresponding to that target camera pose. For example... Figure 10 The image shown displays rendered images of a target object (e.g., a chair) from various shooting perspectives. The data processing of the trained image rendering model based on the target camera pose is similar to the combination of pre-training and final training processes of the image rendering model. Specifically, the image rendering model generates simulated light rays based on the target camera pose, then determines each target sampling point, calculates the color and volume density values ​​of each target sampling point, and obtains the rendered image of the target object from the shooting perspective corresponding to that target camera pose.

[0138] In this embodiment, the image rendering model is trained using the above-described image rendering model training method, which improves the training efficiency of the image rendering model. Then, based on the trained image rendering model, the target object is rendered, and the rendered image of the target object under any shooting angle corresponding to any camera pose can be obtained.

[0139] It should be understood that, although Figure 1 , Figures 4-6 , Figure 8 and Figure 9The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 , Figures 4-6 , Figure 8 and Figure 9 At least some of the steps in the process may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.

[0140] It is understood that the same / similar parts between the various embodiments of the methods described above in this specification can be referred to each other. Each embodiment focuses on the differences from other embodiments, and relevant parts can be referred to the description of other method embodiments.

[0141] Figure 11 This is a block diagram of a training apparatus for an image rendering model according to an exemplary embodiment. (Refer to...) Figure 11 The device includes a pre-training unit 1102, a construction unit 1104, a reconstruction unit 1106, and a training unit 1108.

[0142] The pre-training unit 1102 is configured to acquire training data samples for the target object and pre-train the image rendering model based on the training data samples to obtain an image rendering model that meets the pre-training completion conditions.

[0143] The building unit 1104 is configured to construct point cloud data of the target object based on the depth image corresponding to the target object obtained from the image rendering model that meets the pre-training completion conditions.

[0144] The reconstruction unit 1106 is configured to perform surface reconstruction on the target object based on the point cloud data of the target object, obtain the surface reconstruction result of the target object, and determine the target sampling point corresponding to the target object based on the surface reconstruction result of the target object. The target sampling point is the intersection of the surface reconstruction result of the target object and a preset simulated ray.

[0145] The training unit 1108 is configured to train an image rendering model that meets the pre-training completion conditions based on the target sampling points, thereby obtaining a trained image rendering model.

[0146] In one exemplary embodiment, the pre-training unit 1102 includes:

[0147] The acquisition subunit is configured to acquire a video segment containing a target object; the video segment contains initial image data from multiple preset shooting angles;

[0148] The frame extraction subunit is configured to perform frame extraction processing on the video segment to obtain an image sequence containing initial image data of multiple target initial video frames;

[0149] The first processing subunit is configured to perform background removal processing on each initial image data contained in the image sequence according to a pre-trained object segmentation model to obtain background-removed image data.

[0150] The construction subunit is configured to determine the camera pose corresponding to each of the background-removed image data, and to obtain training data samples of the target object based on each of the background-removed image data and the camera pose corresponding to each of the background-removed image data.

[0151] In one exemplary embodiment, the pre-training unit 1102 includes:

[0152] The second processing subunit is configured to input the training data samples into an image rendering model, process each image data in the training data samples through the image rendering model, and output a rendered image corresponding to each image data.

[0153] The computation subunit is configured to perform loss calculations based on each preset standard rendered image and each of the rendered images to obtain the loss result corresponding to the training data sample;

[0154] A subunit is determined and configured to execute an image rendering model that satisfies the pre-training completion condition when the loss result and the number of model training iterations meet the pre-training completion condition.

[0155] In an exemplary embodiment, the second processing subunit is specifically used to input the training data samples into the image rendering model, generate multiple simulated rays corresponding to the image data under the camera pose based on the camera pose of each image data in the training data samples, and perform data sampling in the ray direction of each simulated ray to obtain multiple sampling points corresponding to the simulated ray.

[0156] The color values ​​and volume density values ​​of multiple sampling points on the simulated light are determined, and the depth image and rendered image corresponding to the image data under the camera pose are determined based on the color values, volume density values ​​and position coordinates of the multiple sampling points.

[0157] In an exemplary embodiment, the second processing subunit is specifically used to determine the rendered image corresponding to the image data under the camera pose based on the color value, volume density value, and preset integration algorithm of multiple sampling points corresponding to each of the simulated light rays;

[0158] Based on the position coordinates, volume density values, and preset integration algorithm of multiple sampling points corresponding to each simulated ray, the depth image corresponding to the image data under the camera pose is determined.

[0159] In one exemplary embodiment, the building unit 1104 includes:

[0160] The fusion and transformation unit is configured to execute the position coordinates of the pixels contained in each depth image obtained based on the image rendering model that meets the pre-training completion conditions and a preset transformation algorithm to fuse and transform the pixels contained in each depth image to obtain initial point cloud data.

[0161] The partitioning unit is configured to divide the three-dimensional space corresponding to the initial point cloud data into a preset number of grids according to a preset spatial partitioning strategy;

[0162] The denoising unit is configured to perform statistical analysis on the number of initial point cloud data in each grid and eliminate the initial point cloud data in grids with a number less than a preset threshold, thereby obtaining the denoised point cloud data of the target object.

[0163] In one exemplary embodiment, an image rendering apparatus is provided. The apparatus includes:

[0164] The rendering unit is configured to input the target camera pose into a trained image rendering model for the target object, and, through processing by the trained image rendering model, output a rendered image of the target object from the shooting perspective corresponding to the target camera pose. The training process of the trained image rendering model includes the execution process of the aforementioned image rendering model training method, which will not be elaborated further in this embodiment.

[0165] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0166] Figure 12 This is a block diagram of an electronic device 1200 for a training method of an image rendering model, according to an exemplary embodiment. For example, the electronic device 1200 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0167] Reference Figure 12 The electronic device 1200 may include one or more of the following components: processing component 1202, memory 1204, power supply component 1206, multimedia component 1208, audio component 1210, input / output (I / O) interface 1212, sensor component 1214, and communication component 1216.

[0168] Processing component 1202 typically controls the overall operation of electronic device 1200, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 1202 may include one or more processors 1220 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 1202 may include one or more modules to facilitate interaction between processing component 1202 and other components. For example, processing component 1202 may include a multimedia module to facilitate interaction between multimedia component 1208 and processing component 1202.

[0169] Memory 1204 is configured to store various types of data to support the operation of electronic device 1200. Examples of such data include instructions for any application or method operating on electronic device 1200, contact data, phonebook data, messages, pictures, videos, etc. Memory 1204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, optical disk, or graphene storage.

[0170] Power supply component 1206 provides power to various components of electronic device 1200. Power supply component 1206 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 1200.

[0171] Multimedia component 1208 includes a screen that provides an output interface between the electronic device 1200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 1208 includes a front-facing camera and / or a rear-facing camera. When the electronic device 1200 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0172] Audio component 1210 is configured to output and / or input audio signals. For example, audio component 1210 includes a microphone (MIC) configured to receive external audio signals when electronic device 1200 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 1204 or transmitted via communication component 1216. In some embodiments, audio component 1210 also includes a speaker for outputting audio signals.

[0173] I / O interface 1212 provides an interface between processing component 1202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0174] Sensor assembly 1214 includes one or more sensors for providing state assessments of various aspects of electronic device 1200. For example, sensor assembly 1214 may detect the on / off state of electronic device 1200, the relative positioning of components such as the display and keypad of electronic device 1200, changes in position of electronic device 1200 or its components, the presence or absence of user contact with electronic device 1200, orientation or acceleration / deceleration of device 1200, and temperature changes of electronic device 1200. Sensor assembly 1214 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 1214 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 1214 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0175] Communication component 1216 is configured to facilitate wired or wireless communication between electronic device 1200 and other devices. Electronic device 1200 can access wireless networks based on communication standards, such as WiFi, carrier networks (such as 2G, 3G, 4G, or 5G), or combinations thereof. In one exemplary embodiment, communication component 1216 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 1216 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0176] In an exemplary embodiment, the electronic device 1200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0177] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 1204 including instructions, which can be executed by a processor 1220 of an electronic device 1200 to perform the above-described method. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0178] In an exemplary embodiment, a computer program product is also provided, the computer program product including instructions that can be executed by a processor 1220 of an electronic device 1200 to perform the above-described method.

[0179] It should be noted that the above-mentioned apparatus, electronic equipment, computer-readable storage medium, computer program product, etc., may also include other implementation methods according to the description of the method embodiments. For specific implementation methods, please refer to the description of the relevant method embodiments, which will not be elaborated here.

[0180] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0181] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A training method for an image rendering model, characterized in that, The method includes: Obtain training data samples for the target object, and pre-train the image rendering model based on the training data samples to obtain an image rendering model that meets the pre-training completion conditions. Based on the image rendering model that meets the pre-training completion conditions, the depth image corresponding to the target object is obtained, and the point cloud data of the target object is constructed. The surface of the target object is reconstructed based on the point cloud data of the target object to obtain the surface reconstruction result of the target object, and the target sampling point corresponding to the target object is determined based on the surface reconstruction result of the target object; the target sampling point is the intersection of the surface reconstruction result of the target object and the preset simulated light ray; Based on the target sampling points, the image rendering model that meets the pre-training completion conditions is trained to obtain the trained image rendering model.

2. The training method for the image rendering model according to claim 1, characterized in that, The process of obtaining training data samples for the target object includes: Acquire a video segment containing the target object; the video segment contains initial image data from multiple preset shooting angles; The video segment is subjected to frame extraction processing to obtain an image sequence containing initial image data of multiple target initial video frames; The background is removed from each initial image data in the image sequence based on a pre-trained object segmentation model to obtain background-removed image data. Determine the camera pose corresponding to each background-removed image data, and obtain training data samples of the target object based on each background-removed image data and the camera pose corresponding to each background-removed image data.

3. The training method for the image rendering model according to claim 1, characterized in that, The step of pre-training the image rendering model based on the training data samples to obtain an image rendering model that meets the pre-training completion conditions includes: The training data samples are input into the image rendering model, and the image rendering model processes each image data in the training data samples to output the rendered image corresponding to each image data. Loss calculations are performed on each preset standard rendered image and each rendered image to obtain the loss result corresponding to the training data sample; When the loss result and the number of model training iterations meet the pre-training completion conditions, an image rendering model that meets the pre-training completion conditions is obtained.

4. The training method for the image rendering model according to claim 3, characterized in that, The step of inputting the training data samples into an image rendering model, processing each image data in the training data samples through the image rendering model, and outputting a depth image and a rendered image corresponding to each image data includes: The training data samples are input into the image rendering model. Based on the camera pose corresponding to each image data in the training data samples, multiple simulated rays corresponding to the image data under the camera pose are generated. Data is sampled in the ray direction of each simulated ray to obtain multiple sampling points corresponding to the simulated ray. The color values ​​and volume density values ​​of multiple sampling points on the simulated light are determined, and the depth image and rendered image corresponding to the image data under the camera pose are determined based on the color values, volume density values ​​and position coordinates of the multiple sampling points.

5. The training method for the image rendering model according to claim 4, characterized in that, The step of determining the depth image and rendered image corresponding to the image data under the camera pose based on the color value, volume density value, and position coordinates of the multiple sampling points includes: Based on the color value, volume density value, and preset integration algorithm of multiple sampling points corresponding to each simulated light ray, the rendered image corresponding to the image data under the camera pose is determined; Based on the position coordinates, volume density values, and preset integration algorithm of multiple sampling points corresponding to each simulated ray, the depth image corresponding to the image data under the camera pose is determined.

6. The training method for the image rendering model according to claim 1, characterized in that, The step of constructing point cloud data of the target object based on the depth image obtained from the image rendering model that meets the pre-training completion conditions includes: Based on the position coordinates of the pixels contained in each depth image obtained by the image rendering model that meets the pre-training completion conditions and the preset conversion algorithm, the pixels contained in each depth image are fused and converted to obtain the initial point cloud data. According to the preset spatial partitioning strategy, the three-dimensional space corresponding to the initial point cloud data is divided into a preset number of grids; The number of initial point cloud data in each grid is counted, and the initial point cloud data in grids with a number less than a preset threshold is eliminated to obtain the denoised point cloud data of the target object.

7. An image rendering method, characterized in that, The method includes: The target camera pose is input into the image rendering model that has been trained for the target object. Through the processing of the trained image rendering model, a rendered image of the target object under the shooting angle corresponding to the target camera pose is output. The training process of the trained image rendering model includes performing the training method of the image rendering model as described in any one of claims 1 to 6.

8. A training device for an image rendering model, characterized in that, The device includes: The pre-training unit is configured to acquire training data samples for the target object and pre-train the image rendering model based on the training data samples to obtain an image rendering model that meets the pre-training completion conditions. The construction unit is configured to execute an image rendering model that satisfies the pre-training completion condition to obtain a depth image corresponding to the target object, and to construct the point cloud data of the target object; The reconstruction unit is configured to perform surface reconstruction of the target object based on the point cloud data of the target object, obtain the surface reconstruction result of the target object, and determine the target sampling point corresponding to the target object based on the surface reconstruction result of the target object; the target sampling point is the intersection of the surface reconstruction result of the target object and a preset simulated ray. The training unit is configured to train the image rendering model that meets the pre-training completion conditions based on the target sampling points, so as to obtain the trained image rendering model.

9. An image rendering apparatus, characterized in that, The device includes: The rendering unit is configured to input the target camera pose into a trained image rendering model for the target object, and through the processing of the trained image rendering model, output a rendered image of the target object from the shooting angle corresponding to the target camera pose; the training process of the trained image rendering model includes performing the training method of the image rendering model as described in any one of claims 1 to 6.

10. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the training method of the image rendering model as described in any one of claims 1 to 6, and the image rendering method as described in claim 7.

11. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the training method of the image rendering model as claimed in any one of claims 1 to 6, and the image rendering method as claimed in claim 7.

Images