A method for generating a light field prediction model and related apparatus

Abstract

A method for generating a light field prediction model and a related device, the method comprising: establishing a cube model surrounding a scene to be photographed according to photographing orientations of a plurality of sample images, wherein the cube model comprises a plurality of small cubes voxels; calculating a plurality of truncated distances of each small cube in the plurality of small cubes according to the plurality of sample images respectively (S402); sampling a space point from the small cube according to the plurality of truncated distances of each small cube, wherein each space point sampled corresponds to a spatial coordinate; and training a light field prediction model according to the spatial coordinates of the space points sampled, wherein the light field prediction model is used to predict a light field of the scene. The method can improve the sampling efficiency of the voxel, thereby improving the generation efficiency of the light field prediction model.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method and apparatus for generating a light field prediction model. Background Technology

[0002] The prospects for virtual reality display technology are very broad. From cardboxes, Vive VR headsets, and Oculus Rift headsets to VR Glass launched in 2019, virtual display hardware devices are becoming simpler, easier to use, and more widespread. However, unlike the rapid improvement of virtual display hardware devices, high-quality virtual reality digital content is very limited. Unlike traditional two-dimensional (2D) digital content, to enhance the immersive experience (such as the displayed content changing with the viewer's movement), virtual reality content requires a three-dimensional light field of the scene. Capturing the three-dimensional light field of a scene requires very complex hardware, limiting the flexibility of acquiring the three-dimensional light field. Therefore, using computer vision algorithms to acquire three-dimensional light fields has become a new research direction. However, how to efficiently and accurately acquire three-dimensional light fields based on computer vision algorithms is a technical problem faced by those skilled in the art. Summary of the Invention

[0003] This application discloses a method and related apparatus for generating a light field prediction model, which can improve the generation efficiency of the light field prediction model.

[0004] In a first aspect, embodiments of this application disclose a method for generating a light field prediction model. The method includes: establishing a cube model surrounding a scene based on the shooting orientations of multiple sample images, wherein the cube model includes multiple small cubes (voxels); then, calculating multiple cutoff distances for each of the multiple small cubes based on the multiple sample images, wherein a cutoff distance for each small cube calculated based on a first sample image includes: determining the cutoff distance based on the distance from the camera to each small cube when the first sample image is captured and the distance from the camera to the surface of an object in the scene, wherein the first sample image is any one of the multiple sample images; then, sampling spatial points from the small cubes based on the multiple cutoff distances for each small cube, wherein each sampled spatial point corresponds to spatial coordinates; and then, training a light field prediction model based on the spatial coordinates of the sampled spatial points, wherein the light field prediction model is used to predict the light field of the scene.

[0005] In the above method, the voxel sampling points used to train the deep learning network (also known as the light field prediction model, used to predict 3D light fields) are obtained based on the depth information of the image. Specifically, a cutoff distance is calculated based on the depth information and the distance from the voxel to the camera. Then, differential sampling is performed according to the size of the cutoff distance. On the one hand, this sampling method can quickly concentrate the sampling on key areas, improving sampling efficiency. On the other hand, the voxels sampled by this sampling method are basically concentrated near the object surface. Therefore, the deep learning network trained based on these voxels can better represent the texture details of the object when performing image prediction, and can reduce blur and structural errors.

[0006] In conjunction with the first aspect, in one optional embodiment of the first aspect, after training the light field prediction model based on the spatial coordinates of the sampled spatial points, the method further includes: predicting the light field of the scene using the light field prediction model. That is, after training a pre-stored light field model, the model training device will also predict the light field using the light field prediction model.

[0007] In conjunction with the first aspect or any of the above-described possible implementations of the first aspect, in another optional scheme of the first aspect, the step of sampling spatial points from the small cubes based on multiple cutoff distances of each small cube includes: performing a fusion calculation on the multiple cutoff distances of each small cube to obtain a fused cutoff distance for each small cube; and sampling spatial points from each small cube based on the fused cutoff distance of each small cube. In this implementation, the fused cutoff distance is calculated for each small cube, and then sampling is performed based on the fused cutoff distance.

[0008] In conjunction with the first aspect or any of the above possible implementations of the first aspect, in another optional scheme of the first aspect, the step of sampling spatial points from the small cubes based on multiple cutoff distances of each small cube includes: determining at least one first small cube among the small cubes whose absolute value of cutoff distance is less than a preset threshold; performing a fusion calculation on the multiple cutoff distances of the first small cube to obtain a fused cutoff distance of the first small cube; and sampling spatial points from the first small cube based on the fused cutoff distance of the first small cube. In this approach, the fused cutoff distance is not calculated for all small cubes, but only for small cubes whose absolute value of cutoff distance is less than the preset threshold. This is because when the cutoff distance is large, the corresponding small cube is farther from the object surface, and the necessity for subsequent sampling of it is smaller. Therefore, this application does not perform a fusion calculation on the cutoff distances of such small cubes, which is equivalent to excluding these small cubes from the sampling scope in advance. This reduces the computational load and improves the generation efficiency of the light field prediction model without significantly reducing the subsequent sampling effect.

[0009] In a further alternative scheme of the first aspect, combining the first aspect or any of the possible implementations described above, during the sampling of spatial points from the small cubes, more spatial points are sampled from the small cubes with smaller fusion cutoff distances. It can be understood that cubes with smaller fusion cutoff distances are closer to the object surface, and the spatial points in these small cubes better represent pixel information compared to spatial points in other small cubes on the same camera ray. Therefore, training the light field prediction model based more on the spatial points in these small cubes is beneficial for the light field prediction model to predict more accurate images subsequently.

[0010] In conjunction with the first aspect or any of the possible implementations of the first aspect, in another optional scheme of the first aspect, the fusion calculation includes a weighted average calculation. It can be understood that the fusion cutoff distance obtained through weighted average calculation can more accurately reflect the distance from the small cube to the object surface.

[0011] In combination with the first aspect or any of the above possible implementations of the first aspect, in another optional scheme of the first aspect, the weight value of the cutoff distance of the second small cube calculated based on the first sample image in the weighted average calculation is negatively correlated with the distance from the second small cube to the camera when the first sample image was captured, and / or positively correlated with the first included angle, where the first included angle is the angle between the field of view line of the camera ray where the second small cube is located and the normal vector of the surface of the object closest to the second small cube, and the second small cube is any small cube in the cube model.

[0012] In this approach, a weight value calculated based on each sample image is used when calculating the fusion cutoff distance. Since this weight value is negatively correlated with the distance to the camera at that time and / or positively correlated with the first included angle, when this weight value is incorporated into the calculation of the fusion cutoff distance, it can more accurately reflect the influence of different orientations on the fusion cutoff distance.

[0013] In conjunction with the first aspect or any of the above possible implementations of the first aspect, in another optional scheme of the first aspect, before calculating the cutoff distance of each small cube in the cube model based on the multiple sample images, the method further includes: when capturing each sample image in the multiple sample images, acquiring depth information from the shooting perspective of each sample image, wherein the depth information is used to characterize the distance from the camera to the surface of the object in the scene.

[0014] In this approach, depth information is collected from the shooting perspective of each sample image, which can more accurately reflect the distance from the camera to the surface of the object in the scene being photographed, thus facilitating the calculation of a more accurate cutoff distance.

[0015] In conjunction with the first aspect or any of the above possible implementations of the first aspect, in another optional scheme of the first aspect, during the calculation of the fusion cutoff distance of the second small cube, the weight value of the cutoff distance of the second small cube calculated based on the first sample image in the weighted average calculation can also be called the weight value w(p) of the second small cube calculated based on the first sample image. This weight value w(p) satisfies the following relationship:

[0016] w(p) = cos(θ) / distance(v)

[0017] Where θ is the first included angle, and distance(v) is the distance from the second small cube to the camera when the first sample image is captured.

[0018] As mentioned earlier, the weight value of the second small cube calculated based on the first sample image is negatively correlated with the distance from the second small cube to the camera when the first sample image was captured, and / or positively correlated with the first included angle. The expression of w(p) here is an optional expression of this idea. Therefore, when this weight value is incorporated into the calculation of the fusion cutoff distance, it can more accurately reflect the influence of different orientations on the fusion cutoff distance.

[0019] In combination with the first aspect or any of the above possible implementations of the first aspect, in another optional scheme of the first aspect, the cutoff distance d(p) of the second small cube calculated based on the first sample image satisfies the following relationship:

[0020] d(p)=sdf(p) / |u|

[0021] Where sdf(p) is the distance from the camera to the first small cube when the first sample image is captured, and the difference between the distance from the camera to the surface of the object in the scene, and u is a preset threshold.

[0022] It is understandable that d(p) here is only one optional formula for calculating the cutoff distance, and there are other ways to express it in practical applications.

[0023] In combination with the first aspect or any of the above possible implementations of the first aspect, in another alternative scheme of the first aspect, if sdf(p) > |u|, then d(p) = 1; if sdf(p) < 0 and |sdf(p)| > |u|, then d(p) = -1.

[0024] In this approach, the cutoff distance of small cubes within one range is assigned a value of 1, while the cutoff distance of small cubes within another range is assigned a value of -1. This facilitates the subsequent application of the same processing to both types of small cubes, thereby improving computational efficiency.

[0025] Secondly, embodiments of this application provide an apparatus for generating a light field prediction model, the apparatus comprising:

[0026] A building unit is used to build a cube model surrounding the scene being photographed based on the shooting positions of multiple sample images, wherein the cube model includes multiple small cubes (voxels).

[0027] The first calculation unit is used to calculate multiple cutoff distances for each of the multiple small cubes based on the multiple sample images. The cutoff distance for each small cube calculated based on the first sample image includes: determining the cutoff distance based on the distance from the camera to each small cube when the first sample image is captured and the distance from the camera to the surface of an object in the scene. The first sample image is any one of the multiple sample images.

[0028] A sampling unit is used to sample spatial points from the small cube based on multiple cutoff distances of each small cube, wherein each sampled spatial point corresponds to spatial coordinates;

[0029] The second computing unit is used to train a light field prediction model based on the spatial coordinates of the sampled spatial points, wherein the light field prediction model is used to predict the light field of the scene.

[0030] In the aforementioned device, the voxel sampling points used to train the deep learning network (also known as the light field prediction model, used to predict 3D light fields) are obtained based on the depth information of the image. Specifically, a cutoff distance is calculated based on the depth information and the distance from the voxel to the camera. Then, differential sampling is performed according to the size of the cutoff distance. On the one hand, this sampling method can quickly concentrate the sampling on key areas, improving sampling efficiency. On the other hand, the voxels sampled by this method are basically concentrated near the object surface. Therefore, when the deep learning network trained based on these voxels performs image prediction, it can better represent the texture details of the object and reduce blur and structural errors.

[0031] In conjunction with the second aspect, in one alternative embodiment of the second aspect, the apparatus further includes:

[0032] The prediction unit is used to predict the light field of the scene using the light field prediction model. That is, after training the pre-stored light field model, the model training device will also predict the light field using the light field prediction model.

[0033] In conjunction with the second aspect or any of the above possible implementations of the second aspect, in another optional scheme of the second aspect, in sampling spatial points from the small cubes based on multiple cutoff distances of each small cube, the sampling unit is specifically used for: performing a fusion calculation on the multiple cutoff distances of each small cube to obtain a fused cutoff distance for each small cube; and sampling spatial points from each small cube based on the fused cutoff distance. In this implementation, the fused cutoff distance is calculated for each small cube, and then sampling is performed based on the fused cutoff distance.

[0034] In conjunction with the second aspect or any of the possible implementations of the second aspect, in another optional scheme of the second aspect, in sampling spatial points from the small cubes based on multiple cutoff distances of each small cube, the sampling unit is specifically used to: determine a first small cube in each small cube whose absolute value of cutoff distance is less than a preset threshold; perform a fusion calculation on the multiple cutoff distances of the first small cube to obtain a fused cutoff distance of the first small cube; and sample spatial points from the first small cube based on the fused cutoff distance of the first small cube. In this method, the fused cutoff distance is not calculated for all small cubes, but only for small cubes whose absolute value of cutoff distance is less than the preset threshold. This is because when the cutoff distance is large, the corresponding small cube is farther from the object surface, and the necessity for subsequent sampling of it is smaller. Therefore, this application does not perform a fusion calculation on the cutoff distances of such small cubes, which is equivalent to excluding these small cubes from the sampling scope in advance. This reduces the computational load and improves the generation efficiency of the light field prediction model without significantly reducing the subsequent sampling effect.

[0035] In a further alternative scheme of the second aspect, combining the second aspect or any of the possible implementations described above, during the sampling of spatial points from the small cubes, more spatial points are sampled from the small cubes with smaller fusion cutoff distances. It can be understood that cubes with smaller fusion cutoff distances are closer to the object surface, and the spatial points in these small cubes better represent pixel information compared to spatial points in other small cubes on the same camera ray. Therefore, training the light field prediction model based more on the spatial points in these small cubes is beneficial for the light field prediction model to predict more accurate images subsequently.

[0036] In conjunction with the second aspect or any of the possible implementations of the second aspect, in another optional scheme of the second aspect, the fusion calculation includes a weighted average calculation. It can be understood that the fusion cutoff distance obtained through weighted average calculation can more accurately reflect the distance from the small cube to the object surface.

[0037] In combination with the second aspect or any of the above possible implementations of the second aspect, in another optional scheme of the second aspect, the weight value of the cutoff distance of the second small cube calculated based on the first sample image in the weighted average calculation is negatively correlated with the distance from the second small cube to the camera when the first sample image was captured, and / or positively correlated with the first included angle, where the first included angle is the angle between the field of view line of the camera ray where the second small cube is located and the normal vector of the surface of the object closest to the second small cube, and the second small cube is any small cube in the cube model.

[0038] In this approach, a weight value calculated based on each sample image is used when calculating the fusion cutoff distance. Since this weight value is negatively correlated with the distance to the camera at that time and / or positively correlated with the first included angle, when this weight value is incorporated into the calculation of the fusion cutoff distance, it can more accurately reflect the influence of different orientations on the fusion cutoff distance.

[0039] In conjunction with the second aspect or any of the possible implementations of the second aspect, in yet another alternative embodiment of the second aspect, the apparatus further includes:

[0040] The acquisition unit is used to acquire depth information from the shooting perspective of each of the multiple sample images when capturing each sample image, wherein the depth information is used to characterize the distance from the camera to the surface of an object in the scene.

[0041] In this approach, depth information is collected from the shooting perspective of each sample image, which can more accurately reflect the distance from the camera to the surface of the object in the scene being photographed, thus facilitating the calculation of a more accurate cutoff distance.

[0042] In conjunction with the second aspect or any of the above possible implementations of the second aspect, in another optional scheme of the second aspect, during the calculation of the fusion cutoff distance of the second small cube, the weight value of the cutoff distance calculated based on the first sample image in the weighted average calculation can also be called the weight value w(p) of the second small cube calculated based on the first sample image. This weight value w(p) satisfies the following relationship:

[0043] w(p) = cos(θ) / distance(v)

[0044] Where θ is the first included angle, and distance(v) is the distance from the second small cube to the camera when the first sample image is captured.

[0045] As mentioned earlier, the weight value of the second small cube calculated based on the first sample image is negatively correlated with the distance from the second small cube to the camera when the first sample image was captured, and / or positively correlated with the first included angle. The expression of w(p) here is an optional expression of this idea. Therefore, when this weight value is incorporated into the calculation of the fusion cutoff distance, it can more accurately reflect the influence of different orientations on the fusion cutoff distance.

[0046] In combination with the second aspect or any of the above possible implementations of the second aspect, in another optional scheme of the second aspect, the cutoff distance d(p) of the second small cube calculated based on the first sample image satisfies the following relationship:

[0047] d(p)=sdf(p) / |u|

[0048] Where sdf(p) is the distance from the camera to the first small cube when the first sample image is captured, and the difference between the distance from the camera to the surface of the object in the scene, and u is a preset threshold.

[0049] It is understandable that d(p) here is only one optional formula for calculating the cutoff distance, and there are other ways to express it in practical applications.

[0050] In combination with the second aspect or any of the above possible implementations of the second aspect, in another alternative scheme of the second aspect, if sdf(p) > |u|, then d(p) = 1; if sdf(p) < 0 and |sdf(p)| > |u|, then d(p) = -1.

[0051] In this approach, the cutoff distance of small cubes within one range is assigned a value of 1, while the cutoff distance of small cubes within another range is assigned a value of -1. This facilitates the subsequent application of the same processing to both types of small cubes, thereby improving computational efficiency.

[0052] Thirdly, embodiments of this application provide a device for generating a light field prediction model, including a processor and a memory, wherein the memory is used to store a computer program, and the computer program, when run on the processor, implements the method described in the first aspect or any optional scheme of the first aspect.

[0053] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed on a processor, implements the method described in the first aspect or any of the optional solutions of the first aspect.

[0054] By implementing the embodiments of this application, the voxel sampling points used to train the deep learning network (also known as the light field prediction model, used to predict three-dimensional light fields) are obtained based on the depth information of the image. Specifically, a cutoff distance is calculated based on the depth information and the distance from the voxel to the camera. Then, differential sampling is performed according to the size of the cutoff distance. On the one hand, this sampling method can quickly concentrate the sampling on key areas, improving sampling efficiency. On the other hand, the voxels sampled by this sampling method are basically concentrated near the surface of the object. Therefore, the deep learning network trained based on these voxels can better represent the texture details of the object when performing image prediction, and can reduce blur and structural errors. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of a scenario for obtaining NeRF provided in an embodiment of this application;

[0056] Figure 2A This is a schematic diagram of a sampling voxel scenario provided in an embodiment of this application;

[0057] Figure 2B This is a schematic diagram illustrating how three-dimensional light field and RGB information change with depth information, provided in an embodiment of this application.

[0058] Figure 3 This is a schematic diagram of a model training architecture provided in an embodiment of this application;

[0059] Figure 4 This is a flowchart illustrating a method for determining a three-dimensional light field in a scene, as provided in an embodiment of this application.

[0060] Figure 5 This is a schematic diagram illustrating the distance from the camera to the voxel and the object surface, provided in an embodiment of this application.

[0061] Figure 6 This is a schematic diagram of a scenario involving a cutoff distance provided in an embodiment of this application;

[0062] Figure 7 This is a schematic diagram of the distribution of cutoff distance provided in an embodiment of this application;

[0063] Figure 8 This is a comparative schematic diagram of the prediction effect of a light field prediction model provided in the embodiments of this application;

[0064] Figure 9 This is a schematic diagram of the structure of a device for generating a light field prediction model according to an embodiment of this application;

[0065] Figure 10This is a schematic diagram of the structure of another light field prediction model generation device provided in the embodiments of this application. Detailed Implementation

[0066] The embodiments of this application are described below with reference to the accompanying drawings.

[0067] Please see Figure 1 , Figure 1 This is a schematic diagram of a scenario for acquiring learning Neural Radiance Fields (NeRF). Figure 1 The method shown employs a sparse image dataset to synthesize the 3D light field of a complex scene. Specifically, as in part (a), for a scene represented by a five-dimensional (5D) coordinate system, a single five-dimensional (5D) coordinate on the camera ray is used. The input is fed into a fully connected deep learning network, where the coordinates... It includes spatial position (x, y, z) and viewpoint direction. Information such as (b) shows the coordinates reconstructed (i.e., output) by the deep learning network. The corresponding RGB information can be represented as RGBσ, including density and color. As shown in part (c), after volume rendering on RGBσ, it is then compared with coordinates... The actual RGB information at the location is compared to derive the rendering loss, as shown in part (d), and the deep learning network is then trained based on this rendering loss. After training the deep learning network using the 5D coordinates of the spatial points collected on the camera ray according to the above steps (a), (b), (c), and (d), the deep learning network can predict the RGB information of the new 5D coordinates. Therefore, for a scene represented by a 5D coordinate system, the deep learning network can predict the view of the scene from any angle, and the set of all viewpoints of the scene is the three-dimensional light field of the scene.

[0068] The inventors of this application are in the process of... Figure 1 After analyzing the operating principle of the deep learning network used in the scene shown, it was found that for a certain field of view (camera ray), if it passes through a point on the object's surface, the RGB information (or pixels) corresponding to that camera ray mainly represents the depth and color of the point on the object's surface. Under this premise, in the process of training the aforementioned deep learning network to obtain the three-dimensional light field, it first needs to uniformly sample from the camera ray to obtain sampling points, such as... Figure 2AAs shown, parts (e) and (f) are both uniform sampling, with part (f) having a smaller sampling granularity. Then, each sampling point is trained and analyzed to obtain sampling points near the object's surface. Based on these sampling points near the object's surface, the approximate range of the "object's surface" can be determined. Therefore, based on this approximate range, further training and analysis are performed to obtain sampling points on the object's surface. Since the depth and color of the sampling points on the object's surface can basically reflect the RGB information (or pixels) corresponding to the camera ray, a deep learning network (or light field prediction model) can be trained based on the sampling points on the object's surface to predict the 3D light field. Figure 2B As shown, the horizontal axis represents the change in image depth information along a camera ray, and the vertical axis represents the change in the 3D light field and the change in RGB information along the ray with depth. Therefore, the weights and RGB information of the deep learning network used to predict the 3D light field have a high degree of overlap in terms of being affected by depth. Since depth information reflects the distance from the camera to the object surface, it can be considered that the weights and RGB information of the deep learning network used to predict the 3D light field have a high degree of overlap in terms of being affected by the object surface.

[0069] In this process, since the deep learning network samples uniformly from the camera ray, for each sampling point on the camera ray, it is necessary to find the sampling points near the object's surface and the sampling points on the object's surface through trial and error. This trial-and-error approach results in a very high computational burden on the deep learning network, leading to a slow convergence speed. Furthermore, this trial-and-error approach cannot accurately locate the sampling points on the camera ray that are located on the object's surface. Therefore, the deep learning network trained based on such sampling points has low accuracy, resulting in a relatively large error in the 3D light field predicted by the deep learning network.

[0070] The inventors of this application believe that depth information reflects the distance from the camera to the object's surface. Therefore, incorporating depth information into the entire deep learning network can reduce ineffective computations and improve the convergence speed and efficiency of the deep learning network. Specifically, this depth information is used for focused sampling and training near the depth values ​​on each camera ray. This ensures that the deep learning network quickly converges to the object's surface on the camera ray during the initial training phase, and concentrates computational power to represent the object's texture details, avoiding blurring and structural errors.

[0071] Please see Figure 3 , Figure 3This is a schematic diagram of a model training architecture provided in an embodiment of this application. The architecture includes a model training device 301 and one or more model-using devices 302. The model training device 301 and the model-using devices 302 communicate via wired or wireless means. Therefore, the model training device 301 can send the trained deep learning network (or light field prediction model) for predicting 3D light fields to the model-using devices 302. Correspondingly, the model-using devices 302 use the received deep learning network to predict the 3D light field in a specific scene. Alternatively, the model training device itself may predict the 3D light field in the scene based on the trained deep learning network.

[0072] Optionally, the model-using device 302 can feed back the prediction results based on the model to the model training device 301, so that the model training device 301 can further train the model based on the prediction results of the model-using device 302; the retrained model can be sent to the model-using device 302 to update the original model.

[0073] The model training device 301 can be a device with strong computing power, such as a server or a server cluster consisting of multiple servers.

[0074] The model uses device 302 as a device that needs to acquire the three-dimensional light field of a specific scene, such as handheld devices (e.g., mobile phones, tablets, PDAs, etc.), in-vehicle devices (e.g., cars, bicycles, electric vehicles, airplanes, ships, etc.), wearable devices (e.g., smartwatches (such as iWatch, smart bracelets, pedometers, etc.), smart home devices (e.g., refrigerators, televisions, air conditioners, electricity meters, etc.), smart robots, workshop equipment, etc.

[0075] The following examples illustrate this using the device 302 as a car and a mobile phone.

[0076] For example, with economic development and the continuous increase in the number of cars worldwide, navigation maps play a crucial role in improving the efficiency of car traffic on roads. On some complex road surfaces, users often find it difficult to obtain comprehensive information about the road surface. However, the method of this application embodiment can predict the three-dimensional light field of a specific scene, thus presenting the effect of viewing complex road surfaces from various angles to the user, which is beneficial for the user to make driving control accordingly and improve the efficiency of car traffic.

[0077] For example, online shopping is very common nowadays. Consumers can learn about the product form of an item by looking at photos of it online. However, many product photos are currently limited, and consumers can only see the effect of viewing the item from a partial angle. However, the method of this application embodiment can predict the three-dimensional light field of the item, so users can view the product form of the item from all angles, which helps users choose a product that is more suitable for them.

[0078] There are many other examples that require the prediction of three-dimensional light fields, such as VR house viewing, VR movies, games, and street scene creation.

[0079] Please see Figure 4 , Figure 4 This is a flowchart illustrating a method for determining a three-dimensional light field in a scene, as provided in an embodiment of this application. This method can be based on... Figure 3 The architecture shown can be implemented, but it can also be implemented based on other architectures. When based on... Figure 3 When implementing the architecture shown, steps S400-S405 can be implemented by the model training device 301, and step S406 can be implemented by the model usage device 302. When implementing based on other architectures, steps S400-S406 can be completed by one device or by multiple devices working together. The application domain of this one or multiple devices is not limited here, as long as they can provide the corresponding computing power and / or communication capabilities. Steps S400-S406 are as follows:

[0080] Step S400: Input multiple sample images and information about the shooting location of the multiple sample images into the deep learning network.

[0081] The sample images are taken by a camera from different angles of the same scene. Optionally, the angle (pose) includes the position coordinates (x, y, z) and the viewpoint direction. For example, if the world coordinate system is used as a reference, x, y, and z in the coordinates (x, y, z) of this location represent longitude, latitude, and altitude, respectively, and the direction of the viewpoint. θ in These represent the horizontal and vertical angles, respectively. Of course, this orientation can also be expressed in other ways.

[0082] Step S401: Establish a cube model surrounding the scene being photographed based on the shooting orientation of multiple sample images.

[0083] Understandably, a cube model encompassing the scene can be constructed based on multiple different angles from which the camera captured these photos. Optionally, the length, width, and height of this cube model can be the maximum values ​​of the scene's length C, width W, and height H calculated from these multiple different angles. The scene itself is not limited; for example, it could be a scene with a person as the main object, a scene with a tree as the main object, a scene with the interior structure of a building as the main subject, and so on. The multiple different angles mentioned here can be the four cardinal directions (east, west, south, and north), or different angles relative to other reference points.

[0084] The cube model is divided into multiple smaller cubes (grid voxels), or simply voxels. Optionally, these voxels can be obtained by dividing the cube model into N equal parts. Their size can be set according to actual needs. It's understood that smaller voxels are beneficial for improving the training accuracy of subsequent deep learning networks. However, smaller voxels also increase the computational burden during deep learning network training. Typically, the size of the voxel is considered in conjunction with accuracy and device computing power. For example, the size of the voxel can be set to 2 centimeters (cm). Generally, for any given voxel, it is assumed that it is either located on the surface of an object in the scene or not on the surface of an object (it can be considered to be located in an empty area of ​​the scene).

[0085] Optionally, if the position of a voxel in the cube model is represented by three-dimensional position coordinates g, i.e., (x, y, z), when the voxels in the cube model are subsequently fed into the deep learning network for training, a single GPU thread can process a voxel at the (x, y) coordinate; that is, one GPU process scans and processes a lattice pillar at the (x, y) coordinate. Optionally, the coordinates of the center point of the voxel are generally used as the coordinates of that voxel. Of course, the coordinates of other points in the voxel can also be used as the coordinates of that voxel, such as the top-left corner, the top-right corner, etc.

[0086] Step S402: Calculate multiple cutoff distances for each of the multiple small cubes based on the multiple sample images.

[0087] The aforementioned sample images were taken by the camera from different angles. Therefore, it is necessary to calculate the cutoff distance of each small cube in the cube model for each sample image taken from each angle. For ease of understanding, the first sample image is used as an example below, and this first sample image is any one of the multiple sample images. The cutoff distance of each small cube calculated based on the first sample image is determined by the distance from the camera to each small cube when the first sample image was taken and the distance from the camera to the surface of the object in the scene. An example is provided below.

[0088] Assuming the first sample image is taken by the camera from the first position, the camera also collects the depth information of the scene when taking the picture from the first position. This depth information reflects the distance from the camera to the surface of the object in the scene. Therefore, in the image obtained by the camera from the first position, each pixel x corresponds to a depth value. The depth value value(x) corresponding to the pixel x reflects the distance value(x) from the voxel on the camera ray located on the surface of the object to the camera.

[0089] The following explanation uses the second small cube (any voxel in the cube model) as an example. Its position is known, and the position of the camera is also known when the camera takes a picture from the first position. Therefore, for the second small cube, the distance from the second small cube to the camera is known when the camera takes a picture from the first position, which can be denoted as distance(v).

[0090] Therefore, when the camera is shooting from the first position, the distance from the second small cube in the cube model to the surface of the object in the scene can be denoted as: sdf(p) = value(x) - distance(v), and the specific scene and geometric relationships are as follows: Figure 5 As shown.

[0091] This application's embodiments focus on voxels near the object's surface. Assuming a distance of no more than a preset threshold u from the object's surface is considered near the surface, whether a voxel is near and to what extent can be represented by a cutoff distance d(p). The cutoff distance d(p) of the second small cube voxel can be calculated as follows:

[0092] d(p)=sdf(p) / |u|

[0093] Optionally, if sdf(p) > |u|, then d(p) = 1; if sdf(p) < 0 and |sdf(p)| > |u|, then d(p) = -1. Figure 6 The functional relationship between the cutoff distance d(p) and u is illustrated. Figure 7The diagram illustrates the distribution of cutoff distances d(p) of some small cube voxels in the cube model corresponding to the scene captured by the camera.

[0094] In an alternative approach, a weight value for each voxel can also be calculated. The weight value of the second small cube, calculated based on the first sample image, is negatively correlated with the distance from the second small cube to the camera when the first sample image was captured, and / or positively correlated with a first angle. The first angle is the angle between the camera ray, the field of view of the second small cube, and the normal vector of the surface of the object closest to the second small cube. This weight value is used for subsequent fusion calculations of the truncation distance of the second small cube. The reason for determining the weight value of each voxel in this way is that when the camera captures from the first position, for any voxel in the cube model, its pixel information (including density, color, etc.) is affected by multiple factors. For example, the closer it is to the camera, the more pixel information it has; the smaller the corresponding first angle (equivalent to the smaller the angle away from the camera), the more pixel information it has.

[0095] Optionally, the expression for the weight value w(p) of the second small cube voxel can be as follows:

[0096] w(p) = cos(θ) / distance(v)

[0097] Where θ is the first included angle, and distance(v) is the distance from the second small cube to the camera when the first sample image was captured.

[0098] Based on the above description, using the first sample image, the cutoff distance d(p) and weight value w(p) of each voxel in the cube model can be calculated. This set of cutoff distance d(p) and weight value w(p) are parameters relative to the first orientation. Using the same calculation principle, the cutoff distance d(p) and weight value w(p) of each voxel in the cube model can also be calculated using other sample images. This set of cutoff distance d(p) and weight value w(p) are parameters relative to another orientation.

[0099] It should be noted that before calculating the cutoff distance d(p) and weight value w(p) for each voxel, the coordinates of the voxel and the camera can be unified into a single coordinate system for easier calculation. For example, based on the size and number of voxels, the position g of each voxel in the cube model can be converted into a position point p in the world coordinate system. Then, based on the camera pose matrix, the mapping point v of position point p in the world coordinate system to the camera coordinate system is determined. The corresponding pixel point x in the depth image is determined based on the camera intrinsic parameter matrix and the mapping point v. Then, the depth value of pixel point x is obtained. The depth value of pixel point x is the distance value(x) from the voxel located on the camera ray at position g in the cube model, which is on the object surface, to the camera. The distance from the mapping point v to the origin of the camera coordinate system is denoted as distance(v). The obtained value(x) and distance(v) can then be used to calculate the cutoff distance d(p) and weight value w(p).

[0100] In one alternative approach, when capturing each of the multiple sample images, depth information is acquired from the shooting perspective of each sample image. This depth information characterizes the distance from the camera to the surface of an object in the scene, i.e., the aforementioned value(x). This depth information can be specifically acquired through a sensor, such as a radar sensor, an infrared sensor, or similar sensor, which can be deployed on or near the camera.

[0101] Step S403: Perform a fusion calculation on the cutoff distances calculated from the multiple sample images to obtain the fused cutoff distance of the small cubes in the cube model.

[0102] In the first alternative, the multiple cutoff distances of each small cube are fused and calculated to obtain the fused cutoff distance of each small cube; during subsequent sampling, spatial points are sampled from each small cube based on the fused cutoff distance of each small cube.

[0103] In the second alternative, it is specifically determined that at least one of the small cubes has a cutoff distance whose absolute value is less than a preset threshold. Then, the cutoff distances of the first small cubes calculated from the multiple sample images are fused to obtain the fused cutoff distance of the first small cube in the cube model. For example, if multiple cutoff distances are calculated for a voxel based on multiple sample images, then if the smallest of these multiple cutoff distances is less than the preset threshold, then that voxel is considered the first small cube. For example, the preset threshold can be set to 1. Subsequent sampling involves sampling spatial points from the first small cube based on the fused cutoff distance of the first small cube.

[0104] The principle of fusion computing will be illustrated below using the second optional scheme as an example.

[0105] Optionally, the principle of fusion calculation can be that the fusion cutoff distance of the second small cube is obtained by weighted averaging of the cutoff distances calculated based on different sample images, and the second small cube is any small cube in the cube model. The weight value of the cutoff distance of the second small cube calculated based on the first sample image in the weighted averaging calculation is equal to the weight value w(p) of the second small cube calculated based on the first sample image. For example, the following operations are performed on multiple sample images: the truncation distance d(p) calculated based on one of the sample images is used as the initial fusion truncation distance D(p), and the weight value w(p) calculated based on that sample image is used as the initial fusion weight value W(p); then, the following operations are performed sequentially on other sample images: the truncation distance d(p) calculated based on the current sample image is integrated into the existing fusion truncation distance D(p) to update the fusion truncation distance D(p), and the weight value w(p) calculated based on the current sample image is integrated into the existing fusion weight value W(p) to update the fusion weight value W(p), until the truncation distance d(p) and weight value w(p) calculated based on each sample image are fused.

[0106] Optionally, the expression for the fusion calculation can be as follows:

[0107] D(p)=(W(p)*D(p)+w(p)d(p)) / (W(p)+w(p))

[0108] W(p) = W(p) + w(p)

[0109] Where D(p) is the fusion cutoff distance of the second small cube, W(p) is the fusion weight value of the second small cube, d(p) is the cutoff distance of the second small cube calculated based on the current sample image, and w(p) is the weight value of the second small cube calculated based on the current sample image.

[0110] It is understandable that this method can be used to calculate the final fusion cutoff distance D(p) and fusion weight value W(p) of all voxels in the cube model. Optionally, the final fusion cutoff distance D(p) and fusion weight value W(p) of all voxels in the cube model can be input into Marching Cube to calculate the triangular faces, thereby representing the cutoff distance field of the cube model.

[0111] Step S404: Sample spatial points from the small cubes based on the fusion cutoff distance of the small cubes in the cube model.

[0112] Specifically, spatial points are sampled from the voxels in the cube model based on the fusion cutoff distance of each voxel. For the first option in step S403, spatial points can be sampled from each voxel based on the fusion cutoff distance of each voxel. For the second option in step S403, spatial points can be sampled from the first small cube (i.e., the first voxel) based on the fusion cutoff distance of the first small cube.

[0113] The sampling approach is to sample more spatial points from smaller cubes in the cube model with smaller fusion cutoff distances. For example, for any phase plane, on the camera ray emitted by that phase plane, the closer the fusion cutoff distance D(p) is to 0, the higher the sampling density and the more times it is sampled. Conversely, if the fusion cutoff distance D(p) is closer to 1 or -1, the sampling density and the number of samples for such voxels are lower. Optionally, the number of samples for cubes in the cube model with fusion cutoff distances greater than or equal to a preset threshold is zero. For example, for voxels with fusion cutoff distances D(p) equal to 1 or equal to -1, no sampling is required.

[0114] Optionally, the number of samples of a voxel on the camera ray and the fusion cutoff distance D(p) of that voxel satisfy the following relationship:

[0115] n∝(1-|D(p)|)

[0116] Where D(p) is the fusion cutoff distance of the voxel, and n is the number of samplings of the voxel.

[0117] It should be noted that each voxel is not a single point in the cubic space, but rather contains a large number of spatial points. Therefore, after calculating the sampling number of each voxel using the above method, a portion of the spatial points can be sampled from the large number of spatial points contained in that voxel. For example, taking 10 voxels out of all voxels, if each voxel contains 1000 spatial points, these 10 voxels are represented as voxel-1, voxel-2, voxel-3, voxel-4, voxel-5, voxel-6, voxel-7, voxel-8, voxel-9, and voxel-10, respectively. The fusion cutoff distance D(p) of these 10 voxels is 0 in sequence. Given the values ​​1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, and 1, the number of spatial points sampled from voxel-1 is 90, from voxel-2 it is 80, from voxel-3 it is 70, from voxel-4 it is 60, from voxel-5 it is 50, from voxel-6 it is 40, from voxel-7 it is 30, from voxel-8 it is 20, from voxel-9 it is 10, and from voxel-10 it is 0. Each sampled spatial point has spatial coordinates (x, y, z).

[0118] In this embodiment, the distance from some voxels on the camera ray to the camera cannot be detected by the relevant sensors. Therefore, the cutoff distance of these voxels cannot be obtained, and sampling based on the cutoff distance is also impossible. For these voxels, traditional uniform sampling (or average sampling) methods can be used for sampling.

[0119] Step S405: Train the deep learning network using the spatial coordinates of sampled spatial points and multiple sample images.

[0120] The multiple sample images correspond to multiple directions. The deep learning network will reconstruct (or calculate) the image of each direction based on the spatial coordinates of the sampled spatial points until the loss (e.g., RGB information difference) between the reconstructed image of each direction and the original sample image of that direction is less than a preset value. When the loss between the reconstructed image of each direction and the original sample image of that direction is less than the preset value, the training of the deep learning network ends.

[0121] The following example illustrates the reconstruction process.

[0122] Optionally, if sample image 1 is taken from orientation 1, then the pixel value of pixel A in image 1 is added to the spatial coordinates of the sampled spatial point on the camera ray corresponding to pixel A. A similar operation is performed on other pixels in image 1. After this operation is completed for all pixels in image 1, an image effect is presented. This image is the reconstructed image A from orientation 1. The ideal reconstruction effect is that the loss (e.g., RGB information difference) between the reconstructed image A from orientation 1 and the sample image 1 is less than a preset value. This relationship is also satisfied for sample images taken from other orientations and the reconstructed images. The deep learning network after this training can also be called a light field prediction model.

[0123] Step S406: Predict the image of the above scene when taken from a new location using the deep learning network (light field prediction model).

[0124] Specifically, the new orientation here can be any orientation other than those mentioned above, and this orientation can be represented using five-dimensional (5D) coordinates, for example, orientation 1. Where (x, y, z) represents the position. This indicates the viewing direction. It's understandable that this method can predict the image of the scene from any angle. Generally, once images of the scene from any angle have been acquired, it's considered that the three-dimensional light field of the scene has been obtained.

[0125] Optionally, the predicted image can be an RGB image.

[0126] Table 1 shows... Figure 1 The illustration shows a comparison between the prediction performance of a pre-trained deep learning network in the prior art and the prediction performance of the pre-trained deep learning network in this application. The PSNR comparison results can be obtained through simulation output:

[0127] Table 1

[0128] Network type Existing technology Examples of this application PSNR-Coarse 33 37 PSNR-Fine 36 37.2 PSNR-Test 35.5 36.5

[0129] In Table 1, Peak Signal-to-Noise Ratio (PSNR) is a metric used to measure prediction performance. PSNR-Coarse represents the performance when performing coarse-grained predictions on images corresponding to the same orientation as existing sample images; PSNR-Fine represents the performance when performing fine-grained predictions on images captured at the same orientation as existing sample images; and PSNR-Test represents the performance when predicting images captured at a new orientation. A higher PSNR value indicates better prediction performance from the trained deep learning network.

[0130] like Figure 8 As shown, Figure 8 This illustration shows a comparison between the prediction performance of a pre-trained deep learning network in the prior art and the prediction performance of the pre-trained deep learning network in this application. The result is presented in image form. Figure 8 In the diagram, (g) is an image of a scene captured in the image; (h) is the result of reconstructing the image using a deep learning network in the prior art; and (i) is the result of reconstructing the image using the deep learning network described in this application's embodiment. It can be seen that this application's embodiment expresses the texture of object surfaces more clearly and effectively, better reflecting the layers of the scene.

[0131] exist Figure 4 In the method shown, the voxel sampling points used to train the deep learning network (for predicting 3D light fields) are obtained based on the depth information of the image. Specifically, a cutoff distance is calculated based on the depth information and the distance from the voxel to the camera. Then, differential sampling is performed according to the size of the cutoff distance. On the one hand, this sampling method can quickly concentrate the sampling on key areas, improving sampling efficiency. On the other hand, the voxels sampled by this method are basically concentrated near the object surface. Therefore, the deep learning network trained based on these voxels can better represent the texture details of the object when performing image prediction, and can reduce blur and structural errors.

[0132] The methods of the embodiments of this application have been described in detail above, and the apparatus of the embodiments of this application is provided below.

[0133] Please see Figure 9 , Figure 9 This is a schematic diagram of the structure of a light field prediction model generation device 90 provided in an embodiment of this application. The device 90 may include a building unit 901, a first calculation unit 902, a sampling unit 903, and a second calculation unit 904, wherein each unit is described in detail below.

[0134] Establishment unit 901 is used to establish a cube model surrounding the scene being photographed based on the shooting positions of multiple sample images, wherein the cube model includes multiple small cubes (voxels);

[0135] The first calculation unit 902 is used to calculate multiple cutoff distances for each of the multiple small cubes based on the multiple sample images. The cutoff distance for each small cube calculated based on the first sample image includes: determining the cutoff distance based on the distance from the camera to each small cube when the first sample image is captured and the distance from the camera to the surface of an object in the scene. The first sample image is any one of the multiple sample images.

[0136] The sampling unit 903 is used to sample spatial points from the small cube according to multiple cut-off distances of each small cube, wherein each sampled spatial point corresponds to spatial coordinates;

[0137] The second computing unit 904 is used to train a light field prediction model based on the spatial coordinates of the sampled spatial points, wherein the light field prediction model is used to predict the light field of the scene.

[0138] In the aforementioned device, the voxel sampling points used to train the deep learning network (also known as the light field prediction model, used to predict 3D light fields) are obtained based on the depth information of the image. Specifically, a cutoff distance is calculated based on the depth information and the distance from the voxel to the camera. Then, differential sampling is performed according to the size of the cutoff distance. On the one hand, this sampling method can quickly concentrate the sampling on key areas, improving sampling efficiency. On the other hand, the voxels sampled by this method are basically concentrated near the object surface. Therefore, when the deep learning network trained based on these voxels performs image prediction, it can better represent the texture details of the object and reduce blur and structural errors.

[0139] In an alternative embodiment, the device 90 further includes:

[0140] The prediction unit is used to predict the light field of the scene using the light field prediction model. That is, after training the pre-stored light field model, the model training device will also predict the light field using the light field prediction model.

[0141] In another alternative embodiment, regarding sampling spatial points from the small cubes based on multiple cutoff distances of each small cube, the sampling unit 903 is specifically configured to: perform a fusion calculation on the multiple cutoff distances of each small cube to obtain a fused cutoff distance for each small cube; and sample spatial points from each small cube based on the fused cutoff distance. In this implementation, the fused cutoff distance is calculated for each small cube, and then sampling is performed based on the fused cutoff distance.

[0142] In another alternative approach, regarding sampling spatial points from the small cubes based on multiple cutoff distances of each small cube, the sampling unit 903 is specifically configured to: determine a first small cube among the small cubes whose absolute cutoff distance is less than a preset threshold; perform a fusion calculation on the multiple cutoff distances of the first small cube to obtain a fused cutoff distance of the first small cube; and sample spatial points from the first small cube based on the fused cutoff distance of the first small cube. In this approach, the fused cutoff distance is not calculated for all small cubes, but only for those whose absolute cutoff distance is less than the preset threshold. This is because when the cutoff distance is large, the corresponding small cube is farther from the object surface, and the necessity for subsequent sampling is less. Therefore, this application does not perform a fusion calculation on the cutoff distances of such small cubes, which is equivalent to excluding these small cubes from the sampling scope in advance. This reduces the computational load and improves the generation efficiency of the light field prediction model without significantly reducing the subsequent sampling effect.

[0143] In another alternative approach, during the sampling of spatial points from the small cubes, the smaller the fusion cutoff distance, the more spatial points are sampled from the small cubes. It can be understood that cubes with smaller fusion cutoff distances are closer to the object surface, and the spatial points in these small cubes better represent pixel information compared to spatial points in other small cubes on the same camera ray. Therefore, training the light field prediction model based more on the spatial points in these small cubes is beneficial for the model to predict more accurate images subsequently.

[0144] In another alternative approach, the fusion calculation includes a weighted average calculation. It is understood that the fusion cutoff distance obtained through a weighted average calculation more accurately reflects the distance of the small cube from the object's surface.

[0145] In another alternative scheme, the weight value of the cutoff distance of the second small cube calculated based on the first sample image in the weighted average calculation is negatively correlated with the distance from the second small cube to the camera when the first sample image was captured, and / or positively correlated with the first included angle, where the first included angle is the angle between the field of view line of the second small cube (camera ray) and the normal vector of the surface of the object closest to the second small cube, and the second small cube is any small cube in the cube model.

[0146] In this approach, a weight value calculated based on each sample image is used when calculating the fusion cutoff distance. Since this weight value is negatively correlated with the distance to the camera at that time and / or positively correlated with the first included angle, when this weight value is incorporated into the calculation of the fusion cutoff distance, it can more accurately reflect the influence of different orientations on the fusion cutoff distance.

[0147] In another alternative embodiment, the device 90 further includes:

[0148] The acquisition unit is used to acquire depth information from the shooting perspective of each of the multiple sample images when capturing each sample image, wherein the depth information is used to characterize the distance from the camera to the surface of an object in the scene.

[0149] In this approach, depth information is collected from the shooting perspective of each sample image, which can more accurately reflect the distance from the camera to the surface of the object in the scene being photographed, thus facilitating the calculation of a more accurate cutoff distance.

[0150] In another alternative scheme, during the calculation of the fusion cutoff distance of the second small cube, the weight value of the cutoff distance calculated based on the first sample image in the weighted average calculation can also be called the weight value w(p) of the second small cube calculated based on the first sample image. This weight value w(p) satisfies the following relationship:

[0151] w(p) = cos(θ) / distance(v)

[0152] Where θ is the first included angle, and distance(v) is the distance from the second small cube to the camera when the first sample image was captured.

[0153] As mentioned earlier, the weight value of the second small cube calculated based on the first sample image is negatively correlated with the distance from the second small cube to the camera when the first sample image was captured, and / or positively correlated with the first included angle. The expression of w(p) here is an optional expression of this idea. Therefore, when this weight value is incorporated into the calculation of the fusion cutoff distance, it can more accurately reflect the influence of different orientations on the fusion cutoff distance.

[0154] In another alternative scheme, the cutoff distance d(p) of the second small cube calculated from the first sample image satisfies the following relationship:

[0155] d(p)=sdf(p) / |u|

[0156] Where sdf(p) is the distance from the camera to the first small cube when the first sample image is captured, and the difference between the distance from the camera to the surface of the object in the scene, and u is a preset threshold.

[0157] It is understandable that d(p) here is only one optional formula for calculating the cutoff distance, and there are other ways to express it in practical applications.

[0158] In another alternative scheme, if sdf(p) > |u|, then d(p) = 1; if sdf(p) < 0 and |sdf(p)| > |u|, then d(p) = -1.

[0159] In this approach, the cutoff distance of small cubes within one range is assigned a value of 1, while the cutoff distance of small cubes within another range is assigned a value of -1. This facilitates the subsequent application of the same processing to both types of small cubes, thereby improving computational efficiency.

[0160] It should be noted that the implementation of each unit can also be referenced accordingly. Figure 4 The corresponding description of the method embodiments shown.

[0161] Please see Figure 10 , Figure 10 This application provides a light field prediction model generation device 100, which includes a processor 1001, a memory 1002 and a communication interface 1003. The processor 1001, the memory 1002 and the communication interface 1003 are interconnected via a bus.

[0162] The memory 1002 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM), and is used for related instructions and data. The communication interface 1003 is used for receiving and sending data.

[0163] The processor 1001 can be one or more central processing units (CPUs). If the processor 1001 is a CPU, the CPU can be a single-core CPU or a multi-core CPU.

[0164] Since the device 100 needs multiple sample images to train the light field prediction model, it needs to acquire these sample images. This can be done by receiving sample images from other devices via the communication interface 1003, or by having a camera (or image sensor, imaging device) on the device 100 capable of capturing the sample images. Optionally, when the device 100 is equipped with a camera, it can also be equipped with a depth sensor to collect the depth of objects in the scene being photographed. The type of depth sensor is not limited here.

[0165] The processor 1001 in the device 100 is used to read the program code stored in the memory 1002 and perform the following operations: First, it establishes a cube model surrounding the scene to be photographed based on the shooting orientation of each of the multiple sample images, wherein the cube model includes multiple small cubes (voxels). Then, it calculates multiple cutoff distances for each of the multiple small cubes based on the multiple sample images, wherein one cutoff distance for each small cube calculated based on the first sample image includes: determining the cutoff distance based on the distance from the camera to each small cube when the first sample image is captured and the distance from the camera to the surface of an object in the scene, wherein the first sample image is any one of the multiple sample images. Next, it samples spatial points from the small cubes based on the multiple cutoff distances for each small cube, wherein each sampled spatial point corresponds to spatial coordinates. Then, it trains a light field prediction model based on the spatial coordinates of the sampled spatial points, wherein the light field prediction model is used to predict the light field of the scene.

[0166] In the above method, the voxel sampling points used to train the deep learning network (also known as the light field prediction model, used to predict 3D light fields) are obtained based on the depth information of the image. Specifically, a cutoff distance is calculated based on the depth information and the distance from the voxel to the camera. Then, differential sampling is performed according to the size of the cutoff distance. On the one hand, this sampling method can quickly concentrate the sampling on key areas, improving sampling efficiency. On the other hand, the voxels sampled by this sampling method are basically concentrated near the object surface. Therefore, the deep learning network trained based on these voxels can better represent the texture details of the object when performing image prediction, and can reduce blur and structural errors.

[0167] In one optional scheme, after training the light field prediction model based on the spatial coordinates of the sampled spatial points, the processor 1001 is specifically used to: predict the light field of the scene using the light field prediction model. That is to say, after training a pre-stored light field model, the model training device will also predict the light field using the light field prediction model.

[0168] In another alternative approach, regarding sampling spatial points from the small cubes based on multiple cutoff distances of each small cube, the processor is specifically configured to: perform a fusion calculation on the multiple cutoff distances of each small cube to obtain a fused cutoff distance for each small cube; and sample spatial points from each small cube based on the fused cutoff distance. In this implementation, the fused cutoff distance is calculated for each small cube, and then sampling is performed based on the fused cutoff distance.

[0169] In another alternative approach, regarding sampling spatial points from the small cubes based on multiple cutoff distances of each small cube, the processor is specifically configured to: determine a first small cube among the small cubes whose absolute cutoff distance is less than a preset threshold; perform a fusion calculation on the multiple cutoff distances of the first small cube to obtain a fused cutoff distance of the first small cube; and sample spatial points from the first small cube based on the fused cutoff distance of the first small cube. In this approach, the fused cutoff distance is not calculated for all small cubes, but only for those whose absolute cutoff distance is less than the preset threshold. This is because when the cutoff distance is large, the corresponding small cube is farther from the object surface, and the necessity for subsequent sampling is less. Therefore, this application does not perform a fusion calculation on the cutoff distances of such small cubes, which is equivalent to excluding these small cubes from the sampling scope in advance. This reduces the computational load and improves the generation efficiency of the light field prediction model without significantly reducing the subsequent sampling effect.

[0170] In another alternative approach, during the sampling of spatial points from the small cubes, the smaller the fusion cutoff distance, the more spatial points are sampled from the small cubes. It can be understood that cubes with smaller fusion cutoff distances are closer to the object surface, and the spatial points in these small cubes better represent pixel information compared to spatial points in other small cubes on the same camera ray. Therefore, training the light field prediction model based more on the spatial points in these small cubes is beneficial for the model to predict more accurate images subsequently.

[0171] In another alternative approach, the fusion calculation includes a weighted average calculation. It is understood that the fusion cutoff distance obtained through a weighted average calculation more accurately reflects the distance of the small cube from the object's surface.

[0172] In another alternative scheme, the weight value of the cutoff distance of the second small cube calculated based on the first sample image in the weighted average calculation is negatively correlated with the distance from the second small cube to the camera when the first sample image was captured, and / or positively correlated with the first included angle, where the first included angle is the angle between the field of view line of the second small cube (camera ray) and the normal vector of the surface of the object closest to the second small cube, and the second small cube is any small cube in the cube model.

[0173] In this approach, a weight value calculated based on each sample image is used when calculating the fusion cutoff distance. Since this weight value is negatively correlated with the distance to the camera at that time and / or positively correlated with the first included angle, when this weight value is incorporated into the calculation of the fusion cutoff distance, it can more accurately reflect the influence of different orientations on the fusion cutoff distance.

[0174] In another alternative scheme, before calculating the cutoff distance of each small cube in the cube model based on the multiple sample images, the processor is further configured to: acquire depth information from the shooting perspective of each sample image when capturing each of the multiple sample images, wherein the depth information is used to characterize the distance from the camera to the surface of an object in the scene.

[0175] In this approach, depth information is collected from the shooting perspective of each sample image, which can more accurately reflect the distance from the camera to the surface of the object in the scene being photographed, thus facilitating the calculation of a more accurate cutoff distance.

[0176] In another alternative scheme, during the calculation of the fusion cutoff distance of the second small cube, the weight value of the cutoff distance calculated based on the first sample image in the weighted average calculation can also be called the weight value w(p) of the second small cube calculated based on the first sample image. This weight value w(p) satisfies the following relationship:

[0177] w(p) = cos(θ) / distance(v)

[0178] Where θ is the first included angle, and distance(v) is the distance from the second small cube to the camera when the first sample image was captured.

[0179] As mentioned earlier, the weight value of the second small cube calculated based on the first sample image is negatively correlated with the distance from the second small cube to the camera when the first sample image was captured, and / or positively correlated with the first included angle. The expression of w(p) here is an optional expression of this idea. Therefore, when this weight value is incorporated into the calculation of the fusion cutoff distance, it can more accurately reflect the influence of different orientations on the fusion cutoff distance.

[0180] In another alternative scheme, the cutoff distance d(p) of the second small cube calculated from the first sample image satisfies the following relationship:

[0181] d(p)=sdf(p) / |u|

[0182] Where sdf(p) is the distance from the camera to the first small cube when the first sample image is captured, and the difference between the distance from the camera to the surface of the object in the scene, and u is a preset threshold.

[0183] It is understandable that d(p) here is only one optional formula for calculating the cutoff distance, and there are other ways to express it in practical applications.

[0184] In combination with the first aspect or any of the above possible implementations of the first aspect, in another alternative scheme of the first aspect, if sdf(p) > |u|, then d(p) = 1; if sdf(p) < 0 and |sdf(p)| > |u|, then d(p) = -1.

[0185] In this approach, the cutoff distance of small cubes within one range is assigned a value of 1, while the cutoff distance of small cubes within another range is assigned a value of -1. This facilitates the subsequent application of the same processing to both types of small cubes, thereby improving computational efficiency.

[0186] It should be noted that the implementation of each operation can also be referenced accordingly. Figure 4 The corresponding description of the method embodiments shown.

[0187] This application embodiment also provides a chip system, the chip system including at least one processor, a memory, and interface circuitry, the memory, the transceiver, and the at least one processor being interconnected via circuitry, the at least one memory storing instructions; when the instructions are executed by the processor, they implement... Figure 4 The method flow is shown.

[0188] This application also provides a computer-readable storage medium storing instructions that, when executed on a processor, implement... Figure 4 The method flow is shown.

[0189] This application also provides a computer program product that, when run on a processor, implements... Figure 4 The method flow is shown.

[0190] In summary, by implementing the embodiments of this application, the voxel sampling points used to train the deep learning network (also known as the light field prediction model, used to predict three-dimensional light fields) are obtained based on the depth information of the image. Specifically, a cutoff distance is calculated based on the depth information and the distance from the voxel to the camera, and then differential sampling is performed according to the size of the cutoff distance. On the one hand, this sampling method can quickly concentrate the sampling on key areas, improving sampling efficiency; on the other hand, the voxels sampled by this sampling method are basically concentrated near the object surface. Therefore, the deep learning network trained based on these voxels can better represent the texture details of the object when performing image prediction, and can reduce blur and structural errors.

[0191] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for generating a light field prediction model, characterized in that, include: A cube model surrounding the scene is constructed based on the shooting positions of multiple sample images, wherein the cube model comprises multiple small cubes (voxels). Based on the multiple sample images, multiple cutoff distances are calculated for each of the multiple small cubes. The cutoff distance for each small cube calculated based on the first sample image includes: determining the cutoff distance based on the distance from the camera to each small cube when the first sample image is captured and the distance from the camera to the surface of an object in the scene. The first sample image is any one of the multiple sample images. Spatial points are sampled from each small cube based on multiple cutoff distances, wherein each sampled spatial point corresponds to spatial coordinates; A light field prediction model is trained based on the spatial coordinates of the sampled spatial points, wherein the light field prediction model is used to predict the light field of the scene.

2. The method according to claim 1, characterized in that, After training the light field prediction model based on the spatial coordinates of the sampled spatial points, the method further includes: The light field of the scene is predicted using the light field prediction model.

3. The method according to claim 1 or 2, characterized in that, The step of sampling spatial points from the small cubes based on multiple cutoff distances of each small cube includes: The multiple cutoff distances of each small cube are fused and calculated to obtain the fused cutoff distance of each small cube; Spatial points are sampled from each small cube based on the fusion cutoff distance of each small cube.

4. The method according to claim 1 or 2, characterized in that, The step of sampling spatial points from the small cubes based on multiple cutoff distances of each small cube includes: In each of the small cubes, at least one first small cube has a cutoff distance whose absolute value is less than a preset threshold. The multiple cutoff distances of the first small cube are fused and calculated to obtain the fused cutoff distance of the first small cube; Spatial points are sampled from the first small cube based on the fusion cutoff distance of the first small cube.

5. The method according to claim 3 or 4, characterized in that, During the process of sampling spatial points from the small cubes, the smaller the fusion cutoff distance, the more spatial points are sampled from the small cubes.

6. The method according to any one of claims 3-5, characterized in that, The fusion calculation includes weighted average calculation.

7. The method according to claim 6, characterized in that, The weight value of the cutoff distance of the second small cube calculated based on the first sample image in the weighted average calculation is negatively correlated with the distance from the second small cube to the camera when the first sample image was captured, and / or positively correlated with the first included angle, which is the angle between the field of view line of the second small cube (camera ray) and the normal vector of the surface of the object closest to the second small cube, and the second small cube is any small cube in the cube model.

8. A device for generating a light field prediction model, characterized in that, include: A building unit is used to build a cube model surrounding the scene being photographed based on the shooting positions of multiple sample images, wherein the cube model includes multiple small cubes (voxels). The first calculation unit is used to calculate multiple cutoff distances for each of the multiple small cubes based on the multiple sample images. The cutoff distance for each small cube calculated based on the first sample image includes: determining the cutoff distance based on the distance from the camera to each small cube when the first sample image was captured and the distance from the camera to the surface of an object in the scene. The first sample image is any one of the multiple sample images. A sampling unit is used to sample spatial points from the small cube based on multiple cutoff distances of each small cube, wherein each sampled spatial point corresponds to spatial coordinates; The second computing unit is used to train a light field prediction model based on the spatial coordinates of the sampled spatial points, wherein the light field prediction model is used to predict the light field of the scene.

9. The apparatus according to claim 8, characterized in that, The device further includes: A prediction unit is used to predict the light field of the scene using the light field prediction model.

10. The apparatus according to claim 8 or 9, characterized in that, In terms of sampling spatial points from the small cubes according to multiple cutoff distances of each small cube, the sampling unit is specifically used for: The multiple cutoff distances of each small cube are fused and calculated to obtain the fused cutoff distance of each small cube; Spatial points are sampled from each small cube based on the fusion cutoff distance of each small cube.

11. The apparatus according to claim 8 or 9, characterized in that, In terms of sampling spatial points from the small cubes according to multiple cutoff distances of each small cube, the sampling unit is specifically used for: In each of the small cubes, at least one first small cube has a cutoff distance whose absolute value is less than a preset threshold. The multiple cutoff distances of the first small cube are fused and calculated to obtain the fused cutoff distance of the first small cube; Spatial points are sampled from the first small cube based on the fusion cutoff distance of the first small cube.

12. The apparatus according to claim 10 or 11, characterized in that, During the process of sampling spatial points from the small cubes, the smaller the fusion cutoff distance, the more spatial points are sampled from the small cubes.

13. The apparatus according to any one of claims 10-12, characterized in that, The fusion calculation includes weighted average calculation.

14. The apparatus according to claim 13, characterized in that, The weight value of the cutoff distance of the second small cube calculated based on the first sample image in the weighted average calculation is negatively correlated with the distance from the second small cube to the camera when the first sample image was captured, and / or positively correlated with the first included angle, which is the angle between the field of view line of the second small cube (camera ray) and the normal vector of the surface of the object closest to the second small cube, and the second small cube is any small cube in the cube model.

15. A device for generating a light field prediction model, characterized in that, It includes a processor and a memory, wherein the memory is used to store a computer program that, when executed on the processor, implements the method of any one of claims 1-7.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed on a processor, implements the method described in any one of claims 1-7.

Images