Adaptive light sampling three-dimensional light field content generation method and device

Abstract

The embodiment of the invention relates to the technical field of three-dimensional light field content generation, and provides an adaptive light sampling three-dimensional light field content generation method and device, and the method comprises the steps: collecting an image of a target three-dimensional scene through employing a multi-path RGB camera, building a three-dimensional world coordinate system, and constructing a three-dimensional sampling space and a voxel space; performing initial sampling on the occupied voxels through an adaptive light sampling strategy to form initial sampling points, and adaptively adjusting a re-sampling mode according to the sufficiency of the initial sampling points; in addition, a neural network model is used for training so as to update the occupancy state of the voxel grid; and finally, the rendering light is calculated through the virtual rendering camera array to generate a composite image, so that the three-dimensional light field content is generated. Therefore, according to the method, the sampling strategy is flexibly adjusted according to the geometric distribution of the scene, and the content generation quality of the three-dimensional light field is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional light field content generation, and in particular to a method and device for generating three-dimensional light field content using adaptive light sampling. Background Art

[0002] With the rapid development of computer vision and computer graphics, their application in the field of 3D light field display has attracted widespread attention. Learning-based methods can achieve efficient reconstruction of 3D scenes by building neural network models and training them with a large amount of data. However, this method still has some disadvantages.

[0003] Learning-based methods often rely on reliable light sampling strategies to search and determine the location of target objects in three-dimensional space, so as to accurately characterize the scene distribution. Traditional three-dimensional light field content generation methods are usually based on a limited sampling space or a fixed sampling strategy, which fails to fully consider the complexity of scene geometry. In highly complex scenes, some areas may be rich in details, while the geometry of other areas is relatively simple. If there is a lack of flexibility in sampling these areas, it may lead to sparse samples or too many sampling points, which will affect the model's ability to recover these areas. Under a fixed sampling strategy, the sampling of light cannot be adjusted adaptively, which can easily lead to inaccurate distribution of sampling points, thereby reducing the effect of light field content generation. For example, uniform sampling or voxel-based sampling strategies may lead to insufficient sampling point density in some complex areas, resulting in missing image details. In open areas, too dense sampling may lead to a waste of computing resources.

[0004] Therefore, the existing sampling strategy not only affects the ability of the neural network model to restore the 3D scene, but also reduces the visual effect of the final generated 3D light field, making it difficult to meet the user's quality expectations in practical applications. Although some improved light sampling methods have been proposed in the prior art, there are still problems such as low sampling efficiency, serious loss of details and low generated image quality. Summary of the invention

[0005] The present invention provides a method and device for generating three-dimensional light field content by adaptive light sampling, so as to solve the problem that the existing sampling strategy in the prior art not only affects the restoration ability of the neural network model to the three-dimensional scene, but also reduces the visual effect of the finally generated three-dimensional light field, making it difficult to meet the user's expectations for quality in practical applications. The method and device can adaptively adjust the sampling mode according to the geometric distribution and detail requirements of the scene, thereby improving the quality of the three-dimensional light field content generation and better adapting to the needs of complex scenes.

[0006] The present invention provides a method for generating three-dimensional light field content by adaptive light sampling, comprising: Acquire multiple original images of a target three-dimensional scene at different viewing angles and camera calibration information of multiple RGB cameras that acquire the multiple original images; Establishing a three-dimensional world coordinate system based on the camera calibration information of the multiple RGB cameras, and constructing a three-dimensional voxel space and a three-dimensional sampling space based on the three-dimensional world coordinate system; In the three-dimensional world coordinate system, a multi-channel virtual rendering camera is set based on the parameters of the three-dimensional light field display device, and a rendering light for calculating a synthetic image is emitted based on the light field coding, wherein each sub-pixel in the synthetic image corresponds to one of the rendering lights; Performing initial sampling of the light emitted by the multi-channel virtual rendering camera in the three-dimensional voxel space based on a preset adaptive light sampling strategy to obtain a plurality of initial sampling points, calculating the point density of the plurality of initial sampling points, and calculating the transparency of the initial sampling points according to the point density of the initial sampling points; According to the set threshold of the number of initial sampling points, the resampling method is adaptively adjusted to perform resampling; The step of adaptively adjusting the resampling mode to perform resampling according to the set threshold of the number of initial sampling points includes: When the number of initial sampling points reaches the number threshold, voxel space resampling is performed according to the cumulative transparency distribution of the multiple initial sampling points to obtain coordinates and direction vectors of the multiple resampling points; When the number of initial sampling points does not reach the threshold, new sampling points are reselected within the three-dimensional sampling space as resampling points to obtain coordinates and direction vectors of multiple resampling points; Based on the coordinates and direction vectors of all resampled points on each rendering light, after position encoding and direction encoding, the colors and point densities of all resampled points on each rendering light are calculated, and the sub-pixel colors corresponding to the rendering light in the synthetic image are obtained after accumulation, so as to generate three-dimensional light field content; A target three-dimensional light field is generated based on the three-dimensional light field content and the three-dimensional light field display device.

[0007] In a possible implementation, the method further includes: The three-dimensional voxel space and the three-dimensional sampling space are constructed based on the three-dimensional world coordinate system, wherein the three-dimensional voxel space represents the spatial range of sampling points obtained by initial sampling or resampling based on the three-dimensional voxel grid, and the three-dimensional sampling space represents the spatial range of new sampling points reselected when the number of the initial sampling points does not reach the number threshold.

[0008] In a possible implementation, the method further includes: In the neural network model optimization training process, based on the pixel colors of the multi-channel RGB cameras and the corresponding original image pixel colors in the original images, the neural network model is optimized and trained with the goal of minimizing a preset error function; During the optimization training of the neural network model, the occupation state of the three-dimensional voxel grid is updated according to the position information of the three-dimensional voxel grid in the three-dimensional voxel space.

[0009] In a possible implementation, the method further includes: Determine a plurality of initial sampling line segments based on the plurality of initial sampling points; The midpoint position of each initial sampling line segment is input into the neural network model to obtain the point density corresponding to each initial sampling point; Calculate the transparency of each initial sampling point according to the point density of each initial sampling point; Accumulate the transparency of each initial sampling point on each rendering ray along the sampling ray to obtain a cumulative distribution of the transparency of the initial sampling point on each rendering ray; Determine the initial sampling line segment area corresponding to each resampling point based on the cumulative distribution of transparency of the multiple initial sampling points; Resampling is performed based on the initial sampling line segment area to obtain coordinates and direction vectors of multiple resampling points.

[0010] In a possible implementation, the method further includes: Setting the number of virtual rendering cameras according to the number of viewpoints of the three-dimensional light field display device; Setting the distance between the virtual rendering camera and the target three-dimensional scene according to the optimal viewing distance of the three-dimensional light field display device; The spacing of the virtual rendering cameras is set according to the maximum viewing angle of the three-dimensional light field display device.

[0011] In a possible implementation, the method further includes: Extracting viewpoint information corresponding to each sub-pixel from the synthetic image; The viewpoint information is output to the three-dimensional light field display device, and target three-dimensional light field content is generated based on the three-dimensional light field display apparatus.

[0012] The present invention also provides an adaptive light sampling three-dimensional light field content generation device, comprising the following modules: An acquisition module is used to acquire multiple original images of a target three-dimensional scene at different viewing angles and camera calibration information of multiple RGB cameras that acquire the multiple original images; An establishing module, used to establish a three-dimensional world coordinate system based on the camera calibration information of the multi-channel RGB cameras, and to construct a three-dimensional voxel space and a three-dimensional sampling space based on the three-dimensional world coordinate system; A setting module, configured to set a multi-channel virtual rendering camera based on parameters of a three-dimensional light field display device in the three-dimensional world coordinate system, and emit rendering rays for calculating a synthetic image based on light field coding, wherein each sub-pixel in the synthetic image corresponds to one of the rendering rays; A sampling module, for performing initial sampling of the light emitted by the multi-channel virtual rendering camera in the three-dimensional voxel space based on a preset adaptive light sampling strategy, obtaining a plurality of initial sampling points, calculating the point density of the plurality of initial sampling points, and calculating the transparency of the initial sampling points according to the point density; adaptively adjusting the resampling mode for resampling according to a set number threshold of the initial sampling points; adaptively adjusting the resampling mode for resampling according to the set number threshold of the initial sampling points, including: when the number of initial sampling points reaches the number threshold, performing voxel space resampling according to the cumulative transparency distribution of the plurality of initial sampling points, obtaining the coordinates and direction vectors of the plurality of resampling points; when the initial sampling points do not reach the number threshold, reselecting new sampling points as resampling points within the three-dimensional sampling space, obtaining the coordinates and direction vectors of the plurality of resampling points; A calculation module is used to calculate the color and point density of all resampled points on each rendering light based on the coordinates and direction vectors of all resampled points on each rendering light after position encoding and direction encoding, and to obtain the sub-pixel color corresponding to the rendering light in the synthetic image after accumulation, so as to generate three-dimensional light field content; A generating module is used to generate a target three-dimensional light field based on the three-dimensional light field content and the three-dimensional light field display device.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the adaptive light sampling three-dimensional light field content generation method as described in any one of the above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for generating three-dimensional light field content by adaptive light sampling as described in any one of the above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method for generating three-dimensional light field content by adaptive light sampling as described in any one of the above is implemented.

[0016] The adaptive light sampling 3D light field content generation method and device provided by the present invention obtain multiple original images of a target 3D scene at different viewing angles and camera calibration information of multiple RGB cameras that collect the multiple original images; establish a 3D world coordinate system based on the camera calibration information of the multiple RGB cameras, and construct a 3D voxel space and a 3D sampling space based on the 3D world coordinate system; set multiple virtual rendering cameras based on parameters of a 3D light field display device in the 3D world coordinate system, emit rendering rays for calculating a synthetic image based on light field coding, and each sub-pixel in the synthetic image corresponds to one rendering ray; perform initial sampling of the rays emitted by the multiple virtual rendering cameras in the 3D voxel space based on a preset adaptive light sampling strategy to obtain multiple initial sampling points, calculate the point density of the multiple initial sampling points, and calculate the transparency of the initial sampling points according to the point density; calculate the transparency of the initial sampling points according to the set initial sampling strategy .... According to the number threshold of the sampling points, the resampling mode is adaptively adjusted to perform resampling; the resampling mode is adaptively adjusted to perform resampling according to the number threshold of the initial sampling points set, including: when the number of the initial sampling points reaches the number threshold, voxel space resampling is performed according to the cumulative transparency distribution of the multiple initial sampling points to obtain the coordinates and direction vectors of the multiple resampling points; when the initial sampling points do not reach the number threshold, new sampling points are reselected as resampling points within the three-dimensional sampling space to obtain the coordinates and direction vectors of the multiple resampling points; based on the coordinates and direction vectors of all the resampling points on each rendering light, after position encoding and direction encoding, the coordinates and direction vectors are input into a neural network model to calculate the color and point density of all the resampling points on each rendering light, and the sub-pixel color corresponding to the rendering light in the synthetic image is obtained after accumulation, so as to generate the three-dimensional light field content; based on the three-dimensional light field content and the three-dimensional light field display device, a target three-dimensional light field is generated. Compared with the existing sampling strategies in the prior art, which not only affect the network model's ability to restore the three-dimensional scene, but also reduce the visual effect of the finally generated three-dimensional light field, making it difficult to meet users' expectations for quality in practical applications, this solution flexibly adjusts the sampling method according to the geometric distribution and detail requirements of the scene, thereby improving the quality of three-dimensional light field content generation and better adapting to the needs of complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 This is one of the flow charts of the method for generating three-dimensional light field content by adaptive light sampling provided by the present invention.

[0019] Figure 2 This is the second flow chart of the method for generating three-dimensional light field content by adaptive light sampling provided by the present invention.

[0020] Figure 3 It is a schematic diagram of the adaptive light sampling method provided by the present invention.

[0021] Figure 4 It is a structural schematic diagram of the adaptive light sampling three-dimensional light field content generation device provided by the present invention.

[0022] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] To facilitate understanding of the embodiments of the present invention, specific embodiments will be further explained below in conjunction with the accompanying drawings. The embodiments do not constitute a limitation on the embodiments of the present invention.

[0025] Figure 1 FIG. 1 is a flow chart of a method for generating three-dimensional light field content by adaptive light sampling provided by the present invention. Figure 1 As shown, the method specifically includes: S11, acquiring a plurality of original images of a target three-dimensional scene at different viewing angles and camera calibration information of a plurality of RGB cameras that acquire the plurality of original images.

[0026] In the embodiment of the present invention, a target three-dimensional scene is photographed from different viewing angles by a multi-channel RGB camera calibrated by a camera, and multiple original image data under multiple viewing angles are obtained. The camera calibration method may select the Zhang Zhengyou calibration method, or the open source calibration software Colmap may be used. After the camera calibration, the multiple original images are subjected to distortion correction and color correction to ensure that the obtained images have good performance in dedistortion and color consistency. The color correction of the multi-view image may be achieved by collecting and color calibrating a 24-color standard color plate in the target scene.

[0027] S12: establishing a three-dimensional world coordinate system based on the camera calibration information of the multiple RGB cameras, and constructing a three-dimensional voxel space and a three-dimensional sampling space based on the three-dimensional world coordinate system.

[0028] Based on the multi-view images collected by multiple RGB cameras, the camera calibration method is used to calculate the camera's intrinsic and extrinsic parameters as camera calibration information, thereby establishing a three-dimensional world coordinate system. In this coordinate system, a three-dimensional sampling space and a three-dimensional voxel space are constructed. At the same time, a three-dimensional voxel grid is constructed in the three-dimensional voxel space, that is, the three-dimensional space is divided into multiple voxels, and each voxel records its occupancy state in the three-dimensional space.

[0029] S13. Setting a multi-channel virtual rendering camera based on parameters of a three-dimensional light field display device in the three-dimensional world coordinate system, and emitting rendering rays for calculating a synthetic image based on light field coding, wherein each sub-pixel in the synthetic image corresponds to one of the rendering rays.

[0030] The number of virtual rendering cameras is set according to the number of viewpoints of the three-dimensional light field display device, the position of the virtual rendering camera from the target three-dimensional scene is set according to the optimal viewing distance of the three-dimensional light field display device, and the spacing of the virtual rendering cameras is set according to the maximum viewing angle of the three-dimensional light field display device.

[0031] Furthermore, rendering rays for calculating the composite image are emitted according to the light field encoding, and each rendering ray corresponds to each sub-pixel in the later composite image.

[0032] S14. Based on a preset adaptive light sampling strategy, initially sampling is performed on the light emitted by the multi-channel virtual rendering camera in the three-dimensional voxel space to obtain a plurality of initial sampling points, and the point density of the plurality of initial sampling points is calculated, and the transparency is calculated according to the point density of the initial sampling points.

[0033] The rays emitted by the virtual rendering camera are sampled using an adaptive ray sampling strategy as they pass through the 3D voxel space.

[0034] The adaptive light sampling strategy first performs initial sampling in the occupied three-dimensional voxel space based on the three-dimensional voxel grid to form the left and right endpoints of the initial sampling line segment, collectively referred to as the initial sampling points, and adaptively adjusts the sampling method according to whether the actual number of initial sampling points reaches the set initial sampling point number threshold (for example, the initial sampling point number threshold is set to 64 or 128). When the initial sampling points are sufficient (the number is greater than or equal to the initial sampling point number threshold), the initial sampling point positions are used to input the neural network model to calculate the cumulative transparency distribution of the initial sampling points, and voxel space resampling is performed based on the cumulative transparency distribution. When the initial sampling points are insufficient, sampling is performed within the three-dimensional sampling space.

[0035] Specifically, the purpose of implementing adaptive light sampling is to flexibly adjust the sampling strategy according to the geometric distribution of the scene. The specific implementation includes initial sampling and resampling processes. First, initial sampling is performed in the three-dimensional voxel space based on the occupancy state of the three-dimensional voxel grid to generate a series of initial sampling segments. The left and right endpoints of these segments are called initial sampling points. By calculating the length and position of each initial sampling segment, its influence range on the surrounding area is determined. After that, the sampling method is adaptively adjusted according to the sufficiency of the initial sampling points. When the initial sampling points are sufficient, the positions of these points are input into the neural network model to calculate the accumulated initial sampling point density, calculate its transparency according to the point density of the initial sampling points, calculate the cumulative distribution of transparency according to the transparency of the initial sampling points, and then perform voxel space resampling. When the initial sampling points are insufficient, the sampling points are reselected within the three-dimensional sampling space. At this time, a method different from the initial sampling of the voxel space or the resampling of the voxel space is adopted to ensure that the resampling points have sufficient spatial span range in the open area. The three-dimensional voxel space and the three-dimensional sampling space are both in the set three-dimensional world coordinate system, and their respective set coverage areas have intersections.

[0036] S15. According to the set threshold value of the number of initial sampling points, adaptively adjust the resampling method to perform resampling.

[0037] In the process of three-dimensional voxel space resampling, the area delimited by the initial sampling segment where each resampling point is located is determined according to the cumulative distribution of transparency of the initial sampling points, and a front end point is defined based on the position of the initial sampling segment where each resampling point is located. The position of the front end point is before the left endpoint of the initial sampling segment, and the length from the front end point to the left endpoint of the initial sampling segment is related to the length from the left endpoint to the right endpoint of the initial sampling segment (for example, the distances are equal, or there is a related calculation relationship), and the position of the resampling point is obtained from the positions of the front end point, the left endpoint, and the right endpoint, and then resampling is performed to obtain multiple resampling points.

[0038] Voxel space resampling is performed on the basis of initial sampling to improve the accuracy of light field content generation. Specifically, the area selected by the initial sampling line segment where each resampling point is located is determined according to the cumulative distribution of transparency of the initial sampling point. Among them, the initial sampling line segment areas where different resampling points are located may be the same or different, and the area where the resampling point is located is determined by the cumulative distribution of transparency of the initial sampling point. A front end point is defined in front of each selected initial sampling line segment, and its position is determined by the length relationship between the left endpoint and the right endpoint. The distance from the front end point to the left endpoint is used as the basis for resampling to ensure that the position of the resampling point is within a reasonable range. The distance from the front end point to the left endpoint can be selected to be slightly less than, equal to, or slightly greater than the distance from the left endpoint to the right endpoint. The resampling result is calculated based on the position information of the front end point, the left endpoint, and the right endpoint. This process will effectively limit the range of the resampling points and prevent the resampling points from re-occupying invalid space areas.

[0039] S16, based on the coordinates and direction vectors of all resampled points on each rendering light, after position encoding and direction encoding, the colors and point density of all resampled points on each rendering light are calculated by inputting them into a neural network model, and the sub-pixel colors corresponding to the rendering light in the synthetic image are obtained after accumulation, so as to generate three-dimensional light field content.

[0040] According to each rendering ray emitted by the virtual rendering camera, the coordinates and direction vector of the resampling point on the rendering ray can be determined.

[0041] The coordinates and direction vector of each resampling point on each rendering ray are encoded (such as hash coding, sine-cosine function coding, etc.) and input into the neural network model to obtain the color and point density of all resampling points, so as to calculate the transparency of the resampling points. The color of the image pixel corresponding to each rendering ray can be calculated by weighted averaging the color of the resampling points according to the transparency of the resampling points.

[0042] The coordinates and direction vector of the resampled points on each ray are encoded and input into the neural network model. In this way, the neural network model can learn how to generate corresponding color and point density information based on the sampling point position and direction during training.

[0043] Optionally, during the neural network optimization training process, the image pixel color of the image pixel coordinates corresponding to each rendering ray can be calculated based on the neural network model, and compared with the original image pixel color of the corresponding image pixel coordinates in the original image, with the goal of calculating and minimizing the error function, to perform optimization training of the neural network model. During the neural network optimization training process, the occupancy state of the three-dimensional voxel grid is updated through the neural network model according to the position information of the three-dimensional voxel grid.

[0044] Specifically, during the neural network optimization training process, the occupancy state of the three-dimensional voxel grid needs to be dynamically updated. Taking the center position of each three-dimensional voxel grid as the reference, add Gaussian random noise of a certain intensity to obtain the corresponding position information. The position information after adding noise is input into the neural network to obtain the corresponding point density, and whether the voxel grid is occupied is judged based on whether the point density reaches the set point density threshold. During the training process, the occupancy state of the voxels is continuously updated according to the point density of the target scene to ensure that the neural network model always maintains an accurate reflection of the target scene. This method can improve the geometric distribution of the dynamically updated scene and improve the effect of adaptive light sampling.

[0045] S17: Generate a target three-dimensional light field based on the three-dimensional light field content and the three-dimensional light field display device.

[0046] During the rendering process, multiple virtual rendering cameras are controlled to emit rendering rays, and the coordinates of the resampling points are calculated using an adaptive ray sampling strategy. This process ensures that rich scene information is captured from all perspectives. Through the neural network model, a synthetic image is calculated to achieve 3D light field content generation. The 3D light field content will integrate information from different perspectives and provide a basis for subsequent 3D light field generation.

[0047] Furthermore, the viewpoint information corresponding to each sub-pixel in the synthesized image and the color information of each sub-pixel are used as the three-dimensional light field content, and a three-dimensional light field display device is used to generate a target three-dimensional light field.

[0048] The adaptive light sampling three-dimensional light field content generation method provided by the present invention obtains a plurality of original images of a target three-dimensional scene at different viewing angles and camera calibration information of a multi-channel RGB camera that collects the plurality of original images; establishes a three-dimensional world coordinate system based on the camera calibration information of the multi-channel RGB camera, and constructs a three-dimensional voxel space and a three-dimensional sampling space based on the three-dimensional world coordinate system; sets a multi-channel virtual rendering camera based on parameters of a three-dimensional light field display device in the three-dimensional world coordinate system, emits rendering light for calculating a synthetic image based on light field coding, and each sub-pixel in the synthetic image corresponds to a rendering light; performs initial sampling of the light emitted by the multi-channel virtual rendering camera in the three-dimensional voxel space based on a preset adaptive light sampling strategy to obtain a plurality of initial sampling points, calculates the point density of the plurality of initial sampling points, and calculates the transparency of the initial sampling points according to the point density; calculates the transparency of the initial sampling points according to the set initial sampling points According to the number threshold of the set initial sampling points, the resampling method is adaptively adjusted to perform resampling; the resampling method is adaptively adjusted to perform resampling according to the number threshold of the set initial sampling points, including: when the number of initial sampling points reaches the number threshold, voxel space resampling is performed according to the cumulative transparency distribution of the multiple initial sampling points to obtain the coordinates and direction vectors of the multiple resampling points; when the initial sampling points do not reach the number threshold, new sampling points are reselected within the three-dimensional sampling space as resampling points to obtain the coordinates and direction vectors of the multiple resampling points; based on the coordinates and direction vectors of all the resampling points on each rendering light, after position encoding and direction encoding, the coordinates and direction vectors are input into a neural network model to calculate the color and point density of all the resampling points on each rendering light, and the sub-pixel color corresponding to the rendering light in the synthetic image is obtained after accumulation, so as to generate three-dimensional light field content; based on the three-dimensional light field content and the three-dimensional light field display device, a target three-dimensional light field is generated. Compared with the existing sampling strategies in the prior art, which not only limit the ability of the neural network model to restore the three-dimensional scene, but also reduce the visual effect of the finally generated three-dimensional light field, making it difficult to meet the user's expectations for the quality of the light field content in practical applications, this method flexibly and adaptively adjusts the sampling method according to the geometric distribution and detail requirements of the scene, thereby improving the quality of the three-dimensional light field content generation and better adapting to the needs of complex scenes.

[0049] Figure 2 FIG. 2 is a flow chart of the method for generating three-dimensional light field content by adaptive light sampling provided by the present invention. Figure 2 As shown, the method specifically includes: S21. Acquire multiple original images of a target three-dimensional scene at different viewing angles and camera calibration information of multiple RGB cameras that acquire the multiple original images.

[0050] In the embodiment of the present invention, a target three-dimensional scene is photographed from different viewing angles by a multi-channel RGB camera calibrated by a camera, and multiple original image data under multiple viewing angles are obtained. The camera calibration method may select the Zhang Zhengyou calibration method, or the open source calibration software Colmap may be used. After the camera calibration, the multiple original images are subjected to distortion correction and color correction to ensure that the obtained images have good performance in de-distortion and color consistency. The color correction of the multi-view image may be achieved by collecting and color calibrating a 24-color standard color plate in the target scene.

[0051] S22: establishing a three-dimensional world coordinate system based on the camera calibration information of the multiple RGB cameras, and constructing a three-dimensional voxel space and a three-dimensional sampling space based on the three-dimensional world coordinate system.

[0052] Based on the multi-view images collected by multiple RGB cameras, the camera calibration method is used to calculate the camera's intrinsic and extrinsic parameters as camera calibration information, thereby establishing a three-dimensional world coordinate system. In this coordinate system, a three-dimensional sampling space and a three-dimensional voxel space are constructed. At the same time, a three-dimensional voxel grid is constructed in the three-dimensional voxel space, that is, the three-dimensional space is divided into multiple voxels, and each voxel records its occupancy state in the three-dimensional space.

[0053] S23. In the three-dimensional world coordinate system, a multi-channel virtual rendering camera is set based on parameters of the three-dimensional light field display device, and rendering rays for calculating a synthetic image are emitted based on light field coding, wherein each sub-pixel in the synthetic image corresponds to one of the rendering rays.

[0054] S24. Based on a preset adaptive light sampling strategy, initially sampling is performed on the light emitted by the multi-channel virtual rendering camera in the three-dimensional voxel space to obtain a plurality of initial sampling points, and the point density of the plurality of initial sampling points is calculated, and the transparency is calculated according to the point density of the initial sampling points.

[0055] S25. Determine a plurality of initial sampling line segments based on the plurality of initial sampling points.

[0056] S26. Input the midpoint position of each initial sampling line segment into the neural network model to obtain the point density corresponding to each initial sampling point.

[0057] S27. Calculate the transparency of each initial sampling point according to the point density of each initial sampling point.

[0058] S28, accumulating the transparency of each initial sampling point on each rendering ray along the sampling ray to obtain a cumulative distribution of the transparency of the initial sampling point on each rendering ray.

[0059] The following is a unified description of S23-S28: The number of virtual rendering cameras is set according to the number of viewpoints of the three-dimensional light field display device, the position of the virtual rendering camera from the target three-dimensional scene is set according to the optimal viewing distance of the three-dimensional light field display device, and the spacing of the virtual rendering cameras is set according to the maximum viewing angle of the three-dimensional light field display device.

[0060] Furthermore, rendering rays for calculating the composite image are emitted according to the light field encoding, and each rendering ray corresponds to each sub-pixel in the later composite image.

[0061] The rays emitted by the virtual rendering camera are sampled using an adaptive ray sampling strategy as they pass through the 3D voxel space.

[0062] like Figure 3 As shown, the adaptive light sampling strategy first performs initial sampling in the occupied three-dimensional voxel space based on the three-dimensional voxel grid to form the left and right endpoints of the initial sampling line segment, collectively referred to as the initial sampling points, and adaptively adjusts the sampling method according to whether the actual number of initial sampling points reaches the set initial sampling point number threshold (for example, the initial sampling point number threshold is set to 64 or 128). When the initial sampling points are sufficient (the number is greater than or equal to the initial sampling point number threshold), the initial sampling point positions are used to input the neural network model to calculate the cumulative transparency distribution of the initial sampling points, and voxel space resampling is performed based on the cumulative transparency distribution. When the initial sampling points are insufficient, sampling is performed within the three-dimensional sampling space.

[0063] Specifically, the purpose of implementing adaptive light sampling is to flexibly adjust the sampling strategy according to the geometric distribution of the scene. The specific implementation includes initial sampling and resampling processes. First, initial sampling is performed in the three-dimensional voxel space based on the occupancy state of the three-dimensional voxel grid to generate a series of initial sampling segments. The left and right endpoints of these segments are called initial sampling points. By calculating the length and position of each initial sampling segment, its influence range on the surrounding area is determined. After that, the sampling method is adaptively adjusted according to the sufficiency of the initial sampling points. When the initial sampling points are sufficient, the positions of these points are input into the neural network model to calculate the accumulated initial sampling point density, calculate its transparency according to the point density of the initial sampling points, calculate the cumulative distribution of transparency according to the transparency of the initial sampling points, and then perform voxel space resampling. When the initial sampling points are insufficient, the sampling points are reselected within the three-dimensional sampling space. At this time, a method different from the initial sampling of the voxel space or the resampling of the voxel space is adopted to ensure that the resampling points have sufficient spatial span range in the open area. The three-dimensional voxel space and the three-dimensional sampling space are both in the set three-dimensional world coordinate system, and their respective set coverage areas have intersections.

[0064] S29: Determine an initial sampling line segment area corresponding to each resampling point based on the cumulative distribution of transparency of the multiple initial sampling points.

[0065] S210 , resampling is performed based on the initial sampling line segment area to obtain coordinates and direction vectors of multiple resampling points.

[0066] In the process of three-dimensional voxel space resampling, the area delimited by the initial sampling segment where each resampling point is located is determined according to the cumulative distribution of transparency of the initial sampling points, and a front end point is defined based on the position of the initial sampling segment where each resampling point is located. The position of the front end point is before the left endpoint of the initial sampling segment, and the length from the front end point to the left endpoint of the initial sampling segment is related to the length from the left endpoint to the right endpoint of the initial sampling segment (for example, the distances are equal, or there is a related calculation relationship), and the position of the resampling point is obtained from the positions of the front end point, the left endpoint, and the right endpoint, and then resampling is performed to obtain multiple resampling points.

[0067] Voxel space resampling is performed on the basis of initial sampling to improve the accuracy of light field content generation. Specifically, the area selected by the initial sampling line segment where each resampling point is located is determined according to the cumulative distribution of transparency of the initial sampling point. Among them, the initial sampling line segment areas where different resampling points are located may be the same or different, and the area where the resampling point is located is determined by the cumulative distribution of transparency of the initial sampling point. A front end point is defined in front of each selected initial sampling line segment, and its position is determined by the length relationship between the left endpoint and the right endpoint. The distance from the front end point to the left endpoint is used as the basis for resampling to ensure that the position of the resampling point is within a reasonable range. The distance from the front end point to the left endpoint can be selected to be slightly less than, equal to, or slightly greater than the distance from the left endpoint to the right endpoint. The resampling result is calculated based on the position information of the front end point, the left endpoint, and the right endpoint. This process will effectively limit the range of the resampling points and prevent the resampling points from re-occupying invalid space areas.

[0068] S211, based on the coordinates and direction vectors of all resampled points on each rendering light, after position encoding and direction encoding, the colors and point density of all resampled points on each rendering light are calculated by inputting them into a neural network model, and the sub-pixel colors corresponding to the rendering light in the synthetic image are obtained after accumulation, so as to generate three-dimensional light field content.

[0069] According to each rendering ray emitted by the virtual rendering camera, the coordinates and direction vector of the resampling point on the rendering ray can be determined.

[0070] The coordinates and direction vector of each resampling point on each rendering ray are encoded (such as hash coding, sine-cosine function coding, etc.) and input into the neural network model to obtain the color and point density of all resampling points, so as to calculate the transparency of the resampling points. The color of the image pixel corresponding to each rendering ray can be calculated by weighted averaging the color of the resampling points according to the transparency of the resampling points.

[0071] The coordinates and direction vector of the resampled points on each ray are encoded and input into the neural network model. In this way, the neural network model can learn how to generate corresponding color and point density information based on the sampling point position and direction during training.

[0072] Optionally, during the neural network optimization training process, the image pixel color of the image pixel coordinates corresponding to each rendering ray can be calculated based on the neural network model, and compared with the original image pixel color of the corresponding image pixel coordinates in the original image, with the goal of calculating and minimizing the error function, to perform optimization training of the neural network model. During the neural network optimization training process, the occupancy state of the three-dimensional voxel grid is updated through the neural network model according to the position information of the three-dimensional voxel grid.

[0073] Specifically, during the neural network optimization training process, the occupancy state of the three-dimensional voxel grid needs to be dynamically updated. Taking the center position of each three-dimensional voxel grid as the reference, add Gaussian random noise of a certain intensity to obtain the corresponding position information. The position information after adding noise is input into the neural network to obtain the corresponding point density, and whether the voxel grid is occupied is judged based on whether the point density reaches the set point density threshold. During the training process, the occupancy state of the voxels is continuously updated according to the point density of the target scene to ensure that the neural network model always maintains an accurate reflection of the target scene. This method can improve the geometric distribution of the dynamically updated scene and improve the effect of adaptive light sampling.

[0074] S212: Extract viewpoint information corresponding to each sub-pixel from the synthesized image.

[0075] S213: Output the viewpoint information to the three-dimensional light field display device to generate a target three-dimensional light field.

[0076] During the rendering process, multiple virtual rendering cameras are controlled to emit rendering rays, and the coordinates of the resampling points are calculated using an adaptive ray sampling strategy. This process ensures that rich scene information is captured from all perspectives. Through the neural network model, a synthetic image is calculated to achieve 3D light field content generation. The 3D light field content will integrate information from different perspectives and provide a basis for subsequent 3D light field generation.

[0077] Furthermore, the viewpoint information corresponding to each sub-pixel in the synthesized image and the color information of each sub-pixel are used as the three-dimensional light field content, and a three-dimensional light field display device is used to generate a target three-dimensional light field.

[0078] The adaptive light sampling three-dimensional light field content generation method provided by the present invention obtains a plurality of original images of a target three-dimensional scene at different viewing angles and camera calibration information of a multi-channel RGB camera that collects the plurality of original images; establishes a three-dimensional world coordinate system based on the camera calibration information of the multi-channel RGB camera, and constructs a three-dimensional voxel space and a three-dimensional sampling space based on the three-dimensional world coordinate system; sets a multi-channel virtual rendering camera based on parameters of a three-dimensional light field display device in the three-dimensional world coordinate system, emits rendering light for calculating a synthetic image based on light field coding, and each sub-pixel in the synthetic image corresponds to a rendering light; performs initial sampling of the light emitted by the multi-channel virtual rendering camera in the three-dimensional voxel space based on a preset adaptive light sampling strategy to obtain a plurality of initial sampling points, calculates the point density of the plurality of initial sampling points, and calculates the transparency of the initial sampling points according to the point density; calculates the transparency of the initial sampling points according to the set initial sampling points The method of adaptively adjusting the resampling mode for resampling according to the number threshold of the set initial sampling points includes: when the number of initial sampling points reaches the number threshold, resampling the voxel space according to the cumulative transparency distribution of the multiple initial sampling points to obtain the coordinates and direction vectors of the multiple resampling points; when the initial sampling points do not reach the number threshold, reselecting new sampling points as resampling points within the three-dimensional sampling space to obtain the coordinates and direction vectors of the multiple resampling points; based on the position encoding and direction encoding of the coordinates and direction vectors of all resampling points on each rendering light, inputting the neural network model to calculate the color and point density of all resampling points on each rendering light, accumulating and obtaining the sub-pixel color corresponding to the rendering light in the synthetic image, and generating the three-dimensional light field content; generating the target three-dimensional light field based on the three-dimensional light field content and the three-dimensional light field display device. According to the method, the sampling mode is flexibly adjusted according to the geometric distribution and detail requirements of the scene, thereby improving the quality of the three-dimensional light field content generation and better adapting to the needs of complex scenes.

[0079] The adaptive light sampling 3D light field content generation device provided by the present invention is described below. The adaptive light sampling 3D light field content generation device described below and the adaptive light sampling 3D light field content generation method described above can be referenced to each other.

[0080] Figure 4 : is a schematic diagram of the structure of the adaptive light sampling three-dimensional light field content generation device provided by the present invention, which specifically includes: The acquisition module 401 is used to acquire multiple original images of the target three-dimensional scene at different viewing angles and camera calibration information of multiple RGB cameras that acquire the multiple original images. For detailed description, please refer to the relevant description corresponding to the above method embodiment, which will not be repeated here.

[0081] The establishment module 402 is used to establish a 3D world coordinate system based on the camera calibration information of the multi-channel RGB camera, and to construct a 3D voxel space and a 3D sampling space based on the 3D world coordinate system. For detailed description, please refer to the relevant description corresponding to the above method embodiment, which will not be repeated here.

[0082] The setting module 403 is used to set a multi-channel virtual rendering camera based on the parameters of the three-dimensional light field display device in the three-dimensional world coordinate system, and emit rendering rays for calculating a synthetic image based on the light field coding, wherein each sub-pixel in the synthetic image corresponds to one of the rendering rays. For detailed description, please refer to the relevant description corresponding to the above method embodiment, which will not be repeated here.

[0083] The sampling module 404 is used to perform initial sampling of the light emitted by the multi-channel virtual rendering camera in the three-dimensional voxel space based on a preset adaptive light sampling strategy to obtain multiple initial sampling points, and calculate the point density of the multiple initial sampling points, and calculate the transparency of the initial sampling points according to the point density; according to the set number threshold of the initial sampling points, adaptively adjust the resampling method to perform resampling; the adaptive adjustment of the resampling method to perform resampling according to the set number threshold of the initial sampling points includes: when the number of initial sampling points reaches the number threshold, voxel space resampling is performed according to the cumulative transparency distribution of the multiple initial sampling points to obtain the coordinates and direction vectors of the multiple resampling points; when the initial sampling points do not reach the number threshold, new sampling points are reselected within the three-dimensional sampling space as resampling points to obtain the coordinates and direction vectors of the multiple resampling points. For detailed descriptions, please refer to the relevant descriptions corresponding to the above method embodiments, which will not be repeated here.

[0084] The calculation module 405 is used to calculate the color and point density of all resampled points on each rendering light based on the coordinates and direction vectors of all resampled points on each rendering light after position encoding and direction encoding, and obtain the sub-pixel color corresponding to the rendering light in the synthetic image after accumulation, so as to generate the three-dimensional light field content. For detailed description, please refer to the relevant description corresponding to the above method embodiment, which will not be repeated here.

[0085] The generating module 406 is used to generate a target three-dimensional light field based on the three-dimensional light field content and the three-dimensional light field display device. For detailed description, please refer to the relevant description corresponding to the above method embodiment, which will not be repeated here.

[0086] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5As shown, the electronic device may include: a processor (processor) 810, a communication interface (Communications Interface) 820, a memory (memory) 830 and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call the logic instructions in the memory 830 to execute the adaptive light sampling three-dimensional light field content generation method, which includes: obtaining multiple original images of the target three-dimensional scene at different viewing angles and camera calibration information of multiple RGB cameras that collect the multiple original images; establishing a three-dimensional world coordinate system based on the camera calibration information of the multiple RGB cameras, and constructing a three-dimensional voxel space and a three-dimensional sampling space based on the three-dimensional world coordinate system; setting a multi-channel virtual rendering camera based on the parameters of the three-dimensional light field display device in the three-dimensional world coordinate system, and emitting rendering light for calculating a synthetic image based on light field coding, wherein each sub-pixel in the synthetic image corresponds to one of the rendering light rays; performing initial sampling of the light emitted by the multi-channel virtual rendering camera in the three-dimensional voxel space based on a preset adaptive light sampling strategy to obtain multiple initial sampling points, and calculating the point density of the multiple initial sampling points, and calculating the transparency of the initial sampling points according to the point density of the initial sampling points. degree; adaptively adjusting the resampling mode to perform resampling according to a set number threshold of initial sampling points; the adaptively adjusting the resampling mode to perform resampling according to the set number threshold of initial sampling points comprises: when the number of initial sampling points reaches the number threshold, performing voxel space resampling according to the cumulative transparency distribution of the plurality of initial sampling points to obtain the coordinates and direction vectors of the plurality of resampling points; when the initial sampling points do not reach the number threshold, reselecting new sampling points as resampling points within the three-dimensional sampling space to obtain the coordinates and direction vectors of the plurality of resampling points; based on the coordinates and direction vectors of all resampling points on each rendering light, after position encoding and direction encoding, inputting into a neural network model to calculate the color and point density of all resampling points on each rendering light, accumulating to obtain the sub-pixel color corresponding to the rendering light in the synthetic image, and generating three-dimensional light field content; generating a target three-dimensional light field based on the three-dimensional light field content and the three-dimensional light field display device.

[0087] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0088] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the adaptive light sampling three-dimensional light field content generation method provided by the above methods, the method including: obtaining multiple original images of a target three-dimensional scene under different viewing angles and camera calibration information of a multi-channel RGB camera that collects the multiple original images; establishing a three-dimensional world coordinate system based on the camera calibration information of the multi-channel RGB camera, and constructing a three-dimensional voxel space and a three-dimensional sampling space based on the three-dimensional world coordinate system; setting a multi-channel virtual rendering camera based on parameters of a three-dimensional light field display device in the three-dimensional world coordinate system, and emitting rendering rays for calculating a synthetic image based on light field coding, wherein each sub-pixel in the synthetic image corresponds to one of the rendering rays; performing initial sampling of the rays emitted by the multi-channel virtual rendering cameras in the three-dimensional voxel space based on a preset adaptive light sampling strategy to obtain multiple initial sampling points, The point density of the multiple initial sampling points is calculated, and the transparency is calculated according to the point density of the initial sampling points; the resampling method is adaptively adjusted according to the set number threshold of the initial sampling points for resampling; the resampling method is adaptively adjusted according to the set number threshold of the initial sampling points for resampling, including: when the number of initial sampling points reaches the number threshold, voxel space resampling is performed according to the cumulative transparency distribution of the multiple initial sampling points to obtain the coordinates and direction vectors of the multiple resampling points; when the initial sampling points do not reach the number threshold, new sampling points are reselected as resampling points within the three-dimensional sampling space to obtain the coordinates and direction vectors of the multiple resampling points; based on the coordinates and direction vectors of all resampling points on each rendering light, after position encoding and direction encoding, the coordinates and direction vectors are input into a neural network model to calculate the color and point density of all resampling points on each rendering light, and the sub-pixel color corresponding to the rendering light in the synthetic image is obtained after accumulation, and the three-dimensional light field content is generated; based on the three-dimensional light field content and the three-dimensional light field display device, a target three-dimensional light field is generated.

[0089] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the adaptive light sampling three-dimensional light field content generation method provided by the above-mentioned methods, the method comprising: obtaining multiple original images of a target three-dimensional scene at different viewing angles and camera calibration information of a multi-channel RGB camera that collects the multiple original images; establishing a three-dimensional world coordinate system based on the camera calibration information of the multi-channel RGB camera, and constructing a three-dimensional voxel space and a three-dimensional sampling space based on the three-dimensional world coordinate system; setting a multi-channel virtual rendering camera based on parameters of a three-dimensional light field display device in the three-dimensional world coordinate system, emitting rendering rays for calculating a synthetic image based on light field coding, wherein each sub-pixel in the synthetic image corresponds to one of the rendering rays; performing initial sampling of the rays emitted by the multi-channel virtual rendering cameras in the three-dimensional voxel space based on a preset adaptive light sampling strategy to obtain multiple initial sampling points, and calculating the points of the multiple initial sampling points Density, calculating the transparency of the initial sampling points according to the point density; adaptively adjusting the resampling method to perform resampling according to the set number threshold of the initial sampling points; the adaptively adjusting the resampling method to perform resampling according to the set number threshold of the initial sampling points includes: when the number of initial sampling points reaches the number threshold, performing voxel space resampling according to the cumulative transparency distribution of the multiple initial sampling points to obtain the coordinates and direction vectors of the multiple resampling points; when the initial sampling points do not reach the number threshold, reselecting new sampling points as resampling points within the three-dimensional sampling space to obtain the coordinates and direction vectors of the multiple resampling points; based on the coordinates and direction vectors of all resampling points on each rendering light, after position encoding and direction encoding, inputting into the neural network model to calculate the color and point density of all resampling points on each rendering light, accumulating to obtain the sub-pixel color corresponding to the rendering light in the synthetic image, and generating three-dimensional light field content; generating a target three-dimensional light field based on the three-dimensional light field content and the three-dimensional light field display device.

[0090] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0091] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating three-dimensional light field content by adaptive light sampling, characterized in that: include: Acquire multiple original images of a target three-dimensional scene at different viewing angles and camera calibration information of multiple RGB cameras that acquire the multiple original images; Establishing a three-dimensional world coordinate system based on the camera calibration information of the multiple RGB cameras, and constructing a three-dimensional voxel space and a three-dimensional sampling space based on the three-dimensional world coordinate system; In the three-dimensional world coordinate system, a multi-channel virtual rendering camera is set based on the parameters of the three-dimensional light field display device, and a rendering light for calculating a synthetic image is emitted based on the light field coding, wherein each sub-pixel in the synthetic image corresponds to one of the rendering lights; Performing initial sampling of the light emitted by the multi-channel virtual rendering camera in the three-dimensional voxel space based on a preset adaptive light sampling strategy to obtain a plurality of initial sampling points, calculating the point density of the plurality of initial sampling points, and calculating the transparency of the initial sampling points according to the point density of the initial sampling points; According to the set threshold of the number of initial sampling points, the resampling method is adaptively adjusted to perform resampling; The step of adaptively adjusting the resampling mode to perform resampling according to the set threshold of the number of initial sampling points includes: When the number of initial sampling points reaches the number threshold, voxel space resampling is performed according to the cumulative transparency distribution of the multiple initial sampling points to obtain coordinates and direction vectors of the multiple resampling points; When the number of initial sampling points does not reach the threshold, new sampling points are reselected within the three-dimensional sampling space as resampling points to obtain coordinates and direction vectors of multiple resampling points; Based on the coordinates and direction vectors of all resampled points on each rendering light, after position encoding and direction encoding, the colors and point densities of all resampled points on each rendering light are calculated, and the sub-pixel colors corresponding to the rendering light in the synthetic image are obtained after accumulation, so as to generate three-dimensional light field content; A target three-dimensional light field is generated based on the three-dimensional light field content and the three-dimensional light field display device.

2. The method according to claim 1, characterized in that The establishing of a three-dimensional world coordinate system based on the camera calibration information of the multi-channel RGB cameras, and the construction of a three-dimensional voxel space and a three-dimensional sampling space based on the three-dimensional world coordinate system include: The three-dimensional voxel space and the three-dimensional sampling space are constructed based on the three-dimensional world coordinate system, wherein the three-dimensional voxel space represents the spatial range of sampling points obtained by initial sampling or resampling based on the three-dimensional voxel grid, and the three-dimensional sampling space represents the spatial range of new sampling points reselected when the number of the initial sampling points does not reach the number threshold.

3. The method according to claim 1, characterized in that The method further comprises: In the neural network model optimization training process, based on the pixel colors of the multi-channel RGB cameras and the corresponding original image pixel colors in the original images, the neural network model is optimized and trained with the goal of minimizing a preset error function; During the optimization training of the neural network model, the occupation state of the three-dimensional voxel grid is updated according to the position information of the three-dimensional voxel grid in the three-dimensional voxel space.

4. The method according to claim 2, characterized in that: When the number of initial sampling points reaches the number threshold, voxel space resampling is performed according to the cumulative transparency distribution of the multiple initial sampling points to obtain the coordinates and direction vectors of the multiple resampling points, including: Determine a plurality of initial sampling line segments based on the plurality of initial sampling points; The midpoint position of each initial sampling line segment is input into the neural network model to obtain the point density corresponding to each initial sampling point; Calculate the transparency of each initial sampling point according to the point density of each initial sampling point; Accumulate the transparency of each initial sampling point on each rendering ray along the sampling ray to obtain a cumulative distribution of the transparency of the initial sampling point on each rendering ray; Determine the initial sampling line segment area corresponding to each resampling point based on the cumulative distribution of transparency of the multiple initial sampling points; Resampling is performed based on the initial sampling line segment area to obtain coordinates and direction vectors of multiple resampling points.

5. The method according to claim 4, characterized in that The method of setting a multi-channel virtual rendering camera based on parameters of a three-dimensional light field display device in the three-dimensional world coordinate system includes: Setting the number of virtual rendering cameras according to the number of viewpoints of the three-dimensional light field display device; Setting the distance between the virtual rendering camera and the target three-dimensional scene according to the optimal viewing distance of the three-dimensional light field display device; The spacing of the virtual rendering cameras is set according to the maximum viewing angle of the three-dimensional light field display device.

6. The method according to claim 5, characterized in that The generating a target three-dimensional light field based on the three-dimensional light field content and the three-dimensional light field display device comprises: Extracting viewpoint information corresponding to each sub-pixel from the synthetic image; The viewpoint information is output to the three-dimensional light field display device, and a target three-dimensional light field is generated based on the three-dimensional light field display apparatus.

7. An adaptive light sampling 3D light field content generation device, characterized in that: include: An acquisition module is used to acquire multiple original images of a target three-dimensional scene at different viewing angles and camera calibration information of multiple RGB cameras that acquire the multiple original images; An establishing module, used to establish a three-dimensional world coordinate system based on the camera calibration information of the multi-channel RGB cameras, and to construct a three-dimensional voxel space and a three-dimensional sampling space based on the three-dimensional world coordinate system; A setting module, configured to set a multi-channel virtual rendering camera based on parameters of a three-dimensional light field display device in the three-dimensional world coordinate system, and emit rendering rays for calculating a synthetic image based on light field coding, wherein each sub-pixel in the synthetic image corresponds to one of the rendering rays; A sampling module, configured to perform initial sampling of the light emitted by the multi-channel virtual rendering camera in the three-dimensional voxel space based on a preset adaptive light sampling strategy to obtain a plurality of initial sampling points, calculate the point density of the plurality of initial sampling points, and calculate the transparency of the initial sampling points according to the point density of the initial sampling points; According to the set number threshold of initial sampling points, the resampling mode is adaptively adjusted to perform resampling; the step of adaptively adjusting the resampling mode to perform resampling according to the set number threshold of initial sampling points includes: when the number of initial sampling points reaches the number threshold, voxel space resampling is performed according to the cumulative transparency distribution of the multiple initial sampling points to obtain the coordinates and direction vectors of the multiple resampling points; when the initial sampling points do not reach the number threshold, new sampling points are reselected within the three-dimensional sampling space as resampling points to obtain the coordinates and direction vectors of the multiple resampling points; A calculation module is used to calculate the color and point density of all resampled points on each rendering light based on the coordinates and direction vectors of all resampled points on each rendering light after position encoding and direction encoding, and to obtain the sub-pixel color corresponding to the rendering light in the synthetic image after accumulation, so as to generate three-dimensional light field content; A generating module is used to generate a target three-dimensional light field based on the three-dimensional light field content and the three-dimensional light field display device.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for generating three-dimensional light field content by adaptive ray sampling according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for generating three-dimensional light field content by adaptive light sampling according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for generating three-dimensional light field content by adaptive light sampling according to any one of claims 1 to 6 is implemented.

Images