Three-dimensional digital content generation method, device, system, equipment, medium and product

Abstract

The present invention discloses a three-dimensional digital content generation method, device, system, equipment, medium and product, which are applied to the field of three-dimensional data processing technology. The method includes generating heterogeneous three-dimensional data by unifying three-dimensional network model data and target three-dimensional data in different data formats under the same coordinate system, and expressing them using a target data structure. Light is projected from the observation position of the image acquisition device along the observation direction of the rendering pixel, and sampling is performed along the light according to preset light parameters; the color value of the sampling point is determined according to the spatial position relationship between the sampling point and the three-dimensional network model data and the target three-dimensional data, the color of the rendering pixel is determined according to the color value of each sampling point, and three-dimensional digital content is generated according to each rendering pixel. The present invention can solve the high cost problem of manually creating three-dimensional digital content, can uniformly process heterogeneous three-dimensional data, and realize efficient and low-cost generation of three-dimensional digital content.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional data processing technology, and in particular to a three-dimensional digital content generation method, device, system, electronic equipment, non-volatile storage medium and computer program product. Background Art

[0002] As artificial intelligence technology is applied to the field of three-dimensional digital technology, two completely different three-dimensional data will appear at the same time, namely, three-dimensional data represented by a neural network model and traditional three-dimensional data, such as three-dimensional point cloud data and triangular meshes.

[0003] The processing process of 3D data represented by neural network models is completely different from that of traditional 3D data, and they cannot be processed uniformly. In order to obtain high-quality 3D digital content, related technologies need to rely on manual labor, such as artists and engineers creating it manually, which is not only costly but also inefficient.

[0004] In view of this, achieving unified processing of three-dimensional data represented by a neural network model and traditional three-dimensional data, and generating three-dimensional digital content efficiently and at low cost, is a technical problem that those skilled in the art need to solve. It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the invention

[0005] The present invention provides a three-dimensional digital content generation method, device, system, electronic device, non-volatile storage medium and computer program product, which can uniformly process three-dimensional data represented by a neural network model and traditional three-dimensional data, and realize efficient and low-cost generation of three-dimensional digital content.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In one aspect, the present invention provides a method for generating three-dimensional digital content, wherein the three-dimensional digital content includes a plurality of target rendering pixels, including:

[0008] Heterogeneous three-dimensional data is generated by unifying three-dimensional network model data and target three-dimensional data in different data formats under the same coordinate system; the three-dimensional network model data is physical light parameters converted from three-dimensional space parameters through a neural network model; the heterogeneous three-dimensional data is represented by a target data structure so as to load the heterogeneous three-dimensional data into a target storage location for calculation; light is projected from an observation position of an image acquisition device along an observation direction of a target rendering pixel, and sampling is performed along the light according to preset light parameters; the color value of each sampling point is determined according to the spatial position relationship between each sampling point of the light and the three-dimensional network model data and the target three-dimensional data, and the color of the target rendering pixel is determined according to the color value of each sampling point, so as to generate three-dimensional digital content according to each target rendering pixel.

[0009] In a first exemplary embodiment, the three-dimensional space parameters are the three-dimensional space position and the line of sight observation direction, the physical light parameters are color information and intensity information, and the method of generating heterogeneous three-dimensional data by unifying the three-dimensional network model data and the target three-dimensional data in different data formats in the same coordinate system includes: inputting the three-dimensional space position and line of sight observation direction data in the target coordinate system into a neural network model, and outputting the neural network model as the color information and intensity information at each spatial position in the target coordinate system to obtain the three-dimensional network model data; and obtaining heterogeneous three-dimensional data by matching the color information and intensity information of the three-dimensional network model data at the corresponding spatial position for each spatial position of the target three-dimensional data in the target coordinate system.

[0010] In a second exemplary embodiment, the use of a target data structure to represent the heterogeneous three-dimensional data includes: determining a root node of a tree structure based on an axis-aligned bounding box of the heterogeneous three-dimensional data, and constructing the tree structure by taking the root node as the starting point and splitting in different directions to obtain child nodes; and storing the target three-dimensional data and the three-dimensional network model data in the heterogeneous three-dimensional data into leaf nodes of the tree structure.

[0011] In a third exemplary embodiment, the tree structure is constructed by taking the root node as the starting point and splitting in different directions to obtain child nodes, including: based on the tree type, splitting the root node bounding box into a corresponding number of child nodes along each coordinate axis direction of the spatial coordinate system; for each child node, splitting it again into a corresponding number of child nodes along each coordinate axis direction of the spatial coordinate system until the maximum side length of the current child node is less than a preset spatial accuracy threshold.

[0012] In a fourth exemplary embodiment, the target three-dimensional data is three-dimensional grid data, and the three-dimensional network model data is multiple, and the target three-dimensional data and the three-dimensional network model data in the heterogeneous three-dimensional data are stored in the leaf nodes of the tree structure, including: based on the spatial position of the grid unit of the target three-dimensional data, each grid unit of the target three-dimensional data is corresponded to the leaf node of the tree structure, each leaf node contains multiple faces; a three-dimensional network model data in the heterogeneous three-dimensional data is stored in a leaf node of the tree structure, and according to the neural network model corresponding to the current leaf node, the data of each spatial position within the spatial range covered by the current leaf node is determined.

[0013] In a fifth exemplary embodiment, the use of a target data structure to represent the heterogeneous three-dimensional data includes: determining each leaf parent node of the tree structure and each corresponding leaf node contained therein; when a heterogeneous three-dimensional data processing request is received, synchronously performing the same data operation on the data of all leaf nodes belonging to the same leaf parent node.

[0014] In a sixth exemplary embodiment, when a heterogeneous three-dimensional data processing request is received, the same data operation is synchronously performed on the data of all leaf nodes belonging to the same leaf parent node, including: when a data loading request is received, the heterogeneous three-dimensional data is loaded into the memory in a manner that the data of all leaf nodes belonging to the same leaf parent node are synchronously loaded into the memory; when a data unloading request is received, the heterogeneous three-dimensional data is deleted from the memory in a manner that the data of all leaf nodes belonging to the same leaf parent node are synchronously unloaded from the memory.

[0015] In a seventh exemplary embodiment, the color value of each sampling point is determined according to the spatial position relationship between each sampling point of the light and the three-dimensional network model data and the target three-dimensional data, including: for each sampling point of the light, if the current sampling point is located in a three-dimensional space area corresponding to the three-dimensional network model data, the color value of the current sampling point is determined according to the intensity value of the three-dimensional network model data; if the current sampling point is located in a two-dimensional plane area corresponding to the target three-dimensional data, the color value of the current sampling point is determined according to the color attributes of the target three-dimensional data and the direction of the light.

[0016] In an eighth exemplary embodiment, the color value of the current sampling point is determined according to the intensity value of the three-dimensional network model data, including: calculating the color value of the current sampling point according to the intensity values ​​of the current sampling point and the previous sampling point adjacent to the current sampling point at corresponding spatial positions of the three-dimensional network model data.

[0017] In a ninth exemplary embodiment, determining the color value of the current sampling point according to the color attributes and light direction of the target three-dimensional data includes: for the current sampling point, determining a target plane closest to the current sampling point in a two-dimensional plane area corresponding to the target three-dimensional data, and determining a target hemispherical surface formed by rotating the target plane in a direction opposite to the observation direction from the observation position of the image acquisition device to the target rendering pixel; and determining the color value of the current sampling point according to the color value of the current sampling point corresponding to the spatial position of the target three-dimensional data and the accumulated value of the color values ​​calculated along different sampling directions for the target hemispherical surface.

[0018] In a tenth exemplary embodiment, the target three-dimensional data is three-dimensional grid data, and determining the color value of each sampling point according to the spatial position relationship between each sampling point of the light and the three-dimensional network model data and the target three-dimensional data includes: for each sampling point of the light, calculating the distance value between the current sampling point and the surface of each grid unit of the target three-dimensional data, and selecting the minimum distance therefrom; determining a preset distance threshold according to a tolerance error allowable value of whether the sampling point position is located in a two-dimensional plane area of ​​the target three-dimensional data; if the minimum distance is greater than the preset distance threshold, the current sampling point is located in the three-dimensional space area corresponding to the three-dimensional network model data; if the minimum distance is less than the preset distance threshold, the current sampling point is located on the grid surface of the target three-dimensional data.

[0019] In an eleventh exemplary embodiment, the sampling along the light according to preset light parameters includes: taking the observation position of the image acquisition device as the starting point of the light, and obtaining a first sampling point according to a preset light sampling step along the observation direction from the observation position of the image acquisition device to the target rendering pixel; when the first sampling point is located in the three-dimensional space area corresponding to the three-dimensional network model data, when the color value of the first sampling point is determined, taking the first sampling point as the current starting point of the light, and obtaining a second sampling point along the observation direction according to the preset light sampling step; when the first sampling point is located in the two-dimensional plane area corresponding to the target three-dimensional data, when the color value of the first sampling point is determined, determining the target plane closest to the first sampling point on the two-dimensional plane area corresponding to the target three-dimensional data, sampling and calculating the optical property information of the target plane at the first sampling point, determining the sampling direction, taking the first sampling point as the current starting point of the light, and obtaining a second sampling point along the sampling direction according to the preset light sampling step; repeating the execution until a preset sampling stop condition is met.

[0020] In a twelfth exemplary embodiment, the target three-dimensional data is three-dimensional grid data, and further includes: spatial sampling of the three-dimensional space area corresponding to the three-dimensional network model data to obtain spatial sampling point information; the spatial sampling point information includes three-dimensional coordinate values ​​and intensity information of each three-dimensional point in the three-dimensional space area; sampling the spatial sampling point information again to obtain spatial regular point information; segmenting the spatial regular point information according to the grid units of the target three-dimensional data to obtain three-dimensional model conversion data of the same data type as the target three-dimensional data; performing physical interaction calculations with the target three-dimensional data based on the three-dimensional model conversion data to identify whether the corresponding three-dimensional entities collide during the three-dimensional digital content generation process.

[0021] In a thirteenth exemplary embodiment, spatial sampling is performed on the three-dimensional spatial area corresponding to the three-dimensional network model data, including: sampling the three-dimensional spatial area corresponding to the three-dimensional network model data according to a first preset spatial sampling step, and selecting spatial sampling points whose intensity values ​​are greater than a first preset light intensity threshold from each spatial sampling point to generate initial spatial sampling point information; determining the spatial area between the maximum coordinate value and the minimum coordinate value in each coordinate axis direction according to the three-dimensional coordinate values ​​of each spatial sampling point of the initial spatial sampling point information, to obtain a horizontal spatial area, a vertical spatial area and a vertical spatial area; taking the intersection of the horizontal spatial area, the vertical spatial area and the vertical spatial area as the minimum enclosing space; sampling the minimum enclosing space according to a second preset spatial sampling step to generate an intermediate sampling point set; selecting spatial sampling points whose intensity values ​​are greater than a second preset light intensity threshold from each spatial sampling point of the intermediate sampling point set to generate spatial sampling point information; wherein the second preset spatial sampling step is smaller than the first preset spatial sampling step.

[0022] In a fourteenth exemplary embodiment, the spatial sampling point information is a spatial sampling point set, and the spatial sampling point information is sampled again to obtain spatial regular point information, including: performing regularized sampling on the spatial sampling point information to obtain an initial spatial regular point set; according to the three-dimensional coordinate values ​​of each initial regular sampling point of the initial spatial regular point set, respectively determining the spatial area between the maximum coordinate value and the minimum coordinate value in each coordinate axis direction to obtain a horizontal regular area, a vertical regular area and a vertical regular area; taking the intersection of the horizontal regular area, the vertical regular area and the vertical regular area as the minimum regular enclosing space of the initial spatial regular point set; sampling the minimum regular enclosing space according to a third preset spatial sampling step size to obtain a spatial regular point set, and the intensity value of each regular sampling point of the spatial regular point set is obtained according to the output of the corresponding neural network model.

[0023] In a fifteenth exemplary embodiment, the spatial regular point information is divided according to the grid units of the target three-dimensional data, including: based on regular sampling points with intensity values ​​greater than a second preset light intensity threshold as real points, isosurface extraction is performed on the spatial regular point set according to the grid units of the target three-dimensional data, and three-dimensional model conversion data is constructed based on the extracted isosurfaces; wherein the real points indicate that there is data at the corresponding spatial position, and the shape of the isosurface is the same as the surface shape of the grid unit.

[0024] Another aspect of the present invention provides a three-dimensional digital content generation device, comprising:

[0025] A heterogeneous data representation module is used to generate heterogeneous three-dimensional data by unifying three-dimensional network model data and target three-dimensional data in different data formats under the same coordinate system; the three-dimensional network model data is physical light parameters converted from three-dimensional space parameters through a neural network model.

[0026] The data representation module is used to represent the heterogeneous three-dimensional data using a target data structure so as to load the heterogeneous three-dimensional data into a target storage location for color rendering calculation.

[0027] The light sampling module is used to project light from the observation position of the image acquisition device along the observation direction of the target rendering pixel, and to sample along the light according to preset light parameters.

[0028] The color calculation module is used to determine the color value of each sampling point according to the spatial position relationship between each sampling point of the light and the three-dimensional network model data and the target three-dimensional data, and determine the color of the target rendering pixel according to the color value of each sampling point, so as to generate three-dimensional digital content according to each target rendering pixel, and the three-dimensional digital content includes multiple target rendering pixels.

[0029] The present invention also provides an electronic device, comprising a processor, wherein the processor is configured to implement the steps of any of the above three-dimensional digital content generation methods when executing a computer program stored in a memory.

[0030] The present invention also provides a non-volatile storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of any of the above three-dimensional digital content generation methods are implemented.

[0031] The present invention also provides a computer program product, comprising a computer program / instruction, which implements the steps of the three-dimensional digital content generation method as described in any of the above when executed by a processor.

[0032] Finally, the present invention also provides a three-dimensional digital content generation system, including a data storage medium, a buffer, a light sampler, a register group, a color renderer and a three-dimensional digital content generator; wherein the data storage medium stores model data of a neural network model, target three-dimensional data and preset light parameters; the buffer loads the model data, the target three-dimensional data and the preset light parameters from the data storage medium, and sends the three-dimensional spatial position information and the line of sight observation direction in the model data to the network input register in the register group, sends the network parameter data and the target three-dimensional data in the model data to the color renderer, and sends the preset light parameters to the light sampler; the light sampler is configured to project light along the observation direction of the target rendering pixel according to the observation position from the image acquisition device, and samples along the light according to the preset light parameters, and sends the sampling points of the light to the light register in the register group; the color renderer includes a position relationship calculator, a parallel calculator, a serial calculator and a color accumulator; the parallel calculator is configured to The three-dimensional spatial position information and the line of sight observation direction are read from the network input register, and the physical light parameters of the three-dimensional network model are calculated according to the three-dimensional spatial position information, the line of sight observation direction and the network parameter data; the position relationship calculator is configured to determine the spatial position relationship between each sampling point of the light and the three-dimensional network model data and the target three-dimensional data, and select whether to use a parallel calculator or a serial calculator to calculate the color value of each sampling point; the parallel calculator is also configured to calculate the color value of each sampling point according to the physical light parameters of the three-dimensional network model data, the serial calculator is configured to calculate the color value of each sampling point according to the target three-dimensional data, the color accumulator is configured to determine the color of the target rendering pixel according to the sampling point color value output by the parallel calculator and the sampling point color value output by the serial calculator, and send the color of the target rendering pixel to the data storage medium; the three-dimensional digital content generator is configured to read the color of each target rendering pixel from the data storage medium, and generate three-dimensional digital content according to the color of each target rendering pixel.

[0033] In a first exemplary embodiment, the parallel calculator includes an intensity calculation unit and a color calculation unit; the intensity calculation unit and the color calculation unit are both multiplication-addition unit arrays; the number of multiplication-addition units of the intensity calculation unit matches the scale of the neural network model.

[0034] In a second exemplary embodiment, it also includes a distance calculator; the distance calculator is configured to read the target three-dimensional data from the buffer, read the sampling points of the light from the light register, and for each sampling point of the light, calculate the distance value between the current sampling point and the surface of each grid unit of the target three-dimensional data, and select the minimum distance therefrom, and store the minimum distance in the distance register in the register group; correspondingly, the position relationship calculator is configured to read the minimum distance from the distance register, if the minimum distance is greater than the preset distance threshold, the current sampling point is located in the three-dimensional space area corresponding to the three-dimensional network model data, and the parallel calculator is used to calculate the color value of the current sampling point; if the minimum distance is less than the preset distance threshold, the current sampling point is located on the grid surface of the target three-dimensional data, and the serial calculator is used to calculate the color value of the current sampling point.

[0035] In a third exemplary embodiment, the serial calculator includes a sampling direction calculation unit, a grid color calculation unit and a sampling color accumulator; wherein the sampling direction calculation unit is configured to determine a target plane closest to a current sampling point on a two-dimensional plane area corresponding to the target three-dimensional data, perform sampling calculations on optical property information of the target plane at the current sampling point, determine a sampling direction, and store each sampling direction in a direction register of the register group; the grid color calculation unit is configured to rotate the target plane in a direction opposite to the observation direction from the observation position of the image acquisition device to the target rendering pixel to form a target hemispherical surface, The sampling direction is read from the direction register, and the color value of the target hemispherical surface along the current sampling direction is determined according to the color value of the current sampling point corresponding to the spatial position of the target three-dimensional data, the light attribute information of the target plane, and the color value of the current incident light, and is stored in the single-direction color result register in the register group; the sampling color accumulator is configured to read the color value of the target hemispherical surface along the current sampling direction from the single-direction color result register, accumulate the color values ​​of the target hemispherical surface along different sampling directions, obtain the color value of the current sampling point, and store the color value of the current sampling point in the sampling point color result register.

[0036] The advantage of the technical solution provided by the present invention is that by placing three-dimensional data of different formats in the same three-dimensional space to combine into a scene, the heterogeneous three-dimensional data can be calculated and processed in a consistent coordinate space, and then the heterogeneous three-dimensional data is represented by the target data structure, thereby realizing the unified processing of the three-dimensional data represented by the neural network model and the traditional three-dimensional data. On this basis, the color of the sampling point is calculated according to whether the light sampling point is in the spatial position of the three-dimensional network model data or in the target three-dimensional data, and finally a consistent color rendering calculation result is achieved by color fusion of the heterogeneous three-dimensional data, so as to realize efficient and low-cost generation of three-dimensional digital content.

[0037] In addition, the present invention also provides corresponding implementation devices, systems, electronic devices, non-volatile storage media and computer program products for the three-dimensional digital content generation method, which further makes the method more practical, and the devices, systems, electronic devices, non-volatile storage media and computer program products have corresponding advantages.

[0038] The technical features mentioned above, the technical features to be mentioned below, and the technical features shown separately in the drawings can be arbitrarily combined with each other, as long as the combined technical features are not contradictory. All feasible feature combinations are technical contents clearly recorded in this article. Any of the multiple sub-features contained in the same sentence can be applied independently, and does not have to be applied together with other sub-features. It should be understood that the above general description and the detailed description below are only exemplary and cannot limit the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 A schematic diagram of a flow chart of a three-dimensional digital content generation method provided by the present invention;

[0041] Figure 2 A schematic diagram of the generation process of the three-dimensional network model data provided by the present invention in an exemplary application scenario;

[0042] Figure 3 A schematic diagram of using a tree structure to represent heterogeneous three-dimensional data provided by the present invention;

[0043] Figure 4 A schematic diagram of the calculation process of the sampling point color provided by the present invention in an exemplary application scenario;

[0044] Figure 5 A structural diagram of an exemplary implementation of a three-dimensional digital content generating device provided by the present invention;

[0045] Figure 6 A structural diagram of a specific implementation of the electronic device provided by the present invention;

[0046] Figure 7 A structural diagram of an exemplary implementation of a three-dimensional digital content generation system provided by the present invention;

[0047] Figure 8 A schematic diagram of a three-dimensional digital content generation process in the three-dimensional digital content generation system provided by the present invention;

[0048] Fig. 9 This is another schematic diagram of a three-dimensional digital content generation process in the three-dimensional digital content generation system provided by the present invention. DETAILED DESCRIPTION

[0049] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Among them, the terms "first", "second", etc. in the specification and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations of the two are intended to cover non-exclusive inclusions. The term "exemplary" means "used as an example, embodiment or illustrative". Any embodiment described here as "exemplary" is not necessarily interpreted as being superior or better than other embodiments.

[0050] Three-dimensional digitization can achieve a series of digital operations such as virtual creation, modification, improvement, and analysis of models by using three-dimensional tools. Three-dimensional digitization processes and splices the shape data of objects, organizes them through modeling, and seamlessly integrates each isolated single-view three-dimensional digital model. After mapping and rendering, a three-dimensional data file is formed. In three-dimensional modeling, mesh data and point cloud data are a common representation method. For example, polygonal meshes can be used to represent three-dimensional data models. With the development of artificial intelligence technology, the three-dimensional data representation form based on neural network models represented by neural radiation fields can realize end-to-end three-dimensional content construction from image to image. Interactive three-dimensional content generation and rendering can be realized directly by collecting images through cameras. There are two completely different three-dimensional data to realize the creation of three-dimensional content.

[0051] In order to obtain high-quality 3D content, it is currently created manually by artists, engineers, etc. The entire process requires a lot of manpower, time and resources, resulting in high production costs and long production cycles for high-quality 3D content. With the application of artificial intelligence technology to the field of 3D digital technology, since the 3D data representation based on neural networks and the traditional 3D point cloud data and mesh data are completely different in processing, achieving unified processing of 3D data represented by neural network models and traditional 3D data is a problem that needs to be solved in the field of 3D data processing and computing under the background of artificial intelligence technology.

[0052] However, there is no processing for the three-dimensional data format represented by the neural network model, and the consistent processing and physical interaction function between the three-dimensional data represented by the neural network model and the traditional three-dimensional data. For example, a related technology discloses the integrated management of the urban full-space three-dimensional model data, and the process includes: integrated fusion management and distributed storage of multi-source three-dimensional models; construction of lightweight full-space three-dimensional models; rapid rendering of multi-source heterogeneous massive three-dimensional data; analysis and application of three-dimensional models through the browser. This method adopts data caching strategy and asynchronous call technology to alleviate the pressure of massive geological model data transmission in large-scale scenes and ensure operational stability; in the process of real-time dynamic management of large-scale complex scene data, the area and level of the target sub-scene data are determined according to the current viewpoint position and line of sight angle, so as to realize real-time dynamic scheduling of data; real-time dynamic scheduling of large-scale three-dimensional geological models, and rapid acquisition of three-dimensional scene data through an efficient web page dynamic scheduling framework, so as to realize real-time scene update and real-time data loading; rapid data rendering is realized through parallel rendering based on data parallelism. It can be seen that this method realizes efficient rendering processing of traditional format three-dimensional data from the perspective of data scheduling and parallel processing, and does not support the processing of three-dimensional data formats represented by neural network models and the physical interaction functions between three-dimensional data represented by neural network and traditional mesh representation. A related technology discloses the integrated management of two-dimensional and three-dimensional data based on massive geographic information data and the organization and management of three-dimensional entity objects using a three-dimensional data engine system. The method performs different operations on databases, data engines and data services with different interfaces, so that data can be correctly accessed and updated. The three-dimensional data engine manages massive geographic information data in a unified manner, and can meet the requirements of multi-user collaborative operation, network sharing, data security, etc., and meet the needs of semantic expression, dynamic update and consistency maintenance and comprehensive analysis of three-dimensional spatial information on the ground, underground, past, present and future. However, this method only has the organization and management functions of three-dimensional data in the field of geographic information, and also does not support the consistent processing calculation and physical interaction functions between three-dimensional data represented by neural network and traditional triangular mesh three-dimensional data. Another related technology discloses a method for constructing a three-dimensional object heterogeneous model, which can quickly construct corresponding three-dimensional heterogeneous modules through the processing operations of three-dimensional heterogeneous elements, and then generate a three-dimensional object heterogeneous model. This method utilizes known three-dimensional models to assemble the required target three-dimensional model, and also fails to realize the consistent processing and interaction functions between the three-dimensional data represented by the neural network and the traditional grid three-dimensional data.

[0053] In view of this, in order to solve the problem that the related art cannot uniformly process the three-dimensional data represented by the neural network model and the traditional three-dimensional data, resulting in high cycle and cost of three-dimensional digital content generation, the present invention combines three-dimensional data of different formats into a scene by placing them in the same three-dimensional space, so as to realize the calculation and processing of heterogeneous three-dimensional data in a consistent coordinate space, and then represents the heterogeneous three-dimensional data by the target data structure, so as to realize the unified processing of three-dimensional data represented by the neural network model and traditional three-dimensional data. On this basis, the color of the sampling point is calculated according to whether the light sampling point is in the spatial position of the three-dimensional network model data or in the target three-dimensional data, and finally, a consistent color rendering calculation result is achieved by color fusion of heterogeneous three-dimensional data, so as to realize efficient and low-cost three-dimensional digital content generation. After introducing the technical scheme of the present invention, various non-limiting embodiments of the present invention are described in detail below. In order to better illustrate the present invention, many specific details are given in the specific embodiments below. Those skilled in the art should understand that the present invention can also be implemented without these specific details. In some other examples, the methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present invention.

[0054] First see Figure 1 , Figure 1 A flowchart of a method for generating three-dimensional digital content is provided in this embodiment. This embodiment may include the following contents:

[0055] S101: Generate heterogeneous three-dimensional data by unifying three-dimensional network model data and target three-dimensional data in different data formats under the same coordinate system.

[0056] In the present invention, the three-dimensional network model data is the physical light parameters converted from the three-dimensional space parameters through the neural network model, that is, the three-dimensional network model data is the three-dimensional data represented by the neural network model. Among them, the neural network model can be the neural network model in the machine learning model of the related technology, the three-dimensional space parameters are the parameters reflecting the three-dimensional physical space, including but not limited to the three-dimensional space position and the line of sight observation direction, the physical light parameters are the parameters required for calculating the color, including but not limited to the color information and the intensity information, the intensity information is similar to a value of transparency or volume density, and the color value can be calculated by weighted accumulation of the color information and the intensity information. The target three-dimensional data is any traditional three-dimensional format data suitable for using the ray casting rendering method, such as three-dimensional mesh data, three-dimensional point cloud data. Among them, the three-dimensional mesh data can be any polygonal mesh such as a triangular mesh, a quadrilateral mesh, etc. Since the processing process of the three-dimensional format data represented by the neural network model is completely different from that of the traditional three-dimensional format data, in order to achieve unified processing of the two, this step puts the three-dimensional network model data and the target three-dimensional data in different formats into the same three-dimensional space, combines them into a scene, and obtains heterogeneous three-dimensional data. The heterogeneous three-dimensional data includes the three-dimensional data represented by the neural network model and the target three-dimensional data. In this way, the two different three-dimensional format data can be calculated and processed in a consistent coordinate space.

[0057] For example, taking the three-dimensional space parameters as the three-dimensional space position and the line of sight observation direction, and the physical light parameters as the color value and the intensity value, the three-dimensional network model data can be obtained by to generate, f represents the neural network model, x, y, z are the three-dimensional space positions, is the line of sight observation direction, is the color value at the corresponding position (x, y, z), is the intensity value at the corresponding position (x, y, z). Figure 2 As shown, taking the neural network model as MLP (Multilayer Perceptron) as an example, each layer has 256 hidden units, and the three-dimensional network model data can be expressed as . Taking the target 3D data as 3D mesh data as an example, polygonal mesh data is a 3D model representation method, usually using triangles or quadrilaterals as basic elements. Polygonal mesh data consists of vertices, edges, and faces. Vertices record 3D coordinate information, edges connect vertices, and faces are surrounded by edges. Its format can be obj (format type), fbx (format type), ply (format type), etc.

[0058] S102: Using a target data structure to represent heterogeneous three-dimensional data, so as to load the heterogeneous three-dimensional data into a target storage location for calculation.

[0059] When the previous step uses heterogeneous 3D data to describe the 3D data represented by the neural network model and the traditional 3D data as a whole, in order to realize the organization and management of heterogeneous 3D data and unified calculation, it is also necessary to represent the heterogeneous 3D data structure as a data structure that can be processed uniformly, such as a graph structure or a tree structure. For the convenience of description, this step defines the data structure selected by technicians in this field according to the actual situation as the target data structure. When heterogeneous 3D data is needed to participate in the calculation process in the process of generating 3D digital content, such as rendering color calculation, the heterogeneous 3D data needs to be loaded into a specified location. For the convenience of description, it can be defined as a target storage location, which can be, for example, a memory or a cache.

[0060] S103: Projecting light from the observation position of the image acquisition device along the observation direction of the target rendering pixel, and sampling along the light according to preset light parameters.

[0061] It is understandable that the three-dimensional digital content needs to be displayed to users, so the three-dimensional data needs to be converted into two-dimensional data, that is, two-dimensional image data, so each three-dimensional point in the three-dimensional data needs to be mapped to a pixel in the two-dimensional image data. Correspondingly, the three-dimensional digital content includes a series of pixels, and then the color value of each pixel needs to be calculated. In order to realize the calculation from heterogeneous three-dimensional data to two-dimensional image pixel values, the present invention needs to first perform sampling, and then calculate the color value of each sampling point through step S104. The image acquisition device is a camera, a video camera, a mobile phone camera, etc. The target rendering pixel is the pixel that currently needs to be rendered color calculation. For example, the camera observation position during the rendering process is , the viewing direction corresponding to the target rendering pixel U is , then you can along Directional cast light The preset light parameters are the way to sample the light, such as uniform sampling and sampling step size. After the preset light parameters are determined, sampling is performed along the light to obtain a series of sampling points.

[0062] S104: Determine the color value of each sampling point according to the spatial position relationship between each sampling point of the light and the 3D network model data and the target 3D data, determine the color of the target rendering pixel according to the color value of each sampling point, and generate 3D digital content according to each target rendering pixel.

[0063] When the light is sampled in the previous step, in order to improve the image rendering efficiency, each time sampling is performed, that is, a sampling point is obtained, the color value of the sampling point can be calculated through S104. Of course, after the sampling is completed, the color calculation can be performed on each sampling point in turn, which does not affect the implementation of the present invention. After all the sampling points of the light are calculated to obtain the color value, the color and intensity of all the sampling points on the light are accumulated to obtain the final color of the light, that is, the color of the pixel in the image corresponding to the light, that is, the color of the target rendering pixel.

[0064] Among them, considering that the three-dimensional data represented by the neural network model will be converted into traditional three-dimensional data, the model representation accuracy will decrease, thereby reducing the quality of the rendered image, so the rendering calculation is directly performed on the basis of heterogeneous three-dimensional data. The three-dimensional network model data and the target three-dimensional data use different formats to represent the three-dimensional object, and the three-dimensional network model data, the target three-dimensional data and the light itself also have color attributes. The color of the sampling point is determined by the three-dimensional network model data, the target three-dimensional data and the light. Therefore, when calculating the color of each sampling point, it is necessary to determine the color calculation according to the spatial position relationship between the sampling point and the three-dimensional network model data and the target three-dimensional data. If the sampling point is located in the spatial model corresponding to the three-dimensional network model data, it indicates that the color of the sampling point is greatly affected by the color of the three-dimensional space model, and if the sampling point is located in the spatial position of the target three-dimensional data, it indicates that it is greatly affected by the color of the target three-dimensional data and the light. Based on the degree of influence of different positions on the color of the sampling point, those skilled in the art can select the color calculation method of the sampling point according to the actual situation, and the present invention does not make any modern. In the process of calculating the color, when the light contacts the target three-dimensional data, such as when it contacts the surface of the triangular mesh model, the rendering color calculation method can be switched.

[0065] In the technical solution provided in this embodiment, three-dimensional data of different formats are combined into a scene in the same three-dimensional space, so as to realize the calculation and processing of heterogeneous three-dimensional data in a consistent coordinate space, and then the heterogeneous three-dimensional data is represented by the target data structure, so as to realize the unified processing of three-dimensional data represented by the neural network model and traditional three-dimensional data. On this basis, the color of the sampling point is calculated according to whether the light sampling point is in the spatial position of the three-dimensional network model data or in the target three-dimensional data, and finally, a consistent color rendering calculation result is achieved by color fusion of heterogeneous three-dimensional data, so as to realize efficient and low-cost generation of three-dimensional digital content.

[0066] It should be noted that there is no strict order of execution between the steps in the present invention. As long as they comply with the logical order, these steps can be executed simultaneously or in a certain preset order. Figure 1This is just a schematic and does not mean that this is the only execution order.

[0067] The above embodiment does not limit how to generate heterogeneous three-dimensional data. This embodiment also provides an exemplary implementation method for obtaining heterogeneous three-dimensional data according to three-dimensional network model data and target three-dimensional data, which may include the following contents:

[0068] The three-dimensional spatial position and line of sight observation direction data in the target coordinate system are input into the neural network model, and the output of the neural network model is used as the color information and intensity information at each spatial position in the target coordinate system to obtain the three-dimensional network model data. Heterogeneous three-dimensional data is obtained by matching the color information and intensity information of the three-dimensional network model data at the corresponding spatial position for each spatial position of the target three-dimensional data in the target coordinate system.

[0069] In this embodiment, the target coordinate system is a predefined coordinate system, and the three-dimensional network model data and the target three-dimensional data are spatially aligned in this coordinate system. That is to say, the three-dimensional positions of the three-dimensional points of the three-dimensional network model data and the target three-dimensional data in the target coordinate system correspond to each other, and the color information of a three-dimensional point includes both the color information and intensity information of the three-dimensional network model data, and the color attribute data of the target three-dimensional data itself.

[0070] As can be seen from the above, this embodiment realizes spatial alignment between heterogeneous data by defining the three-dimensional data represented by the neural network model and the traditional target three-dimensional data in a unified coordinate system, thereby achieving unified processing of three-dimensional data in different formats.

[0071] In the above embodiment, there is no limitation on representing heterogeneous three-dimensional data. In order to further improve the processing efficiency of heterogeneous three-dimensional data, the present invention also provides an implementation method for representing heterogeneous three-dimensional data using a tree structure, which may include the following contents:

[0072] The root node of the tree structure is determined according to the axis-aligned bounding box of the heterogeneous three-dimensional data, and the tree structure is constructed by taking the root node as the starting point and splitting in different directions to obtain child nodes; the target three-dimensional data and three-dimensional network model data in the heterogeneous three-dimensional data are stored in the leaf nodes of the tree structure.

[0073] It can be understood that the tree structure is a finite set of n nodes, with one and only one root node. According to different tree types, such as binary trees, quadtrees, and octrees, each node can be decomposed to obtain multiple child nodes. In order to reflect this relationship, the decomposed node is the parent node, and the decomposed node is the child node, that is, each parent node may include multiple child nodes, and the child node of the last level is the leaf node, and the leaf node belongs to the child node of the leaf parent node. In this embodiment, the axis-aligned bounding box of heterogeneous three-dimensional data is used as the root node of the tree structure, and then the node is decomposed according to different number types to obtain the corresponding child node to construct the entire tree structure. When the tree structure is constructed, heterogeneous three-dimensional data can be stored in the tree structure. In order to efficiently process heterogeneous three-dimensional data, in the tree structure of the present invention, only the leaf nodes store data, that is, the target three-dimensional data and the three-dimensional network model data are stored in the last level node of the tree structure, and the intermediate nodes do not store data. For the case where the target three-dimensional data is three-dimensional mesh data and there are multiple three-dimensional network model data, based on the spatial position of the mesh unit of the target three-dimensional data, that is, the three-dimensional mesh data, each mesh unit of the three-dimensional mesh data is corresponded to a leaf node of the tree structure, and each leaf node contains multiple faces; a three-dimensional network model data in the heterogeneous three-dimensional data is stored in a leaf node of the tree structure, and the data of each spatial position within the spatial range covered by the current leaf node is determined according to the neural network model corresponding to the current leaf node. In other words, each leaf node of the tree structure contains a neural network three-dimensional data, and for the three-dimensional data represented by the neural network, the data at any position within the spatial range covered by this leaf node can be obtained by the neural network model corresponding to the node. For example, if the three-dimensional mesh data is triangular mesh data, then each triangle of the triangular mesh data can correspond to a leaf node of the tree structure according to its spatial position, and each leaf node contains several triangular faces. If the three-dimensional mesh data is quadrilateral mesh data, then each quadrilateral of the quadrilateral mesh data can correspond to a leaf node of the tree structure according to its spatial position, and each leaf node contains several quadrilateral faces.

[0074] Exemplarily, this embodiment also provides a more convenient tree structure generation process. When the axis-aligned bounding box of the scene model represented by heterogeneous three-dimensional data is used as the root node of the tree structure, the root node bounding box is divided into a corresponding number of child nodes along the coordinate axis directions of the spatial coordinate system based on the tree type; for each child node, it is further divided into a corresponding number of child nodes along the coordinate axis directions of the spatial coordinate system until the maximum side length of the current child node is less than the preset spatial accuracy threshold. For example, Figure 3As shown in the figure, the data processing efficiency of the octree structure is shown. Taking the tree type as an octree as an example, the root node bounding box is divided along the X, Y, and Z coordinate axis directions. The dividing point can be, for example, the midpoint position of each edge, and eight child nodes are obtained by division. Of course, the dividing position can also be set according to other rules, which does not affect the implementation of the present invention. For each child node, the division can continue to be performed in the same division method or in a different manner. After each division is completed, it can be verified whether the child node obtained by the current division is the last level. For the convenience of description, it is defined as the current child node. If the maximum side length of the current child node is less than the preset spatial accuracy threshold, it will no longer be divided, that is, the current child node is a leaf node. The spatial accuracy threshold can be specified according to the needs of the actual spatial representation accuracy. The higher the accuracy requirement, the smaller the value.

[0075] In the tree structure of this embodiment, only leaf nodes store data, and intermediate nodes do not store data. The node on the previous layer of the leaf node is the leaf parent node. For a leaf parent node, the data contained in all its leaf nodes are adjacent in space. Correspondingly, all leaf nodes of the same leaf parent node are also adjacent in memory during calculation and loading. In order to achieve locality of data access, the present invention can provide a data processing method implemented in units of leaf parent nodes, which can include the following contents:

[0076] First, each leaf parent node of the tree structure and each corresponding leaf node are determined; when a heterogeneous three-dimensional data processing request is received, the same data operation is synchronously performed on the data of all leaf nodes belonging to the same leaf parent node.

[0077] In this embodiment, the heterogeneous three-dimensional data processing request may include a data loading request and a data unloading request. Accordingly, when a data loading request is received, the heterogeneous three-dimensional data is loaded into the memory in a manner that the data of all leaf nodes belonging to the same leaf parent node are synchronously loaded into the memory; when a data unloading request is received, the heterogeneous three-dimensional data is deleted from the memory in a manner that the data of all leaf nodes belonging to the same leaf parent node are synchronously unloaded from the memory. Taking the octree as an example, when data is loaded based on a data loading request, the data in the eight leaf nodes contained in a leaf parent node are always synchronously loaded into the memory. When the calculation is completed, in order to save memory, when data is unloaded based on a data unloading request, the data in the eight leaf nodes contained in a leaf parent node are also synchronously unloaded from the memory.

[0078] As can be seen from the above, this embodiment uses a tree structure to represent heterogeneous three-dimensional data, thereby performing streaming processing calculations based on the tree structure. The alignment of the three-dimensional network model data and the target three-dimensional data is achieved through leaf nodes, and data operations are performed based on leaf parent nodes, which is not only conducive to improving the efficient and unified processing of heterogeneous three-dimensional data, but also can achieve local data access.

[0079] In the above embodiment, there is no limitation on how to calculate the color process of the sampling point according to different spatial positions. Based on the above embodiment, the present invention also provides an exemplary calculation process of the color of the sampling point, which may include the following contents:

[0080] For each sampling point of the light, if the current sampling point is located in the three-dimensional space area corresponding to the three-dimensional network model data, the color value of the current sampling point is determined according to the intensity value of the three-dimensional network model data; if the current sampling point is located in the two-dimensional plane area corresponding to the target three-dimensional data, the color value of the current sampling point is determined according to the color attributes of the target three-dimensional data and the direction of the light.

[0081] In this embodiment, when the sampling point is located in the spatial model corresponding to the three-dimensional network model data, the color of the sampling point depends on the intensity of each three-dimensional point of the three-dimensional spatial model. In order to further improve the color calculation accuracy, considering the influence of the previous sampling point on the color of the current sampling point, this embodiment can calculate the color value of the current sampling point according to the intensity value of the current sampling point and the previous sampling point adjacent to the current sampling point at the corresponding spatial position of the three-dimensional network model data. Exemplarily, the first color calculation relationship can be pre-stored , current sampling point Color , is the intensity of the previous sampling point, is the intensity of the current sampling point. These two values ​​can be determined by the three-dimensional network model data, that is, calculated by the neural network model itself. If the sampling point is located at the spatial position of the target three-dimensional data, the color of the sampling point depends on the properties of the object itself corresponding to the target three-dimensional data and the direction of the light: for the current sampling point, determine the target plane closest to the current sampling point in the two-dimensional plane area corresponding to the target three-dimensional data, and determine the target hemispherical surface formed by rotating the target plane in the opposite direction of the observation direction from the observation position of the image acquisition device to the target rendering pixel; determine the color value of the current sampling point according to the color value of the current sampling point corresponding to the spatial position of the target three-dimensional data and the cumulative value of the color values ​​calculated along different sampling directions for the target hemispherical surface. In this embodiment, the target three-dimensional data is three-dimensional grid data, and each grid unit surface of the three-dimensional grid data is a two-dimensional plane. The sampling point is located on the grid unit surface. A grid unit surface closest to the sampling point is selected from all the grid unit surfaces of the three-dimensional grid data. For the convenience of description, it is defined as the target plane. The observation direction from the observation position of the image acquisition device to the target rendering pixel is , then the opposite direction is . For example, the current sampling point Color , the second color calculation formula can be pre-stored , is the color value of the current sampling point at the spatial position of the target 3D data, that is, the color value of the target 3D data itself. The target plane is along The integral term is the hemisphere of the direction Along different sampling directions Perform the integral calculation of the calculated color, that is, the accumulated value of the color values ​​calculated along different sampling directions on the target hemisphere. is the optical property information of the target plane, yes Point along direction The color value of the incident light itself, It is the direction and The angle between them. Take the target 3D data as triangular mesh data as an example, Figure 4 As shown, for The color value of the point triangle mesh itself, For distance The plane of the triangle closest to the point is along The hemisphere of the direction, It is the distance The bidirectional scattering distribution function of the triangle surface closest to the point, which represents the reflection, refraction and other attribute information of the object surface, is the predefined information contained in the triangular mesh model itself. yes Point along direction The color value of the incident light itself, It is the direction and After calculating the color of each sampling point on the light, the color value of the rendered pixel U can be expressed as , is the color value of the starting point. The starting point is the position of the observation point. The color of this point is considered to be pure black without any information. , R1 is the color value of the first sampling point, and R2 is the color value of the second sampling point.

[0082] As can be seen from the above, in the process of color rendering calculation for heterogeneous three-dimensional data in this embodiment, the color of the sampling points on the surface of the target three-dimensional data is calculated using the color of the target three-dimensional data itself and the light color, and the color at other spatial positions is calculated using the intensity value output by the neural network model. By fusing and accumulating the colors of the heterogeneous three-dimensional data, a consistent color rendering calculation result is achieved.

[0083] The above embodiment does not limit how to determine the spatial position of the sampling point. The present invention also provides a method for identifying the spatial position of the sampling point based on the distance between the sampling point and the grid unit surface, which may include the following contents:

[0084] For each sampling point of the light, the distance value between the current sampling point and the surface of each grid unit of the target three-dimensional data is calculated, and the minimum distance is selected from them; the preset distance threshold is determined according to the tolerance error allowable value of whether the sampling point position is located in the two-dimensional plane area of ​​the target three-dimensional data; if the minimum distance is greater than the preset distance threshold, the current sampling point is located in the three-dimensional space area corresponding to the three-dimensional network model data; if the minimum distance is less than the preset distance threshold, the current sampling point is located on the grid surface of the target three-dimensional data.

[0085] In this embodiment, the target three-dimensional data is three-dimensional grid data. The distance between the current sampling point and each grid unit surface of the three-dimensional grid data can be calculated by any point-to-surface distance calculation method, and then the minimum distance value is selected from all distances, that is, the current sampling point is obtained. The closest distance dist of all faces of the 3D mesh model corresponding to the 3D mesh data. The preset distance threshold represents the tolerable error in determining whether the sampling point is located on the mesh unit face of the 3D mesh model corresponding to the 3D mesh data. The preset distance threshold can be determined based on the accuracy requirements of the rendering calculation or the modeling accuracy of the 3D mesh model itself corresponding to the 3D mesh data. For example, if the triangle mesh model itself indicates an error of 1 mm, the preset distance threshold can be 1 mm. If dist is greater than the preset distance threshold, the point is considered to be Located in the three-dimensional space represented by the neural network model, if dist is less than or equal to the preset distance threshold, then the point Located on the mesh element surface of the three-dimensional mesh model corresponding to the three-dimensional mesh data.

[0086] In order to further improve the accuracy of color calculation, based on the above embodiment, this embodiment determines the sampling direction of the next sampling point according to the spatial position of the current sampling point. The process of sampling light based on the method for determining the sampling direction may include:

[0087] Taking the observation position of the image acquisition device as the starting point of the light, along the observation direction from the observation position of the image acquisition device to the target rendering pixel, a first sampling point is obtained according to a preset light sampling step; when the first sampling point is located in the three-dimensional space area corresponding to the three-dimensional network model data, when the color value of the first sampling point is determined, the first sampling point is used as the current light starting point, and a second sampling point is obtained along the observation direction according to the preset light sampling step; when the first sampling point is located in the two-dimensional plane area corresponding to the target three-dimensional data, when the color value of the first sampling point is determined, the target plane closest to the first sampling point is determined on the two-dimensional plane area corresponding to the target three-dimensional data, the optical property information of the target plane is sampled and calculated at the first sampling point, the sampling direction is determined, and the first sampling point is used as the current light starting point, and a second sampling point is obtained along the sampling direction according to the preset light sampling step; the execution is repeated until the preset sampling stop condition is met.

[0088] In this embodiment, each sampling point is a point in three-dimensional space, along the light Perform sampling with a step length of the preset light sampling step length s to obtain the first sampling point . Determine the spatial position of the sampling point. If it is located in the three-dimensional space area corresponding to the three-dimensional network model data, As the starting point, , along the direction Calculate the second sampling point according to the preset light sampling step s Location: If it is located on the two-dimensional plane area corresponding to the target three-dimensional data, such as on the grid unit surface of the three-dimensional grid model corresponding to the three-dimensional grid data, then The sampling method such as Monte Carlo sampling, importance sampling, acceptance-rejection sampling, etc. can be used to sample the optical property information of the target plane, so as to determine the sampling direction of the second sampling point. As the starting point, along the calculated sampling direction Calculate the second sampling point according to the preset light sampling step s Location: Taking the three-dimensional grid data as an example, the direction is obtained by using Monte Carlo sampling to calculate the bidirectional scattering distribution function of the triangle face closest to the first sampling point. . After the second sampling point is calculated, the second sampling point is used as the first sampling point, and the above steps are continuously repeated until the preset sampling stop condition is reached. The preset sampling stop condition may be, for example, a preset sampling threshold. Accordingly, the preset sampling stop condition is satisfied when the current sampling times of the light reaches the preset sampling threshold. Of course, the preset sampling stop condition may also be other conditions, such as the total number of sampling points. Accordingly, the preset sampling stop condition is satisfied when the total number of sampling points at the current moment is counted.

[0089] It is understandable that in the process of generating three-dimensional digital content, the generated three-dimensional digital content usually includes multiple three-dimensional models, and different three-dimensional models correspond to different physical entities. In order to ensure that the physical entities do not collide, that is, the three-dimensional digital content does not have at least two three-dimensional models overlapping in an unauthorized situation. Based on the above embodiment, the present invention can convert the heterogeneous three-dimensional data generated by S101 into three-dimensional mesh data, that is, the target three-dimensional data is three-dimensional mesh data, and then implement physical interaction calculation of heterogeneous three-dimensional data according to the physical interaction method of three-dimensional mesh data, which may include the following contents:

[0090] Perform spatial sampling on the three-dimensional space area corresponding to the three-dimensional network model data to obtain spatial sampling point information; perform sampling processing on the spatial sampling point information again to obtain spatial regular point information; divide the spatial regular point information according to the grid units of the target three-dimensional data to obtain three-dimensional model conversion data of the same data type as the target three-dimensional data; perform physical interaction calculation on the three-dimensional model conversion data and the target three-dimensional data to identify whether the corresponding three-dimensional entities collide during the three-dimensional digital content generation process.

[0091] In this embodiment, the three-dimensional physical space represented by the neural network model is sampled to obtain a series of three-dimensional points. Each three-dimensional point (x, y, z) can obtain an intensity through the neural network model. , that is, each three-dimensional point can be expressed as , all three-dimensional points together constitute an intensity field in three-dimensional space. This embodiment defines it as spatial sampling point information, which may include three-dimensional coordinate values ​​and intensity information of each three-dimensional point in the three-dimensional space region. For example, (X, Y, Z, D) may be used to represent a set of spatial sampling points. In order to improve the accuracy of data conversion, after obtaining the spatial sampling point information, it is necessary to continue sampling to obtain spatial regular point information, and finally perform segmentation processing on the spatial regular point information. The segmentation result is the final converted data, that is, the neural network data is converted into three-dimensional grid data. Physical interaction calculation can be achieved through methods such as position-based dynamic calculation. When the physical interaction calculation results are obtained, it is possible to identify whether there is a collision between objects based on the physical interaction results between the three-dimensional objects, and then further calculate the physical properties of the objects such as motion changes and deformation after the collision.

[0092] In order to further improve the accuracy of physical interaction calculation, the present invention adopts progressive sampling from coarse to fine for the three-dimensional network model data, so as to obtain a more accurate set of sampling points in the three-dimensional space corresponding to the three-dimensional network model:

[0093] The three-dimensional space area corresponding to the three-dimensional network model data is sampled according to a first preset spatial sampling step, and spatial sampling points whose intensity values ​​are greater than a first preset light intensity threshold are selected from each spatial sampling point to generate initial spatial sampling point information; according to the three-dimensional coordinate values ​​of each spatial sampling point in the initial spatial sampling point information, the spatial area between the maximum coordinate value and the minimum coordinate value in each coordinate axis direction is determined respectively to obtain a horizontal space area, a vertical space area and a vertical space area; the intersection of the horizontal space area, the vertical space area and the vertical space area is taken as the minimum enclosing space; the minimum enclosing space is sampled according to a second preset spatial sampling step to generate an intermediate sampling point set; and spatial sampling points whose intensity values ​​are greater than a second preset light intensity threshold are selected from each spatial sampling point in the intermediate sampling point set to generate spatial sampling point information.

[0094] In this embodiment, the step sizes of the two samplings are different, and the step size of the latter sampling is smaller than the step size of the former sampling, that is, the sampling step size of the second preset space is smaller than the sampling step size of the first preset space. Uniform sampling of , in order to facilitate the distinction, it is defined as a spatial sampling point, , , Indicates the three-dimensional spatial position of all spatial sampling points obtained by the first sampling. Indicates the intensity values ​​of all sampling spatial points, and the intensity values ​​of the spatial sampling points selected from the sampling point set A are greater than the first preset light intensity threshold The points form a point set A’ (X 1’ , Y 1’ , Z 1’ , D 1’ ), for the convenience of description, it is defined as the initial spatial sampling point information, which can be expressed in the form of a data set, X 1’ , Y 1’ , Z 1’ represents the three-dimensional spatial position of the spatial sampling point selected from A, D 1’ The intensity value of the corresponding spatial sampling point. When the initial spatial sampling point information is obtained, calculate A ’ (X 1’ , Y 1’ , Z 1’ , D 1’ ) in three-dimensional space [X 1’ min , X 1’ max ]∩[Y 1’ min , Y 1’ max ]∩[Z 1’ min , Z 1’ max ], that is, the point set A ’ The spatial coordinates of all points in the X direction are calculated, and the spatial area between the minimum and maximum coordinates in the Y direction, the spatial area between the minimum and maximum coordinates in the Z direction are calculated, and the intersection of the three is taken as A. ’ The minimum enclosing space is uniformly sampled with a step length of the second preset spatial sampling step length, such as d1 / 3. The intensity value of each spatial sampling point can be determined by the neural network model, and all intensity values ​​greater than the second preset light intensity threshold are selected, such as The spatial sampling points are taken as the final sampling point set , that is, to obtain the spatial sampling point information.

[0095] For example, in order to improve the accuracy of the spatial regular point information, based on the above embodiment, the present invention also determines the final included regular sampling points through multiple samplings, which may include the following contents:

[0096] Regularized sampling is performed on the spatial sampling point information to obtain an initial spatial regular point set; according to the three-dimensional coordinate values ​​of each initial regular sampling point of the initial spatial regular point set, the spatial area between the maximum coordinate value and the minimum coordinate value in each coordinate axis direction is determined respectively to obtain a horizontal regular area, a vertical regular area and a vertical regular area; the intersection of the horizontal regular area, the vertical regular area and the vertical regular area is used as the minimum regular enclosing space of the initial spatial regular point set; the minimum regular enclosing space is sampled according to a third preset spatial sampling step to obtain a spatial regular point set, and the intensity value of each regular sampling point of the spatial regular point set is obtained according to the output of the corresponding neural network model.

[0097] In this embodiment, the spatial sampling point information can be represented by a data set, such as in the above embodiment. , perform regularized sampling on the spatial sampling point information to obtain an initial spatial regular point set, and then calculate the minimum enclosing space of the initial spatial regular point set. For this minimum enclosing space, perform uniform sampling along the X, Y, and Z coordinate axis directions with a step length of the third preset spatial sampling step length, for example, d1 / 6 can be selected, and calculate the intensity value of each sampling point to obtain a regularized point set. The third preset spatial sampling step length is smaller than the second preset spatial sampling step length. Those skilled in the art can flexibly select the value of each step length according to actual conditions, and the present invention does not impose any limitation on this.

[0098] Exemplarily, after obtaining the spatial regular point information, regular sampling points with intensity values ​​greater than a second preset light intensity threshold can be taken as real points, and isosurfaces can be extracted from the spatial regular point set according to the grid units of the target three-dimensional data, and three-dimensional model conversion data can be constructed based on the extracted isosurfaces.

[0099] In this embodiment, a real point indicates that the corresponding spatial position has data, and a regular sampling point with a light intensity less than the second preset light intensity threshold is a virtual point, which indicates that there is no data at the position of the point. Any isosurface extraction method can be used to segment the spatial regular point set, and the shape of the isosurface is the same as the surface shape of the grid unit. After segmentation, the three-dimensional grid format data corresponding to the data represented by the neural network is obtained. Taking triangular mesh data as an example, the voxel-level reconstruction method is used to segment the spatial regular point set into a triangular mesh, where all intensity values ​​greater than The points are taken as real points, which indicate that there is data at the position of the points, and the other points are virtual points, which indicate that there is no data at the position of the points. After segmentation, the triangular mesh format data corresponding to the data represented by the neural network is obtained. Among them, the process of segmenting the space regular point set using the voxel-level reconstruction method may include: reading the space regular point set into a specific array, dividing the neural network model into a number of small cubes, namely voxels, each voxel consisting of 8 vertices. It is judged whether the vertex of each voxel is in the neural network model. The vertex has two states: higher than or equal to the isosurface indicates that it is inside the neural network model, and lower than the isosurface indicates that it is outside the neural network model. A state table is generated according to the vertex state of the voxel to record which edges intersect with the isosurface. The position coordinates of the intersection point are calculated by the linear interpolation method, and the normal vector of the vertex is calculated by the central difference method. According to the intersection point and normal vector information, they are connected into triangular facets to generate an approximate representation of the isosurface.

[0100] As can be seen from the above, this embodiment uses a data conversion method based on progressive sampling to convert three-dimensional network model data represented by a neural network model into three-dimensional mesh data, thereby realizing physical interaction calculation based on the unified representation.

[0101] The present invention also provides a corresponding device for the three-dimensional digital content generation method, which further makes the method more practical. Among them, the device can be described from the perspective of functional modules and hardware. The three-dimensional digital content generation device provided by the present invention is introduced below. The device is used to implement the three-dimensional digital content generation method provided by the present invention. In this embodiment, the three-dimensional digital content generation device may include or be divided into one or more program modules, and the one or more program modules are stored in a storage medium and executed by one or more processors to complete the three-dimensional digital content generation method disclosed in the first embodiment. The program module referred to in this embodiment refers to a series of computer program instruction segments that can complete specific functions, which is more suitable for describing the execution process of the three-dimensional digital content generation device in the storage medium than the program itself. The following description will specifically introduce the functions of each program module of this embodiment. The three-dimensional digital content generation device described below and the three-dimensional digital content generation method described above can be referred to in correspondence with each other.

[0102] From the perspective of functional modules, see Figure 5 , Figure 5 The structure diagram of the three-dimensional digital content generation device provided in this embodiment under a specific implementation manner, the device may include:

[0103] The heterogeneous data representation module 501 is used to generate heterogeneous three-dimensional data by unifying the three-dimensional network model data and the target three-dimensional data in different data formats under the same coordinate system; the three-dimensional network model data is the physical light parameters converted from the three-dimensional space parameters through the neural network model.

[0104] The data representation module 502 is used to represent heterogeneous three-dimensional data using a target data structure, so as to load the heterogeneous three-dimensional data into a target storage location for color rendering calculation.

[0105] The light sampling module 503 is used to project light from the observation position of the image acquisition device along the observation direction of the target rendering pixel, and to perform sampling along the light according to preset light parameters.

[0106] The color calculation module 504 is used to determine the color value of each sampling point according to the spatial position relationship between each sampling point of the light and the three-dimensional network model data and the target three-dimensional data, and determine the color of the target rendering pixel according to the color value of each sampling point, so as to generate three-dimensional digital content according to each target rendering pixel, and the three-dimensional digital content includes multiple target rendering pixels.

[0107] Exemplarily, in some implementations of the present embodiment, the above-mentioned heterogeneous data representation module 501 can also be used to: input the three-dimensional spatial position and line of sight observation direction data in the target coordinate system into the neural network model, and the output of the neural network model is used as the color information and intensity information at each spatial position in the target coordinate system to obtain three-dimensional network model data; by matching the color information and intensity information of the three-dimensional network model data at the corresponding spatial position for each spatial position of the target three-dimensional data in the target coordinate system, heterogeneous three-dimensional data is obtained.

[0108] Exemplarily, in some other implementations of the present embodiment, the above-mentioned data representation module 502 can also be used to: determine the root node of the tree structure according to the axis-aligned bounding box of the heterogeneous three-dimensional data, and construct a tree structure by taking the root node as the starting point and splitting along different directions to obtain child nodes; store the target three-dimensional data and three-dimensional network model data in the heterogeneous three-dimensional data into the leaf nodes of the tree structure.

[0109] As an exemplary implementation method of the above embodiment, the above data representation module 502 can also be further used for: based on the tree type, dividing the root node bounding box into a corresponding number of child nodes along the directions of each coordinate axis of the spatial coordinate system; for each child node, dividing it again into a corresponding number of child nodes along the directions of each coordinate axis of the spatial coordinate system until the maximum side length of the current child node is less than a preset spatial accuracy threshold.

[0110] As another exemplary implementation method of the above embodiment, the above data representation module 502 can be further used for: the target three-dimensional data is three-dimensional grid data, and there are multiple three-dimensional network model data; based on the spatial position of the grid units of the target three-dimensional data, each grid unit of the target three-dimensional data is corresponded to a leaf node of the tree structure, and each leaf node contains multiple faces; a three-dimensional network model data in the heterogeneous three-dimensional data is stored in a leaf node of the tree structure, and the data of each spatial position within the spatial range covered by the current leaf node is determined according to the neural network model corresponding to the current leaf node.

[0111] As another exemplary implementation method of the above embodiment, the above data representation module 502 can be further used to: determine each leaf parent node of the tree structure and its corresponding leaf nodes; when a heterogeneous three-dimensional data processing request is received, the same data operation is synchronously performed on the data of all leaf nodes belonging to the same leaf parent node.

[0112] As an exemplary implementation method of the above embodiment, the above data representation module 502 can be further used for: when a data loading request is received, the heterogeneous three-dimensional data is loaded into the memory in a manner that the data of all leaf nodes belonging to the same leaf parent node are synchronously loaded into the memory; when a data unloading request is received, the heterogeneous three-dimensional data is deleted from the memory in a manner that the data of all leaf nodes belonging to the same leaf parent node are synchronously unloaded from the memory.

[0113] Exemplarily, in some other implementations of this embodiment, the above-mentioned color calculation module 504 can also be used for: for each sampling point of the light, if the current sampling point is located in the three-dimensional space area corresponding to the three-dimensional network model data, then the color value of the current sampling point is determined according to the intensity value of the three-dimensional network model data; if the current sampling point is located in the two-dimensional plane area corresponding to the target three-dimensional data, then the color value of the current sampling point is determined according to the color attributes of the target three-dimensional data and the direction of the light.

[0114] As an exemplary implementation of the above embodiment, the color calculation module 504 can be further used to calculate the color value of the current sampling point according to the intensity values ​​of the current sampling point and the previous sampling point adjacent to the current sampling point at corresponding spatial positions of the three-dimensional network model data.

[0115] As another exemplary implementation method parallel to the above-mentioned embodiment, the above-mentioned color calculation module 504 can also be further used to: for the current sampling point, determine the target plane closest to the current sampling point in the two-dimensional plane area corresponding to the target three-dimensional data, and determine the target hemispherical surface formed by rotating the target plane in the opposite direction to the observation direction from the observation position of the image acquisition device to the target rendering pixel; determine the color value of the current sampling point according to the color value of the current sampling point at the spatial position corresponding to the target three-dimensional data and the accumulated value of the color values ​​calculated along different sampling directions for the target hemispherical surface.

[0116] Illustratively, in some other implementations of the present embodiment, the color calculation module 504 may also be used for: if the target three-dimensional data is three-dimensional grid data, for each sampling point of the light, calculating the distance value between the current sampling point and the surface of each grid unit of the target three-dimensional data, and selecting the minimum distance therefrom; determining a preset distance threshold according to a tolerance error allowable value of whether the sampling point position is located in a two-dimensional plane area of ​​the target three-dimensional data; if the minimum distance is greater than the preset distance threshold, the current sampling point is located in the three-dimensional space area corresponding to the three-dimensional network model data; if the minimum distance is less than the preset distance threshold, the current sampling point is located on the grid surface of the target three-dimensional data.

[0117] Exemplarily, in some other implementations of the present embodiment, the above-mentioned light sampling module 503 can also be used to: take the observation position of the image acquisition device as the starting point of the light, and obtain the first sampling point according to the preset light sampling step along the observation direction from the observation position of the image acquisition device to the target rendering pixel; when the first sampling point is located in the three-dimensional space area corresponding to the three-dimensional network model data, when the color value of the first sampling point is determined, take the first sampling point as the current light starting point, and obtain the second sampling point along the observation direction according to the preset light sampling step; when the first sampling point is located in the two-dimensional plane area corresponding to the target three-dimensional data, when the color value of the first sampling point is determined, determine the target plane closest to the first sampling point on the two-dimensional plane area corresponding to the target three-dimensional data, sample and calculate the optical property information of the target plane at the first sampling point, determine the sampling direction, take the first sampling point as the current light starting point, and obtain the second sampling point along the sampling direction according to the preset light sampling step; repeat the execution until the preset sampling stop condition is met.

[0118] Illustratively, in some other implementations of the present embodiment, the above-mentioned device may also include a physical interaction calculator, which may be used to: perform spatial sampling on the three-dimensional space area corresponding to the three-dimensional network model data to obtain spatial sampling point information; the spatial sampling point information includes the three-dimensional coordinate values ​​and intensity information of each three-dimensional point in the three-dimensional space area; perform sampling processing on the spatial sampling point information again to obtain spatial regular point information; divide the spatial regular point information according to the grid units of the target three-dimensional data to obtain three-dimensional model conversion data of the same data type as the target three-dimensional data; perform physical interaction calculations with the target three-dimensional data based on the three-dimensional model conversion data to identify whether the corresponding three-dimensional entities collide during the three-dimensional digital content generation process.

[0119] As an exemplary implementation method of the above embodiment, the above physical interaction calculator can be further used to: sample the three-dimensional spatial area corresponding to the three-dimensional network model data according to the first preset spatial sampling step, and select spatial sampling points whose intensity values ​​are greater than the first preset light intensity threshold from each spatial sampling point to generate initial spatial sampling point information; determine the spatial area between the maximum coordinate value and the minimum coordinate value in each coordinate axis direction according to the three-dimensional coordinate values ​​of each spatial sampling point of the initial spatial sampling point information, and obtain the horizontal spatial area, the vertical spatial area and the vertical spatial area; take the intersection of the horizontal spatial area, the vertical spatial area and the vertical spatial area as the minimum enclosing space; sample the minimum enclosing space according to the second preset spatial sampling step to generate an intermediate sampling point set; select spatial sampling points whose intensity values ​​are greater than the second preset light intensity threshold from each spatial sampling point in the intermediate sampling point set to generate spatial sampling point information; wherein the second preset spatial sampling step is smaller than the first preset spatial sampling step.

[0120] As another exemplary implementation method of the above embodiment, the above physical interaction calculator can be further used to: perform regularized sampling on the spatial sampling point information to obtain an initial spatial regular point set; determine the spatial area between the maximum coordinate value and the minimum coordinate value in each coordinate axis direction according to the three-dimensional coordinate values ​​of each initial regular sampling point of the initial spatial regular point set, and obtain a horizontal regular area, a vertical regular area and a vertical regular area; take the intersection of the horizontal regular area, the vertical regular area and the vertical regular area as the minimum regular enclosing space of the initial spatial regular point set; sample the minimum regular enclosing space according to a third preset spatial sampling step size to obtain a spatial regular point set, and the intensity value of each regular sampling point of the spatial regular point set is obtained according to the output of the corresponding neural network model.

[0121] As an exemplary implementation method of the above embodiment, the above physical interaction calculator can be further used to: based on regular sampling points with intensity values ​​greater than a second preset light intensity threshold as real points, perform isosurface extraction on the spatial regular point set according to the grid units of the target three-dimensional data, and construct three-dimensional model conversion data based on the extracted isosurfaces; wherein, the real point indicates that there is data at the corresponding spatial position, and the shape of the isosurface is the same as the surface shape of the grid unit.

[0122] The three-dimensional digital content generating device mentioned above is described from the perspective of functional modules. Furthermore, the present invention also provides an electronic device, which is described from the perspective of hardware. Figure 6 The following is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention in one implementation manner. Figure 6 As shown, the electronic device includes a memory 60 for storing a computer program; and a processor 61 for implementing the steps of the three-dimensional digital content generation method mentioned in any of the above embodiments when executing the computer program.

[0123] Among them, the processor 61 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the processor 61 may also be a controller, a microcontroller, a microprocessor or other data processing chip. The processor 61 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 61 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 61 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 61 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0124] The memory 60 may include one or more computer non-volatile storage media, which may be non-transitory. The memory 60 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the memory 60 may be an internal storage unit of an electronic device, such as a hard disk of a server. In other embodiments, the memory 60 may also be an external storage device of an electronic device, such as a plug-in hard disk equipped on a server, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Further, the memory 60 may also include both an internal storage unit of an electronic device and an external storage device. The memory 60 may not only be used to store application software and various types of data installed in the electronic device, such as: the code of the program in the process of executing the three-dimensional digital content generation method, but also be used to temporarily store data that has been output or is to be output. In this embodiment, the memory 60 is at least used to store the following computer program 601, wherein, after the computer program is loaded and executed by the processor 61, the relevant steps of the three-dimensional digital content generation method disclosed in any of the aforementioned embodiments can be implemented. In addition, the resources stored in the memory 60 may also include an operating system 602 and data 603, and the storage method may be temporary storage or permanent storage. The operating system 602 may include Windows, Unix, Linux, etc. The data 603 may include but is not limited to data corresponding to the three-dimensional digital content generation result.

[0125] In some embodiments, the electronic device may further include a display screen 62, an input / output interface 63, a communication interface 64 or a network interface, a power supply 65 and a communication bus 66. Among them, the display screen 62 and the input / output interface 63, such as a keyboard, belong to the user interface, and exemplary user interfaces may also include standard wired interfaces, wireless interfaces, etc. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device, etc. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface. The communication interface 64 may exemplarily include a wired interface and / or a wireless interface, such as a WI-FI interface, a Bluetooth interface, etc., which are generally used to establish a communication connection between the electronic device and other electronic devices. The communication bus 66 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0126] Those skilled in the art will understand that Figure 6 The structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than those shown in the figure, for example, it may also include a sensor 67 for realizing various functions.

[0127] It is understandable that if the three-dimensional digital content generation method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes but is not limited to: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM, register, hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), magnetic memory, removable disk, CD-ROM, magnetic disk or optical disk, etc. Various media that can store program codes. Based on this, the present invention also provides a non-volatile storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the three-dimensional digital content generation method recorded in any of the above embodiments are executed.

[0128] It is understandable that if the three-dimensional digital content generation method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, the computer software product does not need to be stored in a physical storage medium. For example, it can be directly transmitted to a computer or other device with information processing capabilities through a wired network or a wireless network to perform all or part of the steps of the methods of various embodiments of the present invention. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product. Based on this, the present invention also provides a computer program product that stores a computer program. When the computer program is executed by a processor, the steps of the three-dimensional digital content generation method described in any of the above embodiments are performed.

[0129] Finally, the present invention also provides a three-dimensional digital content generation system, see Figure 7, the three-dimensional digital content generation system may include a data storage medium 701, a buffer 702, a light sampler 703, a register group 704, a color renderer 705 and a three-dimensional digital content generator 706. Among them, the register group 704 may include multiple registers for storing calculation results generated by different processes. This embodiment will expand the registers included therein according to the description of the three-dimensional digital content generation process. The light sampler 703 and the three-dimensional digital content generator 706 may be solidified based on the computer program corresponding to the corresponding method steps recorded in the above method embodiment, that is, the light sampler 703 and the three-dimensional digital content generator 706 are firmware in the three-dimensional digital content generation system. In addition, the light sampler 703 and the three-dimensional digital content generator 706 may also exist in the three-dimensional digital content generation system in the form of a processor. The color renderer 705 may use any processor or accelerator with high computing performance.

[0130] The data storage medium 701 may be, for example, DDR (Double Data Rate) or HBM (High Bandwidth Memory), which stores the model data of the neural network model, the target three-dimensional data, and the preset light parameters. The target three-dimensional data takes a triangular mesh as an example, which may include a mesh triangular face, a material represented by a bidirectional reflectance distribution function, a texture map, etc. The model data includes network weight data, three-dimensional space position information, and a line of sight observation direction. The preset light parameters may include a light starting point and direction. The buffer 702 loads the model data, the target three-dimensional data, and the preset light parameters from the data storage medium 701, and sends the three-dimensional space position information and the line of sight observation direction in the model data to the network input register in the register group 704, sends the network parameter data and the target three-dimensional data in the model data to the color renderer 705, and sends the preset light parameters to the light sampler 703. The light sampler 703 projects light along the observation direction of the target rendering pixel according to the observation position from the image acquisition device, samples along the light according to the preset light parameters, and sends the sampling point of the light to the light register in the register group 704. The color renderer 705 includes a position relationship calculator, a parallel calculator, a serial calculator and a color accumulator; the parallel calculator reads the three-dimensional spatial position information and the line of sight observation direction from the network input register, and calculates the physical light parameters of the three-dimensional network model based on the three-dimensional spatial position information, the line of sight observation direction and the network parameter data. The position relationship calculator determines the spatial position relationship between each sampling point of the light and the three-dimensional network model data and the target three-dimensional data, and chooses to use a parallel calculator or a serial calculator to calculate the color value of each sampling point; the parallel calculator calculates the color value of each sampling point based on the physical light parameters of the three-dimensional network model data, the serial calculator calculates the color value of each sampling point based on the target three-dimensional data, the color accumulator determines the color of the target rendered pixel based on the sampling point color value output by the parallel calculator and the sampling point color value output by the serial calculator, and sends the color of the target rendered pixel to the data storage medium 701. The three-dimensional digital content generator 706 reads the color of each target rendered pixel from the data storage medium 701, and generates three-dimensional digital content based on the color of each target rendered pixel. Based on the above-mentioned three-dimensional digital content generation system, the generation process of three-dimensional digital content is as follows: Figure 8 As shown, Figure 8 The rectangular box with a triangle shape in the figure represents the corresponding register in the above process.

[0131] For the parallel processor of the color renderer 705, any device with parallel computing function can be used, such as a multiply-add unit array and an FPGA (Field Programmable Gate Array). The multiply-add unit array is suitable for application scenarios with extremely high requirements on computing speed and relatively fixed functions, such as high-performance computing and image processing. FPGA is suitable for application scenarios that require high flexibility and reconfigurability, such as artificial intelligence, machine learning, signal processing, etc., especially when it is necessary to process multiple algorithms and data processing tasks. Based on this, the parallel calculator of the above embodiment can be composed of multiply-add units, such as Figure 8 As shown, it may include an intensity calculation unit and a color calculation unit; the intensity calculation unit and the color calculation unit are both multiplication-addition unit arrays, the number of multiplication-addition units may be 64 or a multiple of 64, the number of multiplication-addition units included in the intensity calculation unit and the number of multiplication-addition units included in the color calculation unit may be different, and the number of multiplication-addition units of the intensity calculation unit matches the scale of the neural network model, such as one multiplication-addition unit can perform the calculation of at least one network hidden layer at a time.

[0132] Furthermore, the present invention can encapsulate the computer program for determining the spatial position of the sampling point by distance calculation in the above method embodiment into a microprocessor or solidify it. For the convenience of description, this embodiment defines it as a distance calculator, such as Fig. 9 As shown, the three-dimensional digital content generation system also includes a distance calculator. The distance calculator reads the target three-dimensional data from the buffer, reads the sampling points of the light from the light register, calculates the distance value between the current sampling point and the surface of each grid unit of the target three-dimensional data for each sampling point of the light, selects the minimum distance from them, and stores the minimum distance in the distance register in the register group; correspondingly, the position relationship calculator reads the minimum distance from the distance register, if the minimum distance is greater than the preset distance threshold, the current sampling point is located in the three-dimensional space area corresponding to the three-dimensional network model data, and a parallel calculator is used to calculate the color value of the current sampling point; if the minimum distance is less than the preset distance threshold, the current sampling point is located on the grid surface of the target three-dimensional data, and a serial calculator is used to calculate the color value of the current sampling point.

[0133] Furthermore, the present invention can encapsulate the computer program for color calculation of the sampling points of the three-dimensional grid model corresponding to the target three-dimensional data in the above method embodiment into a microprocessor or solidify it to form a sampling direction calculation unit, a grid color calculation unit and a sampling color accumulator, such as Fig. 9As shown, correspondingly, the serial calculator of the above embodiment may include a sampling direction calculation unit, a grid color calculation unit and a sampling color accumulator; wherein the sampling direction calculation unit determines the target plane closest to the current sampling point on the two-dimensional plane area corresponding to the target three-dimensional data, performs sampling calculation on the optical property information of the target plane at the current sampling point, determines the sampling direction, and stores each sampling direction in the direction register of the register group; the grid color calculation unit rotates the target plane in the opposite direction of the observation direction from the observation position of the image acquisition device to the target rendering pixel to form a target hemispherical surface, reads the sampling direction from the direction register, determines the color value of the target hemispherical surface along the current sampling direction according to the color value at the spatial position corresponding to the current sampling point in the target three-dimensional data, the optical property information of the target plane, and the color value of the current incident light, and stores it in the single-direction color result register in the register group; the sampling color accumulator reads the color value of the target hemispherical surface along the current sampling direction from the single-direction color result register, accumulates the color values ​​of the target hemispherical surface along different sampling directions, obtains the color value of the current sampling point, and stores the color value of the current sampling point in the sampling point color result register.

[0134] Furthermore, the present invention can encapsulate the computer program corresponding to the physical interaction calculation process of the above method embodiment into a microprocessor or solidify it to form a physical interactor, such as Fig. 9 As shown, the physical interaction calculator performs spatial sampling on the three-dimensional space area corresponding to the three-dimensional network model data to obtain spatial sampling point information; the spatial sampling point information includes the three-dimensional coordinate values ​​and intensity information of each three-dimensional point in the three-dimensional space area; the spatial sampling point information is sampled again to obtain spatial regular point information; the spatial regular point information is segmented according to the grid units of the target three-dimensional data to obtain three-dimensional model conversion data of the same data type as the target three-dimensional data; physical interaction calculation is performed based on the three-dimensional model conversion data and the target three-dimensional data to identify whether the corresponding three-dimensional entities collide during the three-dimensional digital content generation process.

[0135] The above is a detailed introduction to a three-dimensional digital content generation method, device, system, electronic device, non-volatile storage medium and computer program product provided by the present invention. The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can refer to each other. The units and algorithm steps of each example described in each disclosed embodiment are executed in electronic hardware or computer software depending on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods for each specific application to implement the described functions, and such implementation should not be considered to exceed the scope of the present invention. Without departing from the principles of the present invention, the present invention can also be improved and modified in a number of ways, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A method for generating three-dimensional digital content, characterized in that: Three-dimensional digital content includes multiple target rendering pixels, including: Heterogeneous three-dimensional data is generated by unifying three-dimensional network model data and target three-dimensional data in different data formats under the same coordinate system; the three-dimensional network model data is physical light parameters converted from three-dimensional space parameters through a neural network model; Representing the heterogeneous three-dimensional data using a target data structure to load the heterogeneous three-dimensional data into a target storage location for calculation; Projecting light from the observation position of the image acquisition device along the observation direction of the target rendering pixel, and sampling along the light according to preset light parameters; According to the spatial position relationship between each sampling point of the light and the three-dimensional network model data and the target three-dimensional data, the color value of each sampling point is determined, and the color of the target rendering pixel is determined according to the color value of each sampling point, so as to generate three-dimensional digital content according to each target rendering pixel.

2. The three-dimensional digital content generation method according to claim 1, characterized in that: The three-dimensional space parameters are the three-dimensional space position and the sight line observation direction, and the physical light parameters are the color information and the intensity information. The heterogeneous three-dimensional data is generated by unifying the three-dimensional network model data and the target three-dimensional data in different data formats under the same coordinate system, including: Inputting the three-dimensional spatial position and sight line observation direction data in the target coordinate system into the neural network model, the output of the neural network model is used as the color information and intensity information at each spatial position in the target coordinate system, and obtaining the three-dimensional network model data; By matching the color information and intensity information of the three-dimensional network model data at the corresponding spatial position for each spatial position of the target three-dimensional data in the target coordinate system, heterogeneous three-dimensional data is obtained.

3. The three-dimensional digital content generation method according to claim 1, characterized in that: Representing the heterogeneous three-dimensional data using a target data structure includes: Determine a root node of a tree structure according to the axis-aligned bounding box of the heterogeneous three-dimensional data, and construct the tree structure by taking the root node as a starting point and performing splitting in different directions to obtain child nodes; The target three-dimensional data and the three-dimensional network model data in the heterogeneous three-dimensional data are stored in the leaf nodes of the tree structure.

4. The method for generating three-dimensional digital content according to claim 3, characterized in that: The tree structure is constructed by taking the root node as the starting point and splitting in different directions to obtain child nodes, including: Based on the tree type, the bounding box of the root node is divided into a corresponding number of child nodes along the directions of each coordinate axis of the spatial coordinate system; Each child node is divided again into a corresponding number of child nodes along each coordinate axis direction of the spatial coordinate system until the maximum side length of the current child node is less than a preset spatial accuracy threshold.

5. The method for generating three-dimensional digital content according to claim 3, characterized in that: The target three-dimensional data is three-dimensional mesh data, the three-dimensional network model data is multiple, and the target three-dimensional data and the three-dimensional network model data in the heterogeneous three-dimensional data are stored in the leaf nodes of the tree structure, including: Based on the spatial position of the grid unit of the target three-dimensional data, each grid unit of the target three-dimensional data corresponds to a leaf node of the tree structure, each leaf node including a plurality of faces; A three-dimensional network model data in the heterogeneous three-dimensional data is stored in a leaf node of the tree structure, and data of each spatial position within the space covered by the current leaf node is determined according to the neural network model corresponding to the current leaf node.

6. The three-dimensional digital content generation method according to claim 3, characterized in that: Representing the heterogeneous three-dimensional data using a target data structure includes: Determine each leaf parent node of the tree structure and each corresponding leaf node contained therein; When a heterogeneous three-dimensional data processing request is received, the same data operation is synchronously performed on the data of all leaf nodes belonging to the same leaf parent node.

7. The method for generating three-dimensional digital content according to claim 6, characterized in that: When a heterogeneous 3D data processing request is received, the same data operation is synchronously performed on the data of all leaf nodes belonging to the same leaf parent node, including: When a data loading request is received, the heterogeneous three-dimensional data is loaded into the memory in a manner that the data of all leaf nodes belonging to the same leaf parent node are synchronously loaded into the memory; When a data unloading request is received, the heterogeneous three-dimensional data is deleted from the memory in a manner that the data of all leaf nodes belonging to the same leaf parent node are synchronously unloaded from the memory.

8. The three-dimensional digital content generation method according to claim 1, characterized in that: Determining the color value of each sampling point according to the spatial position relationship between each sampling point of the light and the three-dimensional network model data and the target three-dimensional data includes: For each sampling point of the light, if the current sampling point is located in the three-dimensional space area corresponding to the three-dimensional network model data, determining the color value of the current sampling point according to the intensity value of the three-dimensional network model data; If the current sampling point is located in a two-dimensional plane area corresponding to the target three-dimensional data, the color value of the current sampling point is determined according to the color attribute of the target three-dimensional data and the light direction.

9. The method for generating three-dimensional digital content according to claim 8, characterized in that: Determining the color value of the current sampling point according to the intensity value of the three-dimensional network model data includes: The color value of the current sampling point is calculated according to the intensity values ​​of the current sampling point and a previous sampling point adjacent to the current sampling point at corresponding spatial positions of the three-dimensional network model data.

10. The method for generating three-dimensional digital content according to claim 8, characterized in that: Determining the color value of the current sampling point according to the color attribute of the target three-dimensional data and the light direction includes: For a current sampling point, determine a target plane closest to the current sampling point in a two-dimensional plane area corresponding to the target three-dimensional data, and determine a target hemispherical surface formed by rotating the target plane in a direction opposite to the observation direction from the observation position of the image acquisition device to the target rendering pixel; The color value of the current sampling point is determined according to the color value of the current sampling point corresponding to the spatial position of the target three-dimensional data and the accumulated value of the color values ​​calculated along different sampling directions of the target hemispherical surface.

11. The three-dimensional digital content generation method according to claim 1, characterized in that: The target three-dimensional data is three-dimensional grid data, and the color value of each sampling point is determined according to the spatial position relationship between each sampling point of the light and the three-dimensional network model data and the target three-dimensional data, including: For each sampling point of the light, calculate the distance value between the current sampling point and the surface of each grid unit of the target three-dimensional data, and select the minimum distance therefrom; Determine a preset distance threshold according to a tolerance error allowance value of whether the sampling point position is located in the two-dimensional plane area of ​​the target three-dimensional data; If the minimum distance is greater than the preset distance threshold, the current sampling point is located in the three-dimensional space area corresponding to the three-dimensional network model data; if the minimum distance is less than the preset distance threshold, the current sampling point is located on the grid surface of the target three-dimensional data.

12. The method for generating three-dimensional digital content according to claim 1, characterized in that: Sampling along the light according to preset light parameters includes: Taking the observation position of the image acquisition device as the starting point of the light, and obtaining a first sampling point according to a preset light sampling step length along the observation direction from the observation position of the image acquisition device to the target rendering pixel; When the first sampling point is located in the three-dimensional space area corresponding to the three-dimensional network model data, when the color value of the first sampling point is determined, the first sampling point is used as the starting point of the current light, and the second sampling point is obtained along the observation direction according to the preset light sampling step; when the first sampling point is located in the two-dimensional plane area corresponding to the target three-dimensional data, when the color value of the first sampling point is determined, the target plane closest to the first sampling point is determined on the two-dimensional plane area corresponding to the target three-dimensional data, the optical property information of the target plane is sampled and calculated at the first sampling point, the sampling direction is determined, the first sampling point is used as the starting point of the current light, and the second sampling point is obtained along the sampling direction according to the preset light sampling step; and the execution is repeated until the preset sampling stop condition is met.

13. The three-dimensional digital content generation method according to any one of claims 1 to 12, characterized in that: The target three-dimensional data is three-dimensional grid data, and further includes: Performing spatial sampling on the three-dimensional space region corresponding to the three-dimensional network model data to obtain spatial sampling point information; the spatial sampling point information includes three-dimensional coordinate values ​​and intensity information of each three-dimensional point in the three-dimensional space region; The spatial sampling point information is sampled again to obtain spatial regular point information; Divide the spatial regular point information according to the grid units of the target three-dimensional data to obtain three-dimensional model conversion data of the same data type as the target three-dimensional data; Physical interaction calculation is performed based on the three-dimensional model conversion data and the target three-dimensional data to identify whether the three-dimensional entities corresponding to each other collide during the three-dimensional digital content generation process.

14. The method for generating three-dimensional digital content according to claim 13, characterized in that: Performing spatial sampling on the three-dimensional space region corresponding to the three-dimensional network model data includes: Sampling the three-dimensional space region corresponding to the three-dimensional network model data according to a first preset spatial sampling step, and selecting spatial sampling points whose intensity values ​​are greater than a first preset light intensity threshold from each spatial sampling point to generate initial spatial sampling point information; According to the three-dimensional coordinate values ​​of each spatial sampling point of the initial spatial sampling point information, respectively determine the spatial area between the maximum coordinate value and the minimum coordinate value in each coordinate axis direction to obtain the horizontal spatial area, the vertical spatial area and the vertical spatial area; Taking the intersection of the horizontal space region, the vertical space region and the vertical space region as the minimum enclosing space; Sampling the minimum enclosing space according to a second preset spatial sampling step length to generate an intermediate sampling point set; Selecting a spatial sampling point whose intensity value is greater than a second preset light intensity threshold from each spatial sampling point in the intermediate sampling point set to generate spatial sampling point information; The second preset spatial sampling step size is smaller than the first preset spatial sampling step size.

15. The method for generating three-dimensional digital content according to claim 13, characterized in that: The spatial sampling point information is a spatial sampling point set, and the spatial sampling point information is sampled again to obtain spatial regular point information, including: Performing regularized sampling on the spatial sampling point information to obtain an initial spatial regular point set; According to the three-dimensional coordinate values ​​of each initial regular sampling point of the initial spatial regular point set, the spatial area between the maximum coordinate value and the minimum coordinate value in each coordinate axis direction is determined respectively to obtain a horizontal regular area, a vertical regular area and a vertical regular area; Taking the intersection of the horizontal regular region, the vertical regular region and the vertical regular region as the minimum regular enclosing space of the initial spatial regular point set; The minimum regular enclosing space is sampled according to a third preset spatial sampling step to obtain a spatial regular point set; the intensity value of each regular sampling point of the spatial regular point set is obtained according to the output of the corresponding neural network model.

16. The method for generating three-dimensional digital content according to claim 15, characterized in that: The spatial regular point information is divided according to the grid units of the target three-dimensional data, including: Based on the regular sampling points whose intensity values ​​are greater than the second preset light intensity threshold as real points, isosurface extraction is performed on the spatial regular point set according to the grid units of the target three-dimensional data, and three-dimensional model conversion data is constructed according to each extracted isosurface; The solid point indicates that there is data at the corresponding spatial position, and the shape of the isosurface is the same as the surface shape of the grid unit.

17. A three-dimensional digital content generation device, characterized in that: include: A heterogeneous data representation module, used for generating heterogeneous three-dimensional data by unifying three-dimensional network model data and target three-dimensional data in different data formats under the same coordinate system; The three-dimensional network model data is physical light parameters converted from three-dimensional space parameters through a neural network model; A data representation module, used for representing the heterogeneous three-dimensional data using a target data structure, so as to load the heterogeneous three-dimensional data into a target storage location for color rendering calculation; A light sampling module, used to project light from the observation position of the image acquisition device along the observation direction of the target rendering pixel, and to perform sampling along the light according to preset light parameters; The color calculation module is used to determine the color value of each sampling point according to the spatial position relationship between each sampling point of the light and the three-dimensional network model data and the target three-dimensional data, and determine the color of the target rendering pixel according to the color value of each sampling point, so as to generate three-dimensional digital content according to each target rendering pixel, and the three-dimensional digital content includes multiple target rendering pixels.

18. An electronic device, characterized in that: The device comprises a processor and a memory, wherein the processor is configured to implement the steps of the three-dimensional digital content generation method according to any one of claims 1 to 16 when executing a computer program stored in the memory.

19. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the three-dimensional digital content generation method according to any one of claims 1 to 16 are implemented.

20. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the three-dimensional digital content generation method according to any one of claims 1 to 16 are implemented.

21. A three-dimensional digital content generation system, characterized in that: It includes a data storage medium, a buffer, a light sampler, a register group, a color renderer and a three-dimensional digital content generator; Wherein, the data storage medium stores model data of the neural network model, target three-dimensional data and preset light parameters; The buffer loads the model data, the target three-dimensional data and the preset light parameters from the data storage medium, sends the three-dimensional space position information and the sight line observation direction in the model data to the network input register in the register group, sends the network parameter data in the model data and the target three-dimensional data to the color renderer, and sends the preset light parameters to the light sampler; The light sampler is configured to project a light ray along an observation direction of a target rendering pixel according to an observation position from an image acquisition device, and to sample along the light ray according to the preset light parameters, and to send the sampling point of the light ray to a light register in the register group; The color renderer includes a position relationship calculator, a parallel calculator, a serial calculator and a color accumulator; the parallel calculator is configured to read the three-dimensional spatial position information and the line of sight observation direction from the network input register, and calculate the physical light parameters of the three-dimensional network model data according to the three-dimensional spatial position information, the line of sight observation direction and the network parameter data; The position relationship calculator is configured to determine the spatial position relationship between each sampling point of the light and the three-dimensional network model data and the target three-dimensional data, and select whether to use a parallel calculator or a serial calculator to calculate the color value of each sampling point; The parallel calculator is further configured to calculate the color value of each sampling point according to the physical light parameters of the three-dimensional network model data, the serial calculator is configured to calculate the color value of each sampling point according to the target three-dimensional data, and the color accumulator is configured to determine the color of the target rendering pixel according to the sampling point color value output by the parallel calculator and the sampling point color value output by the serial calculator, and send the color of the target rendering pixel to the data storage medium; The three-dimensional digital content generator is configured to read the color of each target rendering pixel from the data storage medium and generate three-dimensional digital content according to the color of each target rendering pixel.

22. The three-dimensional digital content generation system according to claim 21, characterized in that: The parallel calculator includes an intensity calculation unit and a color calculation unit; The intensity calculation unit and the color calculation unit are both multiplication-addition unit arrays; the number of multiplication-addition units of the intensity calculation unit matches the scale of the neural network model.

23. The three-dimensional digital content generation system according to claim 21, characterized in that: Also includes a distance calculator; The distance calculator is configured to read the target three-dimensional data from the buffer, read the sampling points of the light from the light register, calculate the distance value between the current sampling point and the surface of each grid unit of the target three-dimensional data for each sampling point of the light, select the minimum distance therefrom, and store the minimum distance in the distance register in the register group; Correspondingly, the position relationship calculator is configured to read the minimum distance from the distance register. If the minimum distance is greater than a preset distance threshold, the current sampling point is located in the three-dimensional space area corresponding to the three-dimensional network model data, and the parallel calculator is used to calculate the color value of the current sampling point; if the minimum distance is less than the preset distance threshold, the current sampling point is located on the grid surface of the target three-dimensional data, and the serial calculator is used to calculate the color value of the current sampling point.

24. The three-dimensional digital content generation system according to claim 21, characterized in that: The serial calculator includes a sampling direction calculation unit, a grid color calculation unit and a sampling color accumulator; The sampling direction calculation unit is configured to determine a target plane closest to a current sampling point on a two-dimensional plane area corresponding to the target three-dimensional data, perform sampling calculation on the optical property information of the target plane at the current sampling point, determine a sampling direction, and store each sampling direction in a direction register of the register group; The grid color calculation unit is configured to rotate the target plane in the opposite direction to the observation direction from the observation position of the image acquisition device to the target rendering pixel to form a target hemispherical surface, read the sampling direction from the direction register, determine the color value of the target hemispherical surface along the current sampling direction according to the color value of the current sampling point corresponding to the spatial position of the target three-dimensional data, the light attribute information of the target plane, and the color value of the current incident light, and store it in the single-direction color result register in the register group; The sampling color accumulator is configured to read the color value of the target hemisphere along the current sampling direction from the unidirectional color result register, accumulate the color values ​​of the target hemisphere along different sampling directions, obtain the color value of the current sampling point, and store the color value of the current sampling point in the sampling point color result register.

Images