Method, Device, Program Product and Storage Medium for Processing Mixed Three-Dimensional Data

Abstract

The present application discloses a method, device, program product, and storage medium for processing mixed three-dimensional data, relating to the field of three-dimensional data processing technology, including obtaining scene data and rendering parameters; for any rendered pixel, determining the intersection of the projected light with the nearest object surface in the first object set after being emitted; accumulating the color of the projected light path using three-dimensional data in a neural network format to obtain the path color of the projected light, and superimposing it with the starting color of the projected light and the color of the three-dimensional data in a triangular mesh format at the end position of the projected light to obtain the starting color of the reflected light; using the reflected light as the projected light to continue iterating to obtain the final color of the rendered pixel, and generating a rendered image based on the final color of each rendered pixel. The method solves the problem of not supporting the processing of heterogeneous three-dimensional data, and achieves the technical effect of effectively rendering heterogeneous mixed three-dimensional data containing triangular mesh data and three-dimensional data based on a neural network.

Description

Technical Field

[0001] The present application relates to the technical field of three-dimensional data processing, and in particular to a method, device, program product and storage medium for processing mixed three-dimensional data. Background Art

[0002] Three-dimensional data in triangular mesh format is a typical form of three-dimensional data, and a mesh-like three-dimensional model can be constructed through a series of continuous triangles. In addition, with the widespread application of technologies such as artificial intelligence, three-dimensional data in neural network format, represented by neural radiation fields, has been increasingly used. Its representation format realizes the construction of three-dimensional content from image to image, and interactive three-dimensional content generation and rendering can be achieved directly by capturing images through the camera. However, the representation format of three-dimensional data in neural network format and that of three-dimensional data in traditional triangular mesh format are completely different in the processing pipeline. Currently, some software and hardware platforms and tools can accelerate the processing of three-dimensional data in neural network format, or optimize the design and implementation of three-dimensional data in triangular mesh format. However, they do not support the processing of mixed three-dimensional data. In other words, the current solutions can process a single type of three-dimensional data, but have not considered the processing of mixed three-dimensional data.

[0003] In summary, how to effectively achieve unified processing of mixed three-dimensional data including three-dimensional data in triangular mesh format and three-dimensional data in neural network format is a technical problem that technicians in this field urgently need to solve. Summary of the Invention

[0004] The present application provides a method, device, program product and storage medium for processing hybrid three-dimensional data, so as to at least solve the problem that the related art does not support the processing of heterogeneous three-dimensional data.

[0005] This application provides a method for processing mixed three-dimensional data, including:

[0006] Acquiring scene data and rendering parameters reflecting rendering requirements; wherein the scene data represents a three-dimensional scene in a preset three-dimensional space, including a first set of objects represented by three-dimensional data in a triangular mesh format, and a second set of objects represented by three-dimensional data in a neural network format;

[0007] For any rendering pixel in the rendered image, determining a starting position and a projection direction of a light ray of the rendering pixel based on the rendering parameters;

[0008] emitting a current projection ray from the starting position of the ray in the projection direction, and determining an intersection point between the projection ray and the nearest object surface in the first object set after the projection ray is emitted;

[0009] Performing color accumulation on the path of the projection light using the three-dimensional data in a neural network format to obtain the path color of the projection light, and superimposing the starting color of the projection light, the path color of the projection light, and the color of the three-dimensional data in a triangular mesh format at the end position of the projection light, to obtain the result as the starting color of the reflected light of the projection light;

[0010] Using the reflected ray as the current projection ray, returning to determine the intersection point between the projection ray and the nearest object surface in the first object set, until the iteration stops, using the color of the end point of the current projection ray as the final color of the rendered pixel;

[0011] A rendered image is generated based on the resulting final colors of each rendered pixel.

[0012] The present application also provides a computer program product, comprising a computer program, which implements the steps of the above-mentioned method for processing mixed three-dimensional data when executed by a processor.

[0013] The present application also provides an electronic device, comprising:

[0014] Memory for storing computer programs;

[0015] A processor is configured to implement the steps of the above-mentioned method for processing mixed three-dimensional data when executing the computer program.

[0016] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of the method for processing mixed three-dimensional data as described above are implemented.

[0017] In the present application, a unified processing method is designed for mixed 3D data, including 3D data in triangular mesh format and 3D data in neural network format, based on the respective data characteristics. This method, based on ray tracing and spatial segmented sampling, achieves unified color calculation between the mixed 3D data, enabling mutual illumination effects between the mixed 3D data and achieving unified processing. Specifically, after obtaining scene data and rendering parameters reflecting rendering requirements, for any rendered pixel in the rendered image, the starting position and projection direction of the ray for the rendered pixel are determined based on the rendering parameters. Since the starting position and projection direction of the ray are obtained, the first projected ray for the rendered pixel, i.e., the initial ray, can be obtained. Furthermore, the intersection point of the projected ray with the nearest object surface in the first set of objects represented by the 3D data in triangular mesh format can be determined. For the path of the projected ray, the color of the projected ray path can be accumulated using the 3D data in neural network format to obtain the path color of the projected ray. The starting color of the projected ray is then superimposed with the path color of the projected ray and the color of the 3D data in triangular mesh format at the end of the projected ray to obtain the starting color of the reflected ray of the projected ray. The reflected light is then used as the current projection light, and the above process is repeated. By continuously accumulating, when the iteration stops, the color of the end point of the current projection light can be used as the final color of the rendered pixel. It can be seen that the solution of the application can effectively obtain the final color of each rendered pixel, and finally, the generated rendered image is obtained based on the final color of each rendered pixel. In other words, it effectively realizes rendering for mixed 3D data containing 3D data in triangular mesh format and 3D data in neural network format. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0019] Figure 1 A flowchart of a method for processing mixed three-dimensional data provided by a specific embodiment of the present invention;

[0020] Figure 2 Schematic diagram of the representation principle of three-dimensional data in neural network format;

[0021] Figure 3 A schematic diagram of a structure for implementing hybrid three-dimensional data processing based on a first processing unit and a second processing unit in a specific embodiment of the present invention;

[0022] Figure 4A schematic diagram of the overall tasks and data flow of the first processing unit in a specific embodiment of the present invention;

[0023] Figure 5 A schematic diagram of the overall tasks and data flow of the second processing unit in a specific embodiment of the present invention;

[0024] Figure 6 This is a structural diagram of a second processing unit in a specific embodiment of the present invention;

[0025] Figure 7 A schematic structural diagram of an electronic device provided in a specific embodiment of the present invention;

[0026] Figure 8 This is a schematic structural diagram of a computer-readable storage medium of the present invention. DETAILED DESCRIPTION

[0027] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0028] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0029] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Figure 1 , Figure 1 This is a flowchart of a method for processing mixed three-dimensional data provided by a specific embodiment of the present invention. The method for processing mixed three-dimensional data may include the following steps:

[0030] Step S101: Acquire scene data and rendering parameters for reflecting rendering requirements; wherein the scene data represents a three-dimensional scene in a preset three-dimensional space, including a first set of objects represented by three-dimensional data in a triangular mesh format, and a second set of objects represented by three-dimensional data in a neural network format.

[0031] Specifically, in the present application, scene data can be obtained, and the scene data includes three-dimensional data in a triangular mesh format and three-dimensional data in a neural network format. That is, the present application can cope with scenes of mixed three-dimensional data including three-dimensional data in a triangular mesh format and three-dimensional data in a neural network format, and achieve unified processing based on the characteristics of each data.

[0032] The object represented by the three-dimensional data in the triangular mesh format is a mesh-like three-dimensional model composed of a series of continuous triangles. The set of objects represented by the three-dimensional data in the triangular mesh format is referred to as the first object set in this application.

[0033] The object represented by the three-dimensional data in the neural network format is a three-dimensional model mapped by the neural network model. The set of objects represented by the three-dimensional data in the neural network format is called the second object set in this application.

[0034] The basic definition of three-dimensional data in neural network format can be expressed as:

[0035] (r, g, b, σ) = f (x, y, z, θ, ), where x, y, z represent spatial coordinates, θ and Constitute the viewing direction, where θ is the angle between the viewing direction and the xy plane, is the angle between the viewing direction and the xz plane. r (red), g (green), and b (blue) represent the color value at position (x, y, z). σ represents the color intensity at position (x, y, z), reflecting transparency. Function f is implemented as a neural network. A typical implementation method is a multilayer perceptron, such as a neural radiance field. Figure 2 , is a schematic diagram of the representation principle of three-dimensional data in neural network format, which converts the viewpoint position (x, y, z), direction data (θ, ) as input, a position encoding network can first encode the viewpoint data into an implicit feature space. The encoded implicit features are then passed to the color rendering network to calculate the final color. The position encoding process can be implemented using neural networks or methods such as Fourier transforms. The color rendering network can be implemented by a single network or by the collaboration of two or more networks.

[0036] Three-dimensional data in triangular mesh format can be referred to as triangular mesh data, which can be specifically mesh format data. Common formats include obj, fbx, ply, etc.

[0037] The scene data of the present application includes three-dimensional data in triangular mesh format and three-dimensional data in neural network format. That is to say, the present application is a three-dimensional scene in a preset three-dimensional space composed of three-dimensional data in triangular mesh format and three-dimensional data in neural network format. The three-dimensional scene can simultaneously contain multiple models represented by three-dimensional data in neural network format, and multiple models represented by three-dimensional data in triangular mesh format.

[0038] Three-dimensional data can be converted into two-dimensional images through rendering and displayed to users. For example, in a specific scenario, the object set of the three-dimensional data in triangular mesh format consists of {G1, G2, G3, ..., Gn}, which is called the first object set. This means that the first object set is the set of objects represented by the three-dimensional data in triangular mesh format. Here, G1, G2, G3, ..., Gn represent different objects in the three-dimensional data in triangular mesh format, and a single object can be a person or an object. The object set represented by the three-dimensional data in neural network format is denoted as {F1, F2, F3, ..., Fm}, which is called the second object set. Here, F1, F2, F3, ..., Fm represent different objects in the three-dimensional data in neural network format, and a single object can be a person or an object.

[0039] Rendering parameters are used to reflect rendering requirements and may include the resolution (w*h) of the rendered image I, the camera observation position during the rendering process, the light source position, and other rendering-related information.

[0040] Step S102: for any rendering pixel in the rendering image, determine the starting position and projection direction of the light ray of the rendering pixel based on the rendering parameters.

[0041] For any rendered pixel of a rendered image, the final color of the rendered pixel can be obtained according to the principle of the present application scheme. The following text will use the process of determining the final color of a single rendered pixel as an example to explain the principle of the present application scheme in detail.

[0042] The camera observation position can be obtained through the rendering parameters, and then the camera observation position is used as the starting position of the light. In the following text, the starting position of the projected light is recorded as p0 for the convenience of description. It can be understood that when the projected light is the initial projected light, the starting position of the projected light p0 is the camera observation position. From the camera observation position, light can be projected to each pixel to be rendered in the rendered image I. Therefore, for any rendered pixel, the position of the rendered pixel can be determined based on the rendering parameters, and then the observation direction from the camera observation position to the position of the rendered pixel can be determined, that is, the projection direction of the initial projected light is obtained. For example, the observation direction corresponding to a certain rendered pixel U is recorded as , then in this example, we need to start from the camera observation position p0 and follow the observation direction Generate a projection ray, which is recorded as t0.

[0043] Step S103: emitting the current projection light from the starting position of the light in the projection direction, and determining the intersection point between the projection light and the nearest object surface in the first object set after the projection light is emitted.

[0044] According to the description above, for a certain rendering pixel, after obtaining the starting position and projection direction of the light, the current projection light can be obtained, that is, the current projection light is emitted from the starting position of the light in the projection direction. Of course, the projection light is not real natural light, but data in the computer. And it can be understood that for a certain rendering pixel, when the projection light is obtained for the first time, the starting position of the projection light is the camera observation position, and the projection direction of the projection light is the direction from the camera observation position to the rendering pixel position. When reflection occurs subsequently, the starting position of the reflected light is the end position of the previous level light, and the projection direction of the reflected light depends on the direction of the previous level light and the angle of the reflecting surface.

[0045] In the above example, for a certain rendering pixel, when step S103 is triggered for the first time, it starts from the camera observation position p0 and moves along the projection direction Generate the projection ray t0, and then determine the point closest to p0 among all the intersection points of the projection ray t0 and the first object set {G1, G2, G3, ..., Gn} represented by the three-dimensional data in the triangular mesh format. That is, determine the intersection point of the projection ray with the nearest object surface in the first object set after it is emitted. For example, in this example, this point is recorded as p1, and p1 is the end point position of the projection ray generated for the first rendering pixel in this example.

[0046] Step S104: Accumulate the color of the path of the projected light through the three-dimensional data in the neural network format to obtain the path color of the projected light, and superimpose the starting color of the projected light, the path color of the projected light, and the color of the three-dimensional data in the triangular mesh format at the end position of the projected light, and the result is used as the starting color of the reflected light of the projected light.

[0047] Since the intersection of the projected ray and the surface of the object closest to the first object set after it is emitted is obtained, that is, the end position of the projected ray is obtained, the color of the projected ray's path can be accumulated using the three-dimensional data in the neural network format to obtain the path color of the projected ray. In other words, in the above example, the space between p0 and p1 through which the projected ray t0 passes can be sampled according to the method of representing three-dimensional data using a neural network, the color of each sampling point can be obtained, and the colors can be accumulated and calculated to obtain the path color of the projected ray t0. By superimposing the path color of the projected ray t0 with the starting color of the projected ray t0, the color of the projected ray t0 when it reaches p1 can be obtained.

[0048] The starting color of the projection light, the path color of the projection light, and the color of the three-dimensional data in the triangular mesh format at the end position of the projection light are superimposed. The result is the color of the reflected light when it leaves the end position of the projection light after the projection light generates the reflected light, that is, the starting color of the reflected light of the projection light.

[0049] In the above example, by superimposing the path color of the projected light t0 with the starting color of the projected light t0, we can get the color of the projected light t0 when it reaches p1. Adding this color to the color of the three-dimensional data in the triangular mesh format at p1 will be the color of the reflected light of the projected light when it leaves p1.

[0050] In a specific embodiment of the present invention, the step S104 of performing color accumulation on the path of the projected light using the three-dimensional data in a neural network format to obtain the path color of the projected light may specifically include:

[0051] The color of each sampling point on the path of the projected light is calculated through three-dimensional data in a neural network format, and the color is accumulated through integration to obtain the path color of the projected light.

[0052] In the present application, it is necessary to perform color accumulation on the path of the projected light through three-dimensional data in a neural network format to obtain the path color of the projected light. This implementation method takes into account that when performing color accumulation, a more convenient implementation method is to set a number of sampling points on the path of the projected light, and then perform color calculation on each sampling point through the three-dimensional data in a neural network format, and perform color accumulation through integration to obtain the path color of the projected light. Moreover, this implementation method performs color accumulation through integration to obtain the path color of the projected light, and the accuracy is also relatively high, which is conducive to ensuring the rendering effect of the present application.

[0053] Furthermore, in a specific embodiment of the present invention, color calculation is performed on each sampling point on the path of the projected light using three-dimensional data in a neural network format, and color accumulation is performed by integration to obtain the path color of the projected light, which may specifically include:

[0054] The color of each sampling point on the path of the projected light is calculated through the three-dimensional data in the neural network format, and the The color is accumulated by the calculation method to obtain the color when the projection light reaches the end position, which is the superposition of the path color of the projection light and the starting color of the projection light.

[0055] in, is the starting color of the projected light, s represents the sampling point on the path of the projected light, and s∈(p0, p1], p0 is the starting position of the projected light, p1 is the end position of the projected light; T(s) is the color weight of the projected light when it reaches the sampling point s obtained by the three-dimensional data in the neural network format, and ; is the color intensity value at the sampling point s obtained by the three-dimensional data in the neural network format, is the color at the sampling point s obtained by the three-dimensional data in the neural network format, including r (red), g (green), and b (blue). e is the base of the natural logarithm, It is the color obtained by color accumulation when the projection light reaches the end position.

[0056] In this embodiment, color accumulation is performed by integration, and specifically weighted accumulation, which can be expressed as , that is, the color at the sampling point s , when accumulating, the color intensity value at the sampling point s needs to be , and the color weight T(s) when the projected light reaches the sampling point s as a weight to increase the color intensity value The degree of influence on the result makes the path color of the projected light more reasonable.

[0057] The integral result is the path color of the projected light, and the path color of the projected light The starting color of the cast ray Adding them together, we get the color when the projected light reaches the end point.

[0058] In a specific embodiment of the present invention, step S104 describes the process of superimposing the starting color of the projected light, the path color of the projected light, and the color of the three-dimensional data in triangular mesh format at the end position of the projected light, and obtaining the result as the starting color of the reflected light of the projected light, including:

[0059] pass The calculation method is used to obtain the starting color of the reflected light of the projected light.

[0060] Among them, R represents the basic color of the three-dimensional data in triangular mesh format at the end point of the projected light; is the weight coefficient of R in the jth iteration round and Q0=1; j represents the iteration round, when the projection ray is the first projection ray of the rendered pixel, j=1; The color of the 3D data in triangular mesh format at the end point of the projected ray. The starting color of the reflected ray of the cast ray. The color when the projected light reaches the end position. For example, in the above example, the color when the projected light reaches the end position is obtained by color accumulation. .

[0061] Taking the projection ray t0 above as an example, the projection ray starts from the camera observation position p0 and moves along the projection direction. Projection, the intersection point with the nearest object surface in the first object set is p1, then the projection light t0 will be reflected after reaching p1, generating a reflected light of the projection light t0, for example, recorded as reflected light t1 in this example, then the emission color of the reflected light t1 when leaving p1 is the starting color of the reflected light t1, which is the color of the projection light t0 when it reaches p1 superimposed on the color of the three-dimensional data in the triangular mesh format at the position p1.

[0062] In this example, after obtaining the three-dimensional data in the triangular mesh format, the base color R at the end point of the projected light needs to be combined with the weight coefficient Multiply, and the result is the color of the three-dimensional data in triangular mesh format at the end point of the projected light .Will The color when the projected light reaches the end position By superimposing them, we can get the color when the light leaves, that is, the starting color of the reflected light.

[0063] Weight coefficient The specific setting method can be set and adjusted according to actual needs, and usually needs to reflect the relevant attribute information of the intersection position of the projected light and the reflected light in the triangle mesh data.

[0064] In a specific embodiment of the present invention, the weight coefficient is the weight coefficient obtained based on the predefined information of the three-dimensional data in triangular mesh format and satisfies .

[0065] BSTF is the bidirectional scattering distribution function of the triangular surface at the starting position of the reflected light in the three-dimensional data of the triangular mesh format. is the cosine value of the angle between the projected light and the reflected light; B is the scattering probability density function, Q j is the weight coefficient of R in the j+1th iteration round.

[0066] In this embodiment, after the projection light is generated for the first time and the projection light generates the reflected light, it means that it is the first iteration, j=1, that is, the weight coefficient is Specifically, Q0 and Q0=1. In the second iteration, after the reflected light is emitted, it will intersect with the nearest object surface in the first set and generate a reflected light at this position, then j=2, which is the weight coefficient at this time Specifically, Q1 needs to be calculated based on Q0, and so on, Q2 needs to be calculated based on Q1, that is, at the intersection of the projected light and the reflected light, the weight coefficient It will be affected by the weight coefficient in the previous iteration, which is equivalent to the color of the three-dimensional data in the triangular mesh format at the end point of the projected light. It is affected not only by the attribute information of the position, but also by the impact of the incident light, thereby ensuring the color of the three-dimensional data in the triangular mesh format at the end point of the projected light. It is a relatively accurate and reasonable value.

[0067] In this implementation, the BSTF is the bidirectional scattering distribution function of the triangular surface at the starting point of the reflected light in the three-dimensional data in triangular mesh format. This bidirectional scattering distribution function can reflect the reflection, refraction and other properties of the object surface and is predefined information contained in the triangular mesh model itself. B is the scattering probability density function, which is related to the surface material of the triangular mesh model and is also predefined information contained in the triangular mesh model itself.

[0068] Step S105: Use the reflected light as the current projection light, and return to the operation described in step S103 to determine the intersection of the projection light and the nearest object surface in the first object set after it is emitted, until the iteration stops, and execute step S106: Use the color of the end point position of the current projection light as the final color of the rendered pixel.

[0069] Step S107: generating a rendered image based on the obtained final colors of the rendered pixels.

[0070] As can be seen above, color superposition can be used to determine the color of the reflected light at its exit, that is, the color of the reflected light's starting position. It can also be understood that the starting position of the reflected light is the endpoint of the corresponding projected light, and the observation direction of the reflected light (i.e., its exit direction) is determined by the observation direction of the corresponding projected light and the reflecting surface. Therefore, both the starting position and the observation direction of the reflected light can be determined. At this point, the next iteration can proceed, using the reflected light as the current projected light, and returning to the process of determining the intersection of the projected light with the nearest object surface in the first object set, as described in step S103, until the iteration ends.

[0071] The specific conditions for stopping the iteration can be set and adjusted according to actual needs. For example, it can usually be set so that when the projected light is emitted outside the scene during a certain iteration, the iteration can be determined to be stopped. For example, in a specific embodiment of the present invention, when the projected light is emitted outside the scene, or when the iteration round reaches a set round threshold, the iteration can be determined to be stopped. This is because in some scenes, there may be a large number of objects in the triangular mesh data format, and the light will continue to reflect between these objects and will not be emitted outside the scene. As the number of iteration rounds increases, the amount of calculation will continue to increase and the impact on the rendering result will be limited. Therefore, in this embodiment, it can be chosen to stop the iteration when the iteration round reaches the set round threshold.

[0072] By continuously accumulating colors, when the iteration stops, the color at the end point of the current projected ray can be used as the final color of the rendered pixel. According to the principle of the present application scheme, the final color of each rendered pixel can be obtained, and then a rendered image can be generated based on the obtained final colors of each rendered pixel.

[0073] In a specific embodiment of the present invention, it may further include:

[0074] Based on the scene data, determine whether there is an object blocking the light source from the end point of the cast ray. If so, determine the shadow weight.

[0075] The starting color of the projected light is superimposed with the path color of the projected light and the color of the three-dimensional data in the triangular mesh format at the end position of the projected light. The result is adjusted by the shadow weight, and the adjusted color is used as the starting color of the reflected light of the projected light.

[0076] This implementation takes into account that shadow factors can also be considered when rendering an image to ensure better rendering effects. Specifically, when a projected ray reaches its endpoint, it is necessary to determine the position of the light source based on the scene data, and then determine whether there is an object blocking the light source from the endpoint of the projected ray. For example, a shadow ray can be projected from the endpoint of the projected ray toward the light source, and then determine whether the shadow ray touches the object. This can determine whether there is an object blocking the light source from the endpoint of the projected ray.

[0077] If there is an object blocking the light, it is necessary to obtain the shadow weight according to the preset rules. Then, after superimposing the starting color of the projected light, the path color of the projected light, and the color of the three-dimensional data in the triangular mesh format at the end position of the projected light, the shadow weight is used to adjust the superposition result. For example, a simple processing method is to multiply the shadow weight by the superposition result, and then use the adjusted result as the starting color of the reflected light of the projected light. Of course, if there is no object blocking the end position of the projected light and the light source, no adjustment is required.

[0078] In a specific embodiment of the present invention, the method may further include: after the projection light reaches the end position of the projection light, when there is a refracted light of the projection light, determining the refraction color of the refracted light based on three-dimensional data in a triangular mesh format.

[0079] Accordingly, the starting color of the projected light is superimposed with the path color of the projected light and the color of the three-dimensional data in the triangular mesh format at the end position of the projected light, and the result is used as the starting color of the reflected light of the projected light, including:

[0080] The starting color of the projected light is superimposed with the path color of the projected light, the refraction color of the refracted light, and the color of the three-dimensional data in the triangular mesh format at the end position of the projected light, and the result is used as the starting color of the reflected light of the projected light.

[0081] This implementation takes into account that after the projected light reaches the end position, in addition to generating reflected light, refracted light may also be generated in some cases. After the refracted light enters the object of three-dimensional data in triangular mesh format, it will also affect the color of the rendered pixel.

[0082] In this embodiment, after the projected light reaches the end position of the projected light, it can be determined based on the three-dimensional data in the triangular mesh format whether it is necessary to generate a refracted light of the projected light. This can usually be determined based on the material of the position. If it is necessary to generate a refracted light of the projected light, that is, it indicates that there is a refracted light of the projected light, then it is necessary to determine the refraction color of the refracted light based on the three-dimensional data in the triangular mesh format. In addition, it is understood that when determining the refraction color of the refracted light, the color of the starting position of the refracted light is consistent with the color of the starting position of the reflected light generated at that position, and can be determined with reference to the description of the above embodiment.

[0083] After obtaining the refraction color of the refracted light, when superimposing, it is necessary to take the starting color of the projected light, the path color of the projected light, the refraction color of the refracted light, and the color of the three-dimensional data in the triangular mesh format at the end position of the projected light, and use the result as the starting color of the reflected light of the projected light. In this way, when the final color of the rendered pixel is finally obtained, the refracted light at the relevant position is taken into account, so that the final color of the rendered pixel obtained is more reasonable.

[0084] In a specific embodiment of the present invention, step S103 may specifically include: obtaining the current projection light based on the starting position and projection direction of the light through the first processing unit, and determining the intersection of the projection light and the nearest object surface in the first object set after being emitted.

[0085] Step S104 may specifically include: using the second processing unit, accumulating the color of the path of the projected light based on the three-dimensional data in the neural network format to obtain the path color of the projected light.

[0086] The first processing unit superimposes the starting color of the projected light, the path color of the projected light, and the color of the three-dimensional data in the triangular mesh format at the end position of the projected light, and the obtained result is used as the starting color of the reflected light of the projected light.

[0087] See Figure 3 , a schematic diagram of the structure of a specific embodiment of the present invention, which implements hybrid 3D data processing based on a first processing unit and a second processing unit. In this embodiment, two co-processing units, namely the first processing unit and the second processing unit, can be used to complete related calculations under the scheduling of the main processing unit, effectively ensuring the execution efficiency of the present application scheme. The first processing unit and the second processing unit are functionally divided, with the first processing unit responsible for processing the rendering calculations related to the 3D mesh data, and the second processing unit responsible for processing the rendering calculations related to the neural network 3D data.

[0088] Specifically, in actual applications, the main processing unit can be responsible for the overall operation logic control of the task, including loading and parsing scene data, creating data structures, allocating memory space and saving scene data, preparing relevant initialization data according to the image to be rendered, copying data to the first processing unit and the second processing unit, calling the first processing unit and the second processing unit to perform relevant calculations, collecting the calculation results of the first processing unit and the second processing unit, generating the final rendered image, and other functions.

[0089] The units can be specifically connected to each other through PCIE or other protocols, and the memory unit can specifically be a memory unit such as HBM, DDR, etc.

[0090] The first processing unit is responsible for processing the relevant rendering calculations of the three-dimensional mesh data. For example, in one case, refer to Figure 4 , which is a schematic diagram of the overall tasks and data flow of the first processing unit. It mainly includes generating initial rays (i.e., the initial projection rays for each rendered pixel) and initializing the data for each ray; calculating the intersection of the projection rays and the 3D mesh data; calculating secondary rays (reflected rays or refracted rays, depending on the material type at the surface location) at the intersection of the projection rays and the 3D mesh object surface; casting shadow rays from the intersection to the light source to determine whether the intersection is in the shadow area; summarizing the color values of each point on a ray path (including the color accumulation result of each sampling point of the neural network 3D data between two intersection points on the ray path calculated by the second co-processing unit and sent to the first co-processing unit, i.e., the path color of the projection ray); and outputting the final color value of the ray, i.e., the final color of the rendered pixel.

[0091] The second processing unit is responsible for processing the rendering calculation related to the neural network three-dimensional data. For example, in one case, refer to Figure 5 This is a schematic diagram of the overall tasks and data flow of the second processing unit. This primarily involves initializing the projected ray to be tracked based on the sequence of ray and grid intersections output by the first processing unit (i.e., locating the projected ray based on the relevant intersections), traversing the neural network's three-dimensional data voxel space based on the ray's direction, sampling spatial points along the projected ray, invoking the neural network to calculate color and intensity values at each sampled point, and summarizing the color data at each sampled point to obtain the color of the projected ray's path. This aggregated color result is then transmitted back to the first processing unit, enabling it to calculate the total color along the light path.

[0092] Data transmission between the first processing unit and the second processing unit can be transferred through a memory unit implemented by DDR or HBM, or data can be transmitted through a direct link.

[0093] In a specific embodiment of the present invention, the second processing unit includes a position encoding module and a color rendering module; the second processing unit accumulates the color of the path of the projected light using the three-dimensional data to obtain the path color of the projected light, including:

[0094] Dividing the path of the projected light into a plurality of sampling points through a position encoding module of the second processing unit, and obtaining implicit features of each sampling point;

[0095] The implicit features of each sampling point are processed by the color rendering module of the second processing unit to obtain the color value of each sampling point, and the path color of the projected light is obtained through weighted accumulation.

[0096] See Figure 6 , a schematic diagram of the second processing unit in one embodiment, includes a position encoding module and a color rendering module. The position encoding module calculates implicit features and encodes viewpoint data into an implicit feature space. The color rendering module calculates the final color based on the encoded implicit features through a color rendering network.

[0097] It can be seen that in this embodiment, the function of the second processing unit is effectively realized by the position encoding module and the color rendering module, and Figure 6 In the present invention, a customized architecture design is specially carried out for the rendering calculation of three-dimensional data in the neural network format, that is, a customized architecture design is carried out for the position encoding module and the color rendering module, so that efficient and fast processing and calculation of three-dimensional data in the neural network format can be achieved, which ensures the efficiency of the application solution in processing mixed three-dimensional data.

[0098] Specifically, in a specific embodiment of the present invention, please refer to Figure 6 The position encoding module can include: an input submodule, a sampling point generation submodule, a position processing submodule, a weight interpolation submodule, a hash table index submodule and an implicit feature calculation submodule, and each submodule can be implemented based on a hardware circuit.

[0099] The position encoding module of the second processing unit divides the path of the projected light into multiple sampling points, and obtains the implicit features of each sampling point, including:

[0100] Receive input data through the input submodule;

[0101] The sampling point generation submodule processes the input data to obtain the path of the projected light and divides multiple sampling points on the path of the projected light;

[0102] For any sampling point, based on the coordinates of the sampling point, the position processing submodule determines the index parameter corresponding to the coordinates of the sampling point;

[0103] Based on the index parameter, the weight parameter corresponding to the index parameter is determined by the weight interpolation submodule, and the index parameter is hashed by the hash table index submodule to obtain a hash index value;

[0104] Through the implicit feature calculation submodule, the lookup feature corresponding to the hash index value is determined, and the product of the lookup feature and the weight parameter is used as the implicit feature of the sampling point.

[0105] See Figure 6 In this embodiment, the input submodule receives input data. Figure 6 The input submodule specifically uses FIFO, and in actual applications, a single FIFO can be used, or a FIFO array can be used to improve the efficiency of parallel processing. For the input data received by the input submodule, as long as it is input data that can realize the positioning of the projected light, for example, in a specific example, the first processing unit sends the intersection sequence of the projected light and the three-dimensional grid data model to the second processing unit, so that the second processing unit stores it into a FIFO unit. The subsequent sampling point generation submodule can read a point coordinate from the FIFO unit each time, and use the previous point as the starting point of the projected light and the next point adjacent to it as the end point of the projected light, so as to realize the positioning of the projected light. In other words, in the intersection sequence, the line connecting two adjacent points is the path of the projected light, and the sampling point generation submodule can divide the path of the projected light into multiple sampling points.

[0106] When dividing the sampling points, a series of spatial sampling point coordinates are obtained along the path of the projected ray. In practical applications, the sampling method can be uniform sampling, that is, the coordinates of the sampling points are moved according to a fixed sampling step size. For example, batch calculation of the sampling points can be implemented through a multiply-add array. Alternatively, non-uniform sampling can be achieved through methods such as importance sampling. For example, in one scenario, if the path area of the projected ray contains objects with three-dimensional data in a neural network format, more dense sampling can be performed. Conversely, if the path area does not contain objects with three-dimensional data in a neural network format, the spacing between adjacent sampling points can be increased.

[0107] For any sampling point, based on the coordinates of the sampling point, the position processing submodule can determine the index parameter corresponding to the coordinates of the sampling point. Specifically, Figure 6 In an implementation manner, the coordinates (x, y, z) of the sampling point are multiplied by the preset scale parameters and superimposed with the preloaded offset, so as to obtain the index positions of the x, y, and z coordinate components in the spatial position storage table, that is, the index parameters of the coordinates of the sampling point are obtained.

[0108] Input the index parameter into the hash table index submodule, and perform hash calculation on the index parameter, and the result is used as the hash index value. Figure 6 In the implementation method, the functional formula of the hash calculation can be expressed as Here p j Represents the index parameter of a sampling point, symbol Indicates a bitwise exclusive OR operation, that is, if the corresponding binary bits of the two numbers are different, the bit is 1, otherwise the bit is 0. jk Indicates p j The three coordinate components of Ω k represents a large prime number, where Ω1 = 1, Ω2 = 2654435761, and Ω3 = 805459861. mod represents the remainder operation, and N represents the maximum number of features stored in the hash table.

[0109] At the same time, the index parameters are input to the weight interpolation submodule, Figure 6 In the embodiment, specifically, by superimposing the preloaded offset on the index parameter and performing a product calculation, the weight parameter corresponding to the index parameter can be determined, and Figure 6 The preload offset used here is specifically -1.

[0110] After obtaining the hash index value through the hash table index submodule, the hash table can be looked up based on the hash index value to obtain the lookup feature, which is then multiplied by the weight parameter output by the weight interpolation submodule to serve as the implicit feature of the sampling point. It can be cached in the cache submodule of the color rendering module, for example, specifically in the BRAM unit of the color rendering module.

[0111] In a specific embodiment of the present invention, the color rendering module includes: a color calculation unit array, a buffer submodule and an aggregation submodule. Each submodule can be implemented based on a hardware circuit.

[0112] The color rendering module of the second processing unit processes the implicit features of each sampling point to obtain the color value of each sampling point, and obtains the path color of the projected light through weighted accumulation, which may specifically include:

[0113] Based on the pre-stored model parameter data, the implicit features of the sampling points are processed by the color calculation unit array of the second processing unit to obtain the color values of the sampling points and store them in the cache submodule;

[0114] Based on the color values of each sampling point stored in the cache submodule, the path color of the projected light is obtained in a weighted accumulation manner through the aggregation submodule.

[0115] In this embodiment, the implicit features of the sampling points can be read through the color calculation unit array, which can be read individually or in batches. The color calculation unit array can also read the pre-cached neural network model parameter data to realize the processing of implicit features.

[0116] In a specific embodiment of the present invention, the color calculation unit array may include multiple color calculation units. In practical applications, the color calculation unit array may generally include 32 or 64 color calculation units, for example Figure 6 In the embodiment of the present invention, the color calculation unit array includes 32 color calculation units. Each color calculation unit is a multiplication-addition tree structure formed by connecting multiple multiplication-addition units. The multiplication-addition tree structure implemented by the color calculation unit usually has an initial number of 32 or 64 multiplication-addition units. For example, Figure 6 In the implementation method, it is 32.

[0117] By processing the implicit features of the sampling points through the color calculation unit array, according to the principles described above, the color values of the sampling points can be obtained and stored in the cache submodule. The aggregation submodule then uses a weighted accumulation method to obtain the path color of the projected light. The accumulated path color of the projected light can be stored in the I / O cache, from which the first processing unit can read the path color of the projected light via the relevant data link.

[0118] In the present application, a unified processing method is designed for mixed 3D data, including 3D data in triangular mesh format and 3D data in neural network format, based on the respective data characteristics. This method, based on ray tracing and spatial segmented sampling, achieves unified color calculation between the mixed 3D data, enabling mutual illumination effects between the mixed 3D data and achieving unified processing. Specifically, after obtaining scene data and rendering parameters reflecting rendering requirements, for any rendered pixel in the rendered image, the starting position and projection direction of the ray for the rendered pixel are determined based on the rendering parameters. Since the starting position and projection direction of the ray are obtained, the first projected ray for the rendered pixel, i.e., the initial ray, can be obtained. Furthermore, the intersection point of the projected ray with the nearest object surface in the first set of objects represented by the 3D data in triangular mesh format can be determined. For the path of the projected ray, the color of the projected ray path can be accumulated using the 3D data in neural network format to obtain the path color of the projected ray. The starting color of the projected ray is then superimposed with the path color of the projected ray and the color of the 3D data in triangular mesh format at the end of the projected ray to obtain the starting color of the reflected ray of the projected ray. The reflected light is then used as the current projection light, and the above process is repeated. By continuously accumulating, when the iteration stops, the color of the end point of the current projection light can be used as the final color of the rendered pixel. It can be seen that the solution of the application can effectively obtain the final color of each rendered pixel, and finally, the generated rendered image is obtained based on the final color of each rendered pixel. In other words, it effectively realizes rendering for mixed 3D data containing 3D data in triangular mesh format and 3D data in neural network format.

[0119] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0120] Corresponding to the above method embodiments, embodiments of the present invention further provide an electronic device, a computer-readable storage medium, and a computer program product, which may refer to each other in correspondence with the above.

[0121] See also Figure 7 As shown, the electronic device may include:

[0122] Memory 701, used for storing computer programs;

[0123] A processor 702 is configured to execute a computer program to implement the steps of the method for processing mixed three-dimensional data in any of the above embodiments;

[0124] The computer program product includes a computer program, and when the computer program is executed by a processor, the computer program implements the steps of the method for processing mixed three-dimensional data in any of the above embodiments.

[0125] See Figure 8 The computer-readable storage medium 80 stores a computer program 81. When executed by a processor, the computer program 81 implements the steps of the method for processing hybrid 3D data in any of the above-described embodiments. The computer-readable storage medium 80 herein may include, but is not limited to, a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard drive, a magnetic disk, or an optical disk, among other media capable of storing computer programs.

[0126] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0127] The above is a detailed introduction to the processing method, device, program product and storage medium of mixed three-dimensional data provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core ideas of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the present application.

Claims

1. A method for processing mixed three-dimensional data, characterized in that: include: Acquiring scene data and rendering parameters reflecting rendering requirements; wherein the scene data represents a three-dimensional scene in a preset three-dimensional space, including a first set of objects represented by three-dimensional data in a triangular mesh format, and a second set of objects represented by three-dimensional data in a neural network format; For any rendering pixel in the rendered image, determining a starting position and a projection direction of a light ray of the rendering pixel based on the rendering parameters; emitting a current projection ray from the starting position of the ray in the projection direction, and determining an intersection point between the projection ray and the nearest object surface in the first object set after the projection ray is emitted; Performing color accumulation on the path of the projection light using the three-dimensional data in a neural network format to obtain the path color of the projection light, and superimposing the starting color of the projection light, the path color of the projection light, and the color of the three-dimensional data in a triangular mesh format at the end position of the projection light, to obtain the result as the starting color of the reflected light of the projection light; Using the reflected ray as the current projection ray, returning to determine the intersection point between the projection ray and the nearest object surface in the first object set, until the iteration stops, using the color of the end point of the current projection ray as the final color of the rendered pixel; A rendered image is generated based on the resulting final colors of each rendered pixel.

2. The method for processing mixed three-dimensional data according to claim 1, characterized in that: Accumulating the color of the path of the projection light by using the three-dimensional data in a neural network format to obtain the path color of the projection light includes: The color of each sampling point on the path of the projection light is calculated using three-dimensional data in a neural network format, and the color is accumulated by integration to obtain the path color of the projection light.

3. The method for processing mixed three-dimensional data according to claim 2, characterized in that: The color of each sampling point on the path of the projected light is calculated using three-dimensional data in a neural network format, and the color is accumulated by integration to obtain the path color of the projected light, including: The color of each sampling point on the path of the projected light is calculated by the three-dimensional data in the neural network format, and the Perform color accumulation in a calculation method to obtain a color when the projection light reaches the end position, which is formed by superimposing the path color of the projection light and the starting color of the projection light; in, is the starting color of the projection light, s represents the sampling point on the path of the projection light, and s∈(p0, p1], p0 is the starting position of the projection light, p1 is the end position of the projection light; T(s) is the color weight of the projection light when it reaches the sampling point s obtained by the three-dimensional data in the neural network format, and ; is the color intensity value at the sampling point s obtained by the three-dimensional data in the neural network format, is the color at the sampling point s obtained by the three-dimensional data in the neural network format, e is the base of the natural logarithm, It is the color of the projected light when it reaches the end position obtained by color accumulation.

4. The method for processing mixed three-dimensional data according to claim 1, characterized in that: The starting color of the projected light, the path color of the projected light, and the color of the three-dimensional data in triangular mesh format at the end position of the projected light are superimposed to obtain a result as the starting color of the reflected light of the projected light, including: pass The calculation method is to obtain the starting color of the reflected light of the projected light; Wherein, R represents the base color of the three-dimensional data in triangular mesh format at the end point of the projection light; is the weight coefficient of R in the jth iteration round and Q0=1; j represents the iteration round, when the projection ray is the first projection ray of the rendered pixel, j=1; The color of the three-dimensional data in triangular mesh format at the end point of the projection light; is the starting color of the reflected light of the projected light; The color of the projected light when it reaches the end point.

5. The method for processing mixed three-dimensional data according to claim 4, characterized in that: The weight coefficient is the weight coefficient obtained based on the predefined information of the three-dimensional data in triangular mesh format and satisfies ; Wherein, BSTF is the bidirectional scattering distribution function of the triangular surface at the starting position of the reflected light in the three-dimensional data of the triangular mesh format, is the cosine value of the angle between the projected light and the reflected light; B is the scattering probability density function, Q j is the weight coefficient of R in the j+1th iteration round.

6. The method for processing mixed three-dimensional data according to claim 1, characterized in that: Also includes: Based on the scene data, determining whether there is an object blocking the end point of the projected light and the light source; If yes, determine the shadow weight; The starting color of the projection light is superimposed with the path color of the projection light and the color of the three-dimensional data in the triangular mesh format at the end position of the projection light, and the result is adjusted by the shadow weight, and the adjusted color is used as the starting color of the reflected light of the projection light.

7. The method for processing mixed three-dimensional data according to claim 1, characterized in that: Also includes: After the projection light reaches the end position of the projection light, when there is a refracted light of the projection light, determining the refraction color of the refracted light based on the three-dimensional data in a triangular mesh format; The starting color of the projected light, the path color of the projected light, and the color of the three-dimensional data in the triangular mesh format at the end position of the projected light are superimposed to obtain a result as the starting color of the reflected light of the projected light, including: The starting color of the projected light is superimposed with the path color of the projected light, the refraction color of the refracted light, and the color of the three-dimensional data in triangular mesh format at the end position of the projected light, and the result is used as the starting color of the reflected light of the projected light.

8. The method for processing mixed three-dimensional data according to any one of claims 1 to 7, characterized in that: Emitting a current projection ray from the starting position of the ray toward the projection direction, and determining an intersection point between the projection ray and the nearest object surface in the first object set after the projection ray is emitted, comprising: Obtaining, by a first processing unit, a current projection ray based on the starting position of the ray and the projection direction, and determining an intersection point between the projection ray and the nearest object surface in the first object set after the projection ray is emitted; The path of the projection light is color-accumulated by the three-dimensional data in a neural network format to obtain the path color of the projection light, and the starting color of the projection light is superimposed with the path color of the projection light and the color of the three-dimensional data in a triangular mesh format at the end position of the projection light, and the result is used as the starting color of the reflected light of the projection light, including: performing, by a second processing unit, color accumulation on the path of the projection light based on the three-dimensional data in a neural network format to obtain a color of the path of the projection light; Through the first processing unit, the starting color of the projected light is superimposed with the path color of the projected light and the color of the three-dimensional data in triangular mesh format at the end position of the projected light, and the result is used as the starting color of the reflected light of the projected light.

9. The method for processing mixed three-dimensional data according to claim 8, characterized in that: The second processing unit includes a position encoding module and a color rendering module; The method includes: performing color accumulation on the path of the projection light based on the three-dimensional data in a neural network format by a second processing unit to obtain the path color of the projection light, including: Dividing the path of the projection light into a plurality of sampling points by the position encoding module of the second processing unit, and obtaining implicit features of each sampling point; The color rendering module of the second processing unit processes the implicit features of each sampling point to obtain the color value of each sampling point, and obtains the path color of the projection light through weighted accumulation.

10. The method for processing mixed three-dimensional data according to claim 9, characterized in that: The position encoding module includes: an input submodule, a sampling point generation submodule, a position processing submodule, a weight interpolation submodule, a hash table index submodule and an implicit feature calculation submodule; The position encoding module of the second processing unit divides the path of the projected light into a plurality of sampling points, and obtains implicit features of each sampling point, including: receiving input data through the input submodule; Processing the input data through the sampling point generation submodule to obtain the path of the projection light, and dividing a plurality of sampling points on the path of the projection light; For any of the sampling points, based on the coordinates of the sampling point, the position processing submodule determines an index parameter corresponding to the coordinates of the sampling point; Based on the index parameter, determining a weight parameter corresponding to the index parameter by the weight interpolation submodule, and performing a hash calculation on the index parameter by the hash table index submodule to obtain a hash index value; The implicit feature calculation submodule determines the table lookup feature corresponding to the hash index value, and uses the product of the table lookup feature and the weight parameter as the implicit feature of the sampling point.

11. The method for processing mixed three-dimensional data according to claim 10, characterized in that: The color rendering module includes: a color calculation unit array, a cache submodule and an aggregation submodule; The color rendering module of the second processing unit processes the implicit features of each sampling point to obtain the color value of each sampling point, and obtains the path color of the projection light by weighted accumulation, including: Based on pre-stored model parameter data, the color calculation unit array of the second processing unit processes the implicit features of the sampling point to obtain the color value of the sampling point and stores it in the cache submodule; Based on the color value of each sampling point stored in the cache submodule, the path color of the projection light is obtained in a weighted accumulation manner through the aggregation submodule.

12. The method for processing mixed three-dimensional data according to claim 11, characterized in that: The color calculation unit array includes a plurality of color calculation units, and each of the color calculation units is a multiplication-addition tree structure formed by connecting a plurality of multiplication-addition units.

13. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method for processing mixed three-dimensional data according to any one of claims 1 to 12 are implemented.

14. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the method for processing mixed three-dimensional data according to any one of claims 1 to 12 when executing the computer program.

15. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the method for processing mixed three-dimensional data according to any one of claims 1 to 12 are implemented.

Images