An occlusion body generation method and device, electronic equipment and medium

Abstract

Embodiments of the present application provide a kind of occlusion generation method, device, electronic equipment and medium, the method comprises: obtaining the grid model of the object to be processed, and the projection profile of the grid model is input into the neural network model pre-trained;According to the vertex offset of the face sheet in the preset starting grid by the offset information output by the neural network model, obtain the starting occlusion for the object to be processed;According to the starting occlusion, determine the target occlusion for the object to be processed.According to the embodiments of the present application, the face sheet of starting grid is used as the starting structure of occlusion, so that the generated occlusion has the ability to form various shapes freely, and is not constrained by topological relationship, so it has the potential to form very good occlusion with very few face sheet number, can cope with various topological shape inputs, and the occlusion generated by the present application can achieve higher occlusion performance with much fewer face numbers compared with prior art.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an occlusion generation method, an occlusion generation apparatus, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Real-time rendering of complex scenes relies on occlusion culling techniques to reduce rendering overhead for invisible objects. One approach to occlusion culling is to use an occluder that approximates the original object's projection but has a much smaller face count. The occluder is used only to quickly calculate occlusion culling relationships and is not displayed. Before actual rendering, the occluder is used to calculate which objects are occluded, and these objects are then removed during rendering to reduce rendering overhead without affecting the actual rendering effect.

[0003] Therefore, an occlusion volume needs to not exceed the projection of the original object; otherwise, it will remove some normally visible objects as occluded objects, leading to incorrect rendering results. Furthermore, the more of the original object's projection the occlusion volume covers, the better; a larger projection coverage area allows for the culling of more invisible objects. Finally, the fewer faces the occlusion volume should have, the better. Fewer faces result in faster occlusion culling operations and require less storage space. Summary of the Invention

[0004] In view of the above problems, embodiments of the present invention are proposed to provide an occlusion generation method and a corresponding occlusion generation apparatus, an electronic device, and a computer-readable storage medium to overcome or at least partially solve the above problems.

[0005] This invention discloses a method for generating an occlusion, the method comprising:

[0006] Obtain the mesh model of the object to be processed, and input the projected contour map of the mesh model into a pre-trained neural network model;

[0007] The vertex offset of the facets in the preset starting mesh is performed according to the offset information output by the neural network model to obtain the starting occlusion body for the object to be processed; the starting mesh is composed of a preset number of facets evenly distributed in three-dimensional space; the starting occlusion body is composed of facets after vertex offset.

[0008] Based on the initial occlusion body, a target occlusion body is determined for the object to be processed.

[0009] Optionally, the faces in the initial mesh are triangular faces; the vertices of each triangular face in the initial mesh are independent of each other, and the positions of all vertices in the same triangular face are the same; the offset information is the offset value of the vertices of the triangular faces in the initial mesh, and the step of offsetting the vertices of the faces in the preset initial mesh according to the offset information output by the neural network model to obtain the initial occlusion body for the object to be processed includes:

[0010] The vertices in the triangular facets of the starting mesh are offset according to the offset value output by the neural network model to obtain the starting occlusion body for the object to be processed.

[0011] Optionally, inputting the projected contour map of the mesh model into a pre-trained neural network model includes:

[0012] The mesh model is orthogonally projected using multiple uniformly distributed viewpoints to obtain the first orthogonal projection contour map corresponding to each viewpoint.

[0013] The first orthogonal projection contour map is input into the pre-trained neural network model.

[0014] Optionally, determining the target occluder for the object to be processed based on the initial occluder includes:

[0015] The initial occlusion volume is projected using a differentiable renderer, and the loss function value between the mesh model and the initial occlusion volume is calculated based on the projection result.

[0016] The loss function value is backpropagated to the initial occlusion body for gradient optimization to obtain the target occlusion body for the object to be processed.

[0017] Optionally, the step of projecting the initial occlusion volume using a differentiable renderer and calculating the loss function value between the mesh model and the initial occlusion volume based on the projection result includes:

[0018] A differentiable renderer is used to orthogonally project the initial occlusion body according to the same multiple viewpoints to obtain the second orthogonal projection contour map corresponding to each of the multiple viewpoints;

[0019] The loss function value between the mesh model and the initial occlusion body is calculated based on the first orthogonal projection contour map and the second orthogonal projection contour map.

[0020] Optionally, calculating the loss function value between the mesh model and the initial occlusion body based on the first orthogonal projection contour map and the second orthogonal projection contour map includes:

[0021] Based on the first orthogonal projection contour map and the second orthogonal projection contour map, a first loss function value, a second loss function value, and a third loss function value are calculated using a preset calculation method. The first loss function value represents the average number of pixels that are background pixels in the first orthogonal projection contour map but foreground pixels in the second orthogonal projection contour map. The second loss function value represents the average number of pixels that are foreground pixels in both the first and second orthogonal projection contour maps. The third loss function value represents the average area of ​​the triangular facets contained in the initial occlusion body.

[0022] Based on the first loss function value, the second loss function value, and the third loss function value, the total loss function value between the mesh model and the initial occlusion body is calculated.

[0023] Optionally, the step of backpropagating the loss function value to the initial occlusion for gradient optimization to obtain the target occlusion for the object to be processed includes:

[0024] The loss function value is backpropagated to the initial occlusion body for gradient optimization to obtain the optimized intermediate occlusion body.

[0025] The triangular facets that meet the preset culling conditions among the triangular facets contained in the intermediate occluder are culled to obtain the target occluder for the object to be processed.

[0026] This invention also discloses an occlusion generation device, the device comprising:

[0027] The acquisition and input module is used to acquire the mesh model of the object to be processed and input the projected contour map of the mesh model into a pre-trained neural network model;

[0028] The vertex offset module is used to offset the vertices of the facets in the preset starting mesh according to the offset information output by the neural network model, so as to obtain the starting occlusion body for the object to be processed; the starting mesh consists of a preset number of facets evenly distributed in three-dimensional space; the starting occlusion body consists of the facets after vertex offset;

[0029] The determination module is used to determine the target occluder for the object to be processed based on the initial occluder.

[0030] Optionally, the facets in the starting mesh are triangular facets; the vertices of each triangular facet in the starting mesh are independent of each other, and the positions of all vertices in the same triangular facet are the same; the offset information is the offset value for the vertices of the triangular facets in the starting mesh, and the vertex offset module includes:

[0031] The offset submodule is used to offset the vertices in the triangular facets of the starting mesh according to the offset value output by the neural network model, so as to obtain the starting occlusion body for the object to be processed.

[0032] Optionally, the acquisition and input module includes:

[0033] The projection submodule is used to orthogonally project the mesh model using multiple uniformly distributed viewpoints to obtain the first orthogonal projection contour map corresponding to each of the multiple viewpoints.

[0034] The input submodule is used to input the first orthogonal projection contour map into the pre-trained neural network model.

[0035] Optionally, the determining module includes:

[0036] The projection and calculation submodule is used to project the initial occlusion body using a differentiable renderer, and calculate the loss function value between the mesh model and the initial occlusion body based on the projection result;

[0037] The optimization submodule is used to backpropagate the loss function value to the initial occlusion body for gradient optimization, so as to obtain the target occlusion body for the object to be processed.

[0038] Optionally, the projection and calculation submodule includes:

[0039] The projection unit is used to orthogonally project the initial occlusion body using a differentiable renderer according to the same multiple viewpoints, so as to obtain the second orthogonal projection contour map corresponding to the multiple viewpoints respectively.

[0040] The calculation unit is used to calculate the loss function value between the mesh model and the initial occlusion body based on the first orthogonal projection contour map and the second orthogonal projection contour map.

[0041] Optionally, the computing unit includes:

[0042] The first calculation subunit is used to calculate a first loss function value, a second loss function value, and a third loss function value based on the first orthogonal projection contour map and the second orthogonal projection contour map using a preset calculation method; wherein, the first loss function value represents the average number of pixels that belong to the background in the first orthogonal projection contour map but belong to the foreground in the second orthogonal projection contour map; the second loss function value represents the average number of pixels that belong to the foreground in both the first and second orthogonal projection contour maps; and the third loss function value represents the average area of ​​the triangular facets contained in the initial occlusion body.

[0043] The second calculation subunit is used to calculate the total loss function value between the mesh model and the initial occlusion body based on the first loss function value, the second loss function value and the third loss function value.

[0044] Optionally, the optimization submodule includes:

[0045] An optimization unit is used to backpropagate the loss function value to the initial occlusion body for gradient optimization, thereby obtaining the optimized intermediate occlusion body.

[0046] The culling unit is used to cull the triangular facets that meet the preset culling conditions among the triangular facets contained in the intermediate occluder, so as to obtain the target occluder for the object to be processed.

[0047] This invention also discloses an electronic device, including: a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the steps of an occlusion generation method as described above.

[0048] This invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of an occlusion generation method as described above.

[0049] The embodiments of the present invention have the following advantages:

[0050] In this embodiment of the invention, the projected contour map of the mesh model of the object to be processed is used as the input of the neural network model. The neural network model outputs corresponding offset information, and the vertex offsets of the faces in the initial mesh can be performed according to the output offset information to obtain the initial occlusion body composed of the facets after vertex offset. By adopting the above method, using the facets of the initial mesh as the initial structure of the occlusion body allows the generated occlusion body to freely form various shapes without being constrained by topological relationships. Therefore, it has the potential to form a very good occlusion body with a very small number of facets and can cope with inputs of various topological shapes. Using the neural network model to obtain the initial occlusion body can significantly reduce the time required to generate the occlusion body and increase the practicality of the algorithm. Compared with the prior art, the occlusion body generated by this invention can achieve higher occlusion performance with a much smaller number of facets. Attached Figure Description

[0051] Figure 1 This is a schematic diagram illustrating the generation effect of an occlusion.

[0052] Figure 2 This is a flowchart of the steps of a method for generating an occlusion according to an embodiment of the present invention;

[0053] Figure 3 This is a flowchart of another method for generating an occlusion provided in an embodiment of the present invention;

[0054] Figure 4 This is a schematic diagram of the projected outline of a mesh model according to an embodiment of the present invention;

[0055] Figure 5 This is a schematic diagram of an initial grid according to an embodiment of the present invention;

[0056] Figure 6 This is a flowchart of a method for generating an occlusion according to an embodiment of the present invention;

[0057] Figure 7 This is a structural block diagram of an occlusion generation device provided in an embodiment of the present invention. Detailed Implementation

[0058] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art are within the scope of protection of the present invention.

[0059] Among the currently used algorithms for generating occlusion volumes, the OccluderSimplification using Planar Section algorithm proposed by Ari Silvennoinen performs well. This algorithm first voxels the input mesh model, then uses planes of multiple directions and positions to cut voxels. Each plane cuts a polygon on the model voxel. The planar polygon that best increases the projected area is selected to form the desired occlusion volume. After selecting the polygons, they are triangulated, and then a greedy algorithm is used to remove unnecessary triangular faces to obtain the occlusion volume. (See reference...) Figure 1 The image shows a schematic diagram of the generation effect of an occlusion object. The left side represents the input mesh model, and the right side represents the output occlusion object.

[0060] Occluder simplification using Planar Section constructs the occlusion volume using facets on voxels. However, since not every part of a facet contributes to the final projection, some redundancy exists. Furthermore, this technique first selects the smallest triangular facet from each polygon facet and then removes those that contribute the least to the projection. However, the smallest triangular facet among the polygon facets is not necessarily the one that contributes the least to the projection, thus this facet reduction process can lead to suboptimal solutions. Considering these two points, the number of faces in the generated occlusion volume can still be compressed, or the projected area can be higher with the same number of faces.

[0061] One of the core concepts of this invention is that the projected contour map of the mesh model of the object to be processed is used as the input of a neural network model. The neural network model outputs corresponding offset information, and the vertex offsets of the faces in the initial mesh can be performed according to the output offset information to obtain an initial occlusion body composed of faces with offset vertices. By adopting the above method, using the faces of the initial mesh as the initial structure of the occlusion body allows the generated occlusion body to freely form various shapes without being constrained by topological relationships. Therefore, it has the potential to form a very good occlusion body with a very small number of faces, and can cope with inputs of various topological shapes. Using a neural network model to obtain the initial occlusion body can significantly reduce the time required to generate the occlusion body and increase the practicality of the algorithm. Compared with the prior art, the occlusion body generated by this invention can achieve higher occlusion performance with a much smaller number of faces.

[0062] Reference Figure 2 The diagram illustrates a flowchart of a method for generating an occlusion according to an embodiment of the present invention, which may specifically include the following steps:

[0063] Step 201: Obtain the mesh model of the object to be processed, and input the projected contour map of the mesh model into the pre-trained neural network model.

[0064] In game development, culling is a crucial technique. Culling helps minimize unnecessary rendering overhead, significantly improving frame rates. Occlusion culling is the most challenging and varied to implement. Occlusion culling refers to omitting rendering an object when it is occluded by other objects and outside the camera's view. Occlusion culling can use a model with a small facet that approximates the original object's projection as an occluder, which can be used to quickly calculate occlusion culling relationships.

[0065] In this embodiment of the invention, a mesh model of the object to be processed can be obtained first. A mesh model is a model that approximates a three-dimensional object using a series of polygons (usually triangles) of similar size and shape. For example, a house in a three-dimensional game can be converted into a corresponding polygon mesh model.

[0066] The mesh model of the object to be processed is projected to obtain a projected contour map of the mesh model. This projected contour map can then be used as input to a pre-trained neural network model for computation. The pre-trained neural network model has the ability to generate the initial occlusion volume.

[0067] Step 202: Based on the offset information output by the neural network model, the vertex offset of the face in the preset starting mesh is performed to obtain the starting occlusion body for the object to be processed.

[0068] The initial mesh consists of a predetermined number of facets evenly distributed in three-dimensional space; the initial occlusion volume consists of facets after vertex offset.

[0069] In this embodiment of the invention, after the projected contour map of the mesh model is input into a pre-trained neural network model, the neural network model outputs offset information for offsetting the vertices of the faces in the preset initial mesh. The output value of the convolutional neural network is used to offset the vertices of the initial mesh to obtain an initial occlusion.

[0070] In one example, the facets in the starting mesh of this invention can be scattered, unconnected triangular facets. Common mesh structures are topologies formed by connected triangular facets. Connection means that the triangular facets share some vertices, while the triangular facets in the starting mesh of this invention do not share vertices.

[0071] Step 203: Determine the target occluder for the object to be processed based on the initial occluder.

[0072] The initial occlusion volume obtained from the convolutional neural network only provides preliminary occlusion performance, which is insufficient to meet the requirements for occlusion recall and precision. Therefore, this initial occlusion volume can be further optimized and processed to obtain a target occlusion volume that meets the requirements for occlusion recall and precision. After obtaining the target occlusion volume, operations such as occlusion culling relationship calculation can be performed using it. In one example, differentiable rendering can be used to optimize the initial occlusion volume, and after processing through pre-defined post-processing steps, the target occlusion volume can be obtained.

[0073] In summary, in this embodiment of the invention, the projected contour map of the mesh model of the object to be processed is used as the input of the neural network model. The neural network model outputs corresponding offset information, and the vertex offsets of the faces in the initial mesh can be performed according to the output offset information to obtain the initial occlusion body composed of the facets after vertex offset. By adopting the above method, using the facets of the initial mesh as the initial structure of the occlusion body allows the generated occlusion body to freely form various shapes without being constrained by topological relationships. Therefore, it has the potential to form a very good occlusion body with a very small number of facets and can cope with inputs of various topological shapes. Using a neural network model to obtain the initial occlusion body can significantly reduce the time required to generate the occlusion body and increase the practicality of the algorithm. Compared with the prior art, the occlusion body generated by this invention can achieve higher occlusion performance with a much smaller number of facets.

[0074] Reference Figure 3 The diagram illustrates a flowchart of another method for generating an occlusion according to an embodiment of the present invention, which may specifically include the following steps:

[0075] Step 301: Obtain the mesh model of the object to be processed, and input the projected contour map of the mesh model into the pre-trained neural network model;

[0076] In this embodiment of the invention, a mesh model of the object to be processed can be obtained, and the projected contour map obtained by projecting the mesh model is input into a pre-trained neural network model for calculation. In the projected contour map, each pixel has a value of 0 or 1, where 1 represents the foreground and 0 represents the background.

[0077] In an alternative embodiment, for step 301, the following steps may be performed:

[0078] Sub-step S11 involves orthogonally projecting the mesh model using multiple uniformly distributed viewpoints to obtain first orthogonal projection contour maps corresponding to the multiple viewpoints.

[0079] Sub-step S12: Input the first orthogonal projection contour map into the pre-trained neural network model.

[0080] In this embodiment of the invention, the mesh model can be orthogonally projected using multiple uniformly distributed viewpoints. Orthogonal projections of the mesh model can be obtained by viewing it from multiple angles towards the origin of the network model in three-dimensional space, resulting in orthogonal projection contour maps. For ease of distinction, these are referred to here as the first orthogonal projection contour map. One viewpoint corresponds to one first orthogonal projection contour map, and all first orthogonal projection contour maps need to be input into the convolutional neural network of the neural network model for computation.

[0081] Reference Figure 4The figure shown is a schematic diagram of the projected outline of a mesh model according to an embodiment of the present invention. The left side of the figure is the mesh model of the object to be processed, and the right side of the figure is the projected outline obtained by orthogonally projecting the mesh model according to different viewpoints.

[0082] In practice, the size of the projected outline can be set to 224x224, but the size within a general range has no significant impact on the effect. Additionally, the number of viewing angles can be set to 45; theoretically, the more viewing angles, the better the effect, but the computational load will increase.

[0083] The initial mesh consists of a predetermined number of faces evenly distributed in three-dimensional space. After the projected contour map of the mesh model is input into the neural network model, offset information is output to offset the vertices of the faces in the predetermined initial mesh. The vertices of the initial mesh can be offset according to the offset information to obtain the initial occlusion body composed of the faces after the vertex offset.

[0084] In this embodiment of the invention, the facets in the starting mesh are triangular facets; the vertices of each triangular facet in the starting mesh are independent of each other, and the positions of each vertex in the same triangular facet are the same. The independence of vertices means that the triangular facets are not connected and are separated from each other; the offset information output by the neural network model can be the offset value of the vertex of the triangular facet of the starting mesh, and at this time step 302 can be executed.

[0085] Step 302: Offset the vertices in the triangular facets of the starting mesh according to the offset value output by the neural network model to obtain the starting occlusion body for the object to be processed.

[0086] The following explains how to train a neural network model to generate the initial occlusion.

[0087] Training convolutional neural networks requires collecting a certain number of grid models. For example, the ShapeNetSem dataset can be used, as it has a sufficient number of grid models and covers a wide range of types. Other datasets with sufficient quantity and variety can also be used for neural network training.

[0088] Once the dataset is determined, the mesh model in the dataset can be orthogonally projected using multiple uniformly distributed viewpoints to obtain a projected profile map. Pixels in the projected profile map can be divided into foreground pixels or background pixels.

[0089] Using the contour projection map as input to the convolutional neural network model, the output value of the trained convolutional neural network can be used to offset the vertices of a starting grid to obtain the starting occlusion.

[0090] Therefore, for a trained neural network model, the vertex offset of the starting mesh can be directly calculated based on the offset values ​​of the vertices of the triangle facets of the starting mesh, thus generating the starting occlusion volume.

[0091] In practice, the initial mesh consists of 1000 unconnected faces evenly distributed in three-dimensional space. Each face consists of three vertices. Unconnectedness means that the vertices of different faces are independent and not shared with other faces; the total number of vertices equals the number of faces multiplied by three. A similar effect can be achieved as long as the face density is sufficient. The initial positions of the three vertices within the same face are equal, ensuring that the initial area of ​​each face is zero. (Refer to...) Figure 5 The diagram shown is a schematic representation of a starting mesh according to an embodiment of the present invention. A point in the diagram represents a facet, which is also a vertex within that facet.

[0092] The advantage of setting the starting position of the vertices of each face in the initial grid to be equal is that when the vertex offset of a face output by the convolutional neural network is 0, the area of ​​that face in the resulting occlusion is 0. Therefore, it is easy to control which faces of the initial grid are used to construct the output occlusion, so as to achieve the purpose of dynamically adjusting the number of faces according to the input.

[0093] Step 303: Determine the target occluder for the object to be processed based on the initial occluder.

[0094] For step 303, the following steps can be performed:

[0095] Sub-step S21: Project the initial occlusion volume using a differentiable renderer, and calculate the loss function value between the mesh model and the initial occlusion volume based on the projection result.

[0096] Sub-step S22: Backpropagate the loss function value to the initial occlusion body for gradient optimization to obtain the target occlusion body for the object to be processed.

[0097] After obtaining the initial occlusion volume, the outline of the initial occlusion volume can be projected using differentiable rendering. Based on the projection result, the loss between the mesh model and the initial occlusion volume is calculated using a pre-designed loss function calculation method. Then, the gradient is backpropagated to optimize the shape of the initial occlusion volume.

[0098] For sub-step S21, the following steps can be performed:

[0099] A differentiable renderer is used to orthogonally project the initial occlusion body according to the same multiple viewpoints to obtain second orthogonal projection contour maps corresponding to the multiple viewpoints respectively; the loss function value between the mesh model and the initial occlusion body is calculated based on the first orthogonal projection contour map and the second orthogonal projection contour map.

[0100] In this embodiment of the invention, a differentiable renderer can be used to orthogonally project the initial occlusion body from the same viewpoint as the mesh model projection to obtain an orthogonal projection contour map, which is referred to here as the second orthogonal projection contour map for easy distinction. The loss function value between the mesh model and the initial occlusion body is determined based on the difference between the first and second orthogonal projection contour maps.

[0101] Specifically, the step of calculating the loss function value between the mesh model and the initial occlusion body based on the first orthogonal projection contour map and the second orthogonal projection contour map can be performed using the following steps:

[0102] Based on the first orthogonal projection contour map and the second orthogonal projection contour map, the first loss function value, the second loss function value and the third loss function value are calculated using a preset calculation method; based on the first loss function value, the second loss function value and the third loss function value, the total loss function value between the mesh model and the initial occlusion body is calculated.

[0103] Wherein, the first loss function value represents the average number of pixels that belong to the background in the first orthogonal projection contour map but belong to the foreground in the second orthogonal projection contour map; the second loss function value represents the average number of pixels that belong to the foreground in the first orthogonal projection contour map and also belong to the foreground in the second orthogonal projection contour map; and the third loss function value represents the average area of ​​the triangular facets contained in the initial occlusion volume.

[0104] Assume the first orthogonal projection contour is Si∈R W×H×C The second orthogonal projection contour is Sn∈R W×H×C , where R represents a real number. W×H×C The matrix represents a real number matrix of size W*H*C, which is C W*H projected contour maps, where WH represents the length and width of the projected contour map, and C represents the number of viewing angles.

[0105] The loss function value consists of three parts. The first part is the contour overflow loss: loss_o = mean((1-Si)·Sn), where mean represents averaging over image pixels, and · represents multiplying pixel by pixel. Therefore, loss_o actually calculates the average number of background pixels in the first orthogonal projection contour map that are foreground pixels in the second orthogonal projection contour map. A lower loss_o indicates that the output initial occluder exceeds the contour of the original mesh model less. The second part is the contour recall loss: loss_i = mean(Si·Sn), where loss_i calculates the average number of foreground pixels in the first orthogonal projection contour map that are foreground pixels in the second orthogonal projection contour map. A higher loss_i indicates that the output initial occluder occludes more of the original mesh model's contour. The third part is the facet penalty loss_f, which is the average area of ​​the triangular faces. A lower loss_f indicates that fewer triangular faces were used to form the occluder. The total loss is loss = -loss_i + α*loss_o + β*loss_f. β is used to adjust the number of facets; the larger β is, the fewer facets are used to form the occlusion. α is used to adjust recall and precision; the larger α is, the more important precision is, and the recall of the occlusion will decrease while precision will increase. Therefore, α and β can be set according to the actual desired effect.

[0106] For sub-step S22, the following steps can be performed:

[0107] The loss function value is backpropagated to the initial occlusion body for gradient optimization to obtain the optimized intermediate occlusion body; the triangular faces that meet the preset culling conditions in the triangular faces contained in the intermediate occlusion body are culled to obtain the target occlusion body for the object to be processed.

[0108] In one alternative embodiment, a convolutional neural network can be trained to generate intermediate occlusions. During training, after calculating the loss function value, the gradient of the model weights relative to the loss can be calculated, thereby optimizing the model weights using a gradient descent algorithm to allow the model to learn. This learning process is repeated multiple times until convergence. Once the convolutional neural network is well-learned, it can be used to generate intermediate occlusions, meaning the learning process only needs to be performed once, and subsequent generation of intermediate occlusions does not require repeated training. It should be noted that the convolutional neural network used to generate intermediate occlusions here is different from the convolutional neural network used to generate the initial occlusion described above.

[0109] After obtaining the optimized intermediate occlusion body, redundant faces can be removed to obtain the desired target occlusion body. In one example, triangular faces with an area less than a preset area threshold can be removed from the intermediate occlusion body. The threshold for face removal can be customized according to different requirements for face redundancy.

[0110] To enable those skilled in the art to better understand steps 301 to 303 of the embodiments of the present invention, an example is provided below:

[0111] Reference Figure 6 The diagram shows a flowchart of an occlusion generation method according to an embodiment of the present invention. The specific process is as follows:

[0112] 1. Orthogonally project the mesh model of the object to be processed into a projected outline;

[0113] 2. Input the projected contour map into the convolutional neural network. The output value of the convolutional neural network is used to offset the vertices of the starting mesh to obtain an initial occlusion volume.

[0114] 3. The initial occlusion volume obtained by the convolutional neural network only has a preliminary occlusion volume performance, which does not meet the required occlusion recall and precision. Therefore, this initial occlusion volume is orthogonally projected using differentiable rendering. The projection viewpoint is consistent with the mesh model. Then, the projection contour map of the mesh model and the initial occlusion volume is compared, and the loss function value is calculated according to the set loss function calculation method.

[0115] 4. Then, another trained convolutional neural network is used to determine the gradient of the vertex position of the initial occluder relative to the loss function. The vertex position is optimized through hundreds of iterations until the loss converges, and the optimized intermediate occluder is obtained.

[0116] 5. After optimization, first remove triangular faces with an area less than 0.01. For the remaining triangular faces, calculate the outline projection area of ​​the occlusion body formed after removal one by one. If the outline projection area does not decrease after deleting a face, the face can be removed; if the outline projection area decreases by more than 0 after deleting a face, the face is retained. This step can remove redundant triangular faces. After completion, the occlusion body mesh to be output is obtained.

[0117] In summary, in this embodiment of the invention, the projected contour map of the mesh model of the object to be processed is used as the input of the neural network model. The neural network model outputs corresponding offset information, and the vertex offset of the facets in the initial mesh can be performed according to the output offset information to obtain the initial occlusion body composed of facets after vertex offset. By adopting the above method, using uniformly distributed unconnected facets as the initial structure of the occlusion body allows the generated occlusion body to freely form various shapes without being constrained by topological relationships. Therefore, it has the potential to form a very good occlusion body with a very small number of facets and can cope with inputs of various topological shapes. By using differentiable rendering and optimizing with a well-designed loss function, the output occlusion body can be optimized to the optimal occlusion performance and redundant facets can be removed. The number of facets of the occlusion body can be adaptively adjusted according to different models. Using a convolutional neural network to obtain the initial occlusion body can significantly reduce the number of iterations required for differentiable rendering optimization, significantly reduce the time required to generate the occlusion body, and increase the practicality of the algorithm. Compared with the prior art, the occlusion body generated by this invention can achieve higher occlusion performance with a much smaller number of facets.

[0118] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0119] Reference Figure 7 The diagram shows a structural block diagram of an occlusion generation device provided by an embodiment of the present invention, which may specifically include the following modules:

[0120] The acquisition and input module 701 is used to acquire the mesh model of the object to be processed and input the projected contour map of the mesh model into a pre-trained neural network model;

[0121] Vertex offset module 702 is used to offset the vertices of the facets in the preset starting mesh according to the offset information output by the neural network model, so as to obtain the starting occlusion body for the object to be processed; the starting mesh is composed of a preset number of facets evenly distributed in three-dimensional space; the starting occlusion body is composed of facets after vertex offset;

[0122] The determination module 703 is used to determine the target occlusion for the object to be processed based on the initial occlusion.

[0123] In this embodiment of the invention, the faces in the initial mesh are triangular faces; the vertices of each triangular face in the initial mesh are independent of each other, and the positions of all vertices in the same triangular face are the same; the offset information is the offset value for the vertices of the triangular faces in the initial mesh, and the vertex offset module includes:

[0124] The offset submodule is used to offset the vertices in the triangular facets of the starting mesh according to the offset value output by the neural network model, so as to obtain the starting occlusion body for the object to be processed.

[0125] In this embodiment of the invention, the acquisition and input module includes:

[0126] The projection submodule is used to orthogonally project the mesh model using multiple uniformly distributed viewpoints to obtain the first orthogonal projection contour map corresponding to each of the multiple viewpoints.

[0127] The input submodule is used to input the first orthogonal projection contour map into the pre-trained neural network model.

[0128] In this embodiment of the invention, the determining module includes:

[0129] The projection and calculation submodule is used to project the initial occlusion body using a differentiable renderer, and calculate the loss function value between the mesh model and the initial occlusion body based on the projection result;

[0130] The optimization submodule is used to backpropagate the loss function value to the initial occlusion body for gradient optimization, so as to obtain the target occlusion body for the object to be processed.

[0131] In this embodiment of the invention, the projection and calculation submodule includes:

[0132] The projection unit is used to orthogonally project the initial occlusion body using a differentiable renderer according to the same multiple viewpoints, so as to obtain the second orthogonal projection contour map corresponding to the multiple viewpoints respectively.

[0133] The calculation unit is used to calculate the loss function value between the mesh model and the initial occlusion body based on the first orthogonal projection contour map and the second orthogonal projection contour map.

[0134] In this embodiment of the invention, the computing unit includes:

[0135] The first calculation subunit is used to calculate a first loss function value, a second loss function value, and a third loss function value based on the first orthogonal projection contour map and the second orthogonal projection contour map using a preset calculation method; wherein, the first loss function value represents the average number of pixels that belong to the background in the first orthogonal projection contour map but belong to the foreground in the second orthogonal projection contour map; the second loss function value represents the average number of pixels that belong to the foreground in both the first and second orthogonal projection contour maps; and the third loss function value represents the average area of ​​the triangular facets contained in the initial occlusion body.

[0136] The second calculation subunit is used to calculate the total loss function value between the mesh model and the initial occlusion body based on the first loss function value, the second loss function value and the third loss function value.

[0137] In this embodiment of the invention, the optimization submodule includes:

[0138] An optimization unit is used to backpropagate the loss function value to the initial occlusion body for gradient optimization, thereby obtaining the optimized intermediate occlusion body.

[0139] The culling unit is used to cull the triangular facets that meet the preset culling conditions among the triangular facets contained in the intermediate occluder, so as to obtain the target occluder for the object to be processed.

[0140] In summary, in this embodiment of the invention, the projected contour map of the mesh model of the object to be processed is used as the input of the neural network model. The neural network model outputs corresponding offset information, and the vertex offsets of the faces in the initial mesh can be performed according to the output offset information to obtain the initial occlusion body composed of the facets after vertex offset. By adopting the above method, using the facets of the initial mesh as the initial structure of the occlusion body allows the generated occlusion body to freely form various shapes without being constrained by topological relationships. Therefore, it has the potential to form a very good occlusion body with a very small number of facets and can cope with inputs of various topological shapes. Using a neural network model to obtain the initial occlusion body can significantly reduce the time required to generate the occlusion body and increase the practicality of the algorithm. Compared with the prior art, the occlusion body generated by this invention can achieve higher occlusion performance with a much smaller number of facets.

[0141] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0142] This invention also provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, it implements the various processes of the above-described occlusion generation method embodiment and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0143] This invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described occlusion generation method embodiment and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0144] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0145] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0146] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0147] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0148] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0149] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0150] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0151] The foregoing has provided a detailed description of the occlusion generation method, occlusion generation device, electronic device, and computer-readable storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for generating an occlusion, characterized in that, The method includes: Obtain the mesh model of the object to be processed, and perform orthogonal projection on the mesh model using multiple uniformly distributed viewpoints to obtain the first orthogonal projection contour map corresponding to each of the multiple viewpoints; The first orthogonal projection contour map is input into a pre-trained neural network model; The vertex offset of the facets in the preset starting mesh is performed according to the offset information output by the neural network model to obtain the starting occlusion body for the object to be processed; the starting mesh is composed of a preset number of facets evenly distributed in three-dimensional space; the starting occlusion body is composed of facets after vertex offset. The initial occlusion volume is projected using a differentiable renderer, and the loss function value between the mesh model and the initial occlusion volume is calculated based on the projection result. The loss function value is backpropagated to the initial occlusion body for gradient optimization to obtain the target occlusion body for the object to be processed.

2. The method according to claim 1, characterized in that, The faces in the initial mesh are triangular faces; the vertices of each triangular face in the initial mesh are independent of each other, and the positions of all vertices in the same triangular face are the same; the offset information is the offset value of the vertices of the triangular faces in the initial mesh; the step of offsetting the vertices of the faces in the preset initial mesh according to the offset information output by the neural network model to obtain the initial occlusion body for the object to be processed includes: The vertices in the triangular facets of the starting mesh are offset according to the offset value output by the neural network model to obtain the starting occlusion body for the object to be processed.

3. The method according to claim 1, characterized in that, The step of projecting the initial occlusion volume using a differentiable renderer and calculating the loss function value between the mesh model and the initial occlusion volume based on the projection result includes: A differentiable renderer is used to orthogonally project the initial occlusion body according to the same multiple viewpoints to obtain the second orthogonal projection contour map corresponding to each of the multiple viewpoints; The loss function value between the mesh model and the initial occlusion body is calculated based on the first orthogonal projection contour map and the second orthogonal projection contour map.

4. The method according to claim 3, characterized in that, The calculation of the loss function value between the mesh model and the initial occlusion volume based on the first orthogonal projection contour map and the second orthogonal projection contour map includes: Based on the first orthogonal projection contour map and the second orthogonal projection contour map, a first loss function value, a second loss function value, and a third loss function value are calculated using a preset calculation method. The first loss function value represents the average number of pixels that are background pixels in the first orthogonal projection contour map but foreground pixels in the second orthogonal projection contour map. The second loss function value represents the average number of pixels that are foreground pixels in both the first and second orthogonal projection contour maps. The third loss function value represents the average area of ​​the triangular facets contained in the initial occlusion body. Based on the first loss function value, the second loss function value, and the third loss function value, the total loss function value between the mesh model and the initial occlusion body is calculated.

5. The method according to claim 4, characterized in that, The step of backpropagating the loss function value to the initial occlusion body for gradient optimization to obtain the target occlusion body for the object to be processed includes: The loss function value is backpropagated to the initial occlusion body for gradient optimization to obtain the optimized intermediate occlusion body. The triangular facets that meet the preset culling conditions among the triangular facets contained in the intermediate occluder are culled to obtain the target occluder for the object to be processed.

6. A device for generating an obstruction, characterized in that, The device includes: The acquisition and input module is used to acquire the mesh model of the object to be processed, and to perform orthogonal projection on the mesh model using multiple uniformly distributed viewpoints to obtain the first orthogonal projection contour map corresponding to each of the multiple viewpoints; and to input the first orthogonal projection contour map into a pre-trained neural network model. The vertex offset module is used to offset the vertices of the facets in the preset starting mesh according to the offset information output by the neural network model, so as to obtain the starting occlusion body for the object to be processed; the starting mesh consists of a preset number of facets evenly distributed in three-dimensional space; the starting occlusion body consists of the facets after vertex offset; The projection and calculation submodule is used to project the initial occlusion body using a differentiable renderer, and calculate the loss function value between the mesh model and the initial occlusion body based on the projection result; The optimization submodule is used to backpropagate the loss function value to the initial occlusion body for gradient optimization, so as to obtain the target occlusion body for the object to be processed.

7. An electronic device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the steps of a method for generating an occlusion as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the steps of the occlusion generation method as described in any one of claims 1 to 5.

Images