Method, device and equipment for generating three-dimensional model and storage medium

Abstract

This application discloses a method, apparatus, device, and storage medium for generating a 3D model, relating to the field of computer graphics technology. The method includes: sampling P mapping points from each pixel of a texture image; mapping the P pixels into a 3D space to obtain P mapping points in the 3D space; determining the number of meshes assigned to each of the P mapping points, wherein at least two of the P mapping points have different numbers of meshes assigned; and establishing a connection relationship between the P mapping points based on the number of meshes assigned to each of the P mapping points to generate a 3D model. This method reduces the complexity of the 3D model while maintaining its detailed representation.

Description

Technical Field

[0001] This application relates to the field of computer graphics technology, and in particular to a method, apparatus, device, and storage medium for generating three-dimensional models. Background Technology

[0002] A 3D model is a polygonal representation of an object. Both real objects in nature and fictional objects created by artists can be represented using 3D models. 3D models have a wide range of applications; for example, rendered 3D models are commonly seen in games and videos.

[0003] In 3D models, areas with denser meshes produce better rendering results, but also have higher complexity. Further research is needed on how to construct 3D models reasonably. Summary of the Invention

[0004] This application provides a method, apparatus, device, and storage medium for generating three-dimensional models. The technical solutions provided by this application are as follows:

[0005] According to one aspect of the embodiments of this application, a method for generating a three-dimensional model is provided, the method comprising:

[0006] P pixels are sampled from each pixel in the texture image. The P pixels are not uniformly distributed in the texture image, and P is an integer greater than 1.

[0007] The P pixels are mapped to a three-dimensional space to obtain P mapped points in the three-dimensional space;

[0008] The number of grids assigned to each of the P mapping points is determined, wherein at least two of the P mapping points are assigned different numbers of grids.

[0009] Based on the number of grids allocated to the P mapping points, the connection relationship of the P mapping points is established to generate the three-dimensional model. In the generated three-dimensional model, the number of associated grids of the i-th mapping point among the P mapping points is equal to the number of grids allocated to the i-th mapping point. The associated grids of the i-th mapping point refer to the grids with the i-th mapping point as vertices, where i is a positive integer less than or equal to P.

[0010] According to one aspect of the embodiments of this application, a three-dimensional model generation apparatus is provided, the apparatus comprising:

[0011] The sampling module is used to sample P pixels from each pixel in the texture image, wherein the P pixels are non-uniformly distributed in the texture image, and P is an integer greater than 1.

[0012] The mapping module is used to map the P pixels to a three-dimensional space to obtain P mapped points in the three-dimensional space.

[0013] An allocation module is used to determine the number of grids allocated to each of the P mapping points, wherein at least two of the P mapping points are allocated different numbers of grids.

[0014] The generation module is used to establish the connection relationship of the P mapping points according to the number of grids allocated to each of the P mapping points, so as to generate the three-dimensional model. In the generated three-dimensional model, the number of associated grids of the i-th mapping point among the P mapping points is equal to the number of grids allocated to the i-th mapping point. The associated grids of the i-th mapping point refer to the grids with the i-th mapping point as vertices, where i is a positive integer less than or equal to P.

[0015] According to one aspect of the present application, a computer device is provided, the computer device including a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the above-described method for generating a three-dimensional model.

[0016] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, the computer program being loaded and executed by a processor to implement the above-described method for generating a three-dimensional model.

[0017] According to one aspect of the embodiments of this application, a computer program product is provided, the computer program product including a computer program stored in a computer-readable storage medium, and a processor reading from the computer-readable storage medium and executing the computer program to implement the above-described method for generating a three-dimensional model.

[0018] The technical solutions provided in this application have at least the following beneficial effects:

[0019] 3D models are generated by sampling pixels from texture images. On one hand, the sampled pixels are not uniformly distributed in the texture image; they are more concentrated in certain areas. The 3D model generated based on these pixels can achieve more detailed representation in those areas. On the other hand, in 3D space, connections are established between mapping points derived from the pixels. During the 3D model generation process, these connections are established according to the number of meshes allocated to each mapping point. This ensures that in the generated 3D model, for each mapping point, the number of meshes with that mapping point as a vertex equals the number of meshes allocated to that mapping point. Since at least two mapping points may have different numbers of meshes allocated to them, the number of associated meshes (meshes with that mapping point as a vertex) for each mapping point in the final generated 3D model is not entirely consistent. This results in a denser mesh on the surface of the 3D model at some vertices (i.e., a more detailed mesh structure in the corresponding area) and a relatively smaller mesh at other vertices (i.e., a simpler mesh structure in the corresponding area). This reduces the complexity of the 3D model while maintaining its detailed representation. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of a method for generating a three-dimensional model provided in one embodiment of this application;

[0021] Figure 2 This is a schematic diagram of a computer system provided in one embodiment of this application;

[0022] Figure 3 This is a flowchart of a method for generating a three-dimensional model according to an embodiment of this application;

[0023] Figure 4 This is a schematic diagram illustrating the connection relationship of a set of mapping points provided in one embodiment of this application;

[0024] Figure 5 This is a flowchart of a method for generating a three-dimensional model provided in another embodiment of this application;

[0025] Figure 6 This is a schematic diagram of different three-dimensional models generated based on the same mapping points according to one embodiment of this application;

[0026] Figure 7 This is a flowchart of a method for generating a three-dimensional model provided in another embodiment of this application;

[0027] Figure 8 This is a flowchart of a method for adjusting a 3D model based on a 3D convolutional neural network according to an embodiment of this application;

[0028] Figure 9 This is a schematic diagram of voxel representation provided in one embodiment of this application;

[0029] Figure 10 This is a schematic diagram of a three-dimensional convolutional neural network model provided in one embodiment of this application;

[0030] Figure 11 This is a flowchart of a method for adjusting a 3D model based on a 3D convolutional neural network, provided in another embodiment of this application;

[0031] Figure 12 This is a schematic diagram of a rendering method for a three-dimensional model provided in one embodiment of this application;

[0032] Figure 13 This is a block diagram of a three-dimensional model generation apparatus provided in one embodiment of this application;

[0033] Figure 14 This is a structural block diagram of a computer device provided in one embodiment of this application. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0035] Before introducing the technical solutions of this application, some terms involved in this application will be explained. The following related explanations are optional solutions and can be arbitrarily combined with the technical solutions of the embodiments of this application, all of which fall within the protection scope of the embodiments of this application. The embodiments of this application include at least some of the following contents.

[0036] 3D model: A polygonal representation of an object. A 3D model can represent real-world entities or virtual objects.

[0037] Mesh: Also known as mesh face or patch, it is the basic unit that makes up a 3D model. A 3D model includes at least two meshes, and the vertices of a 3D model are the vertices of the meshes of the 3D model.

[0038] Texture image: A two-dimensional image used to describe the texture of a 3D model. Each pixel in the texture image can be mapped to different locations on the 3D model, thus providing information such as the model's color, texture, bumps, and lighting. During the rendering process, mapping the texture image onto the surface of the 3D model allows it to present a more realistic effect.

[0039] Contour: The region in a texture image where pixel values ​​change abruptly; it can also be called the edge in a texture image.

[0040] Position vector: The position vector is used to indicate the position of a point in three-dimensional space. For a certain point, the position vector of the point is the vector pointing from the origin of the coordinate system in three-dimensional space to the point. In the embodiments of this application, the position of the origin of the coordinate system in three-dimensional space is not limited. For example, the origin of the coordinate system can be set outside the three-dimensional model in three-dimensional space, or the origin of the coordinate system can be set at the center point of the three-dimensional model in three-dimensional space.

[0041] 3DCNN (3D Convolutional Neural Networks) model: A neural network model that can perform three-dimensional convolution operations.

[0042] As mentioned above, a 3D model is composed of a mesh, and the number and density of the mesh determine the computational cost of rendering the 3D model and its display effect. Generally, the denser the mesh of a 3D model, the greater the computational cost of rendering, and the better the display effect. Currently, 3D models are mostly modeled manually, and technicians can construct the mesh of the 3D model according to their needs. However, manual modeling of 3D models is inefficient and not suitable for many scenarios. Based on this, embodiments of this application provide a method for generating 3D models. This method automatically generates 3D models based on texture images and can balance the display effect and computational cost of the 3D model.

[0043] Please refer to Figure 1 The diagram illustrates a method for generating a three-dimensional model according to an embodiment of this application, the method comprising at least one of the following steps:

[0044] 1. Pixel sampling: Sample P pixels from the texture image to generate the 3D model, such as... Figure 1 As shown, these P pixels are not evenly distributed in the texture image.

[0045] 2. Mesh Allocation and 3D Model Generation: P pixels are mapped to P mapping points in 3D space, and the number of meshes is adaptively allocated to these P mapping points (i.e., how many meshes are allocated to a mapping point as a vertex). This results in different mesh densities in different regions of the 3D model (i.e., at least two mapping points are assigned different numbers of meshes). Based on the allocated meshes, the connection relationships between the P mapping points are established to generate the 3D model. This 3D model is more detailed in some areas and simpler in others, which can optimize the computational cost of operations (such as rendering) performed on the 3D model while maintaining its detail.

[0046] 3. Mesh Structure Adjustment: In this step, the mesh structure of the 3D model will be optimized and adjusted according to the set strategy, thereby further optimizing the performance of the 3D model (such as reducing complexity and improving expressiveness).

[0047] Ultimately, using texture image rendering to render the 3D model obtained through the above steps can achieve better rendering results and computational efficiency.

[0048] For steps not described in detail in the above method, please refer to the embodiments below.

[0049] Please refer to Figure 2 The diagram illustrates a computer system provided in one embodiment of this application. The computer system includes at least one of the following: a terminal device 10 and a server 20.

[0050] Terminal device 10 may be an electronic device such as a mobile phone, tablet computer, multimedia playback device, PC (Personal Computer), wearable device, in-vehicle terminal device, VR (Virtual Reality) device, AR (Augmented Reality) device, MR (Mixed Reality) device, etc. In some embodiments, terminal device 10 may run a client of a first type of application. The first type of application is an application used to build a 3D model, such as a modeling application, game engine application, etc. In some embodiments, a texture image is loaded into the first type of application to build the 3D model. In some embodiments, terminal device 10 may run a client of a second type of application, which is an application that requires the use of a 3D model, such as a game application, video application, etc. In some embodiments, the texture image is exported from the second type of application. In some embodiments, if the client of the second type of application fails to render the 3D model using the texture image, the texture image is exported to the client of the first type of application to regenerate the 3D model.

[0051] Server 20 can be used to provide background services for clients of the aforementioned first-type or second-type applications. For example, server 20 may be a background server for the aforementioned first-type or second-type applications. In some embodiments, the client of the first-type application (or second-type application) uploads a texture image (such as a texture image corresponding to a failed 3D model) to server 20. Server 20 generates a 3D model based on the texture image and sends the generated 3D model back to terminal device 10. In some embodiments, terminal device 10 uses the texture image to render the 3D model sent back by server 20 and displays the rendered 3D model.

[0052] Terminal device 10 can communicate with server 20 via a network, such as a wireless or wired network.

[0053] Furthermore, in this embodiment, the implementation form of the target application is not limited. For example, it can be an application that needs to be downloaded and installed, a mini-program that does not require installation, a web application, etc.

[0054] Please refer to Figure 3 The diagram illustrates a flowchart of a method for generating a three-dimensional model according to an embodiment of this application. The execution entity for each step of this method is a computer device, which can be the terminal device 10 in the above embodiment or a server 20. The method includes at least one of the following steps 310 to 340.

[0055] Step 310: Sample P pixels from each pixel in the texture image.

[0056] The P pixels are non-uniformly distributed in the texture image, where P is an integer greater than 1.

[0057] In some embodiments, non-uniform distribution of P pixels in a texture image means that after dividing the texture image into M image regions of the same size, there are at least two image regions where the number of sampled pixels (belonging to P pixels) is different, where M is an integer greater than 1.

[0058] In some embodiments, the P pixels include: at least two first-type pixels uniformly distributed in the texture image and at least two second-type pixels distributed on the contours contained in the texture image. Step 310 includes at least one of the following sub-steps 312 to 318 ( Figure 3 (Not shown in the image):

[0059] Sub-step 312 divides the texture image into M image regions of the same size.

[0060] For example, assuming a rectangular image of size W1 × H1, the texture image can be divided into m × n = M image regions of the same size, each region having a size of Δx × Δy. Here, W1 and H1 are both greater than 0, and their units can be pixels or other length units. m and n are both integers greater than 1.

[0061] Sub-step 314: Sample the same number of pixels from each image region to obtain at least two first-class pixels.

[0062] For example, from each of the above m×n image regions of the same size, one pixel (such as the center pixel) is sampled, and the coordinates of the sampled pixel are: Where a = 0, 1, ..., m-1; b = 0, 1, ..., n-1.

[0063] Sub-step 316: Extract at least two key pixels from the texture image. Key pixels are pixels that belong to the contour contained in the texture image.

[0064] To extract key pixels from a texture image, it is necessary to identify key pixels from the texture image.

[0065] In some embodiments, key pixels are identified from a texture image through the following steps:

[0066] 1. For any pixel in the texture image, the feature value of the pixel is obtained based on the difference between the first value and the second value. The feature value of the pixel is used to reflect the degree of abrupt change in the pixel value at the pixel in the texture image. The first value is the product of the first derivative value and the second derivative value, and the second value is the square of the third derivative value. The first derivative value is the second derivative value of the texture image in the first direction at the pixel. The second derivative value is the second derivative value of the texture image in the second direction at the pixel. The third derivative value is the mixed second derivative value of the texture image in the first and second directions at the pixel. The first direction and the second direction are perpendicular.

[0067] The first direction and the second direction are two mutually perpendicular directions. For example, in the case of a rectangular texture image, the first direction is along the width of the texture image, and the second direction is along the height of the texture image.

[0068] The pixel value of a pixel is used to characterize the color, brightness, etc. of the pixel, but this application does not limit this.

[0069] In some embodiments, the second derivative of the texture image is represented in the form of a second-order matrix:

[0070] That is, a second-order matrix

[0071] For a pixel (x, y) in a texture image, L xx (x,y,σ) is the value of the first derivative, L yy (x,y,σ) is the value of the second derivative, L xy (x,y,σ) is the third derivative value, where σ is the standard deviation of the Gaussian function used for Gaussian filtering of the texture image before calculating the second derivative of the texture image.

[0072] The eigenvalues ​​of the aforementioned pixels are the same as the determinant of the aforementioned second-order matrix:

[0073] That is, the feature value of pixel (x,y) is det(H) = L xx (x,y,σ)·L yy (x,y,σ)-(L xy (x,y,σ)) 2

[0074] 2. If the feature value of a pixel is greater than or equal to the first threshold, the pixel is identified as a key pixel.

[0075] The first threshold shall be set by the technician as needed, and this application does not limit it.

[0076] In some embodiments, the feature value of a pixel is positively correlated with the degree of abrupt change in pixel value at that pixel in the texture image. In other words, the above method can identify pixels with a significant abrupt change in pixel value as key pixels.

[0077] In the above embodiment, key pixels are detected based on the second derivative of the texture image. Given a fixed second-order matrix, the determinant of the second-order matrix is ​​used as the feature value of the pixel. Therefore, the feature value of a single pixel can be obtained through simple multiplication and subtraction operations (see the feature value calculation method above), thereby enabling the rapid and effective detection of key pixels in the texture image.

[0078] In some embodiments, key pixels are identified from a texture image through the following steps:

[0079] 1. For any pixel in the texture image, determine the magnitude of the gradient at that pixel.

[0080] The gradient at a certain pixel in a texture image refers to the gradient of the pixel value of each pixel in the texture image at that pixel.

[0081] Given an image I, its gradient at pixel (x,y) can be expressed as: The magnitude of the gradient can be expressed as In some embodiments, the partial derivative at (x,y) and It is obtained based on the pixel values ​​of the neighboring pixels, for example. Where I(x+1,y) is the pixel value of pixel (x+1,y), I(x-1,y) is the pixel value of pixel (x-1,y), I(x,y+1) is the pixel value of pixel (x,y+1), and I(x,y-1) is the pixel value of pixel (x,y-1).

[0082] 2. If the gradient magnitude at a pixel is greater than or equal to the sixth threshold, the pixel is identified as a key pixel.

[0083] The sixth threshold shall be set by the technician as needed, and this application does not limit it.

[0084] The magnitude of a pixel gradient can also reflect the degree of abrupt changes in pixel values ​​in a texture image, thus it can be used to detect key pixels in a texture image.

[0085] In some embodiments, key pixels are identified from a texture image through the following steps:

[0086] 1. Apply Gaussian filtering to the texture image to obtain the filtered image.

[0087] That is I smooth (x,y)=I(x,y)*G(x,y,σ), where, I smooth (x,y) is the filtered image I smooth The pixel value at (x,y) is the pixel value of the texture image at (x,y), and G(x,y,σ) is the Gaussian function used for Gaussian filtering.

[0088] 2. For the first extreme point in the filtered image, if the magnitude of the gradient at the first extreme point in the filtered image is greater than or equal to the second threshold, the first extreme point is determined as a pixel belonging to the contour contained in the filtered image.

[0089] The first extreme point is the extreme point in the filtered image (i.e., the pixel point where the pixel value reaches the extreme value). In some embodiments, the first extreme point is the extreme point along the gradient direction in the first image. That is, in the filtered image, the pixel value of the first extreme point reaches the extreme value in the gradient direction at the first extreme point.

[0090] In some embodiments, the filtered image is convolved with the Sobel operator in the first direction and the Sobel operator in the second direction to obtain the magnitude of the gradient of each pixel in the filtered image in the first direction and the magnitude of the gradient in the second direction. The square root of the sum of the squares of the magnitudes of the gradients in the first direction and the second direction is then taken to obtain the magnitude of the gradient of the pixel. The first direction and the second direction are two mutually perpendicular directions.

[0091] In some embodiments, the magnitude of the gradient at the first extreme point (x, y) in the filtered image is calculated using the following formula: G x =I smooth *S x G y =I smooth *S y , in, S is the magnitude of the gradient at (x,y). x and S y It is the Sobel operator.

[0092] 3. If the gradient magnitude at the first extreme point of the filtered image is greater than or equal to the third threshold and less than the second threshold, and if there are pixels in the filtered image adjacent to the first extreme point whose gradient magnitude is greater than or equal to the second threshold, then the first extreme point is determined to be a pixel belonging to the contour contained in the filtered image.

[0093] The third threshold is less than the second threshold. For example, if the second threshold is 100 and the third threshold is 80, then in the filtered image, extreme points with a gradient magnitude greater than or equal to 100 will be regarded as pixels on the contour. For extreme points with a gradient magnitude between 80 and 100, they will only be regarded as pixels on the contour if there are pixels with a gradient magnitude greater than or equal to 100 in each of the adjacent pixels of the extreme point.

[0094] 4. Based on the pixels that belong to the contours contained in the filtered image, determine the key pixels in the texture image.

[0095] Since the filtered image is obtained by filtering the texture image, and the size of the filtered image is the same as that of the texture image, each pixel in the filtered image can be found in the texture image.

[0096] In some embodiments, the position of a pixel belonging to the contour contained in the filtered image in the filtered image is the same as the position of the key pixel corresponding to that pixel in the texture image. In other words, the position of a pixel belonging to the contour contained in the filtered image in the filtered image can directly represent the position of the key pixel in the texture image.

[0097] For example, if a pixel belonging to the contour contained in the filtered image has coordinates (x, y) in the filtered image, then the key pixel corresponding to that pixel also has coordinates (x, y) in the texture image.

[0098] In the above embodiment, after Gaussian filtering of the texture image, the pixels on the contour are determined from the extreme points in the filtered image, suppressing the non-extreme points in the filtered image, thereby improving the efficiency and accuracy of contour detection. In addition, two thresholds, a second threshold and a third threshold, are set to distinguish the pixels on the contour. The connection relationship between pixels whose gradient magnitude is between the high and low thresholds and pixels whose gradient magnitude is greater than or equal to the high threshold is considered, so that the contour of the filtered image can be detected more comprehensively, and key pixels can be identified from the texture image efficiently.

[0099] It should be noted that the key pixels identified by the various methods provided in the above embodiments can all be used as sampling sources for the second type of pixels, and this application does not limit them.

[0100] Sub-step 318: Sample at least two second-class pixels from at least two key pixels.

[0101] In some embodiments, during the traversal of at least two associated pixels, a second-type pixel is sampled every time a number of key pixels are traversed for a sampling interval. For example, if there are 10,000 key pixels and the sampling interval is 5, then a second-type pixel is sampled every 5 key pixels (e.g., the first key pixel in every 5 key pixels is sampled as a second-type pixel), resulting in a final sample of 2,000 second-type pixels.

[0102] In some embodiments, the traversal order of at least two key pixels is determined based on the respective positions of at least two key pixels in the texture image, and the at least two key pixels are traversed in that traversal order.

[0103] For example, in a rectangular texture image, at least two key pixels are traversed in a top-to-bottom, left-to-right order. For example, the traversal order of the position vectors of the at least two key pixels is obtained by sorting the moduli of their respective position vectors. The position vectors of the key pixels point from the origin of the coordinate system on the plane containing the texture image to the key pixel itself, indicating the position of the key pixel in the texture image. The origin of the coordinate system can be inside or outside the texture image; this application does not limit this.

[0104] The number of sampling intervals determines the sampling density of the second type of pixels, which is set by technicians as needed, and this application does not impose any restrictions on it.

[0105] In the above embodiment, the P pixels include at least two first-type pixels and at least two second-type pixels. The at least two first-type pixels are evenly distributed in the texture image, so that in three-dimensional space, the mapping points derived from the first-type pixels can restore the basic structure of the three-dimensional model to be rendered by the texture image. The at least two second-type pixels are distributed on the contours contained in the texture image, so that in three-dimensional space, the mapping points derived from the second-type pixels can reflect the texture details of the texture image.

[0106] Step 320: Map P pixels into three-dimensional space to obtain P mapped points in three-dimensional space.

[0107] In some embodiments, each pixel in the texture image has a predefined mapping position in three-dimensional space, set by a technician, and different pixels have different predefined mapping positions in three-dimensional space.

[0108] In some embodiments, P pixels are mapped to three-dimensional space according to their respective set mapping positions to obtain P mapping points.

[0109] In some embodiments, after a target application fails to render a constructed 3D model using a texture image, the texture image and mapping data are exported from the target application. The mapping data is used to indicate the set mapping positions of each pixel in the texture image. In some embodiments, the failure of the target application to render the constructed 3D model means that after rendering the constructed 3D model, there are unrendered or incorrectly rendered meshes in the constructed 3D model (e.g., pixels that should be mapped to the mesh according to the set mapping positions are not mapped to the mesh). Excessive complexity of the constructed 3D model may cause this problem. In the case of failure to render the constructed 3D model, the technical solutions provided in the embodiments of this application can be used to reconstruct the 3D model.

[0110] In some embodiments, the texture image has a defined mapping rule (such as a mapping function) that maps pixels from the texture image to a defined mapping position in three-dimensional space. That is, the mapping rule converts the two-dimensional coordinates of pixels in the texture image into the three-dimensional coordinates of the defined mapping position in three-dimensional space. Following this mapping rule, P pixels can be mapped to three-dimensional space, resulting in P mapped points in three-dimensional space.

[0111] Step 330: Determine the number of grids assigned to each of the P mapping points, where at least two mapping points are assigned different numbers of grids.

[0112] The number of meshes assigned to a mapping point indicates the number of associated meshes for that mapping point after the 3D model is generated. Associated meshes for a mapping point are meshes with that mapping point as a vertex. For example, if a mapping point is assigned 3 meshes, then after the 3D model is generated, the number of associated meshes for that mapping point will also be 3.

[0113] In this application, at least two of the P mapping points are assigned different numbers of meshes. Compared to directly assigning the same number of meshes to each of the P mapping points, this allows for adaptive setting of mesh density for different mapping points in the 3D model. This not only improves the detail of the 3D model but also reduces its complexity.

[0114] Step 340: Based on the number of grids assigned to each of the P mapping points, establish the connection relationship between the P mapping points to generate a 3D model.

[0115] In the generated 3D model, the number of associated meshes for the i-th mapping point out of P mapping points is equal to the number of meshes assigned to the i-th mapping point. The associated meshes for the i-th mapping point refer to the meshes with the i-th mapping point as vertices, where i is a positive integer less than or equal to P.

[0116] In some embodiments, step 340 includes: establishing a connection relationship between P mapping points, such that the number of grids with the i-th mapping point as the mapping point is equal to the number of grids allocated to the i-th mapping point, so as to generate a three-dimensional model.

[0117] Those skilled in the art should understand that, during the generation of a 3D model, given a certain number of mapping points, the number of associated meshes for each mapping point can be controlled during the process of establishing the connection relationships between mapping points. For example, please refer to... Figure 4 For a set of mapping points in three-dimensional space, when the first connection relationship is adopted, mapping point 41 has 5 associated mesh surfaces, while when the second connection relationship is adopted, mapping point 41 has 4 associated mesh surfaces.

[0118] In some embodiments, triangulation is used to establish the connection relationship of P mapping points, that is, the mesh in the generated 3D model is a triangular mesh. For P mapping points in 3D space, triangulation refers to the process of connecting P mapping points to form at least two non-intersecting triangular meshes. In other words, the mesh of the 3D model generated by triangulation is a triangular mesh.

[0119] In some embodiments, in a three-dimensional model, the circumcircle of any triangular mesh does not contain any other mapping points besides the mapping points of that triangular mesh.

[0120] In some embodiments, the mesh of the 3D model is a triangular mesh; please refer to [reference needed]. Figure 5 Step 340 includes: for each of the P mapping points, connecting the mapping point to a first number of other mapping points to generate a 3D model, where the first number is the number of meshes assigned to the mapping point.

[0121] In some embodiments, the first number of other mapping points mentioned above are the first number of mapping points that are arranged first after sorting each other mapping point in ascending order of its distance from the above mapping point.

[0122] For example, please refer to Figure 6This illustration shows a schematic diagram of different 3D models generated based on the same mapping points according to an embodiment of this application. In this example, in case 1, the number of meshes assigned to mapping points A, B, C, D, E, and F is the same, all being 4. Therefore, each of the mapping points A, B, C, D, E, and F is connected to 4 other mapping points, thereby generating the 3D model as shown in case 1. In case 2, the number of meshes assigned to mapping points A and B is 3, the number of meshes assigned to mapping points C and D is 5, and the number of meshes assigned to mapping points E and F is 4. Therefore, mapping points A and B are each connected to 3 other mapping points, mapping points C and D are each connected to 5 other mapping points, and mapping points E and F are each connected to 4 other mapping points, thereby generating the 3D model as shown in case 2. As can be seen, in case 2, there are mapping points C and D with relatively more associated meshes, as well as mapping points A and B with relatively fewer associated meshes. Compared with case 1, without changing the total number of meshes (8) of the 3D model and the total number of meshes (24) allocated to each mapping point, a more detailed mesh structure is achieved at some vertices (C and D).

[0123] In the above embodiment, since triangulation is used to generate a three-dimensional model for P mapping points, the number of meshes allocated to a mapping point can directly indicate the number of other mapping points connected to that mapping point, thereby enabling the efficient generation of a three-dimensional model that matches the number of meshes allocated to each mapping point.

[0124] In some embodiments, a texture image is applied to render the 3D model to obtain the rendered 3D model.

[0125] The technical solution provided in this application sample pixels from a texture image to generate a 3D model. On one hand, the sampled pixels are non-uniformly distributed in the texture image, meaning they are more concentrated in certain areas. The 3D model generated based on these pixels can achieve more detailed representation in the corresponding areas. On the other hand, in 3D space, connections are established between mapping points derived from the pixels. During the 3D model generation process, these connections are established according to the number of meshes allocated to each mapping point. This ensures that in the generated 3D model, for each mapping point, the number of meshes with that mapping point as a vertex is equal to the number of meshes allocated to that mapping point. Since at least two mapping points are allocated different numbers of meshes, the number of associated meshes (i.e., meshes with that mapping point as a vertex) for each mapping point in the final generated 3D model is not entirely consistent. This results in a denser mesh on the surface of the 3D model at some vertices (i.e., a more detailed mesh structure in the corresponding area) and a relatively smaller mesh at other vertices (i.e., a simpler mesh structure in the corresponding area). This reduces the complexity of the 3D model while maintaining its detailed representation.

[0126] Please refer to Figure 7 The diagram illustrates a flowchart of a method for generating a three-dimensional model according to another embodiment of this application. The execution entity for each step of this method is a computer device, which can be the terminal device 10 in the above embodiment or a server 20. The method includes at least one of the following steps 510 to 550.

[0127] Step 510: Sample P pixels from each pixel in the texture image.

[0128] The P pixels are not uniformly distributed in the texture image.

[0129] Step 520: Map P pixels into three-dimensional space to obtain P mapped points in three-dimensional space.

[0130] Step 530: Based on the feature information of each of the P mapping points, determine the weight of the grid to be assigned to each of the P mapping points.

[0131] The feature information of the i-th mapping point among P mapping points is used to reflect the features of the pixel corresponding to the i-th mapping point in the texture image.

[0132] In some embodiments, the number of grids allocated to the i-th mapping point is positively correlated with the weight of the grids allocated to the i-th mapping point. The feature information of the i-th mapping point includes at least one of the following: the feature value of the pixel corresponding to the i-th mapping point in the texture image, the magnitude of the gradient at the pixel corresponding to the i-th mapping point in the texture image, and the size of the segmentation region to which the pixel corresponding to the i-th mapping point in the texture image belongs.

[0133] The calculation methods for the feature values ​​of the aforementioned pixels and the magnitude of the gradient can be found in the embodiments described above, and will not be repeated here.

[0134] The segmented regions are obtained by segmenting the texture image based on the pixel values ​​of each pixel. The texture image includes U segmented regions, where U is an integer greater than 1. Among the U segmented regions, at least two segmented regions have different pixel densities (out of P pixels), and the pixels belonging to the same segmented region have similar characteristics. This application does not limit the method for segmenting regions from the texture image.

[0135] In some embodiments, the texture image is first divided into different statistical units, and the distribution of pixel values ​​(or gradient magnitudes of pixels) in different statistical units is statistically analyzed. Then, each statistical unit is classified according to the statistical results, thereby dividing the texture image into at least two segmentation regions.

[0136] In some embodiments, based on deep learning technology, a neural network model is used to segment regions from the texture image. For example, the neural network model is a UNet model. The texture image is input into the UNet model, and the UNet model outputs a segmentation result, which indicates the category of each pixel in the texture image. Based on the segmentation result, the texture image can be segmented into at least two regions.

[0137] The weights assigned to the grid at the i-th mapping point are positively correlated with the feature values ​​of the corresponding pixel. The weights assigned to the grid at the i-th mapping point are positively correlated with the magnitude of the gradient at the corresponding pixel. The weights assigned to the grid at the i-th mapping point are negatively correlated with the size of the segmented region to which the corresponding pixel belongs.

[0138] The higher the feature value (the higher the gradient magnitude) at a given pixel, the greater the probability that the pixel belongs to a contour in the texture image. Furthermore, after segmenting a texture image, the larger segmented regions are typically the smoother, less detailed non-contour areas within the texture image. Therefore, by designing the aforementioned correlation, mapping points derived from pixels on more detailed contours in the texture image receive greater weight when allocating mesh elements, thereby enriching the details of the corresponding regions of the contours contained in the texture image on the surface of the 3D model.

[0139] For example, the weight of the grid assigned to the i-th mapping point is determined by the following formula:

[0140]

[0141] Among them, w i This assigns grid weights to the i-th mapping point; α and β are adjustable parameters. S is the magnitude of the gradient at the pixel corresponding to the i-th mapping point. i This is the feature value of the pixel corresponding to the i-th mapping point. For a detailed explanation of the feature value, please refer to sub-step 316 in the above embodiment; it will not be repeated here. It should be noted that the above formula is derived from two of the three types of feature information. This is merely an example of applying feature information to calculate weights. Those skilled in the art can modify this formula according to the weights and the relationship between the three types of feature information provided in the above embodiment; this application does not limit this modification. For example, the above formula can be changed to w i =β·S i That is, the weights are obtained from only one type of feature information. For example, the above formula can be changed to Where c is an adjustable parameter, and u is used to represent the size (such as length, width, area, etc.) of the segmentation region to which the pixel corresponding to the i-th mapping point belongs. That is, the weight can be obtained from all three types of feature information.

[0142] Step 540: Based on the set total number of grids and the weights of the grids assigned to the P mapping points, determine the number of grids assigned to each of the P mapping points.

[0143] The total number of grid cells is set by technical personnel as needed, and this application does not impose any restrictions on this.

[0144] In some embodiments, the weight of the grid assigned to the i-th mapping point is divided by the sum of the weights of the grids assigned to the P mapping points respectively to obtain the first intermediate weight. The total number of grids is set and the first intermediate weight is multiplied to obtain the number of grids assigned to the i-th mapping point.

[0145] Right now Where N is the set total number of grid cells, N i This is the number of grid cells allocated to the i-th mapping point. That is, the first intermediate weight.

[0146] Step 550: Based on the number of grids assigned to each of the P mapping points, establish the connection relationship between the P mapping points to generate a 3D model.

[0147] In the generated 3D model, the number of associated meshes for the i-th mapping point out of P mapping points is equal to the number of meshes assigned to the i-th mapping point. The associated meshes for the i-th mapping point are those that contain the i-th mapping point, where i is a positive integer less than or equal to P.

[0148] In the technical solution provided in this application embodiment, the weight of the mesh assigned to the mapping point is obtained based on the feature information of the mapping point, which reflects the features of the corresponding pixel in the texture image. Therefore, the weight actually reflects the importance of the mapping point, that is, the degree of influence of the mapping point on the final rendering effect when applying the texture image to render the 3D model. Therefore, the above solution realizes the weighted allocation of the mesh quantity of each of the P mapping points based on the importance of the mapping points, thereby ensuring that more meshes are allocated to the high-importance and high-complexity regions of the 3D model, and thus enabling the 3D model to obtain a more detailed mesh structure in the above regions.

[0149] After generating a 3D model using the technical solutions provided in the above embodiments, the structure of the 3D model can be adjusted using the following technical solutions to improve its display effect or reduce its complexity.

[0150] In some embodiments, the method for generating the above-described three-dimensional model further includes at least one of the following steps:

[0151] 1. Determine the optimized side length based on the segmentation parameters of the pixel corresponding to the first vertex of the 3D model in the texture image and the segmentation parameters of the pixel corresponding to the second vertex of the 3D model in the texture image.

[0152] In the texture image, each defined region has set parameters that are related to the density of pixels sampled from the segmented region. These parameters can be used to represent the mesh edge length desired by the technician in the region corresponding to the segmented region on the surface of the 3D model. In some embodiments, the set parameters are negatively correlated with the density of pixels sampled from the segmented region.

[0153] In some embodiments, the optimized side length is determined based on the weights of the mesh assigned to the first vertex, the setting parameters of the segmentation region to which the pixel corresponding to the first vertex of the 3D model in the texture image belongs, the weights of the mesh assigned to the second vertex, and the setting parameters of the segmentation region to which the pixel corresponding to the second vertex of the 3D model in the texture image belongs.

[0154] In some embodiments, the absolute value of the difference between the weights assigned to the first vertex and the weights assigned to the second vertex is divided by the sum of the weights assigned to the first vertex and the weights assigned to the second vertex to obtain a third value. The product of the first adjustable parameter and the third value is incremented by 1 to obtain a fourth value. The average of the set parameters of the segmentation regions corresponding to the pixels of the first vertex and the second vertex of the 3D model in the texture image is multiplied by the fourth value to obtain the optimized side length.

[0155] That is, the optimized side length is determined by the following formula:

[0156]

[0157] Among them, l 12 γ is the optimized side length, l1 is the setting parameter of the segmentation region to which the pixel corresponding to the first vertex of the 3D model belongs in the texture image, l2 is the setting parameter of the segmentation region to which the pixel corresponding to the second vertex of the 3D model belongs in the texture image, γ is an adjustable parameter (the first adjustable parameter), w1 is the weight of the mesh assigned to the first vertex, and w2 is the weight of the mesh assigned to the second vertex.

[0158] 2. Adjust the position of at least one of the first vertex and the second vertex in three-dimensional space to adjust the length of the edge connecting the first vertex and the second vertex in the three-dimensional model to the optimized edge length.

[0159] In the above embodiments, the process of adjusting the edge length of the edge connecting the first vertex and the second vertex is only described by way of example. In the embodiments of this application, the edge length of each edge in the 3D model can be adjusted using the above method, thereby realizing regional edge length adjustment on the mesh surface of the 3D model. After adjustment according to the above method, the mesh edge length of the region corresponding to the segmentation region of high-density sampled pixels on the surface of the 3D model is shorter, that is, the mesh is denser, and the mesh edge length of the region corresponding to the segmentation region of low-density sampled pixels on the surface of the 3D model is longer, that is, the mesh is sparser. Ultimately, the mesh distribution of the 3D model is more reasonable and can match the smooth regions and contours on the texture image.

[0160] In some embodiments, the above-described method for generating a three-dimensional model further includes: adjusting the position of each vertex of the three-dimensional model in three-dimensional space based on the curvature of each vertex of the three-dimensional model, wherein the curvature of the vertex is used to reflect the degree of bending of the three-dimensional model at the vertex.

[0161] The curvature of each vertex of a 3D model can intuitively represent the surface shape and curvature of the 3D model. Therefore, adjusting the position of each vertex of a 3D model in 3D space based on curvature can effectively improve the expressiveness of the surface features of the 3D model.

[0162] In some embodiments, the above method includes at least one of the following steps:

[0163] 1. For the third vertex of the 3D model, determine the curvature of the third vertex based on the angles of each corner of the 3D model with the third vertex as the vertex.

[0164] The third vertex can be any vertex in the 3D model. The other vertices in the 3D model can also be adjusted in the same way as the third vertex. This application does not limit this.

[0165] In some embodiments, the curvature of a vertex includes Gaussian curvature and mean curvature.

[0166] In some embodiments, the average curvature of the third vertex is obtained by subtracting the angles of each angle with the third vertex as the vertex from 2π.

[0167] For the third vertex, the Gaussian curvature is calculated using the following formula:

[0168] K i =2π-Σθ it

[0169] Among them, K i It is the Gaussian curvature of the third vertex, ∑θ it In a 3D model, θ is the sum of the angles of all angles with the third vertex as the vertex. it That is, the angle with i as its vertex.

[0170] In some embodiments, for an edge including a third vertex, the cotangent function values ​​of the two angles on the edge with the third vertex as the vertex are added together to obtain the cotangent function sum of the edge. The cotangent function sum of the edge is multiplied by the difference between the position vector of the other vertex on the edge and the position vector of the third vertex to obtain the curvature vector of the edge. The curvature vectors of all edges including the third vertex are summed to obtain the sum vector. The sum vector is divided by twice the total area of ​​the associated mesh of the third vertex to obtain the average curvature.

[0171] For the third vertex, the mean curvature is calculated using the following formula:

[0172]

[0173] Among them, G i It is the Gaussian curvature of the third vertex, v i A is the position vector of the third vertex. i v is the total area of ​​the associated mesh at the third vertex. s α is the position vector of the adjacent vertex s of the third vertex. is and β is These are two angles on the edge formed by the third vertex and the adjacent vertex s, with the third vertex as the vertex of these two angles.

[0174] 2. Adjust the position of the third vertex in three-dimensional space according to the curvature of the third vertex.

[0175] With the vertex curvature known, technicians can change the position of the vertices as needed, thereby altering the surface shape of the 3D model.

[0176] In some embodiments, the position vector of the third vertex and the average curvature are weighted and summed to obtain the adjusted position vector of the third vertex.

[0177] In some embodiments, the product of the Gaussian curvature and the parameter vector is added to the position vector of the third vertex to obtain the adjusted position vector of the third vertex. The parameter vector is set by the technician as needed.

[0178] For example, for the third vertex, its position vector can be changed according to the following formula: v′ i =v i +λ·H i v′ i =v i +v0·K i , where v′ i λ is the adjusted position vector of the third vertex, λ is the adjustable weight parameter, and v0 is the adjustable parameter vector.

[0179] It should be noted that in the above embodiments, the process of adjusting the position of the third vertex is only used as an example. In the embodiments of this application, the position of each vertex in the three-dimensional model can be adjusted using the above method.

[0180] In the above method, the curvature of the third vertex can be accurately and efficiently determined based on the angles of each corner with the third vertex as the vertex. Furthermore, adjusting the position of the third vertex based on its curvature allows the surface features of the 3D model at the third vertex to be presented more accurately and vividly, thereby improving the display effect of the 3D model.

[0181] In some embodiments, the above-described method for generating a three-dimensional model further includes: merging the first mesh and the second mesh when the angle between the normal vector of the first mesh and the normal vector of the second mesh in the three-dimensional model is less than a fourth threshold, wherein the first mesh and the second mesh have overlapping edges.

[0182] The fourth threshold is set by the technician as needed, and this application does not limit it.

[0183] In some embodiments, the first mesh and the second mesh are merged if the normal vectors of the first mesh and the second mesh satisfy the following conditions:

[0184] cosθ=n1·n2≥cosθ threshold

[0185] Where θ is the angle between the normal vectors of the first and second grids, n1 is the normal vector of the first grid, n2 is the normal vector of the second grid, and θ threshold That is, the fourth threshold.

[0186] It should be noted that in the above embodiments, the merging process of the first and second meshes is only used as an example. In the embodiments of this application, all meshes with overlapping edges in the three-dimensional model can be determined to merge using the above method.

[0187] The above method merges two meshes only when the angle between the normal vectors of two adjacent meshes is small, that is, when the angle between the two meshes is large (i.e., the surface of the 3D model is relatively smooth). This can reduce the mesh complexity of the 3D model while avoiding the loss of important details (such as edges and corners).

[0188] In some embodiments, the method for generating the above-described three-dimensional model further includes at least one of the following steps:

[0189] 1. Based on the 3D model, the position vectors of the R vertices of the Q meshes are formed, and the position vectors of the fitted vertices of the R vertices are also formed, thus obtaining the first judgment value.

[0190] Each of the Q grids has an edge that overlaps with at least one other grid in the Q grids. Fitted vertices are used to determine the grid after merging the Q grids. The position vectors of the vertices are used to indicate the position of the vertices in 3D space. Q is an integer greater than 1 and R is an integer greater than 3.

[0191] The fitted vertex is located on top of the merged Q mesh, and thus can be used to determine the merged Q mesh. In some embodiments, the fitted vertex of a given vertex is the point on the merged Q mesh that is closest to that vertex. In some embodiments, in three-dimensional space, R vertices are adjusted to the same plane to obtain the fitted vertices of each of the R vertices (i.e., the fitted vertices of each of the R vertices are obtained by adjusting the R vertices).

[0192] In some embodiments, for each of the R vertices, the difference between the position vector of the vertex and the position vector of the fitted vertex is calculated to obtain the difference vector of the vertex. The magnitudes of the difference vectors of the R vertices are added together to obtain the first determination value.

[0193] Right now Among them, v r It is the position vector of one of the R vertices. E is the fitted vertex position vector of that vertex. R It is the first judgment value.

[0194] 2. If the first judgment value is less than or equal to the fifth threshold, merge the Q grids.

[0195] In some embodiments, at least 3 of the R vertices are adjusted to be vertices of the mesh after merging the Q meshes, that is, among the fitted vertices of each of the R vertices, there are at least 3 fitted vertices that are vertices of the mesh after merging the Q meshes.

[0196] The fifth threshold shall be set by the technician as needed, and this application does not limit it.

[0197] It should be noted that in the above embodiment, the Q meshes can be any part of the mesh in the 3D model, and there can be multiple sets of Q meshes in the 3D model. The above method can be used to determine whether to merge these multiple sets of Q meshes.

[0198] The first judgment value in the above method is obtained from the original position vectors of R vertices and the position vectors of the fitted vertices of each of the R vertices. It can be used to reflect the difference between the Q meshes and the mesh after merging the Q meshes. When the first judgment value is large, it means that merging the Q meshes into one mesh will result in the loss of more geometric information on the surface of the 3D model. Therefore, the above method only merges the Q meshes when the first judgment value is small, which can reduce the mesh complexity of the 3D model with less change to the geometry of the 3D model.

[0199] It should be noted that after adjusting the 3D model using at least one of the above methods, the position of the vertices in the 3D model may change or the number of vertices may decrease. However, for each vertex in the 3D model, it is known which vertex out of the P vertices it was adjusted from, that is, it is still known which pixel in the texture image the vertex in the 3D model was mapped from (the correspondence between the vertices of the 3D model and the pixels in the texture image).

[0200] Please refer to Figure 8 This application also provides a method for adjusting a 3D model based on 3DCNN. The execution subject of each step of this method is a computer device, which can be the terminal device 10 in the above embodiment or a server 20. The method includes at least one of the following steps 610 to 620.

[0201] It should be noted that all the above-described steps for adjusting the 3D model are performed before the 3D model adjustment method based on 3DCNN provided in this application embodiment. This ensures that even if the position or number of vertices in the 3D model changes compared to the original P mapping points (position or number) mapped from P pixels, 3DCNN can still match the 3D model with the texture image. In some embodiments, after generating the 3D model, the 3D model can also be directly adjusted using the 3DCNN model; this application does not limit this approach.

[0202] Step 610: Input the voxel representation of the 3D model into the 3DCNN model, and output the adjustment parameters of the 3D model from the 3DCNN model.

[0203] Voxel representation is used to indicate the position of each vertex of a 3D model in 3D space, and to represent the pixel value of each pixel of a texture image in 3D space. In some embodiments, voxel representation is also used to indicate the position of each edge of a 3D model in 3D space. The specific form of voxel representation in this application will be described below.

[0204] A two-dimensional image can be represented as an H×W two-dimensional matrix. The elements in the matrix are the pixels of the two-dimensional image. The position of a pixel in the matrix indicates its position on the two-dimensional image. The value of a pixel in the matrix is ​​its pixel value. W and H are both integers greater than 1, representing the width and height of the two-dimensional image.

[0205] Similarly, please refer to Figure 9 In this application, the voxel representation can be a three-dimensional tensor of size H×W×D, where H, W, and D are all integers greater than 1, representing the height, width, and depth of the three-dimensional space. Elements in the tensor can be called voxels, and the position of a voxel in the tensor indicates its position in three-dimensional space. In some embodiments, the voxel value of a voxel in the voxel representation indicates two pieces of information: first, whether a vertex (or edge) of the three-dimensional model exists at that voxel location; for example, 0 indicates no vertex (or edge), and 1 indicates the presence of a vertex (or edge); or, for example, 0 indicates no vertex, 1 indicates the presence of a vertex, and 2 indicates the presence of an edge. Second, it indicates the pixel value at that voxel location after the texture image is represented in three-dimensional space. For example, in... Figure 9 In the voxel value of voxel 61, 1 indicates that there is a vertex or edge of the 3D model at that voxel (0 if there is no vertex or edge), and 80 indicates that the pixel value of the pixel mapped to that voxel by the texture image is 80 (0 if no pixel is mapped to that voxel). For how to map (represent) the pixels in the texture image to 3D space, please refer to step 230 in the above embodiment; this application will not repeat it further.

[0206] In some embodiments, please refer to Figure 10 The 3DCNN model 800 includes a convolutional layer 80, a pooling layer 85, and a fully connected layer 90. Step 610 includes at least one of the following sub-steps 612-616 (…). Figure 8 (Not shown in the image):

[0207] Sub-step 612: Input the voxel representation into the convolutional layer, and the convolutional layer performs a convolution operation on the voxel representation to output the convolutional data.

[0208] In some embodiments, the 3DCNN model further includes an input layer, where voxel representations are input to the convolutional layer from the input layer.

[0209] In some embodiments, a convolutional layer includes at least two sub-convolutional layers, and the operation performed by each sub-convolutional layer is represented by the following formula:

[0210]

[0211] Where C represents the number of input channels in the l-th sub-convolutional layer. It is the convolution kernel of the l-th sub-convolutional layer, b lσ represents the bias term of the l-th sub-convolutional layer, D′ is the depth of the input data of the l-th sub-convolutional layer, H′ is the height of the input data of the l-th sub-convolutional layer, and W′ is the width of the input data of the l-th sub-convolutional layer. Y is the output data of the l-th sub-convolutional layer. l The value at a depth of i", a height of j", and a width of k". It is the input data of the c-th input channel of the l-th sub-convolutional layer. The value at depth i"+d, height j"+h, and width k"+w. C is a positive integer, and D′, H′, and W′ are all integers greater than 1.

[0212] In other words, each sub-convolutional layer performs a convolution operation on the input data and uses an activation function to perform non-linear processing on the sum of the convolution result and the bias term, thereby obtaining the output data of the sub-convolutional layer. The input data of the first convolutional layer in at least two convolutional layers is represented by voxels, and the output data of the last sub-convolutional layer in at least two sub-convolutional layers is the convolutional data.

[0213] In some embodiments, please refer to Figure 10 The convolutional layer 80 includes at least two convolutional branches 81 and a fusion layer 82, and the sub-step 612 above includes at least one of the following steps:

[0214] 1. Input the voxel representation into at least two convolution branches, perform convolution operations on the voxel representation by the at least two convolution branches, and output the convolution results of the at least two convolution branches respectively.

[0215] At least two convolutional branches have different convolutional kernels.

[0216] In some embodiments, the kernel sizes of at least two convolutional branches are different. For example, please refer to... Figure 10 The convolutional layer 80 includes three convolutional branches 81. The kernel size of the first convolutional branch is 3×3×3, the kernel size of the second convolutional branch is 5×5×5, and the kernel size of the third convolutional branch is 7×7×7.

[0217] It should be noted that the operations performed by the convolution branch on the voxel representation are not limited to convolution operations. Similar to the sub-convolutional layer, it can also perform steps such as adding bias terms and non-linear activation. This application does not limit this.

[0218] 2. Input the convolution results of at least two convolution branches into the fusion layer, and the fusion layer merges the convolution results of at least two convolution branches to output the convolutional data.

[0219] In some embodiments, a fusion layer concatenates the convolution results of at least two convolutional branches to output convolutional data.

[0220] In some embodiments, the fusion layer performs a weighted sum of the convolution results of at least two convolutional branches to output the convolutional data. That is:

[0221]

[0222] Among them, Y fusion Y represents the data after convolution. s This represents the output feature map of the s-th convolutional branch, where s is an integer greater than 1, and α... s is the fusion weight of the s-th convolutional branch.

[0223] In the above embodiment, the convolutional data is obtained by fusing the convolution results of voxel representations with convolution kernels of different sizes. Therefore, in the process of generating adjustment parameters, CDCNN can comprehensively consider the local features (corresponding to smaller convolution kernels) and global features (corresponding to larger convolution kernels) of the voxel representation, thereby capturing the geometric features of the 3D model at different scales. Adjusting the 3D model based on the adjustment parameters output by the 3DCNN with this structure can make the adjusted 3D model more balanced in detail and overall shape, and more expressive.

[0224] Sub-step 614: Input the convolutional data into the pooling layer, and the pooling layer downsamples the convolutional data and outputs the pooled data.

[0225] In some embodiments, the convolutional data is input into a pooling layer, which then performs max pooling to downsample the convolutional data and outputs pooled data.

[0226] In some embodiments, the convolutional data is input into a pooling layer, which then performs average pooling to downsample the convolutional data and outputs pooled data.

[0227] For example, when the pooling window size is p×p×p (p is an integer greater than 1), if max pooling is used, for a 3D feature map of size p×p×p in the convolutional data, the largest eigenvalue in the 3D feature map will be used as the pooled eigenvalue, thus downsampling p×p×p eigenvalues ​​into 1 eigenvalue. If average pooling is used, for a 3D feature map of size p×p×p in the convolutional data, the average eigenvalue of the 3D feature map will be used as the pooled eigenvalue, thus downsampling p×p×p eigenvalues ​​into 1 eigenvalue.

[0228] Sub-step 616: Input the pooled data into the fully connected layer, which then maps the pooled data to the dimension containing the adjustment parameters and outputs the adjustment parameters.

[0229] In some embodiments, the flattened pooled data is input into a fully connected layer, which maps the pooled data to the dimension where the adjustment parameters are located and outputs the adjustment parameters.

[0230] In the above embodiment, the 3DCNN model includes convolutional layers, pooling layers, and fully connected layers. The fully connected layers output adjustment parameters based on the features (pooled data) of the voxel representations extracted by the convolutional and pooling layers. Thus, the fully connected layers can focus on the more critical information in the voxel representations and output adjustment parameters accurately and efficiently.

[0231] Step 620: Based on the adjustment parameters, adjust the 3D model to obtain the adjusted 3D model.

[0232] In the above embodiment, voxels are used to indicate the positions of each vertex of the 3D model in 3D space, and to represent the pixel values ​​of each pixel of the texture image in 3D space. 3DCNN thus learns the structure of the generated 3D model and the mapping position of the texture image in 3D space. 3DCNN can then match and align the texture image and the 3D model in 3D space, outputting adjustment parameters to make the 3D model and texture image more compatible.

[0233] In some embodiments, the adjustment parameters include the displacement vectors of each vertex of the 3D model. The displacement vector of the k-th vertex of the 3D model is used to adjust the position of the k-th vertex in 3D space, where k is a positive integer. Please refer to [reference needed]. Figure 11 Step 620 includes at least one of the following sub-steps 622 to 624:

[0234] Sub-step 622: For the kth vertex of the 3D model, add the position vector of the kth vertex to the displacement vector of the kth vertex to obtain the updated position vector of the kth vertex. The position vector of the kth vertex is used to indicate the position of the kth vertex in 3D space.

[0235] That is, v′ k =v k +Δv k , where v′ k Let v be the updated position vector of the k-th vertex. k It is the position vector of the kth vertex, Δv k It is the displacement vector of the k-th vertex.

[0236] Sub-step 624: Based on the updated position vectors of each vertex of the 3D model, adjust the position of each vertex of the 3D model in the 3D space to obtain the adjusted 3D model.

[0237] Based on the updated position vectors of each vertex in the 3D model, the positions of each vertex in the 3D model in the 3D space are adjusted so that they match the updated position vectors of each vertex.

[0238] For example, for the k-th vertex, the position of the k-th vertex in the three-dimensional space is adjusted to match the updated position vector of the k-th vertex.

[0239] In the above embodiment, 3DCNN directly outputs translation vectors to adjust the vertex positions of the 3D model, which can optimize the surface shape of the 3D model and make the vertex positions of the 3D model more compatible with the texture image.

[0240] In some embodiments, the adjustment parameters further include the density factors of each mesh in the 3D model, where the density factor of the j-th mesh is used to adjust the edge length of the j-th mesh, and j is a positive integer. Before step 624 above, the method further includes: for the j-th mesh, determining the adjusted length of the edge of the j-th mesh based on the length of the edge to be adjusted and the density factor of the j-th mesh.

[0241] In some embodiments, the length of the edge to be adjusted in the j-th grid is divided by the square root of the density factor of the j-th grid to obtain the adjusted length of the edge to be adjusted in the j-th grid.

[0242] In some embodiments, Among them, l j ′ is the length to which the edge to be adjusted in the j-th grid is adjusted, l j It is the length of the edge to be adjusted in the j-th grid, ρ j It is the density factor of the j-th grid, and a grid has at least one edge to be adjusted.

[0243] In some embodiments, sub-step 624 includes: adjusting the position of each vertex of the 3D model in the 3D space based on the updated position vector of each vertex of the 3D model and the length to which the edge to be adjusted of each mesh of the 3D model is adjusted, to obtain the adjusted 3D model.

[0244] In some embodiments, sub-step 624 includes at least one of the following steps:

[0245] 1. Based on the updated position vectors of each vertex of the 3D model, adjust the position of each vertex in the 3D space to obtain the updated 3D model.

[0246] In some embodiments, the positions of each vertex of the 3D model in 3D space are adjusted to match the updated position vectors of each vertex, thereby obtaining the updated 3D model.

[0247] 2. Adjust the length of each edge to be adjusted in the 3D model to obtain the adjusted 3D model.

[0248] In some embodiments, the positions of each vertex of the updated 3D model in 3D space are adjusted so that the side length of each mesh of the updated 3D model matches the length of the side to be adjusted of each mesh of the 3D model, thus obtaining the adjusted 3D model.

[0249] In the above embodiment, on the one hand, adjusting the 3D model based on the updated position vectors of each vertex ensures that the overall structure of the 3D model matches the texture image. On the other hand, the 3DCNN model outputs a density factor to guide the adjustment of the mesh side length of the 3D model. This ensures that the 3D model has shorter mesh side lengths and denser meshes in complex textured surface areas, and longer mesh side lengths and sparser meshes in simple textured surface areas. Thus, while the overall structure of the 3D model matches the texture image, the local structure of the 3D model also matches the texture details (such as contours) in the texture image.

[0250] In some embodiments, after adjusting the 3D model based on the adjustment parameters to obtain the adjusted 3D model, the following steps are further included: using the voxel representation of the adjusted 3D model as the voxel representation of the 3D model, and starting again from the step of inputting the voxel representation of the 3D model into the 3DCNN model and having the 3DCNN model output the adjustment parameters of the 3D model.

[0251] In some embodiments, the adjustment parameters include the displacement vectors of each vertex of the 3D model; please refer to [reference needed]. Figure 11 After adjusting the 3D model based on the adjustment parameters to obtain the adjusted 3D model, the process further includes at least one of the following steps 630 to 660:

[0252] Step 630: Based on the displacement vectors of each vertex of the 3D model, obtain the first loss.

[0253] The first loss is used to reflect the displacement difference of each adjacent vertex pair in the 3D model. An adjacent vertex pair includes two adjacent vertices, and the displacement difference of an adjacent vertex pair is the difference in the displacement vectors of the two vertices included in the adjacent vertex pair.

[0254] In some embodiments, the squares of the magnitudes of the displacement difference vectors of each adjacent vertex pair are added together to obtain the first loss. The displacement difference vector of an adjacent vertex pair is obtained by subtracting the displacement vectors of the two vertices included in the adjacent vertex pair.

[0255] That is, the first loss E smooth =∑ <r,t> |Δv r -Δv t | 2 ,in,<r,t> It is an adjacent vertex pair, Δv r and Δv t It is the displacement vector of the two vertices in an adjacent vertex pair.

[0256] Step 640: Based on the updated position vectors of each vertex of the 3D model, obtain the second loss.

[0257] The second loss is used to reflect the degree of matching between the adjusted 3D model and the texture image.

[0258] In some embodiments, a second loss is obtained based on the updated position vectors of each vertex of the 3D model and the position vectors used to indicate the set mapping positions of the corresponding pixels of each vertex on the texture image in 3D space.

[0259] In some embodiments, the differential Euclidean distances of each vertex are summed to obtain a second loss, where the differential Euclidean distance of a vertex is the Euclidean distance between the updated position vector of the vertex and the position vector indicating the set mapping position of the corresponding pixel on the texture image in three-dimensional space.

[0260] That is, the second loss Where, v′ k It is the position vector of the k-th vertex. It is a position vector used to indicate the set mapping position of the pixel corresponding to the vertex in the texture image in three-dimensional space.

[0261] Step 650: Based on the first loss and the second loss, adjust the parameters of the 3DCNN model to obtain the adjusted 3DCNN model.

[0262] In some embodiments, the first loss and the second loss are weighted and summed to obtain the total loss, which is used to reflect the performance of 3DCNN.

[0263] That is, E total =E data +λ·E smooth , where λ is an adjustable weight parameter.

[0264] In some embodiments, the parameters of the 3DCNN model are adjusted with the goal of minimizing the total loss, resulting in an adjusted 3DCNN model.

[0265] In some embodiments, prior to this step, a third loss is obtained based on the density factors of each grid in the 3D model, the third loss being used to reflect the accuracy of the density factors output by the 3DCNN model.

[0266] In some embodiments, a third loss is obtained based on the density factor of each mesh in the 3D model and the set density factor of each mesh.

[0267] The set density factor of a mesh is related to the set density factor of the surface region of the 3D model to which the mesh belongs. The set density factor of a mesh is used to reflect the density factor expected to be obtained for that mesh. In some embodiments, the technician divides the surface of the 3D model into at least two surface regions and sets a set density factor for each surface region. The set density factor of a surface region is used to reflect the mesh density adjustment strategy within that surface region (e.g., a set density factor greater than 1 indicates that more meshes need to be added within that surface region, and a set density factor less than 1 indicates that fewer meshes need to be added within that surface region). When a mesh belongs to a surface region of the 3D model, the set density factor of that mesh is the set density factor of that surface region. When a mesh belongs to at least two surface regions, the set density factor of that mesh is obtained by a weighted sum of the set density factors of the at least two surface regions.

[0268] In some embodiments, the difference between the ratio of the density factor of the j-th mesh to the set density factor of the j-th mesh and 1 is obtained to get the loss value of the j-th mesh. The squares of the loss values ​​of each mesh in the 3D model are summed to obtain the third loss.

[0269] In some embodiments, the third loss is calculated using the following formula:

[0270]

[0271] Among them, E density It is the third loss, ρ j ρ is the density factor of the j-th grid. target It is the set density factor of the j-th grid.

[0272] In some embodiments, step 650 includes: adjusting the parameters of the 3DCNN model based on the first loss, the second loss, and the third loss to obtain the adjusted 3DCNN model.

[0273] In some embodiments, the first loss, the second loss, and the third loss are weighted and summed to obtain the total loss. The parameters of the 3DCNN model are adjusted to minimize the total loss, resulting in an adjusted 3DCNN model.

[0274] By designing the aforementioned third loss, 3DCNN can learn to accurately output density factors, thereby ensuring that the mesh distribution density in each region of the adjusted 3D model meets the expectations of the technician. Furthermore, in the above embodiment, since the set density factor of the mesh is related to the set density factor of the surface region of the 3D model to which the mesh belongs, the set density factor of the surface region of the 3D model can be arbitrarily adjusted by the technician according to the actual structure of the 3D model. When adjusting the parameters of the 3DCNN using a multi-round iterative approach, the 3DCNN can gradually learn how to reasonably control the mesh density of the 3D model.

[0275] Step 660: The adjusted voxel representation of the 3D model is used as the voxel representation of the 3D model, and the adjusted 3DCNN model is used as the 3DCNN model. The process starts again from the step of inputting the voxel representation of the 3D model into the 3DCNN model and outputting the adjusted parameters of the 3D model from the 3DCNN model.

[0276] Reaching this step signifies the completion of one iteration of adjusting the 3D model using the CDCNN model.

[0277] In some embodiments, prior to this step, the method further includes: determining an evaluation metric for the adjusted 3D model, wherein the evaluation metric is obtained based on at least one of a first loss, a second loss, a third loss, smoothness error, and computational cost. For example, the evaluation metric is any one of the above terms, or, for example, the evaluation metric is obtained by a weighted sum of at least two of the above terms.

[0278] Smoothness error is used to reflect the smoothness of the adjusted 3D model. In some embodiments, the smoothness error is obtained based on the magnitude of the difference vector of each adjacent mesh pair in the adjusted 3D model. An adjacent mesh pair includes two meshes with overlapping edges in the adjusted 3D model. The difference vector of an adjacent mesh pair is obtained by subtracting the normal vectors of the two meshes in the adjacent mesh pair.

[0279] In some embodiments, the smoothness error is obtained by summing the magnitudes of the difference vectors of each adjacent mesh pair in the adjusted 3D model.

[0280] In some embodiments, the smoothness error is calculated using the following formula:

[0281]

[0282] Among them, E smoothnessFor smoothness error,<va,vb> They are adjacent grid pairs, n va and n vb It is the normal vector of the two grids in an adjacent grid pair.

[0283] Computational overhead can be measured by the number of meshes, vertices, and rendering time of the adjusted 3D model, but this application does not limit this.

[0284] In some embodiments, prior to step 650 above, the strategy for adjusting the 3D model using the 3DCNN model is redefined based on the above evaluation metrics, so as to better meet the needs of technicians for the 3D model.

[0285] In some embodiments, based on the above evaluation metrics, at least one of the following is adjusted: the network architecture of the 3DCNN model, the weights of each function term in the loss function of the 3DCNN model (e.g., λ as described above), the number of iterations for adjusting the 3D model using the 3DCNN model, and the learning rate of the 3DCNN model. For example, if the above evaluation metrics fall within a first defined range, the number of sub-convolutional layers or convolutional branches of the 3DCNN is changed. For example, if the above evaluation metrics fall within a second defined range, the weights of at least one function term in the loss function are adjusted. For example, if the above evaluation metrics fall within a third defined range, the number of iterations for adjusting the 3D model using the 3DCNN model is changed. The above first, second, and third defined ranges are all set by those skilled in the art as needed, and this application does not limit them.

[0286] In some embodiments, the learning rate of the 3DCNN model is adjusted according to the evaluation metrics described above. For example, the learning rate of the 3DCNN is adjusted using the following formula: Where t is a positive integer, η t The learning rate η is the value used when adjusting the parameters of the 3DCNN model during the t-th iteration of adjusting the 3D model. t+1 The learning rate is the one used when adjusting the parameters of the 3DCNN model during the (t+1)th iteration. The second loss is obtained during the first iteration. The second loss is obtained during the t-th iteration. The second loss in the above formula can also be replaced by the first loss, the third loss, the smoothness error, etc., and this application does not limit this.

[0287] In some embodiments, step 660 includes: if the termination condition is not met, using the voxel representation of the adjusted 3D model as the voxel representation of the 3D model and the adjusted 3DCNN model as the 3DCNN model, and starting again from the step of inputting the voxel representation of the 3D model into the 3DCNN model and outputting the adjusted parameters of the 3D model from the 3DCNN model.

[0288] In some embodiments, the termination condition includes: the above evaluation indicators meet the termination range. The termination range is set by the technicians as needed, such as being greater than a certain threshold, less than a certain threshold, or belonging to a certain interval, etc., and this application does not limit it in this regard.

[0289] In some embodiments, the termination condition includes: the iterative process of adjusting the 3D model using the CDCNN model reaches a set number of iterations.

[0290] In the above embodiment, the 3D model is iteratively adjusted by the 3DCNN model. The adjustment result of the previous round will affect the voxel representation input to the 3DCNN in the next round. Furthermore, during this iteration, the parameters of the 3DCNN model are also dynamically adjusted. That is, the quality of the adjusted parameters output by the 3DCNN model will directly affect the parameters of the 3DCNN in the next round of iteration, enabling the 3DCNN model to gradually achieve better performance, thereby ensuring that the final adjusted 3D model has high quality. In addition, the second loss mentioned above enables the 3DCNN model to learn the ability to match the texture image with the 3D model by adjusting the vertices of the 3D model during the iteration process. The first loss mentioned above constrains the difference in the displacement vectors of adjacent vertices when adjusting the position of the 3D model vertex, so that the difference in the position adjustment strategies of two adjacent vertices is not too large. Thus, during the iteration process, the surface smoothness of the 3D model is guaranteed, and self-intersection of the 3D model mesh is avoided.

[0291] In some embodiments, the texture image is applied to render the adjusted 3D model to obtain the rendered 3D model.

[0292] Please refer to Figure 12 This diagram illustrates a rendering method for a 3D model provided in one embodiment of this application. The rendering method includes the following four stages:

[0293] 1. Pixel Sampling: P pixels are sampled from the texture image to generate the 3D model. These P pixels are then mapped into 3D space to obtain P mapped points for generating the 3D model. On one hand, the P pixels include at least two first-type pixels evenly distributed in the texture image, ensuring the generated 3D model has a basic structure consistent with the texture image. On the other hand, the P pixels include second-type pixels distributed along the contours in the texture image, ensuring the generated 3D model better reflects the texture details in the texture image.

[0294] 2. Patch Number Allocation and 3D Model Generation: Since the sampling process described above results in P pixels being densely distributed along the contours of the texture image, it is necessary to adaptively allocate the number of meshes to the P mapping points (i.e., how many meshes are allocated to share a mapping point), so that different regions of the 3D model have different mesh densities. Based on the allocated mesh numbers, the connection relationships between the P mapping points are established to generate the 3D model. This can reduce the computational cost of performing operations (such as rendering) on ​​the 3D model while maintaining its detail. Simultaneously, preliminary adjustments can be made to the generated 3D model during this stage, such as adjusting mesh edge lengths, adjusting mapping point positions based on curvature, and merging meshes.

[0295] 3. Optimizing 3D Models with 3DCNN: The voxel representation of the 3D model is input into a 3DCNN, and the 3DCNN model outputs adjustment parameters for adjusting the 3D model. Since the voxel representation is used to indicate the position of each vertex of the 3D model in 3D space, and to represent the pixel value of each pixel of the texture image in 3D space, 3DCNN can optimize the structure of the 3D model, thereby matching and aligning the texture image and the 3D model as closely as possible in 3D space, improving the expressiveness of the 3D model.

[0296] 4. Applying texture image rendering to the adjusted 3D model: Mapping the texture image onto the surface of the adjusted 3D model to render the adjusted 3D model. Rendering the adjusted 3D model obtained through the above method can achieve good rendering results with high rendering efficiency and low computing resources.

[0297] The following are embodiments of the apparatus of this application. For details not described in the apparatus embodiments of this application, please refer to the method embodiments above.

[0298] Please refer to Figure 13This diagram illustrates a block diagram of a three-dimensional model generation apparatus according to an embodiment of this application. The apparatus has the function of implementing the aforementioned three-dimensional model generation method; this function can be implemented in hardware or by hardware executing corresponding software. The apparatus can be a computer device or can be installed within a computer device. The apparatus 1300 may include: a sampling module 1310, a mapping module 1320, an allocation module 1330, and a generation module 1340.

[0299] The sampling module 1310 is used to sample P pixels from each pixel of the texture image, wherein the P pixels are non-uniformly distributed in the texture image.

[0300] The mapping module 1320 is used to map the P pixels into a three-dimensional space to obtain P mapped points in the three-dimensional space, where P is an integer greater than 1.

[0301] The allocation module 1330 is used to determine the number of grids allocated to each of the P mapping points, wherein at least two of the P mapping points are allocated different numbers of grids.

[0302] The generation module 1340 is used to establish the connection relationship of the P mapping points according to the number of grids allocated to the P mapping points respectively, so as to generate the three-dimensional model. In the generated three-dimensional model, the number of associated grids of the i-th mapping point among the P mapping points is equal to the number of grids allocated to the i-th mapping point. The associated grids of the i-th mapping point refer to the grids with the i-th mapping point as vertices, where i is a positive integer less than or equal to P.

[0303] In some embodiments, the allocation module 1330 is configured to determine the weight of the grid to be allocated to each of the P mapping points based on the feature information of each of the P mapping points, wherein the feature information of the i-th mapping point is used to reflect the features of the pixel corresponding to the i-th mapping point in the texture image; and to determine the number of grids to be allocated to each of the P mapping points based on the set total number of grids and the weight of the grids allocated to each of the P mapping points.

[0304] In some embodiments, the number of grids allocated to the i-th mapping point is positively correlated with the weight of the grids allocated to the i-th mapping point; the feature information of the i-th mapping point includes at least one of the following: the feature value of the pixel corresponding to the i-th mapping point in the texture image, wherein the weight of the grids allocated to the i-th mapping point is positively correlated with the feature value of the pixel corresponding to the i-th mapping point, and the feature value of the pixel is used to reflect the degree of abrupt change in pixel value at the pixel in the texture image; the magnitude of the gradient at the pixel corresponding to the i-th mapping point in the texture image, wherein the weight of the grids allocated to the i-th mapping point is positively correlated with the magnitude of the gradient at the pixel corresponding to the i-th mapping point; the size of the segmented region to which the pixel corresponding to the i-th mapping point belongs in the texture image, wherein the weight of the grids allocated to the i-th mapping point is negatively correlated with the size of the segmented region to which the pixel corresponding to the i-th mapping point belongs, wherein the segmented region is obtained from the texture image based on the pixel values ​​of each pixel in the texture image.

[0305] In some embodiments, the P pixels include: at least two first-class pixels uniformly distributed in the texture image, and at least two second-class pixels distributed on the contour contained in the texture image.

[0306] The sampling module 1310 includes: a first sampling submodule and a second sampling submodule.

[0307] The first sampling submodule is used to divide the texture image into M image regions of the same size, where M is an integer greater than 1; and to sample the same number of pixels from each of the image regions to obtain the at least two first-type pixels.

[0308] The second sampling submodule is used to extract at least two key pixels from the texture image, wherein the key pixels are pixels belonging to the contour contained in the texture image; and to sample the at least two second-type pixels from the at least two key pixels.

[0309] In some embodiments, the second sampling submodule is further configured to, for any pixel in the texture image, obtain a feature value of the pixel based on the difference between a first value and a second value. The feature value of the pixel is used to reflect the degree of abrupt change in the pixel value at the pixel in the texture image. The first value is the product of a first derivative value and a second derivative value, the second value is the square of a third derivative value, the first derivative value is the second derivative value of the texture image at the pixel in a first direction, the second derivative value is the second derivative value of the texture image at the pixel in a second direction, and the third derivative value is the mixed second derivative value of the texture image at the pixel in the first and second directions, where the first and second directions are perpendicular. If the feature value of the pixel is greater than or equal to a first threshold, the pixel is identified as the key pixel.

[0310] In some embodiments, the second sampling submodule is further configured to perform Gaussian filtering on the texture image to obtain a filtered image; for a first extreme point in the filtered image, if the magnitude of the gradient of the filtered image at the first extreme point is greater than or equal to a second threshold, the first extreme point is determined as a pixel belonging to the contour contained in the filtered image; if the magnitude of the gradient of the filtered image at the first extreme point is greater than or equal to a third threshold and less than the second threshold, if there are pixels in the filtered image adjacent to the first extreme point whose gradient magnitude is greater than or equal to the second threshold, then the first extreme point is determined as a pixel belonging to the contour contained in the filtered image; based on the pixels belonging to the contour contained in the filtered image, the key pixel in the texture image is determined.

[0311] In some embodiments, the mesh of the three-dimensional model is a triangular mesh. The generation module 1340 is configured to connect each of the P mapping points to a first number of other mapping points to generate the three-dimensional model, where the first number is the number of meshes allocated to the mapping points.

[0312] In some embodiments, the device 1300 further includes: a network control module and a first adjustment module. Figure 13 (Not shown in the image).

[0313] The network control module is used to input the voxel representation of the three-dimensional model into the three-dimensional convolutional neural network 3DCNN model, and the 3DCNN model outputs the adjustment parameters of the three-dimensional model. The voxel representation is used to indicate the position of each vertex of the three-dimensional model in the three-dimensional space, and to represent the pixel value of each pixel of the texture image in the three-dimensional space.

[0314] The first adjustment module is used to adjust the three-dimensional model based on the adjustment parameters to obtain the adjusted three-dimensional model.

[0315] In some embodiments, the 3DCNN model includes convolutional layers, pooling layers, and fully connected layers. The network control module includes a convolutional submodule, a pooling submodule, and a fully connected submodule.

[0316] The convolution submodule is used to input the voxel representation into the convolutional layer, and the convolutional layer performs a convolution operation on the voxel representation to output the convolutional data.

[0317] The pooling submodule is used to input the convolutional data into the pooling layer, and the pooling layer downsamples the convolutional data to output the pooled data.

[0318] The fully connected submodule is used to input the pooled data into the fully connected layer, and the fully connected layer maps the pooled data to the dimension where the adjustment parameter is located, and outputs the adjustment parameter.

[0319] In some embodiments, the convolutional layer includes at least two convolutional branches and a fusion layer.

[0320] The convolution submodule is used to input the voxel representation into the at least two convolution branches respectively, perform convolution operations on the voxel representation by the at least two convolution branches respectively, and output the convolution results of the at least two convolution branches respectively, wherein the convolution kernels of the at least two convolution branches are different; input the convolution results of the at least two convolution branches respectively into the fusion layer, and fuse the convolution results of the at least two convolution branches respectively, and output the convolutional data.

[0321] In some embodiments, the adjustment parameters include the displacement vectors of each vertex of the three-dimensional model, wherein the displacement vector of the k-th vertex of the three-dimensional model is used to adjust the position of the k-th vertex in the three-dimensional space, and k is a positive integer.

[0322] The first adjustment module is used to add the position vector of the kth vertex to the displacement vector of the kth vertex to obtain the updated position vector of the kth vertex, wherein the position vector of the kth vertex is used to indicate the position of the kth vertex in the three-dimensional space; and adjust the position of each vertex of the three-dimensional model in the three-dimensional space based on the updated position vector of each vertex of the three-dimensional model to obtain the adjusted three-dimensional model.

[0323] In some embodiments, the device 1300 further includes: a training module and an iteration module. Figure 13(Not shown in the image).

[0324] The training module is used to obtain a first loss based on the displacement vectors of each vertex of the 3D model. The first loss reflects the displacement difference between each pair of adjacent vertices in the 3D model. Each pair of adjacent vertices includes two adjacent vertices, and the displacement difference between the two adjacent vertices is the difference in their respective displacement vectors. Based on the updated position vectors of each vertex of the 3D model, a second loss is obtained. The second loss reflects the degree of matching between the adjusted 3D model and the texture image. Based on the first loss and the second loss, the parameters of the 3DCNN model are adjusted to obtain the adjusted 3DCNN model.

[0325] The iterative module is used to take the voxel representation of the adjusted 3D model as the voxel representation of the 3D model and the adjusted 3DCNN model as the 3DCNN model, and then start executing again from the step of inputting the voxel representation of the 3D model into the 3DCNN model and having the 3DCNN model output the adjustment parameters of the 3D model.

[0326] In some embodiments, the adjustment parameters further include the density factor of each mesh of the three-dimensional model, wherein the density factor of the j-th mesh of the three-dimensional model is used to adjust the side length of the j-th mesh, where j is a positive integer.

[0327] The first adjustment module is further configured to, for the j-th mesh, determine the length to which the edge to be adjusted of the j-th mesh is to be adjusted based on the length of the edge to be adjusted of the j-th mesh and the density factor of the j-th mesh; and adjust the position of each vertex of the three-dimensional model in the three-dimensional space based on the updated position vector of each vertex of the three-dimensional model and the length to which the edge to be adjusted of each mesh of the three-dimensional model is to be adjusted, to obtain the adjusted three-dimensional model.

[0328] In some embodiments, the training module is further configured to obtain a third loss based on the density factors of each grid of the 3D model, the third loss being used to reflect the accuracy of the density factors output by the 3DCNN model; and to adjust the parameters of the 3DCNN model based on the first loss, the second loss, and the third loss to obtain the adjusted 3DCNN model.

[0329] In some embodiments, the device 1300 further includes: a second adjustment module ( Figure 13 (Not shown in the image).

[0330] The second adjustment module is used to determine the optimized side length based on the setting parameters of the segmentation region to which the pixel corresponding to the first vertex of the 3D model in the texture image belongs, and the setting parameters of the segmentation region to which the pixel corresponding to the second vertex of the 3D model in the texture image belongs, wherein the segmentation region is obtained by segmenting from the texture image based on the pixel values ​​of each pixel in the texture image, and the setting parameters are related to the density of the pixels sampled from the segmentation region; and to adjust the position of at least one of the first vertex and the second vertex in the 3D space so as to adjust the length of the side connecting the first vertex and the second vertex in the 3D model to the optimized side length.

[0331] In some embodiments, the device 1300 further includes: a third adjustment module ( Figure 13 (Not shown in the image).

[0332] The third adjustment module is used to adjust the position of each vertex of the three-dimensional model in the three-dimensional space based on the curvature of each vertex of the three-dimensional model. The curvature of the vertex is used to reflect the degree of bending of the three-dimensional model at the vertex.

[0333] In some embodiments, the third adjustment module is used to determine the curvature of the third vertex of the three-dimensional model based on the angles of each angle in the three-dimensional model with the third vertex as the vertex; and to adjust the position of the third vertex in the three-dimensional space according to the curvature of the third vertex.

[0334] In some embodiments, the device 1300 further includes: a merging module ( Figure 13 (Not shown in the image).

[0335] The merging module is used to merge the first mesh and the second mesh in the 3D model when the angle between the normal vector of the first mesh and the normal vector of the second mesh is less than a fourth threshold, wherein the first mesh and the second mesh have overlapping edges.

[0336] In some embodiments, the merging module is used to obtain a first determination value based on the position vectors of R vertices forming Q meshes in the three-dimensional model, and the position vectors of fitted vertices of the R vertices, wherein each of the Q meshes has an edge overlapping with at least one other mesh in the Q meshes, the fitted vertices are used to determine the mesh after merging the Q meshes, the position vectors of the vertices are used to indicate the position of the vertices in the three-dimensional space, Q is an integer greater than 1, and R is an integer greater than 3; if the first determination value is less than or equal to a fifth threshold, the Q meshes are merged.

[0337] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0338] Please refer to Figure 14 The diagram illustrates a structural block diagram of a computer device provided in one embodiment of this application.

[0339] Typically, computer device 1400 includes a processor 1401 and a memory 1402.

[0340] Processor 1401 may include one or at least two processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1401 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1401 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1401 may integrate a GPU, which is responsible for rendering and drawing the content required to be displayed on the screen. In some embodiments, processor 1401 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0341] The memory 1402 may include one or at least two computer-readable storage media, which may be tangible and non-transitory. The memory 1402 may also include high-speed random access memory and non-volatile memory, such as one or at least two disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1402 stores a computer program that is loaded and executed by the processor 1401 to implement the above-described method for generating the three-dimensional model.

[0342] Those skilled in the art will understand that Figure 14The structure shown does not constitute a limitation on the computer device 1400, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0343] In some embodiments, a computer-readable storage medium is also provided, wherein a computer program is stored therein, the computer program being loaded and executed by a processor to implement the above-described method for generating a three-dimensional model.

[0344] Optionally, the computer-readable storage medium may include: ROM (Read-Only Memory), RAM (Random-Access Memory), SSD (Solid State Drives), or optical disc, etc. The random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0345] In some embodiments, a computer program product is also provided, the computer program product including a computer program stored in a computer-readable storage medium, and a processor reading from the computer-readable storage medium and executing the computer program to implement the above-described method for generating a three-dimensional model.

[0346] It should be understood that "at least two" as mentioned herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order. In some other embodiments, the steps may not be executed in numerical order, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this.

[0347] The above are merely exemplary embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application shall be included within the protection scope of this application.

Claims

1. A method of generating a three-dimensional model, characterized by, The method includes: P pixels are sampled from each pixel in the texture image. The P pixels are not uniformly distributed in the texture image, and P is an integer greater than 1. The P pixels are mapped to a three-dimensional space to obtain P mapped points in the three-dimensional space; The number of grids assigned to each of the P mapping points is determined, wherein at least two of the P mapping points are assigned different numbers of grids. Based on the number of grids allocated to the P mapping points, the connection relationship of the P mapping points is established to generate the three-dimensional model. In the generated three-dimensional model, the number of associated grids of the i-th mapping point among the P mapping points is equal to the number of grids allocated to the i-th mapping point. The associated grids of the i-th mapping point refer to the grids with the i-th mapping point as vertices, where i is a positive integer less than or equal to P.

2. The method of claim 1, wherein, The determination of the number of grids allocated to each of the P mapping points includes: Based on the feature information of each of the P mapping points, the weight of the grid is determined for each of the P mapping points. The feature information of the i-th mapping point is used to reflect the features of the pixel corresponding to the i-th mapping point in the texture image. Based on the set total number of grids and the weights assigned to the grids for the P mapping points, the number of grids assigned to each of the P mapping points is determined.

3. The method of claim 2, wherein, The number of grids allocated to the i-th mapping point is positively correlated with the weight of the grids allocated to the i-th mapping point; The feature information of the i-th mapping point includes at least one of the following: The feature value of the pixel corresponding to the i-th mapping point in the texture image is positively correlated with the weight of the grid assigned to the i-th mapping point. The feature value of the pixel is used to reflect the degree of sudden change in pixel value at the pixel in the texture image. The magnitude of the gradient at the pixel corresponding to the i-th mapping point in the texture image is positively correlated with the magnitude of the gradient at the pixel corresponding to the i-th mapping point. The size of the segmentation region to which the pixel corresponding to the i-th mapping point belongs in the texture image, and the weight of the grid assigned to the i-th mapping point are negatively correlated with the size of the segmentation region to which the pixel corresponding to the i-th mapping point belongs, wherein the segmentation region is obtained from the texture image based on the pixel values ​​of each pixel in the texture image.

4. The method according to any one of claims 1 to 3, characterized in that, The P pixels include: At least two first-class pixels are evenly distributed in the texture image, and at least two second-class pixels are distributed on the contour contained in the texture image; The step of sampling P pixels from each pixel in the texture image includes: The texture image is divided into M image regions of the same size, where M is an integer greater than 1; From each of the image regions, the same number of pixels are sampled to obtain the at least two first-type pixels; Extract at least two key pixels from the texture image, wherein the key pixels are pixels that belong to the contour contained in the texture image; The at least two second-type pixels are obtained by sampling from the at least two key pixels.

5. The method of claim 4, wherein, The method further includes: For any pixel in the texture image, a feature value of the pixel is obtained based on the difference between a first value and a second value. The feature value of the pixel is used to reflect the degree of abrupt change in the pixel value at the pixel in the texture image. The first value is the product of a first derivative value and a second derivative value, and the second value is the square of a third derivative value. The first derivative value is the second derivative value of the texture image at the pixel in a first direction, the second derivative value is the second derivative value of the texture image at the pixel in a second direction, and the third derivative value is the mixed second derivative value of the texture image at the pixel in the first and second directions. The first and second directions are perpendicular. If the feature value of a pixel is greater than or equal to a first threshold, the pixel is identified as the key pixel.

6. The method according to claim 4 or 5, characterized in that, The method further includes: The texture image is Gaussian filtered to obtain the filtered image; For the first extreme point in the filtered image, if the magnitude of the gradient of the filtered image at the first extreme point is greater than or equal to the second threshold, the first extreme point is determined as a pixel belonging to the contour contained in the filtered image. If the magnitude of the gradient at the first extreme point in the filtered image is greater than or equal to the third threshold and less than the second threshold, and if there are pixels in the filtered image that are adjacent to the first extreme point and have a gradient magnitude greater than or equal to the second threshold, then the first extreme point is determined to be a pixel belonging to the contour contained in the filtered image. The key pixels in the texture image are determined based on the pixels that belong to the contours contained in the filtered image.

7. The method according to any one of claims 1 to 6, characterized in that, The mesh of the 3D model is a triangular mesh; the step of establishing the connection relationship of the P mapping points according to the number of meshes allocated to the P mapping points to generate the 3D model includes: For each of the P mapping points, the mapping point is connected to a first number of other mapping points to generate the 3D model, where the first number is the number of meshes allocated to the mapping point.

8. The method according to any one of claims 1 to 7, characterized in that, After establishing the connection relationship of the P mapping points based on the number of grids allocated to each of the P mapping points to generate the 3D model, the process further includes: The voxel representation of the three-dimensional model is input into a three-dimensional convolutional neural network (3DCNN) model, and the 3DCNN model outputs the adjustment parameters of the three-dimensional model. The voxel representation is used to indicate the position of each vertex of the three-dimensional model in the three-dimensional space and to represent the pixel value of each pixel of the texture image in the three-dimensional space. Based on the adjustment parameters, the three-dimensional model is adjusted to obtain the adjusted three-dimensional model.

9. The method of claim 8, wherein, The 3DCNN model includes convolutional layers, pooling layers, and fully connected layers; The step of inputting the voxel representation of the 3D model into a 3DCNN model, and having the 3DCNN model output the adjustment parameters of the 3D model, includes: The voxel representation is input into the convolutional layer, the convolutional layer performs a convolution operation on the voxel representation, and outputs the convolutional data. The convolutional data is input into the pooling layer, which downsamples the convolutional data and outputs the pooled data. The pooled data is input into the fully connected layer, which maps the pooled data to the dimension where the adjustment parameter is located and outputs the adjustment parameter.

10. The method of claim 9, wherein, The convolutional layer includes at least two convolutional branches and a fusion layer; The step of inputting the voxel representation into the convolutional layer, having the convolutional layer perform a convolution operation on the voxel representation, and outputting the convolutional data includes: The voxel representations are respectively input into the at least two convolution branches, and the at least two convolution branches perform convolution operations on the voxel representations respectively, and output the convolution results of the at least two convolution branches respectively, wherein the convolution kernels of the at least two convolution branches are different; The convolution results of each of the at least two convolutional branches are input into the fusion layer, which then merges the convolution results of the at least two convolutional branches to output the convolutional data.

11. The method according to any one of claims 8 to 10, characterized in that, The adjustment parameters include the displacement vectors of each vertex of the three-dimensional model. The displacement vector of the kth vertex of the three-dimensional model is used to adjust the position of the kth vertex in the three-dimensional space, where k is a positive integer. The step of adjusting the 3D model based on the adjustment parameters to obtain the adjusted 3D model includes: For the k-th vertex, the position vector of the k-th vertex is added to the displacement vector of the k-th vertex to obtain the updated position vector of the k-th vertex, wherein the position vector of the k-th vertex is used to indicate the position of the k-th vertex in the three-dimensional space; Based on the updated position vectors of each vertex of the 3D model, the positions of each vertex of the 3D model in the 3D space are adjusted to obtain the adjusted 3D model.

12. The method of claim 11, wherein, The method further includes: Based on the displacement vectors of each vertex in the 3D model, a first loss is obtained. The first loss is used to reflect the displacement difference of each pair of adjacent vertices in the 3D model. The pair of adjacent vertices includes two adjacent vertices, and the displacement difference of the pair of adjacent vertices is the difference in the displacement vectors of the two vertices in the pair of adjacent vertices. Based on the updated position vectors of each vertex of the 3D model, a second loss is obtained, which is used to reflect the degree of matching between the adjusted 3D model and the texture image. Based on the first loss and the second loss, the parameters of the 3DCNN model are adjusted to obtain the adjusted 3DCNN model; The process begins again from the step of inputting the voxel representation of the 3D model into the 3DCNN model and outputting the adjustment parameters of the 3D model by the 3DCNN model, using the adjusted 3D CNN model as the 3D CNN model.

13. The method according to claim 11 or 12, characterized in that, The adjustment parameters also include the density factor of each mesh in the 3D model, wherein the density factor of the j-th mesh in the 3D model is used to adjust the side length of the j-th mesh, where j is a positive integer; the method further includes: For the j-th grid, based on the length of the edge to be adjusted in the j-th grid and the density factor of the j-th grid, the length to which the edge to be adjusted in the j-th grid is adjusted is determined; The process of adjusting the positions of each vertex of the 3D model in the 3D space based on the updated position vectors of each vertex of the 3D model to obtain the adjusted 3D model includes: Based on the updated position vectors of each vertex of the 3D model and the adjusted lengths of each edge of each mesh of the 3D model, the positions of each vertex of the 3D model in the 3D space are adjusted to obtain the adjusted 3D model.

14. The method of claim 13, wherein, The method further includes: Based on the density factor of each grid in the 3D model, a third loss is obtained, which is used to reflect the accuracy of the density factor output by the 3DCNN model. The step of adjusting the parameters of the 3DCNN model based on the first loss and the second loss to obtain the adjusted 3DCNN model includes: Based on the first loss, the second loss, and the third loss, the parameters of the 3DCNN model are adjusted to obtain the adjusted 3DCNN model.

15. The method according to any one of claims 8 to 14, characterized in that, Before inputting the voxel representation of the 3D model into the 3DCNN model and having the 3DCNN model output the adjustment parameters of the 3D model, the method further includes: The optimized side length is determined based on the setting parameters of the segmentation region to which the pixel corresponding to the first vertex of the 3D model in the texture image belongs, and the setting parameters of the segmentation region to which the pixel corresponding to the second vertex of the 3D model in the texture image belongs. The segmentation region is obtained by segmenting the texture image based on the pixel value of each pixel in the texture image, and the setting parameters are related to the density of the pixels sampled from the segmentation region. Adjust the position of at least one of the first vertex and the second vertex in the three-dimensional space to adjust the length of the edge connecting the first vertex and the second vertex in the three-dimensional model to the optimized edge length.

16. The method according to any one of claims 8 to 15, characterized in that, Before inputting the voxel representation of the 3D model into the 3DCNN model and having the 3DCNN model output the adjustment parameters of the 3D model, the method further includes: Based on the curvature of each vertex of the 3D model, the position of each vertex of the 3D model in the 3D space is adjusted, and the curvature of the vertex is used to reflect the degree of bending of the 3D model at the vertex.

17. The method of claim 16, wherein, The step of adjusting the position of each vertex of the 3D model in the 3D space based on the curvature of each vertex of the 3D model includes: For the third vertex of the three-dimensional model, the curvature of the third vertex is determined based on the angles of each angle in the three-dimensional model with the third vertex as the vertex. The position of the third vertex in the three-dimensional space is adjusted according to the curvature of the third vertex.

18. The method according to any one of claims 8 to 17, characterized in that, Before inputting the voxel representation of the 3D model into the 3DCNN model and having the 3DCNN model output the adjustment parameters of the 3D model, the method further includes: If the angle between the normal vector of the first mesh and the normal vector of the second mesh in the 3D model is less than a fourth threshold, the first mesh and the second mesh are merged, and the first mesh and the second mesh have overlapping edges.

19. A computer device, comprising: The computer device includes a processor and a memory, the memory storing a computer program that is loaded and executed by the processor to implement the method for generating a three-dimensional model as described in any one of claims 1 to 18.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the method for generating a three-dimensional model as described in any one of claims 1 to 18.

Images