3D model automatic style conversion method and device, equipment and medium
By building and optimizing the texture training set, using deep learning network to achieve 3D model style conversion, the time-consuming and labor-consuming problem of traditional methods is solved, and seamless migration and efficient conversion between different game styles are achieved.
Patent Information
- Application Number
- CN202310676220.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-08
- Publication Date
- 2024-12-10
AI Technical Summary
The traditional 3D model style conversion method requires manual creation of new models, which is time-consuming and labor-intensive, making it difficult to achieve easy migration between different game styles.
By constructing a reference texture training set, dividing areas and filling color blocks of different colors, generating a reference matching training set and a target matching training set, and using deep learning multi-layer neural network to optimize the target texture training set to achieve 3D model style conversion.
Without manual creation of new models, the seamless migration of 3D game entities between different game styles is achieved, improving the efficiency and accuracy of style conversion.
Smart Images
Figure CN119097908A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of virtual reality technology, and in particular to a 3D model automatic style conversion method, device, equipment and medium. Background Art
[0002] With the development of 3D games, various games of different styles have gradually emerged. Game developers hope to easily convert and migrate game entities, such as characters, props, and items, between games of different styles. However, traditional style transfer methods usually manually create new models based on the style to be converted, which is time-consuming and labor-intensive.
[0003] Therefore, it is necessary to propose a 3D model automatic style conversion method, device, equipment and medium to solve the above problems. Summary of the invention
[0004] The purpose of the present invention is to provide a 3D model automatic style conversion method, device, equipment and medium to improve the time-consuming and labor-intensive problems of existing manual style conversion methods.
[0005] In a first aspect, the present application provides a method for automatic style conversion of a 3D model, the method comprising: constructing a reference texture training set of the reference game model according to UV maps of a reference game model and a game model to be converted, dividing the reference texture training set into a plurality of different regions, and filling different regions with blocks of different colors; applying the reference texture training set to the UV coordinates of the reference game model to generate a 3D model of the reference game, generating a reference matching training set from the 3D model of the reference game, wherein the reference matching training set is a 2D reference image of different perspectives; pre-generating a target texture training set of the game model to be converted for each block of the region, applying the target texture training set to the UV coordinates of the game model to be converted to generate a 3D model of the game to be converted, generating a target matching training set from the 3D model of the game to be converted, wherein the target matching training set is a 2D target image that corresponds one-to-one to the 2D reference image; reducing the area of the region into which the reference texture training set is divided, and repeating the above steps until a target texture training set of a desired style is obtained.
[0006] The beneficial effects of the method provided by the present invention are: constructing a reference texture training set of a reference game model, dividing and color-filling the reference texture training set to obtain a plurality of regions filled with different color blocks, generating a reference matching training set according to the reference texture training set and the UV coordinates of the reference game model, pre-generating a target texture training set according to the color blocks of each of the regions, generating a target matching training set according to the target texture training set and the UV coordinates of the game model to be converted, increasing the number of regions by reducing the area of the regions divided by the reference texture training set, and repeating the above steps so that a 2D target image in the target texture training set is more similar to a 2D reference image in the reference matching training set, thereby obtaining a target texture training set of a desired style, and the automatic style conversion algorithm of the present invention can realize seamless migration of 3D game entities between different game styles without manually creating a new model.
[0007] In a possible embodiment, after generating a target matching training set from the 3D model of the game to be converted, the method further includes:
[0008] The reference matching training set and the target matching training set are input into the deep learning multi-layer neural network to obtain the optimized target texture training set.
[0009] In a possible embodiment, after obtaining the optimized target texture training set, the method further includes:
[0010] A loss function is used to measure the similarity between a reference matching training set and a target matching training set. Based on the similarity measurement result, an optimizer is used to optimize the weights of the deep learning multi-layer neural network to minimize the loss function.
[0011] In a possible embodiment, before inputting the reference matching training set and the target matching training set into the deep learning multi-layer neural network, the method further includes:
[0012] Establishing a deep learning multi-layer neural network, wherein the deep learning multi-layer neural network may be a deep convolutional neural network including a plurality of convolutional layers, activation layers, pooling layers, and fully connected layers; or, the multi-layer neural network is a Transformer neural network including a plurality of attention layers;
[0013] The deep learning multi-layer neural network is trained using a training set, and during the training process, hyperparameters are adjusted to improve the performance of the deep learning multi-layer neural network, and the performance of the deep learning multi-layer neural network is verified on a validation set, wherein the hyperparameters include but are not limited to a learning rate, a batch size, and the number of network layers.
[0014] In a possible embodiment, during the process of training the deep learning multi-layer neural network, the weights of the deep learning multi-layer neural network are updated according to the loss function in each training iteration.
[0015] In the second aspect, the present application also provides a 3D model automatic style conversion device, which includes a module / unit that executes any possible design method of the first aspect. These modules / units can be implemented by hardware, or by hardware executing corresponding software implementations.
[0016] In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory, wherein the memory is used to store one or more computer programs; when the one or more computer programs stored in the memory are executed by the processor, the electronic device can implement any possible design method of the first aspect.
[0017] In a fourth aspect, a computer-readable storage medium is also provided in an embodiment of the present application. The computer-readable storage medium includes a computer program. When the computer program runs on an electronic device, the electronic device executes any possible design method of the first aspect mentioned above.
[0018] In a fifth aspect, an embodiment of the present application further provides a method comprising a computer program product, which, when executed on an electronic device, enables the electronic device to execute any possible design of any of the above aspects.
[0019] For the beneficial effects of the second to fifth aspects, reference may be made to the description of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 Schematic diagram of the process of the automatic style conversion method of 3D model of the present invention.
[0021] Figure 2 Schematic diagram of the 3D model automatic style conversion device of the present invention.
[0022] Figure 3 It is a schematic structural diagram of the electronic device of the present invention. DETAILED DESCRIPTION
[0023] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Unless otherwise defined, the technical terms or scientific terms used herein should be understood by people with general skills in the field to which the present invention belongs. "Including" and similar words used in this article mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects.
[0024] Some technical terms in the present invention are explained below.
[0025] 1. UV is the abbreviation of u, v texture mapping coordinates (it is similar to the X, Y, and Z axes of the spatial model). It defines the position information of each point on the image. These points are interconnected with the 3D model to determine the position of the surface texture map. UV is to accurately correspond each point on the image to the surface of the model object. The gaps between points are processed by the software for smooth image interpolation. The "UV" in three-dimensional modeling can be understood as the "skin" of the three-dimensional model. The "skin" is unfolded and then drawn on a two-dimensional plane and assigned to the object. The texture coordinates usually have two coordinate axes, U and V, so they are called UV coordinates. U represents the distribution on the horizontal coordinate and V represents the distribution on the vertical coordinate.
[0026] 2. 2D is the abbreviation of "2 Dimensions" in English, which means two dimensions in Chinese. 2D is also called plane graphics. The content of 2D graphics only has the horizontal X-axis and the vertical Y-axis.
[0027] 3. 3D is the abbreviation of "3 Dimensions" in English. In Chinese, it means three dimensions, three coordinates, namely length, width and height. In other words, it is three-dimensional. Three dimensions refer to a space system formed by adding a direction vector to a two-dimensional plane system. Three dimensions are the three axes of the coordinate axis, namely the x-axis, y-axis and z-axis, where x represents the left and right space, y represents the front and back space, and z represents the up and down space, forming a three-dimensional visual sense for people.
[0028] In view of the problems existing in the prior art, an embodiment of the present invention provides a 3D model automatic style conversion method. Figure 1 This is a flow chart of the automatic style conversion method for 3D models of the present invention, see Figure 1 , the method comprises the following steps:
[0029] S101: constructing a reference texture training set of the reference game model according to the UV maps of the reference game model and the game model to be converted, dividing the reference texture training set into a plurality of different regions, and filling different regions with blocks of different colors.
[0030] S102: Apply the reference texture training set to the UV coordinates of the reference game model to generate a 3D model of the reference game, and generate a reference matching training set from the 3D model of the reference game, wherein the reference matching training set is a 2D reference image of different viewing angles.
[0031] S103: For each color block in the area, pre-generate a target texture training set of the game model to be converted, apply the target texture training set to the UV coordinates of the game model to be converted to generate a 3D model of the game to be converted, and generate a target matching training set from the 3D model of the game to be converted, wherein the target matching training set is a 2D target image that corresponds one-to-one to the 2D reference image.
[0032] S104: reducing the area of the reference texture training set divided region, and repeating the above steps until a target texture training set of a desired style is obtained.
[0033] In a preferred embodiment, after generating a target matching training set from the 3D model of the game to be converted, the method further includes: inputting the reference matching training set and the target matching training set into a deep learning multi-layer neural network to obtain an optimized target texture training set. In this embodiment, the target texture training set can be automatically optimized through the deep learning multi-layer neural network, thereby improving the accuracy and reliability of style conversion.
[0034] In a preferred embodiment, after obtaining the optimized target texture training set, the method further includes: using a loss function to measure the similarity between the reference matching training set and the target matching training set, and using an optimizer to optimize the weights of the deep learning multi-layer neural network based on the similarity measurement results to minimize the loss function. In this embodiment, the similarity between the generated reference matching training set and the target matching training set is measured and evaluated, and the generated target texture training set can be verified. Based on the similarity measurement results, the weights of the deep learning multi-layer neural network are optimized to ensure the accuracy and reliability of style conversion.
[0035] In a preferred embodiment, before inputting the reference matching training set and the target matching training set into the deep learning multi-layer neural network, the method further includes:
[0036] A deep learning multi-layer neural network is established, wherein the multi-layer neural network may be a deep convolutional neural network and includes multiple convolutional layers, activation layers, pooling layers, and fully connected layers; or, the multi-layer neural network is a Transformer neural network and includes multiple attention layers; the Transformer neural network abandons traditional convolutional neural networks (CNN) and recurrent neural networks (RNN), and the entire network structure is a combination of full connection (or one-dimensional convolution) plus attention.
[0037] The deep learning multi-layer neural network is trained using a training set, and during the training process, hyperparameters are adjusted to improve the performance of the deep learning multi-layer neural network, and the performance of the deep learning multi-layer neural network is verified on a validation set, wherein the hyperparameters include but are not limited to a learning rate, a batch size, and the number of network layers.
[0038] In this embodiment, the deep learning multi-layer neural network is established and trained, and the deep learning multi-layer neural network automatically identifies and learns the characteristics of different game styles to optimize the target texture training set. The performance of the deep learning multi-layer neural network is verified on the validation set to prevent overfitting.
[0039] In a specific embodiment, during the process of training the deep learning multi-layer neural network, the weights of the deep learning multi-layer neural network are updated according to the loss function in each training iteration.
[0040] In a specific embodiment, the trained deep learning multi-layer neural network is evaluated on the test set, and the performance indicators are calculated, including but not limited to accuracy, recall rate and F1 score. Among them, the F1 score is an indicator used in statistics to measure the accuracy of a binary classification model. It takes into account both the accuracy and recall rate of the classification model. The F1 score can be regarded as a harmonic mean of the model accuracy and recall rate, and its maximum value is 1 and its minimum value is 0. If the performance of the deep learning multi-layer neural network can meet the requirements, it can be applied to the actual 3D model style conversion task.
[0041] In a specific embodiment, the loss function may be a mean square error loss function, and the optimizer may be an Adam optimizer or an SGD optimizer. The Adam optimizer is a gradient-based adaptive optimizer that can adaptively adjust the learning rate of each parameter. Stochastic Gradient Descent (SGD) is a common optimizer and one of the most basic optimization algorithms in deep learning models. It is an implementation of the gradient descent algorithm and is often used for weight updates in neural networks.
[0042] In a preferred embodiment, after generating a reference matching training set from a 3D model of a reference game, the method further includes: for a movable model, generating a skeleton of a game model to be converted according to the skeleton and skeleton parameters of the reference game model;
[0043] After obtaining the target texture training set of the required style, it also includes: applying the target texture training set of the required style to the UV coordinates of the game model to be converted to generate a UV map of the game model to be converted after the style is converted, adding the skeleton of the game model to be converted to the UV map of the game model to be converted after the style is converted, and obtaining a movable game model to be converted, thereby realizing the motion effect of the skeleton-driven model moving.
[0044] When training deep learning multi-layer neural networks, skeleton-related training content is added to ensure that the model supports motion scenarios.
[0045] In this embodiment, when the style of game A needs to be transferred to game B, the UV map of the entity of game A and the UV map of the entity of game B can be used to automatically generate a target texture training set of the style of game B through the 3D model automatic style transfer algorithm of the present invention.
[0046] The 3D model automatic style conversion method of the present invention proposes a 3D model style conversion and multi-view 2D image matching algorithm based on UV mapping. According to the UV mapping of the reference game model and the game model to be converted, a reference texture training set is constructed, and a reference matching training set and a target matching training set are generated. There is a one-to-one mapping relationship between the 2D reference image in the reference matching training set and the 2D target image in the target matching training set. Through the similar matching of the 2D target image and the corresponding 2D reference image in two dimensions, it can be confirmed that the 3D model of the game to be converted generated according to the target texture training set of the required style will also be similarly matched with the 3D model of the reference game generated according to the reference texture training set in three-dimensional space, that is, the style conversion of the 3D model of the game to be converted is realized. The present invention uses deep learning technology to automatically identify and learn the characteristics of different game styles, realize the automatic conversion and migration of 3D game entities between different game styles, improve the efficiency of model style conversion, and does not require manual creation of new models, which is more intelligent and convenient.
[0047] In addition, this application also proposes a 3D model automatic style conversion device. Figure 2 is a schematic diagram of the 3D model automatic style conversion device of the present invention, see Figure 2 The device includes: a reference texture training set construction unit 201, which is used to construct a reference texture training set of the reference game model according to the UV maps of the reference game model and the game model to be converted, divide the reference texture training set into a plurality of different regions, and fill the different regions with blocks of different colors; a reference matching training set generation unit 202, which is used to apply the reference texture training set to the UV coordinates of the reference game model to generate a 3D model of the reference game, and generate a reference matching training set from the 3D model of the reference game, wherein the reference matching training set is a 2D reference image of different perspectives; a target matching training set generation unit 203, which is used to pre-generate a target texture training set of the game model to be converted for each block of the region, apply the target texture training set to the UV coordinates of the game model to be converted to generate a 3D model of the game to be converted, and generate a target matching training set from the 3D model of the game to be converted, wherein the target matching training set is a 2D target image that corresponds one-to-one to the 2D reference image; a divided region reduction unit 204, which is used to reduce the area of the divided region of the reference texture training set and trigger the reference texture training set construction unit. All relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here.
[0048] In other embodiments of the present application, the present application discloses an electronic device, Figure 3 FIG. 1 is a schematic diagram of the structure of the electronic device of the present invention, see Figure 3 , the electronic device may include: one or more processors 301; memory 302; display 303; one or more applications (not shown); and one or more computer programs 304, and the above components may be connected via one or more communication buses 305. The one or more computer programs 304 are stored in the above memory 302 and configured to be executed by the one or more processors 301, and the one or more computer programs 304 include instructions, which can be used to execute the following: Figure 1 and Figure 2 And each step in the corresponding embodiment.
[0049] Through the description of the above implementation methods, technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0050] Each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units.
[0051] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as flash memory, mobile hard disk, read-only memory, random access memory, disk or optical disk.
[0052] Although the embodiments of the present invention are described in detail above, it is obvious to those skilled in the art that various modifications and variations can be made to these embodiments. However, it should be understood that such modifications and variations are within the scope and spirit of the present invention as described in the claims. Moreover, the present invention described herein may have other embodiments and may be implemented or realized in a variety of ways.
Claims
1. A 3D model automatic style transfer method, It is characterized in that The method comprises: According to the UV maps of the reference game model and the game model to be converted, a reference texture training set of the reference game model is constructed, the reference texture training set is divided into a plurality of different regions, and different regions are filled with blocks of different colors; Applying the reference texture training set to the UV coordinates of the reference game model to generate a 3D model of the reference game, and generating a reference matching training set from the 3D model of the reference game, wherein the reference matching training set is a 2D reference image of different viewing angles; For each color block in the region, a target texture training set of the game model to be converted is pre-generated, the target texture training set is applied to the UV coordinates of the game model to be converted to generate a 3D model of the game to be converted, and the 3D model of the game to be converted is used to generate a target matching training set, wherein the target matching training set is a 2D target image that corresponds one-to-one to the 2D reference image; The area of the reference texture training set divided region is reduced, and the above steps are repeated until a target texture training set of a desired style is obtained.
2. The method according to claim 1, It is characterized in that After generating the target matching training set from the 3D model of the game to be converted, it also includes: The reference matching training set and the target matching training set are input into the deep learning multi-layer neural network to obtain the optimized target texture training set.
3. The method according to claim 2, It is characterized in that After obtaining the optimized target texture training set, it also includes: A loss function is used to measure the similarity between a reference matching training set and a target matching training set. Based on the similarity measurement result, an optimizer is used to optimize the weights of the deep learning multi-layer neural network to minimize the loss function.
4. The method according to claim 2, It is characterized in that Before inputting the reference matching training set and the target matching training set into the deep learning multi-layer neural network, it also includes: Establishing a deep learning multi-layer neural network, wherein the multi-layer neural network may be a deep convolutional neural network and includes a plurality of convolutional layers, activation layers, pooling layers, and fully connected layers; or, the multi-layer neural network is a Transformer neural network and includes multiple attention layers; The deep learning multi-layer neural network is trained using a training set, and during the training process, hyperparameters are adjusted to improve the performance of the deep learning multi-layer neural network, and the performance of the deep learning multi-layer neural network is verified on a validation set, wherein the hyperparameters include but are not limited to a learning rate, a batch size, and the number of network layers.
5. A 3D model automatic style conversion device, It is characterized in that The device comprises: A reference texture training set construction unit, used to construct a reference texture training set of the reference game model according to the UV maps of the reference game model and the game model to be converted, divide the reference texture training set into a plurality of different areas, and fill different areas with blocks of different colors; A reference matching training set generating unit, configured to apply the reference texture training set to the UV coordinates of the reference game model to generate a 3D model of the reference game, and to generate a reference matching training set from the 3D model of the reference game, wherein the reference matching training set is a 2D reference image of different viewing angles; a target matching training set generating unit, for pre-generating a target texture training set of the game model to be converted for each color block in the region, applying the target texture training set to the UV coordinates of the game model to be converted to generate a 3D model of the game to be converted, and generating a target matching training set from the 3D model of the game to be converted, wherein the target matching training set is a 2D target image that corresponds one-to-one to the 2D reference image; The divided area reduction unit is used to reduce the area of the divided area of the reference texture training set and trigger the reference texture training set construction unit.
6. The device according to claim 5, It is characterized in that Also includes: The optimization unit is used to input the reference matching training set and the target matching training set into the deep learning multi-layer neural network to obtain an optimized target texture training set.
7. The device according to claim 6, It is characterized in that Also includes: The similarity measurement unit is used to obtain the optimized target texture training set, and then use the loss function to measure the similarity between the reference matching training set and the target matching training set. According to the similarity measurement result, the optimizer is used to optimize the weights of the deep learning multi-layer neural network to minimize the loss function.
8. The device according to claim 6, It is characterized in that Also includes: A neural network building unit, used to build a deep learning multi-layer neural network, wherein the multi-layer neural network may be a deep convolutional neural network and include multiple convolutional layers, activation layers, pooling layers, and fully connected layers; or, the multi-layer neural network is a Transformer neural network and includes multiple attention layers; A neural network training unit is used to train the deep learning multi-layer neural network using a training set, improve the performance of the deep learning multi-layer neural network by adjusting hyperparameters during the training process, and verify the performance of the deep learning multi-layer neural network on a validation set, wherein the hyperparameters include but are not limited to a learning rate, a batch size, and the number of network layers.
9. An electronic device, It is characterized in that include: A processor and a memory, wherein the memory is used to store a computer program; The processor is configured to execute the computer program stored in the memory so as to enable the electronic device to perform the method according to any one of claims 1 to 4.
10. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.