A method and apparatus for enabling instantiation by replacing similar assets

Abstract

The present application provides a method for enabling instantiation of similar asset replacement, including: traversing static meshes in a scene, generating different 2D images based on 14 different angles, then using ResNet for image feature extraction and clustering to automatically complete the recognition of similar assets; counting the assets with the most references in the same category, and then replacing other assets with them; calculating the most similar angle of the asset to be replaced according to the asset before replacement for placement. By replacing assets, the present application can achieve asset instantiation rendering and reduce the DrawCall generated in scene production.

Description

Technical Field

[0001] The present application relates to the field of scene instantiation, and more particularly to a method for similar asset replacement to enable instantiation. The present application also relates to a device for similar asset replacement to enable instantiation. Background Art

[0002] The game scene production process is:

[0003] Concept design is used to determine the overall layout of the scene design atmosphere, reducing unnecessary early refinement and modification. After the concept design is approved, the line draft will generally be refined, down to the detailed modeling design, to facilitate more accurate model production and restoration. Finally, the final draft will be refined and colored.

[0004] 3D model production, model production is carried out through precise game camera angles to achieve accurate game perspective angles, improve modeling restoration, use material balls to express the required texture, use lighting to set off the local atmosphere and express delicate light sources and shadows. Generally, vray renderer is used for rendering, which will be softer.

[0005] Integration editing is the link that presents the final game screen, so it is responsible for the final effect, mainly showing the connection and enrichment of details, the setting of environmental color details and light, and the final expression of the overall atmosphere. The editing content of different types of scenes is different. The outdoor mountain scenes are mainly matched with plants and landscaping, while the indoor scenes are mainly light.

[0006] Currently, there are two situations in scene production:

[0007] The first is that, for the sake of scene execution efficiency, developers are very restrictive in the use of scene assets, limiting the diversity and specifications of assets, resulting in the inability to display good results.

[0008] The second is that in order to create a richer and more diverse effect, developers will relax the constraints on assets. The consequence of this is that there may be many visually similar but different mesh art assets.

[0009] When the second situation occurs, these assets are treated as different meshes in the engine rendering pipeline, and instanced rendering cannot be enabled, resulting in high DrawCall costs.

[0010] In addition, since these assets need to be manually selected and placed, the production workload is large and the efficiency is low. Existing technical solutions do not effectively solve these problems. Summary of the invention

[0011] The purpose of this application is to overcome the problem of high DrawCall in scene production in the prior art, and provide a method for enabling instantiation by replacing similar assets. This application also relates to a device for enabling instantiation by replacing similar assets.

[0012] This application provides a method for enabling instantiation by replacing similar assets, including:

[0013] Traverse the static meshes in the scene, generate different 2D images based on 14 different angles, and then use ResNet for image feature extraction and clustering to automatically complete the recognition of similar assets;

[0014] Count the asset with the most references in the same category, and then replace other assets with it;

[0015] Place the asset according to the most similar angle of the asset to be replaced calculated based on the asset before replacement.

[0016] Optionally, the static mesh includes: components of a geometric body composed of polygons.

[0017] Optionally, a CS architecture is adopted, where the UE plugin itself is regarded as the client, and the server scene data is used for calculation.

[0018] Optionally, the server includes:

[0019] Python backend: Implement similarity calculation by combining blender and torch vgg;

[0020] C++ backend: Implement similarity calculation by combining libtorch and opengl, and add multi-task concurrent calculation.

[0021] Optionally, in the CS architecture, GRPC standard protocol is used for communication.

[0022] This application also provides a device for enabling instantiation by replacing similar assets, including:

[0023] An identification module, configured to traverse the static meshes in the scene, generate different 2D images based on 14 different angles, and then use ResNet for image feature extraction and clustering to automatically complete the recognition of similar assets;

[0024] A replacement module, configured to count the asset with the most references in the same category, and then replace other assets with it;

[0025] A placement module, configured to place the asset according to the most similar angle of the asset to be replaced calculated based on the asset before replacement.

[0026] Optionally, the static mesh includes: components of a geometry composed of polygons.

[0027] Optionally, in the CS architecture, the UE plugin itself is regarded as the client, and the server scene data is used for calculation.

[0028] Optionally, the server includes:

[0029] Python backend: Implement similarity calculation by combining Blender and torch VGG;

[0030] C++ backend: Implement similarity calculation by combining libtorch and OpenGL, and add multi-task concurrent calculation.

[0031] Optionally, in the CS architecture, GRPC standard protocol is used for communication.

[0032] Advantages and beneficial effects of this application:

[0033] This application provides a method for enabling instantiation of similar asset replacement, including: traversing static meshes in the scene, generating different 2D images based on 14 different angles, then using ResNet for image feature extraction and clustering to automatically complete the recognition of similar assets; counting the assets with the most references in the same category, and then replacing other assets with it; calculating the most similar angle of the asset to be replaced according to the asset before replacement and placing it. By replacing assets, this application can achieve asset instantiation rendering and reduce DrawCall generated in scene production. Description of the Drawings

[0034] Figure 1 is a schematic diagram of enabling instantiation of similar asset replacement in this application.

[0035] Figure 2 is a schematic diagram of the residual neural structure in this application.

[0036] Figure 3 is a schematic diagram of the C / S call process in this application.

[0037] Figure 4 is a schematic diagram of the Server processing process in this application.

[0038] Figure 5 is a schematic diagram of the device for enabling instantiation of similar asset replacement in this application. Detailed Implementation Manner

[0039] The following further describes this application with reference to the drawings and specific embodiments, so that those skilled in the art can better understand this application and implement it.

[0040] The following content is an example of the specific implementation process provided to elaborate on the technical solution to be protected by this application. However, this application can also be implemented in other ways different from the described ones. Those skilled in the art can implement this application using different technical means under the guidance of the concept of this application. Therefore, this application is not limited by the following specific embodiments.

[0041] This application relates to the field of scene instantiation. The technical problem solved by this application is to achieve asset instantiation rendering and reduce the DrawCall generated in scene production.

[0042] The overall idea of this application is:

[0043] Use the Asset (asset) most frequently referenced by StaticMeshActor in the scene to replace the Asset with fewer references. Its implementation mainly utilizes blender + torch. After rendering into pictures, calculate the picture similarity, find the most similar Asset, and replace the Asset according to the calculated Translate, Rotation, and Scale parameters, tightening the Asset set in the scene, thereby achieving improved rendering performance.

[0044] This application provides a method for enabling instantiation by replacing similar assets, including:

[0045] Traverse the static meshes in the scene, generate different 2D pictures based on 14 different angles, and then use ResNet for picture feature extraction and clustering to automatically complete the identification of similar assets;

[0046] Count the asset with the most references in the same category, and then replace other assets with it;

[0047] Calculate the most similar angle of the asset to be replaced according to the asset before replacement, and place it.

[0048] By replacing assets, this application can achieve asset instantiation rendering and reduce the DrawCall generated in scene production.

[0049] Figure 1 It is a schematic diagram of enabling instantiation by replacing similar assets in this application.

[0050] Please refer to Figure 1 As shown, in S101, traverse the static meshes in the scene, generate different 2D pictures based on 14 different angles, and then use ResNet for picture feature extraction and clustering to automatically complete the identification of similar assets.

[0051] The Static Mesh described in this application is a basic unit in the Unreal Engine, used to create the geometry of a scene, which is a component of the geometry composed of a series of polygons. The Static Mesh can be cached in the video memory and further rendered using the graphics card to generate the scene image.

[0052] Since the Static Mesh is cached in the video memory, it can be transformed such as translated, rotated, and scaled, but animations cannot be set for the vertices of the Static Mesh.

[0053] By traversing to obtain all the Static Meshes, and then using the Static Meshes as the targets, different 2D pictures are generated at 14 different angles to generate the picture data processed by the neural network.

[0054] Based on the ResNet network, the 2D pictures are input, feature extraction and clustering are performed to obtain the set of 2D pictures based on feature clustering.

[0055] As Figure 2 shown, the ResNet network (Residual Neural Network) includes a sub-grid composed of two processing branches, namely the main branch and the branch. The sub-grids are stacked to produce the ResNet network.

[0056] Based on the extraction of 2D image features by the ResNet network, clustering is performed based on the extracted features to obtain the clustered features.

[0057] In this application, the clustering includes: splitting a data set into different classes or clusters according to a set standard. Among them, the similarity of data objects within the same cluster is as large as possible, and the difference of data objects not in the same cluster is also as large as possible.

[0058] This application adopts a clustering method as follows: assign values to the features and calculate the similarity based on different features:

[0059]

[0060] Among them, the i A i and max B max are respectively the i-th of the n features extracted from two of the 2D pictures. When the max i max < n, the feature values from i max to n are 0.

[0061] After the above comparison, when the

[0061] S = 0, the two pictures are exactly the same, otherwise further calculations are performed:

[0062]

[0063] The "T" i refers to each category of the clustering. The "min~min" x refers to selecting the features corresponding to the values from the minimum value to the pre-trial threshold. The

[0064] difference of each feature of two 2D images.

[0065] Finally, asset identification is performed based on the features after clustering.

[0066] Please refer to Figure 1 as shown. S102 counts the asset with the most citation times under the same category, and then replaces other assets with it.

[0067] Specifically, an asset using another asset in a certain way is the asset reference (or having a reference) described in this application.

[0068] For example: If a cube Actor uses a color material, then the Actor references the material.

[0069] The meaning of the "asset" is: texture, mesh, blueprint, and other similar data. Further, files with the suffixes of uasset and umap belong to assets.

[0070] In this application, the "asset" refers to the above-mentioned static meshes, and other data for rendering.

[0071] The above has completed the asset identification after clustering based on the 2D image features. In this step, first, according to the categories of the clustering, the number of asset citations is counted, and the asset with the most citation times is selected based on the number of times.

[0072] To view the references of an asset, in one implementation, click on the asset in the Content Browser of the Unreal Engine. Then, from the context menu that appears, select the reference viewing interface. A new window will open, showing the visual effect of the asset reference.

[0073] Preferably, this application directly reads the references using computer software.

[0074] Please refer to Figure 1 as shown. S103 calculates the most similar angle of the asset to be replaced according to the asset before replacement, and arranges it.

[0075] Replace the asset with the most citation times with other assets. Developers do not need to manually select and arrange a large number of similar assets.

[0076] This application can achieve instantiated rendering by replacing assets with similar ones, treating visually similar but different meshes as the same assets, thus significantly improving the rendering efficiency.

[0077] Please refer to Figure 3 、 Figure 4 As shown, based on the above method and the corresponding designed software, its scene optimization service can adopt the CS architecture. The UE plugin itself is regarded as the Client side, and the main scene data of the Server side is used for calculation, including two versions:

[0078] Python backend: Combining blender and torch vgg to achieve similarity calculation, currently only supporting single-task calculation.

[0079] C++ backend: Combining libtorch and opengl to achieve similarity calculation. Compared with the Python backend, it has added multi-task concurrent calculation and improved calculation performance.

[0080] CS communication adopts the GRPC standard protocol.

[0081] This application also provides a device for enabling instantiation of similar asset replacement, including:

[0082] Recognition module 301, used to traverse the static meshes in the scene, generate different 2D images based on 14 different angles, and then use ResNet for image feature extraction and clustering to automatically complete the recognition of similar assets;

[0083] Replacement module 302, used to count the asset with the most reference times in the same category, and then replace other assets with it;

[0084] Placement module 303, used to calculate the most similar angle of the asset to be replaced according to the asset before replacement and place it.

[0085] Figure 5 It is a schematic diagram of the device for enabling instantiation of similar asset replacement in this application.

[0086] Please refer to Figure 5 As shown, in the recognition module 301:

[0087] The static mesh described in this application is a basic unit in the Unreal Engine, used to create the geometry of the scene and is a component of the geometry composed of a series of polygons. The static mesh can be cached in the video memory and further rendered using the graphics card to generate the scene image.

[0088] The static mesh is cached in the video memory, so it can be transformed such as translated, rotated, and scaled, but animations cannot be set for the vertices of the static mesh.

[0089] All the static meshes are obtained by traversal, and then, taking the static meshes as the targets, different 2D pictures are generated at 14 different angles to generate picture data for neural network processing.

[0090] Based on the ResNet network, the 2D pictures are input, feature extraction and clustering are performed, and a set of 2D pictures based on feature clustering is obtained.

[0091] As Figure 2 shown, the ResNet network (residual neural network) includes a sub-mesh composed of two processing branches, namely the main branch and the branch. The sub-meshes are stacked to produce the ResNet network.

[0092] Based on the extraction of 2D image features by the ResNet network, clustering is performed based on the extracted features to obtain the clustered features.

[0093] In this application, the clustering includes: splitting a data set into different classes or clusters according to a set criterion. Among them, the similarity of data objects within the same cluster is as large as possible, and the difference between data objects not in the same cluster is also as large as possible.

[0094] This application adopts a clustering method as follows: assign values to the features and calculate the similarity based on different features:

[0095]

[0096] Among them, the A i , B i are respectively the i-th of n features extracted from two of the 2D pictures. When the i max < n, the feature values of the i max to n are 0.

[0097] After the above comparison, when S = 0, the two pictures are exactly the same, otherwise further calculations are performed:

[0098]

[0099] The T i refers to each class of the clustering, and min~min x refers to selecting the features corresponding to the values from the minimum value to the pre-reviewed threshold. The

[0100] The difference of each feature between two 2D images.

[0101] Finally, assets are identified based on the clustered features.

[0102] Please refer to Figure 5 as shown, in the replacement module 302:

[0103] Specifically, when one asset uses another asset in a certain way, it is the asset reference (or has a reference) described in this application.

[0104] For example: If a cube Actor uses a color material, then the Actor references the material.

[0105] The meaning of the asset is: texture, mesh, blueprint, and other similar data. Further, files with the suffixes uasset and umap belong to assets.

[0106] In this application, the asset refers to the above-mentioned static meshes and other data for rendering.

[0107] For the assets identified after clustering based on the 2D image features, in this step, first, according to the categories of the clustering, count the number of times the assets are referenced, and filter out the asset with the most reference times based on the number of times.

[0108] To view the references of an asset, one implementation is to click on the asset in the Content Browser of the Unreal Engine. Then, from the context menu that appears, select the reference viewing interface. A new window will open, showing the visual effect of the asset references.

[0109] Preferably, this application directly reads the references using computer software.

[0110] Please refer to Figure 5 as shown, in the placement module 303:

[0111] Replace the asset with the most reference times with other assets. Developers do not need to manually select and place a large number of similar assets.

[0112] This application can regard visually similar but different meshes as the same asset through the replacement of similar assets, thus realizing instance rendering, and the rendering efficiency is significantly improved.

[0113] Please refer to Figure 3 、 Figure 4 as shown, based on the above method and the corresponding designed software, its scene optimization service can adopt the CS architecture, where the UE plugin itself is regarded as the Client side (client), and the Server side (server) mainly uses the scene data for calculation, including two versions:

[0114] Python backend: Implement similarity calculation by combining Blender and Torch VGG. Currently, only single-task calculation is supported.

[0115] C++ backend: Implement similarity calculation by combining LibTorch and OpenGL. Compared with the Python backend, multi-task concurrent calculation is added, and the calculation performance is improved.

[0116] CS communication adopts the GRPC standard protocol.

[0117] It should be noted that although the above device only shows the recognition module 301, the replacement module 302, and the placement module 303, in the specific implementation process, the device may also include other components necessary for normal operation.

[0118] In addition, those skilled in the art can understand that the above device may also only include the components necessary to implement the solution of the embodiments of this specification, and do not have to include all the components shown in the figure.

[0119] The specific embodiments of this specification have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve the desired result.

[0120] In addition, the processes depicted in the figures do not necessarily require the specific order or sequential order shown to achieve the desired result. In certain embodiments, multi-tasking and parallel processing are also possible or may be advantageous.

[0121] One or more embodiments of this specification are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims.

[0122] Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of this specification shall be included in the protection scope of this disclosure.

Claims

1. A method for enabling instantiation by replacing similar assets, characterized in that, It includes: Traverse the static meshes in the scene, generate different 2D images based on 14 different angles, then use ResNet for image feature extraction and clustering to automatically complete the recognition of similar assets; Count the assets with the most citation times in the same category, and then replace other assets with it; Place the assets according to the most similar angle of the assets to be replaced calculated based on the assets before replacement.

2. The method for enabling instantiation by replacing similar assets according to claim 1, characterized in that, The static meshes include: components of a geometric body composed of polygons.

3. The method for enabling instantiation by replacing similar assets according to claim 1, characterized in that, Among them, Adopt the CS architecture, where the UE plugin itself is regarded as the client, and the server scene data is used for calculation.

4. The method for enabling instantiation by replacing similar assets according to claim 3, characterized in that, The server includes: Python backend: Implement similarity calculation by combining blender and torch vgg; C++ backend: Implement similarity calculation by combining libtorch and opengl, and add multi-task concurrent calculation.

5. The method for enabling instantiation by replacing similar assets according to claim 3, characterized in that, In the CS architecture, GRPC standard protocol is used for communication.

6. A device for enabling instantiation by replacing similar assets, characterized in that, It includes: Recognition module, which is used to traverse the static meshes in the scene, generate different 2D images based on 14 different angles, then use ResNet for image feature extraction and clustering to automatically complete the recognition of similar assets; Replacement module, which is used to count the assets with the most citation times in the same category, and then replace other assets with it; Placement module, which is used to place the assets according to the most similar angle of the assets to be replaced calculated based on the assets before replacement.

7. The device for enabling instantiation by replacing similar assets according to claim 6, characterized in that, The static meshes include: components of a geometric body composed of polygons.

8. The device for enabling instantiation by replacing similar assets according to claim 6, characterized in that, Among them, Adopt the CS architecture, where the UE plugin itself is regarded as the client, and the server scene data is used for calculation.

9. The device for enabling instantiation by replacing similar assets according to claim 8, characterized in that, The server includes: Python backend: Implement similarity calculation by combining blender and torch vgg; C++ backend: Implement similarity calculation by combining libtorch and opengl, and add multi-task concurrent calculation.

10. The device for enabling instantiation by replacing similar assets according to claim 8, characterized in that, In the CS architecture, GRPC standard protocol is used for communication.

Images