Renderer material intercommunication method, device and equipment based on deep learning

Abstract

The present disclosure provides a renderer material intercommunication method, device and equipment based on deep learning. The present disclosure relates to the field of computer technology, and in particular to the field of data processing and three-dimensional design software technology. The method comprises: utilizing an image encoder of a material conversion model to obtain image features based on a map containing a target material and a corresponding rendering; utilizing a text encoder of the material conversion model to obtain text features based on a text instruction containing instruction information; utilizing a connection layer of the material conversion model to fuse the image features and the text features to obtain fused features; utilizing a decoder of the material conversion model to generate a structured material file containing the material information of the target material and the path information of the map based on the fused features. According to the present disclosure, the target material of a first material system can be converted to a second material system to achieve material intercommunication.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to the field of data processing and three-dimensional design software technology. Background Art

[0002] At present, various mainstream rendering engines usually have their own rendering material systems, and each is independent. There are solid barriers between different material systems, and it is difficult to achieve simple conversion using a regularized approach. For example, the material materials generated by the VRay rendering engine cannot be used directly in the C4D renderer and often require manual conversion by the creator. Summary of the Invention

[0003] The present disclosure provides a renderer material intercommunication method, apparatus, device, and storage medium based on deep learning to solve or alleviate one or more technical problems in the prior art.

[0004] In a first aspect, the present disclosure provides a renderer material interoperability method based on deep learning, comprising:

[0005] Using the image encoder of the material conversion model, the image features are obtained based on the texture containing the target material and the corresponding rendering;

[0006] Using the text encoder of the material conversion model, a text feature is obtained according to the text instruction containing the instruction information;

[0007] Using the connection layer of the material conversion model, the image feature and the text feature are fused to obtain a fused feature;

[0008] A structured material file including material information of the target material and path information of the texture is generated according to the fusion feature using a decoder of the material conversion model.

[0009] In a second aspect, the present disclosure provides a method for training a material conversion model, comprising:

[0010] Obtaining a training data set, wherein the training data set includes multiple sets of sample maps containing sample materials, corresponding renderings, and target results of structured material files to be obtained;

[0011] Using an image encoder of a material conversion model, feature extraction is performed on the sample texture and the sample rendering to obtain an image feature vector;

[0012] Using the text encoder of the material conversion model, a text feature vector is obtained according to the text instruction containing the instruction information;

[0013] Using the connection layer of the material conversion model, the image feature vector and the text feature vector are subjected to feature fusion to generate a fused feature vector;

[0014] Inputting the fused feature vector into the decoder of the material conversion model and outputting the prediction result of the structured material file;

[0015] According to the loss value between the predicted result and the target result, the model parameters of the material conversion model are updated to obtain a trained material conversion model.

[0016] In a third aspect, the present disclosure provides a renderer material intercommunication device based on deep learning, comprising

[0017] An image encoding module, configured to utilize an image encoder of a material conversion model to obtain image features based on a texture containing a target material and a corresponding rendering;

[0018] a text encoding module, configured to obtain text features according to text instructions containing instruction information using a text encoder of the material conversion model;

[0019] a fusion module, configured to fuse the image features and the text features using a connection layer of the material conversion model to obtain a fused feature;

[0020] A generation module is used to generate a structured material file containing material information of the target material and path information of the map according to the fusion feature by using the decoder of the material conversion model.

[0021] In a fourth aspect, the present disclosure provides a training device for a material conversion model, comprising:

[0022] An acquisition module is used to acquire a training data set, wherein the training data set includes multiple sets of sample maps containing sample materials, corresponding renderings, and target results of structured material files to be obtained;

[0023] An image feature extraction module is used to extract features from the sample texture and the sample rendering using an image encoder of a material conversion model to obtain an image feature vector;

[0024] A text feature extraction module, configured to obtain a text feature vector according to a text instruction containing instruction information using a text encoder of the material conversion model;

[0025] A feature fusion module, configured to perform feature fusion on the image feature vector and the text feature vector using the connection layer of the material conversion model to generate a fused feature vector;

[0026] A prediction module, configured to input the fused feature vector into a decoder of the material conversion model and output a prediction result of a structured material file;

[0027] A training module is used to update the model parameters of the material conversion model according to the loss value between the prediction result and the target result to obtain a trained material conversion model.

[0028] According to a fifth aspect, an electronic device is provided, including:

[0029] at least one processor; and

[0030] a memory communicatively connected to the at least one processor; wherein,

[0031] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any method in the embodiments of the present disclosure.

[0032] In a sixth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0033] In a seventh aspect, a computer program product is provided, comprising a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0034] The beneficial effects of the technical solution provided by the present disclosure include at least: material intercommunication can be achieved between renderers of different material systems.

[0035] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar components or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments provided in accordance with the present disclosure and should not be regarded as limiting the scope of the present disclosure.

[0037] Figure 1 This is a flowchart of a renderer material intercommunication method based on deep learning provided according to an embodiment of the present disclosure;

[0038] Figure 2 is a structural block diagram of a material conversion model provided according to an embodiment of the present disclosure;

[0039] Figure 3 is a flowchart of a method for training a material conversion model according to an embodiment of the present disclosure;

[0040] Figure 4 It is a structural diagram of a renderer material intercommunication device based on deep learning provided by the present disclosure;

[0041] Figure 5 is a structural diagram of a training device for a material conversion model provided by the present disclosure;

[0042] Figure 6 is a block diagram of an electronic device for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0043] The present disclosure will be described in further detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

[0044] In addition, numerous specific details are provided in the following detailed description to better illustrate the present disclosure. Those skilled in the art will appreciate that the present disclosure can be practiced without certain specific details. In some instances, methods, means, components, circuits, etc. well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present disclosure.

[0045] In related technologies, material parameters for different renderers are not based on a unified standard design. Directly uploading a model from one renderer to another will render the material ineffective and the model may turn white. Methods like Generating Procedural Materials from Text or Image Prompts (GPMTI) are unable to reproduce image details, resulting in poor modeling material performance due to a lack of detail.

[0046] Figure 1 This is a flow chart of a renderer material intercommunication method based on deep learning provided by an embodiment of the present disclosure. Figure 1 As shown, the method comprises at least the following steps:

[0047] S110 , using an image encoder of a material conversion model to obtain image features according to a texture containing a target material and a corresponding rendering.

[0048] In the disclosed embodiments, a rendered image can be rendered in a first rendering engine using a texture containing a target material. During rendering, the texture can be applied to any model, such as a material ball model. The first rendering engine uses a first material system, and the target material also belongs to the first material system. The first material system can be understood as any material system that is not interoperable with the target material system, or a material system that cannot be directly used by the target rendering engine.

[0049] A texture is image data applied to the surface of a 3D model, primarily used to simulate the visual appearance of a material (such as color, bumps, and transparency). Common texture types include diffuse, normal, roughness, metalness, and ambient occlusion.

[0050] The Diffuse Map is used to define the surface color of an object.

[0051] Normal Map is used to simulate surface bump details.

[0052] Metallic Map is used to define the metallic properties of a material.

[0053] Roughness Map is used to control the roughness of the material surface.

[0054] The attribute map clearly defines the pixel-level numerical distribution of each channel (for example, the XYZ vector of each pixel in the normal map directly indicates the direction of convexity and concavity). The absolute values ​​of the core properties of the material (metallicity / roughness, etc.) are stored in the map, and the rendering is the result of the lighting calculation (for example, the specular highlight area is the result of the light product of the metalness and roughness parameters). In other words, the rendering is obtained by combining all the mapping parameters and lighting calculations. The rendering contains material details including visual features such as highlight areas, concave and convex structures, and shadow interactions. Multi-channel map input (such as using the Normap Map as a separate input channel) provides fine-grained data to make up for the physical information that may be mixed and compressed in the rendering.

[0055] The rendered image and texture are fed into the pre-trained material conversion model's image encoder. Convolutional and pooling layers extract a multidimensional image feature vector, which the material conversion model then analyzes to derive material information. The rendered image captures the material's overall appearance under specific lighting and viewing angles (e.g., highlight placement reflects roughness, and shadow transitions reveal normal details). The image encoder extracts high-level semantic features (e.g., the distribution of metallic reflections on an object's surface) through convolutional layers, establishing a correspondence between visual features and physical properties.

[0056] S120 , using a text encoder of a material conversion model to obtain text features according to a text instruction containing instruction information.

[0057] Construct text instructions that contain instructions such as task goals (such as material parameter conversion), output requirements (such as material file format), background information (such as target rendering engine type), etc.

[0058] refer to Figure 2 As shown, the text instruction Text instruct is input into the text encoder TextEncoder of the material conversion model for encoding, and a text feature vector is output.

[0059] S130. Utilize the connection layer of the material conversion model to fuse the image features and the text features to obtain fused features.

[0060] The text unit sequence in the image feature is paired with the instruction sequence in the text feature. The paired features are fused using a cross-attention mechanism to generate a fused feature vector.

[0061] S140 , using the decoder of the material conversion model to generate a structured material file including material information of the target material and path information of the texture according to the fusion features.

[0062] The fused features are fed into a Transformer-based decoder to generate an initial sequence. The grammar rule state machine is applied to insert material parameters or texture paths based on the current generation state. For example, when the state machine is in the texture path field, the texture path extracted in step S101 is forcibly inserted.

[0063] Structured material files can be generated based on a target format, such as the Physically Based Rendering (PBR) material format. PBR materials are currently the mainstream material format for 3D design, so PBR materials can be used as the final conversion target, i.e., the second material system.

[0064] After obtaining the structured material file, the generated structured material file can be imported into the target rendering engine, and the path information and material information of the texture in the material file can be parsed to achieve material reuse across rendering engines.

[0065] According to the solution of the embodiment of the present disclosure, the material conversion model can be used to deconstruct the visual information and underlying texture data of the rendering image, map them to PBR parameters, and obtain a structured material file of the target material, thereby converting the target material of the first material system into the second material system.

[0066] In one possible implementation, step S140 utilizes the decoder of the material conversion model to generate a structured material file containing the material information of the target material and the path information of the texture according to the fusion features, further comprising the steps of:

[0067] S141. Input the fused features into the decoder of the material conversion model to obtain a generated sequence output by the decoder.

[0068] S142. Using the grammar rule state machine, according to the grammar state of the current generation sequence, insert the material information of the target material or the path information of the texture to obtain a structured material file of the target material.

[0069] In the disclosed embodiment, the decoder of the material conversion model can be an encoder based on the visual transformer (ViT). ViT divides the input image into image blocks (Patch) of fixed size. Each Patch is mapped to a vector through a linear layer to form a set of token sequences. The token sequence is processed through multi-head self-attention (Multi-head Self-Attention) and a feed-forward network (FFN), and the final output still maintains the sequence structure.

[0070] The Grammar Rule State Machine (GRSM) is primarily used to ensure that output field information conforms to pre-defined grammatical and logical rules. This GRSM can be a grammar rule library built on PBR materials, defining constraints such as JSON key-value structures, data types, and nesting levels. A finite state automaton is generated to describe legal grammar paths. During the decoder's sequence output, grammar rules are applied in real time, and a masking mechanism is used to filter out invalid token candidates to select the optimal token.

[0071] According to the solution of the embodiment of the present disclosure, the grammatical correctness of the material structured file can be guaranteed through the dynamic grammar guidance mechanism, and the corresponding material information and map path information can be inserted into the specified field, which not only ensures the output compliance but also retains the generation capability of the deep learning model.

[0072] In a possible implementation, the material intercommunication method further includes the steps of:

[0073] S210 : When the number of textures is greater than 1, create an independent key-value projection matrix group for the first texture in the textures.

[0074] S220. Utilize the attention layer of the material conversion model to dedicate most of the attention channels to processing the key-value projection matrix group of the first texture, and allocate the remaining few attention channels to other textures.

[0075] The path field of the structured material file of the target material is the path information of the first map.

[0076] In the disclosed embodiment, when there are multiple textures, it is necessary to ensure that the model's attention remains on the primary texture (the first texture). One approach is to create a separate key-value projection matrix group (K1, V1) for the primary texture (labeled as the first texture). A shared key-value projection matrix (K2, V2) is assigned to the remaining textures. In each attention head of the material conversion model, 60% of the attention channels are dedicated to processing the key-value pairs of the first texture, and the remaining 40% of the channels are used to process the remaining textures. A separate key-value projection matrix group is created for the first texture, and the remaining 40% of the projection matrix dimensions are shared by the remaining textures.

[0077] According to the solution of the embodiment of the present disclosure, the high-frequency detail encoding capability is retained for the main texture, which significantly improves the reliability and execution efficiency of the material conversion system.

[0078] In one possible implementation, S110 utilizes an image encoder of a material conversion model to obtain image features according to a texture containing a target material and a corresponding rendering, further comprising the steps of:

[0079] S111: Rendering is performed in a first rendering engine using a texture containing a target material to obtain a rendered image, wherein the first rendering engine uses a first material system.

[0080] S112. Extracting image features from the rendering and the texture through the image encoder of the material conversion model.

[0081] In the embodiment of the present disclosure, a first rendering engine uses a texture containing a target material to render a rendered image. The first rendering engine uses a first material system, and the target material also belongs to the first material system.

[0082] The rendered image and texture are fed into the image encoder of the pre-trained material conversion model. The image encoder extracts high-level semantic features through convolutional layers, establishing a correspondence between visual features and physical properties. The image features of each image are independent of each other, and each image can be uniquely identified. For example, the rendered image is Image-0, the first texture is Image-1, and so on for the remaining textures. This facilitates pairing with text instructions.

[0083] In a possible implementation, S120 utilizes a text encoder of a material conversion model to obtain text features according to a text instruction containing instruction information, further comprising the steps of:

[0084] S121. Obtain a text instruction based on at least one instruction information among task objectives, output requirements, background information, input specifications, processing flow, and constraint conditions.

[0085] S122. Encode the text instructions through the text encoder of the material conversion model to obtain text features.

[0086] In the embodiment of the present disclosure, the text instruction may be composed of multiple paragraphs consisting of one or more of task objectives, output requirements, background information, input specifications, processing procedures, and constraints.

[0087] In one example, the first paragraph of text instructions may include background information, input specifications, task objectives, and output requirements. For example, "You are a PBR material reverse rendering system that generates a rendering JSON based on a given image." The background information is used to inform the model that it is a material conversion system, and the task objectives are used to clarify that the system's core function is to infer material parameters from the input image through reverse rendering. The output requirements are used to specify the format of the generated file, such as JSON format. The input specifications are used to clarify the input-output relationship, and the input content is an image. Therefore, it can be considered that the first paragraph of text instructions is used to cause the system to reversely generate the initial render JSON based on the input map (such as inferring material parameters from a normal map).

[0088] The second paragraph of text instructions can include another paragraph of input-output relationships and constraints, such as "This image is generated by the renderer. Please analyze the image and generate a rendering JSON for the renderer." Among them, this instruction describes the operation steps in a specific scenario (analyzing the image and generating JSON), implicitly constraining the source of the input image. The image is an image generated by the renderer, that is, a rendering. The system is required to analyze its quality (such as lighting and material performance) and generate or adjust JSON parameters. Therefore, it can be considered that the second paragraph of text instructions is used to enable the system to analyze the properties of the input rendering image (such as lighting, shadow effects, etc.) and optimize the JSON parameters by fine-tuning the algorithm (such as adjusting the lighting parameters to match the rendering effect).

[0089] According to the solution of the embodiment of the present disclosure, the model can clearly indicate the task purpose and background information, and better enable the model to generate a formatted file based on the provided map. During the generation, the JSON parameters can be optimized or adjusted by analyzing the material information in the rendering.

[0090] In one possible implementation, S130 utilizes the connection layer of the material conversion model to fuse the image features and the text features to obtain the fused features, further comprising the steps of:

[0091] S131. Pair the text unit sequence in the image feature with the text unit sequence in the text feature.

[0092] S132: Perform feature fusion on the paired multiple text unit sequences to obtain fused features.

[0093] In the disclosed embodiment, both the rendering and the texture have corresponding prompt words, i.e., two segments of prompt words. However, during input, the image and text instructions enter different encoders respectively. When the text encoder encodes the text instructions, it does not know which of the multiple images the descriptions such as "rendering", "this image", and "analyzed image" mentioned therein refer to. Therefore, it is necessary to pair the image and the corresponding prompt words before merging them.

[0094] In a possible implementation, S131 pairs the text unit sequence in the image feature with the text unit sequence in the text feature, further comprising the steps of:

[0095] A first sequence of text units in the image feature is paired with a second sequence of text units in the text feature.

[0096] At least one third sequence of text units in the image feature is paired with a fourth sequence of text units in the text feature.

[0097] The first text unit sequence is obtained according to the rendering image, and the third text unit sequence is obtained according to the texture image.

[0098] In the embodiment of the present disclosure, a text unit sequence refers to a token generated by an encoder, a second text unit sequence refers to a text unit sequence generated by a text instruction that mentions a rendered image, and a fourth text unit sequence refers to a text unit sequence generated by a text instruction that mentions a texture. In one example, a specific pairing method includes:

[0099] Perform feature source encoding on the input data:

[0100] Rendering (Image-0) additional metadata identifier <render>, add the prefix tag [System Prompt] to the corresponding prompt word;

[0101] Additional level identifiers for texture groups (Image-1, 2, 3...)<TEXTURE_LEVEL=1> , corresponding to the prompt word mark [Material Prompt].

[0102] Inject positioning information during the feature encoding stage:

[0103] When the image encoder outputs features, a learnable positioning embedding (such as<IMG_ANCHOR=0> );

[0104] When encoding the prompt word, the text encoder inserts a matching semantic anchor at the beginning of the sentence (such as<TXT_ANCHOR=0> ).

[0105] Implement a strong association strategy between rendering features and system prompts:

[0106] When calculating the cross attention layer, <render>Features impose bidirectional attention constraints on [System Prompt] features.

[0107] An orthogonal constraint is imposed on the first 128-dimensional features of the rendered image feature sequence (Image-0) to ensure that they are aligned with the system prompt word feature space.

[0108] Dynamically allocate attention heads based on the map identifier:

[0109] The first map is processed by the majority of attention heads, while the other maps are processed by the minority of attention heads.

[0110] According to the solution of the embodiment of the present disclosure, the images indicated by different paragraphs in the text instruction are paired with the image features generated by the images, thereby improving the accuracy of cross-modal registration.

[0111] In a possible implementation, step S140 utilizes the decoder of the material conversion model to generate a structured material file including the material information of the target material and the path information of the texture according to the fusion feature, further comprising the steps of:

[0112] The fused features are fed into the decoder of the material conversion model.

[0113] The decoder of the material conversion model generates output results that conform to the preset grammatical rules.

[0114] According to the output results, a structured material file of the target material is obtained.

[0115] In the embodiment of the present disclosure, a material grammar rule library based on Extended Backus-Naur Form (EBNF) may be constructed to define:

[0116] Field generation order constraints (e.g., textures fields must be generated before parameters fields);

[0117] Data type matching rules (path fields must comply with URI specifications, and numeric fields must have a limited range of values);

[0118] Hierarchical nesting relationship (subsurface_scattering is a child node of optical_properties).

[0119] Maintaining the state stack in real time during the decoding process:

[0120] When a "{" symbol is generated, a new nested level state is created;

[0121] When encountering the material_type: "PBR" declaration, activate the metalness / roughness parameter generation substate;

[0122] When the texture_path field is detected, the path validity check state is forced to begin.

[0123] At each decoding step execute:

[0124] Generate a set of allowed tokens based on the current grammar state;

[0125] Apply mask filtering to the original probability distribution output by the decoder to retain only legal candidate tokens.

[0126] When a potential syntax conflict is detected:

[0127] Enable look-ahead parsing to predict the validity of the next 3-5 tokens;

[0128] Roll back the generation path that causes syntax errors and reselect the optimal candidate sequence;

[0129] Strengthen the generation weight of grammatical key positions through attention redirection technology.

[0130] Implement post-processing optimization on generated results:

[0131] Automatically complete required closing symbols (such as unclosed brackets);

[0132] Standardized indentation and line break formats improve readability;

[0133] Verify path validity and perform relative path conversion.

[0134] According to the solution of the embodiment of the present disclosure, the conversion mechanism based on the parameter mapping table ensures that the generated file can be directly loaded and used in the target rendering engine. This embodiment achieves the precise generation of structured material files by combining the grammar rule state machine with the constraint decoding strategy.

[0135] In a possible implementation, the material intercommunication method further includes the steps of:

[0136] The structured material file is imported into the second rendering engine so that the second rendering engine obtains material information of the target material and path information of the texture, wherein the second rendering engine uses the second material system.

[0137] In the disclosed embodiment, after obtaining a structured material file, the file is imported into a second rendering engine, which can then obtain the target material's material information and texture path information. Through the synergy of semantic mapping and path management, high-fidelity material conversion across rendering engines is achieved.

[0138] Figure 3 FIG. 1 is a flow chart of a method for training a material conversion model according to an embodiment of the present disclosure. Figure 3 As shown, the method comprises at least the following steps:

[0139] S310: Obtain a training data set, where the training data set includes multiple groups of sample maps containing sample materials, corresponding renderings, and target results of structured material files to be obtained.

[0140] In the embodiment of the present disclosure, a training data set is constructed that includes the following elements:

[0141] Sample texture set: Contains multiple types of textures such as diffuse maps, normal maps, specular maps, etc., covering material categories such as metal, fabric, plastic, etc., stored in PNG / TIFF format.

[0142] Corresponding rendering: Use the first rendering engine (such as V-Ray) to perform physically accurate rendering of the sample map to generate an RGB image containing lighting and shadow information.

[0143] Structured target file: A standard JSON file containing structured data such as material parameters (metalness, roughness), texture path references, UV mapping parameters, etc. Structured target files can be obtained by manually converting sample materials.

[0144] S320 , using the image encoder of the material conversion model to perform feature extraction on the sample texture and the sample rendering to obtain an image feature vector.

[0145] A multi-branch image encoder is used to extract local texture features from each map and global material appearance features from the rendering.

[0146] S330 , using the text encoder of the material conversion model to obtain a text feature vector according to the text instruction containing the instruction information.

[0147] Implement layered encoding processing on text instructions to extract the semantics of operation instructions in the text.

[0148] S340: Utilize the connection layer of the material conversion model to perform feature fusion on the image feature vector and the text feature vector to generate a fused feature vector.

[0149] Feature fusion optimization is implemented at the connection layer to establish a many-to-many attention mapping between image regions and text instruction fragments, focusing on enhancing feature interactions related to key material properties (such as metallicity).

[0150] S350: Input the fused feature vector into the decoder of the material conversion model, and output the prediction result of the structured material file.

[0151] The decoder workflow includes:

[0152] State-aware decoding: Maintains the grammatical state of the current generation position (such as field type and nesting level) and constrains the candidate token set through a finite state machine.

[0153] Dynamic masking mechanism: Illegal characters are hard-masked according to preset grammatical rules (such as JSON format specifications) to ensure the structural legality of the generated sequence.

[0154] S360: Update the model parameters of the material conversion model according to the loss value between the predicted result and the target result to obtain a trained material conversion model.

[0155] Model training can adopt a multi-objective joint optimization strategy:

[0156] Semantic matching loss: Calculates the cross entropy loss between the predicted material parameters and the target file to optimize parameter accuracy.

[0157] Structural compliance loss: A structural error penalty generated by a grammatical rule checker to enforce format compliance.

[0158] According to the solution of the embodiment of the present disclosure, accurate reconstruction of the optical properties of the material is ensured by inputting maps and renderings; a dynamic masking strategy based on state-aware decoding effectively eliminates structural errors in JSON files; and through the combination of image reverse engineering and data-driven learning, a breakthrough is made in the dilemma of guessing physical parameters in traditional material conversion, achieving automated cross-engine compatibility.

[0159] In one possible implementation, S350 inputs the fused feature vector into a decoder of a material conversion model, and outputs a prediction result of a structured material file, including:

[0160] S351. Input the fused features into the decoder of the material conversion model to obtain a generated sequence output by the decoder.

[0161] S352. Using the grammar rule state machine, according to the grammar state of the current generation sequence, insert the material information of the sample material or the path information of the texture to obtain the prediction result of the structured material file of the sample material.

[0162] In the embodiment of the present disclosure, the implementation steps of S351~S352 can be implemented with reference to steps S141~142, and will not be repeated here.

[0163] In one possible implementation, S352 utilizes a grammar rule state machine to insert material information of a sample material or path information of a texture according to the grammar state of the current generation sequence to obtain a prediction result of a structured material file of the sample material, further comprising:

[0164] Get the probability distribution of the decoder over the set of predicted text units for the next position in the generated sequence.

[0165] The grammar rule state machine is used to determine the next optional text unit set that complies with the preset grammar rule according to the grammar state of the current generated sequence. The grammar rule state machine is established according to the target format.

[0166] The predicted text unit set is filtered according to the optional text unit set to retain text units that comply with preset grammatical rules.

[0167] The text at the next position is determined in the filtered candidate set until a complete sequence is generated.

[0168] The prediction results of the material structured file are obtained based on the complete sequence.

[0169] In the embodiment of the present disclosure, the specific steps include:

[0170] 1. Generate current token

[0171] At each decoder time step, the model generates a raw token prediction based on the context vector, the historical sequence, and the positional encoding. The model outputs a raw probability distribution containing the predicted scores for all possible tokens. For example, when generating material parameter names, this might include valid parameter names such as "roughness" and "metallic."

[0172] 2. Query Syntax State Machine

[0173] The grammar rule engine tracks the generation process in real time and implements dynamic constraints through state transitions:

[0174] 1. State node definition: Convert the grammatical rules of the target format (such as JSON Schema) into a finite state automaton, where each state node represents a specific grammatically allowed state. For example:

[0175] State A: Waiting to generate the object start character "{"

[0176] Status B: The key name has been generated, waiting for the colon to be generated:"

[0177] State C: Floating point value needs to be generated (range 0-1)

[0178] State D: Path information of the texture needs to be inserted.

[0179] 2. Path constraint detection:

[0180] Infer the current grammar state based on the generated sequence. For example, when "roughness": is generated, the next state must accept floating-point numbers.

[0181] 3. Legal path prediction:

[0182] Generate all possible legal transfer paths in advance.

[0183] Preemptively exclude options that would result in a syntax error (such as prematurely closing brackets when generating an array).

[0184] 3. Generate a set of legal token candidates

[0185] Based on the current grammar state, dynamically generate the set of tokens that are allowed to appear:

[0186] 1. Basic type constraints:

[0187] When a numeric value is required, only tokens such as numbers and decimal points are retained;

[0188] When a string is required, filter out non-alphabetic characters.

[0189] 2. Context-sensitive constraints:

[0190] Key name generation phase: only material parameter names supported by the current rendering engine are allowed;

[0191] Numeric range control: If the parameter is defined as a range of 0-1, exclude negative numbers or numeric characters greater than 1.

[0192] 3. Structural integrity constraints:

[0193] Bracket matching: When a "{" is generated but not closed, it is mandatory to have a "}" following it.

[0194] Delimiter requirements: When commas must appear between array elements, reserve the comma generation position in advance.

[0195] 4. Applying Probabilistic Masking

[0196] Modifying the probability distribution of the model's original output can be done by increasing the probability of tokens that conform to specific grammatical rules (such as increasing the probability of required symbols), or by setting the probability of illegal tokens to negative infinity or 0 to ensure that these tokens are not selected.

[0197] 5. Choosing the best token

[0198] Select the token with the highest probability or the highest score from the filtered set of valid candidates. You can also use greedy strategies, beam search optimization, and other methods to determine the optimal token.

[0199] According to the solution of the embodiment of the present disclosure, the grammatical correctness of the material structured file can be guaranteed through the dynamic grammar guidance mechanism, and the corresponding material information and map path information can be inserted into the specified field, which not only ensures the output compliance but also retains the generation capability of the deep learning model.

[0200] In a possible implementation, the preset grammar rule includes associations between multiple preset fields and corresponding data types.

[0201] Among them, the multiple preset fields include a mapping path field, an optical characteristic field, a surface detail field, and a texture mapping field.

[0202] In the disclosed embodiment, multiple preset fields cover multiple dimensions such as the optical properties of materials, surface details, texture mapping, etc. The following are some of the preset fields and their corresponding data types:

[0203] Material opacity: FloatTextureType

[0204] Diffuse color: ColorTypeTextureType

[0205] Self-illumination intensity self illumination:EmissionTextureType

[0206] Reflection color reflect: ColorTypeTextureType

[0207] Refraction color refract: ColorTextureType

[0208] Bump map bump: BumpTextureType

[0209] Normal map normal: NormalTextureType

[0210] Displacement map displacement: DisplacementTextureType.

[0211] In one possible implementation, S340 utilizes the connection layer of the material conversion model to perform feature fusion on the image feature vector and the text feature vector to generate a fused feature vector, including:

[0212] When the number of stickers is greater than 1, the corresponding image feature vectors of the multiple stickers are determined based on the sticker number codes in the text feature vector, wherein the sticker number codes are obtained based on the sticker numbers included in the indication information.

[0213] According to the sticker numbers of multiple stickers, the corresponding attention mask matrix is ​​generated.

[0214] The mask matrix is ​​fused with the original attention score to obtain weight parameters of multiple maps, where the first map among the multiple maps has a greater weight than the remaining maps.

[0215] According to the weight parameters, the text feature vector and the image feature vectors corresponding to multiple maps are fused to obtain a fused feature vector.

[0216] In the disclosed embodiments, in a physically based rendering (PBR) system, primary maps (such as diffuse maps) typically carry the core visual properties of a material (base color and texture distribution), while other maps (such as normal maps and roughness maps) are used to describe detail enhancement or physical properties. Model-first parsing of the primary map establishes the material's basic semantic framework, and auxiliary maps add details within this framework to avoid feature confusion.

[0217] Joint processing of multiple tiles can easily lead to distracted attention. By using attention masking mechanisms (e.g., dynamic weight allocation), the model can focus the majority (e.g., 70%-80%) of its computational resources on extracting key features from the primary tile, while allocating the remaining resources to auxiliary tiles as needed. This significantly reduces redundant computation, improving processing speed and memory efficiency.

[0218] The main texture is usually directly associated with the core requirement in the text instruction (such as "generate basic material parameters"). The attention strategy ensures that the model strictly follows the user's intention and avoids the noise interference of secondary textures that causes semantic drift. To this end, the texture function identifier can be explicitly marked in the text instruction (such as<diffuse_map_1> 、<normal_map_2> ), establish encoding associations in the following ways:

[0219] During the text encoding stage, the label description fragments are extracted to generate the label embedding vector.

[0220] Attach a learnable label identifier (such as<TEX_ID=1> )

[0221] Create a label-texture mapping table to record the correspondence between each texture feature and the text label description.

[0222] Generate hierarchical attention masks based on label priority:

[0223] Main map enhancement mechanism: for the<diffuse_map> The main map of is used to generate a full-pass mask, allowing it to participate in the computation of all attention heads.

[0224] Auxiliary map constraints: Auxiliary maps (such as<height_map> ) imposes spatial locality constraints, limiting its attention range to relevant feature areas.

[0225] Global attention interactions are performed between primary texture features and textual instruction features to capture core material properties. Scaled dot-product attention is used to compute raw scores, and a label-guided mask matrix is ​​superimposed to generate modified attention weights. Grouped attention is applied to auxiliary texture features, with each group corresponding to specific material detail properties (e.g., normal details, roughness distribution). The primary and secondary features are weighted concatenated to generate a fused feature vector, which retains the key information of the original features through residual connections.

[0226] It's important to note that auxiliary texture paths can also be included in the JSON file. The auxiliary texture's parameter definitions (e.g., normal strength, roughness range) often rely on the primary texture's metadata (e.g., resolution, color space). The explicit declaration of the primary texture in the JSON file provides an anchor point for subsequent parameter parsing.

[0227] When the target environment does not support multiple textures, the main texture can be used to independently form the minimum runnable material. The prominent mark of the main path in the JSON file facilitates the system to automatically implement downgrade processing (such as only loading the main texture and ignoring other ones).

[0228] In the case that the material file must explicitly specify the base texture path, other texture paths (such as normalTexture) are optional fields. The main texture path is a required parameter and is the entry point for material loading.

[0229] The solution of the disclosed embodiment balances computational efficiency and generation quality in multi-texture scenarios, and the focused attention strategy improves the accuracy of master material parameter generation. Furthermore, users can quickly switch material styles by modifying the master texture path without having to reconstruct the entire JSON file.

[0230] In one possible implementation, combining Figure 2 As shown, the renderer material interoperability method includes the following steps:

[0231] 1. Get the rendered image and corresponding texture of the non-PBR material system renderer. The non-PBR material system rendering engine renders a rendering image with material characteristics and obtains the corresponding texture. The rendered image has the necessary characteristics of the target rendering material.

[0232] Second, the material rendering and texture images are passed through an image encoder to extract feature maps from the rendered and texture images. Options for image encoders include convolutional layers, Transformer encoders, and multi-layer perceptrons.

[0233] 3. Text instructions in the input text instruct The text features are extracted through the text encoder. The text encoder options include: Tokenizer layer, Input_ids layer, and Transformer decoder.

[0234] Fourth, due to the information gap between image features and text features, it is necessary to use a connection layer connector to pair image features with text features, fuse the information features of text and image into a unified output. The connector options include: multi-layer perceptron.

[0235] 5. Load the output of the connection layer into the large language model decoder LLM decoder layer, and output the information features into the required PBR json through the decoder layer. The decoder can use the Transformer decoder.

[0236] During training, the input is the rendered image from another renderer, the corresponding attribute map, and the preset prompt. Training uses gradient descent to learn the correct parameters. The loss function can use the commonly used cross-entropy loss function.

[0237] Figure 4 This is a structural diagram of a renderer material intercommunication device based on deep learning provided by the present disclosure. Figure 4 As shown, the renderer material intercommunication device 400 includes:

[0238] An image encoding module 401 is configured to utilize an image encoder of a material conversion model to obtain image features based on a texture containing a target material and a corresponding rendering;

[0239] A text encoding module 402 is configured to obtain text features based on text instructions containing instruction information using a text encoder of a material conversion model;

[0240] A fusion module 403 is configured to fuse image features and text features using a connection layer of a material conversion model to obtain fused features;

[0241] The generation module 404 is configured to generate a structured material file including material information of the target material and path information of the texture according to the fusion features using the decoder of the material conversion model.

[0242] In a possible implementation, the generating module 404 is configured to:

[0243] Input the fused features into the decoder of the material conversion model to obtain the generated sequence output by the decoder;

[0244] By utilizing the grammar rule state machine, the material information of the target material or the path information of the texture is inserted according to the grammar state of the current generation sequence to obtain a structured material file of the target material.

[0245] In one possible implementation, an attention module is further included, configured to:

[0246] When the number of textures is greater than 1, an independent key-value projection matrix group is created for the first texture in the textures;

[0247] Using the attention layer of the material conversion model, most of the attention channels are dedicated to processing the key-value projection matrix group of the first texture, and the remaining few attention channels are allocated to other textures;

[0248] The path field of the structured material file of the target material is the path information of the first map.

[0249] In a possible implementation, the image encoding module 401 is configured to:

[0250] Rendering using a map containing a target material in a first rendering engine to obtain a rendering image; wherein the first rendering engine uses a first material system;

[0251] Extract image features from renderings and textures through the image encoder of the material conversion model.

[0252] In one possible implementation, the text encoding module 402 is used to:

[0253] Obtaining text instructions based on at least one of task objectives, output requirements, background information, input specifications, processing procedures, and constraints;

[0254] The text instructions are encoded through the text encoder of the material conversion model to obtain text features.

[0255] In a possible implementation, the fusion module 403 is configured to:

[0256] Pairing the text unit sequence in the image feature with the text unit sequence in the text feature;

[0257] The features of multiple paired text unit sequences are fused to obtain fused features.

[0258] In a possible implementation, the fusion module 403 is configured to:

[0259] pairing a first sequence of text units in the image feature with a second sequence of text units in the text feature;

[0260] pairing at least one third sequence of text units in the image feature with a fourth sequence of text units in the text feature;

[0261] The first text unit sequence is obtained according to the rendering image, and the third text unit sequence is obtained according to the texture image.

[0262] In a possible implementation, an import module is further included for:

[0263] The structured material file is imported into the second rendering engine, so that the second rendering engine obtains material information of the target material and path information of the texture; wherein the second rendering engine uses the second material system.

[0264] Figure 5 Schematic diagram of a training device for a material conversion model provided by the present disclosure. Figure 5 As shown, the training device 500 includes:

[0265] The acquisition module 501 is used to acquire a training data set, which includes multiple groups of sample maps containing sample materials, corresponding renderings, and target results of structured material files to be obtained.

[0266] The image feature extraction module 502 is used to extract features from the sample texture and the sample rendering using the image encoder of the material conversion model to obtain an image feature vector.

[0267] The text feature extraction module 503 is configured to obtain a text feature vector according to the text instruction containing the instruction information by using the text encoder of the material conversion model.

[0268] The feature fusion module 504 is used to perform feature fusion on the image feature vector and the text feature vector using the connection layer of the material conversion model to generate a fused feature vector.

[0269] The prediction module 505 is used to input the fused feature vector into the decoder of the material conversion model and output the prediction result of the structured material file.

[0270] The training module 506 is used to update the model parameters of the material conversion model according to the loss value between the prediction result and the target result to obtain a trained material conversion model.

[0271] In one possible implementation, the prediction module 505 is configured to:

[0272] Input the fused features into the decoder of the material conversion model to obtain the generated sequence output by the decoder;

[0273] By using the grammar rule state machine, the material information of the sample material or the path information of the texture is inserted according to the grammar state of the current generation sequence, and the prediction result of the structured material file of the sample material is obtained.

[0274] In one possible implementation, the prediction module 505 is configured to:

[0275] Obtain the probability distribution of the decoder's predicted text unit set for the next position in the generated sequence;

[0276] Using a grammar rule state machine, according to the grammar state of the current generated sequence, a set of optional text units that conform to the preset grammar rules at the next position is determined; wherein the grammar rule state machine is established according to the target format;

[0277] Filtering the predicted text unit set according to the optional text unit set to retain text units that comply with preset grammatical rules;

[0278] Determine the text at the next position in the filtered candidate set until a complete sequence is generated;

[0279] The prediction results of the material structured file are obtained based on the complete sequence.

[0280] In one possible implementation, the preset grammar rule includes an association relationship between a plurality of preset fields and corresponding data types;

[0281] Among them, the multiple preset fields include a mapping path field, an optical characteristic field, a surface detail field, and a texture mapping field.

[0282] In one possible implementation, the feature fusion module 504 is configured to:

[0283] When the number of stickers is greater than 1, determining corresponding image feature vectors of the plurality of stickers according to the coding of the sticker numbers in the text feature vector; wherein the coding of the sticker numbers is obtained according to the sticker numbers included in the indication information;

[0284] Generate corresponding attention mask matrices according to the sticker numbers of multiple stickers;

[0285] The mask matrix is ​​fused with the original attention score to obtain weight parameters of multiple maps; wherein the first map among the multiple maps has a greater weight than the remaining maps;

[0286] According to the weight parameters, the text feature vector and the image feature vectors corresponding to multiple maps are fused to obtain a fused feature vector.

[0287] Figure 6 FIG. 1 is a structural block diagram of an electronic device according to an embodiment of the present disclosure. Figure 6 As shown, the electronic device includes: a memory 610 and a processor 620. The memory 610 stores a computer program that can be executed on the processor 620. The number of memory 610 and processor 620 can be one or more. The memory 610 can store one or more computer programs. When the one or more computer programs are executed by the electronic device, the electronic device performs the method provided by the above method embodiment. The electronic device may also include: a communication interface 630 for communicating with external devices and performing data exchange.

[0288] If the memory 610, processor 620, and communication interface 630 are implemented independently, the memory 610, processor 620, and communication interface 630 can be connected to each other via a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0289] Optionally, in a specific implementation, if the memory 610, the processor 620 and the communication interface 630 are integrated on a chip, the memory 610, the processor 620 and the communication interface 630 can communicate with each other through an internal interface.

[0290] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor that supports the Advanced RISC Machines (ARM) architecture.

[0291] Furthermore, optionally, the above-mentioned memory may include a read-only memory and a random access memory, and may also include a non-volatile random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM).

[0292] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present disclosure is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, data subscriber line (DSL)) or wireless (e.g., infrared, Bluetooth, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, or a magnetic tape), an optical medium (e.g., a digital versatile disc (DVD)), or a semiconductor medium (e.g., a solid-state drive (SSD)). It is worth noting that the computer-readable storage medium mentioned in the present disclosure may be a non-volatile storage medium, in other words, a non-transient storage medium.

[0293] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0294] In the description of the embodiments of the present disclosure, reference to the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present disclosure. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art may combine and integrate different embodiments or examples described in this specification, as well as features of different embodiments or examples, unless otherwise inconsistent.

[0295] In the description of the embodiments of the present disclosure, unless otherwise specified, " / " means or. For example, A / B can mean A or B. "And / or" in this document is simply a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0296] In the description of the embodiments of the present disclosure, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of the present disclosure, unless otherwise specified, "plurality" means two or more.

[0297] The above description is merely an exemplary embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the scope of protection of the present disclosure.< / render> < / render>

Claims

1. A renderer material interoperability method based on deep learning, comprising: Using the image encoder of the material conversion model, the image features are obtained based on the texture containing the target material and the corresponding rendering; Using the text encoder of the material conversion model, a text feature is obtained according to the text instruction containing the instruction information; Using the connection layer of the material conversion model, the image feature and the text feature are fused to obtain a fused feature; Inputting the fused features into a decoder of the material conversion model to obtain a generated sequence output by the decoder; The grammar rule state machine is used to insert the material information of the target material or the path information of the texture according to the grammar state of the current generation sequence to obtain a structured material file of the target material.

2. The method according to claim 1, further comprising: When the number of the textures is greater than 1, establishing an independent key-value projection matrix group for a first texture in the textures; Using the attention layer of the material conversion model, a majority of the attention channels are dedicated to processing the key-value projection matrix group of the first texture, and a remaining minority of the attention channels are allocated to other textures; The path field of the structured material file of the target material is the path information of the first map.

3. The method according to claim 1, wherein The image encoder using the material conversion model obtains image features based on a texture containing a target material and a corresponding rendering, including: Rendering using a map containing a target material in a first rendering engine to obtain a rendering image; wherein the first rendering engine uses a first material system; Image features are extracted from the rendering and the texture using an image encoder of a material conversion model.

4. The method according to claim 1, wherein The text encoder using the material conversion model obtains text features according to the text instruction containing the instruction information, including: Obtaining text instructions based on at least one of task objectives, output requirements, background information, input specifications, processing procedures, and constraints; The text instructions are encoded by the text encoder of the material conversion model to obtain text features.

5. The method according to claim 1, wherein The method of utilizing the connection layer of the material conversion model to fuse the image features and the text features to obtain fused features includes: pairing a sequence of text units in the image feature with a sequence of text units in the text feature; The features of multiple paired text unit sequences are fused to obtain fused features.

6. The method according to claim 5, wherein: Pairing the text unit sequence in the image feature with the text unit sequence in the text feature, comprising: pairing a first sequence of text units in the image feature with a second sequence of text units in the text feature; pairing at least one third sequence of text units in the image feature with a fourth sequence of text units in the text feature; The first text unit sequence is obtained according to the rendering image, and the third text unit sequence is obtained according to the texture image.

7. The method according to claim 1, further comprising: Importing the structured material file into a second rendering engine so that the second rendering engine obtains the material information of the target material and the path information of the map; wherein the second rendering engine uses a second material system.

8. A method for training a material conversion model, comprising: Obtaining a training data set, wherein the training data set includes multiple sets of sample maps containing sample materials, corresponding renderings, and target results of structured material files to be obtained; Using an image encoder of a material conversion model, feature extraction is performed on the sample texture and the sample rendering to obtain an image feature vector; Using the text encoder of the material conversion model, a text feature vector is obtained according to the text instruction containing the instruction information; Using the connection layer of the material conversion model, the image feature vector and the text feature vector are subjected to feature fusion to generate a fused feature vector; Inputting the fused features into a decoder of the material conversion model to obtain a generated sequence output by the decoder; Using a grammar rule state machine, inserting the material information of the sample material or the path information of the texture according to the grammar state of the current generation sequence, to obtain a prediction result of the structured material file of the sample material; According to the loss value between the predicted result and the target result, the model parameters of the material conversion model are updated to obtain a trained material conversion model.

9. The method according to claim 8, wherein The method of using a grammar rule state machine to insert the material information of the sample material or the path information of the texture according to the grammar state of the current generation sequence to obtain a prediction result of the structured material file of the sample material includes: Obtaining a probability distribution of a set of predicted text units for the next position of the generated sequence by the decoder; Determining a set of optional text units that conform to preset grammatical rules at the next position according to the grammatical state of the current generated sequence using a grammatical rule state machine; wherein the grammatical rule state machine is established according to the target format; Filtering the predicted text unit set according to the optional text unit set to retain text units that comply with preset grammatical rules; Determine the text at the next position in the filtered candidate set until a complete sequence is generated; A prediction result of the material structured file is obtained according to the complete sequence.

10. The method according to claim 9, wherein: The preset grammar rules include association relationships between multiple preset fields and corresponding data types; Among them, the multiple preset fields include a mapping path field, an optical characteristic field, a surface detail field, and a texture mapping field.

11. The method according to claim 8, wherein The image feature vector and the text feature vector are subjected to feature fusion by utilizing the connection layer of the material conversion model to generate a fused feature vector, including: When the number of the tiles is greater than 1, determining corresponding image feature vectors of the plurality of tiles based on the coding of the tile numbers in the text feature vector; wherein the coding of the tile numbers is obtained based on the tile numbers included in the indication information; Generate a corresponding attention mask matrix according to the tile numbers of the multiple tiles; Fusing the mask matrix with the original attention score to obtain weight parameters of the multiple maps; wherein a first map among the multiple maps has a weight greater than that of the remaining maps; According to the weight parameter, the text feature vector and the image feature vectors corresponding to the multiple maps are fused to obtain a fused feature vector.

12. A renderer material intercommunication device based on deep learning, comprising An image encoding module, configured to utilize an image encoder of a material conversion model to obtain image features based on a texture containing a target material and a corresponding rendering; a text encoding module, configured to obtain text features according to text instructions containing instruction information using a text encoder of the material conversion model; a fusion module, configured to fuse the image features and the text features using a connection layer of the material conversion model to obtain a fused feature; A generation module, configured to input the fusion features into a decoder of the material conversion model to obtain a generation sequence output by the decoder; The grammar rule state machine is used to insert the material information of the target material or the path information of the texture according to the grammar state of the current generation sequence to obtain a structured material file of the target material.

13. A training device for a material conversion model, comprising: An acquisition module is used to acquire a training data set, wherein the training data set includes multiple sets of sample maps containing sample materials, corresponding renderings, and target results of structured material files to be obtained; An image feature extraction module is used to extract features from the sample texture and the sample rendering using an image encoder of a material conversion model to obtain an image feature vector; A text feature extraction module, configured to obtain a text feature vector according to a text instruction containing instruction information using a text encoder of the material conversion model; A feature fusion module, configured to perform feature fusion on the image feature vector and the text feature vector using the connection layer of the material conversion model to generate a fused feature vector; a prediction module, configured to input the fused features into a decoder of the material conversion model to obtain a generated sequence output by the decoder; Using a grammar rule state machine, inserting the material information of the sample material or the path information of the texture according to the grammar state of the current generation sequence, to obtain a prediction result of the structured material file of the sample material; A training module is used to update the model parameters of the material conversion model according to the loss value between the prediction result and the target result to obtain a trained material conversion model.

14. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 11.

15. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-11.

16. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 11.

Images