Map generation method and device, electronic equipment and storage medium

Abstract

The invention provides a map generation method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a to-be-processed text; based on an image generation model, performing semantic analysis on the to-be-processed text to obtain text features of the to-be-processed text, obtaining image features associated with the text features according to the text features, generating a target image according to the image features, and scanning the to-be-processed text according to the target image and a target scanning configuration file. A target map corresponding to the target image is generated according to the target scanning configuration file, the target scanning configuration file comprises scanning parameters needed for scanning the target image, and scanning of the target image is used for obtaining target image information for generating the target map, so that the target image with more texture map features can be obtained, and the image quality is improved. Therefore, a high-quality target map can be accurately generated based on the target image.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a texture generation method, device, electronic device and storage medium. Background Art

[0002] Physically-Based Rendering (PBR) materials can well express the information of the object surface, can render more realistic images, and create more realistic effects in the scene. Materials are generated by maps, which can include texture and other information. PBR materials are currently being applied to more and more fields, and the requirements for PBR material maps are also higher. How to obtain higher quality maps is very important. Summary of the invention

[0003] The embodiments of the present disclosure at least provide a method, device, electronic device and storage medium for generating a texture.

[0004] In a first aspect, an embodiment of the present disclosure provides a method for generating a map, comprising:

[0005] Get the text to be processed;

[0006] Based on the image generation model, semantic analysis is performed on the text to be processed to obtain text features of the text to be processed, and based on the text features, image features associated with the text features are obtained, and a target image is generated based on the image features;

[0007] A target map corresponding to the target image is generated according to the target image and a target scanning configuration file, wherein the target scanning configuration file includes scanning parameters required for scanning the target image, and the scanning of the target image is used to obtain target image information for generating the target map.

[0008] In a second aspect, the present disclosure also provides a map generation device, including:

[0009] An acquisition module is used to obtain the text to be processed;

[0010] An image generation module, configured to perform semantic analysis on the text to be processed based on an image generation model, obtain text features of the text to be processed, obtain image features associated with the text features based on the text features, and generate a target image based on the image features;

[0011] A map generation module is used to generate a target map corresponding to the target image according to the target image and a target scanning configuration file, wherein the target scanning configuration file includes scanning parameters required for scanning the target image, and the scanning of the target image is used to obtain target image information for generating the target map.

[0012] In a third aspect, an optional implementation of the present disclosure further provides an electronic device, comprising a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is used to execute the machine-readable instructions stored in the memory. When the machine-readable instructions are executed by the processor, the processor executes the above-mentioned first aspect, or the steps in any possible implementation of the first aspect.

[0013] In a fourth aspect, an optional implementation of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-mentioned first aspect, or the steps in any possible implementation of the first aspect.

[0014] In the disclosed embodiments, a text to be processed is obtained; based on an image generation model, a semantic analysis is performed on the text to be processed to obtain text features of the text to be processed, and based on the text features, image features associated with the text features are obtained, a target image is generated based on the image features, and a target map corresponding to the target image is generated based on the target image and a target scanning configuration file. In this way, a target image corresponding to the text to be processed can be generated based on the image generation model, and then based on the target scanning configuration file, a target map can be generated by scanning the target image, thereby improving the map generation efficiency.

[0015] In addition, in the embodiment of the present disclosure, the image generation model is trained based on a set of image samples rendered based on materials generated by texture samples. Therefore, based on the image generation model, a target image with more material texture characteristics can be obtained, and thus a set of high-quality target textures can be accurately generated based on the target image, thereby meeting the needs of material production and improving generation effect and quality.

[0016] For a description of the effects of the above-mentioned texture generation device, electronic device, and computer-readable storage medium, please refer to the description of the above-mentioned texture generation method, which will not be repeated here.

[0017] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present disclosure.

[0018] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.

[0020] Figure 1 A flowchart of a method for generating a map provided by an embodiment of the present disclosure is shown;

[0021] Figure 2 A flowchart of another method for generating a map provided by an embodiment of the present disclosure is shown;

[0022] Figure 3 A schematic diagram of a map generating device provided by an embodiment of the present disclosure is shown;

[0023] Figure 4 A schematic diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0024] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0025] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure is not intended to limit the scope of the present disclosure claimed for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

[0026] To facilitate the understanding of the technical solution of the present disclosure, the technical terms in the embodiments of the present disclosure are first explained:

[0027] Physically-Based Rendering (PBR): refers to a lighting and rendering method that can more accurately depict the interaction between light and surfaces. PBR imitates the actual behavior of light rather than simulating the appearance of light, making the rendered image more realistic.

[0028] Texture: Texture can be understood as an attribute parameter in a shader. Shader is responsible for combining the input vertex data with the input texture or color in a specified way, and then outputting it. The drawing unit can then draw the image onto the screen based on this output. The texture can include information such as texture and UV coordinates.

[0029] Material: Input textures or colors, etc., plus the corresponding Shader, and the specific parameter settings for the Shader. These contents are packaged and stored together to obtain a material. Of course, the packaged package will also include other parameters. The material can then be assigned to the 3D model for rendering and output. For ease of understanding, it can be considered that the material is the product ultimately used by the engine, the Shader is the processing method for producing this product, and the texture is the raw material of the product. In addition, in the disclosed embodiment, the PBR material refers to the material based on physical rendering, and the PBR map refers to the map used to generate the PBR material.

[0030] According to research, more realistic images can be obtained based on PBR material rendering, and more realistic effects can be created in the scene. Materials are generated by textures. PBR materials are currently applied to more and more fields, and the texture requirements for PBR materials are also higher. How to obtain higher-quality textures more efficiently is very important. In addition, in the related art, images can be converted into PBR textures through conversion tools, but the images used in the related art are usually obtained arbitrarily, and the obtained images may be of low quality and cannot meet the texture requirements of PBR materials. In addition, a complete set of textures is usually required when generating PBR materials, such as normal maps, height maps, etc., and in the related art, since the images used may not meet the requirements, it is difficult to ensure the generation of a complete set of textures, which also reduces the efficiency and accuracy of texture generation.

[0031] Based on the above research, the present disclosure provides a texture generation method, which obtains a text to be processed, generates a target image corresponding to the text to be processed based on an image generation model, and generates a target texture corresponding to the target image according to the target image and a target scanning configuration file. In this way, the target image is generated based on the image generation model, and is scanned through the target scanning configuration file to generate a target texture, which improves the texture generation efficiency. Moreover, since the image generation model is obtained by training based on image samples after material rendering, a target image with more material texture characteristics can be obtained based on the image generation model, so that a high-quality target texture can be accurately generated based on the target image, and a set of target textures required to generate a material can be obtained to meet the needs when making materials.

[0032] The defects existing in the above solutions are the results obtained by the inventor after practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed by the present disclosure for the above problems below should be the contributions made by the inventor to the present disclosure during the disclosure process.

[0033] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0034] To facilitate understanding of this embodiment, a method for generating a sticker disclosed in the embodiment of the present disclosure is first introduced in detail. The execution subject of the method for generating a sticker provided in the embodiment of the present disclosure is generally an electronic device with certain computing capabilities, and the electronic device includes, for example: a terminal device or a server or other processing device. The terminal device may be a user equipment (UE), a mobile device, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc., wherein a personal digital assistant is a handheld electronic device with certain functions of an electronic computer, which can be used to manage personal information, browse the Internet, send and receive emails, etc. It is generally not equipped with a keyboard and may also be called a handheld computer. In some possible implementations, the method for generating a sticker may be implemented by a processor calling computer-readable instructions stored in a memory.

[0035] The following describes the method for generating a texture provided by the embodiment of the present disclosure by taking the execution subject as a terminal device as an example.

[0036] See also Figure 1 FIG. 1 is a flowchart of a method for generating a map according to an embodiment of the present disclosure, the method comprising:

[0037] S101: Obtain the text to be processed.

[0038] In the disclosed embodiments, the main purpose is to obtain a high-quality set of PBR maps, which can be applied to different scenes to generate materials and render them. For example, based on the PBR maps, special effects in scenes such as games, short videos, and film and television productions can be obtained through material rendering.

[0039] In the embodiments of the present disclosure, a map can be generated based on image conversion, and the image generation method can adopt the implementation method of the Vincent map, wherein the text to be processed for the Vincent map method is not limited, and the text representing the image description information can be input as required, for example, the input text to be processed is "a texture map of mahogany".

[0040] S102: Based on the image generation model, semantic analysis is performed on the text to be processed to obtain text features of the text to be processed, and based on the text features, image features associated with the text features are obtained, and a target image is generated based on the image features.

[0041] The image generation model is obtained by training according to a training sample set, the training sample set includes multiple training samples, and each training sample includes an image sample rendered based on a material obtained from a map sample, and a text label corresponding to the image sample.

[0042] The image generation model in the disclosed embodiment can be implemented in combination with a diffusion model for converting text to image and a fine-tuning model for adjusting the diffusion model, wherein the diffusion model is a general basic model, and directly using the diffusion model in a specific task scenario may not achieve good results, while the fine-tuning model is a small model, which requires a small training data set and small video memory resources, and the fine-tuning model can control the content generation of many specific scenarios, and is suitable for training the style and features of images, so it can well meet the training generation of images for map generation. In the disclosed embodiment, a fine-tuning model can be added on the basis of the diffusion model to train to obtain an image generation model.

[0043] And when training the image generation model, training can be performed based on a training sample set of a specific task scenario, that is, image samples marked with corresponding text labels and rendered based on materials obtained from texture samples. During the fine-tuning model training process, parameter training and adjustment are performed on the cross-attention layer in the diffusion model to better associate text and images, thereby improving the accuracy of generating target images with material map features.

[0044] S103: Generate a target map corresponding to the target image according to the target image and the target scanning configuration file, wherein the target scanning configuration file includes scanning parameters required for scanning the target image, and scan the target image to obtain target image information for generating the target map.

[0045] Execute step S103, the present disclosure provides a specific implementation method:

[0046] 1) Determine the material category of the target image based on the text to be processed.

[0047] Among them, the material category is, for example, wood, paper, metal, etc., which is not limited in the embodiments of the present disclosure. For example, if the text to be processed is "a texture map of mahogany", it can be determined that the material category of the corresponding target image is wood.

[0048] 2) According to the material category, determine the target scanning configuration file corresponding to the material category.

[0049] Among them, the target scanning profile includes scanning parameters required for scanning the target image. In the embodiment of the present disclosure, the map can be understood as image information rendered onto the material. By scanning an image, image information such as texture and color represented by the image can be obtained, and then different types of maps can be obtained by transformation. In the embodiment of the present disclosure, different material categories require different scanning parameters. A scanning profile can be created for each material category in advance. When scanning the target image, the target scanning profile required for scanning the target image can be determined first, and then based on the scanning parameters in the target scanning profile, the target image can be scanned to obtain the required target image information.

[0050] 3) Based on the scanning parameters in the target scanning configuration file, the target image is scanned and analyzed to obtain target image information of the target image, and a target map corresponding to the target image is generated according to the target image information.

[0051] In the disclosed embodiment, based on the target scanning configuration file, the target image information of the target image is scanned and analyzed, and then a corresponding set of target maps can be generated, such as a normal map, an albedo map, a roughness map, a height map, etc.

[0052] In the process of generating a target map, different methods can be used for different types of target maps. Specifically, for example, when generating a normal map, the target image is grayscale processed, and the R channel, G channel and B channel of the normal map are obtained through high-pass filtering, and then the data of the R channel, G channel and B channel are used to generate the normal map. For example, when generating an albedo map, the albedo texture can usually be extracted from the diffuse texture generated by scanning. For example, it can be applied to the color gradient node through the color map, and then the roughness map can be generated by adjusting the contrast, etc.

[0053] Of course, the method of generating different types of target maps is not limited in the embodiments of the present disclosure, and there are usually multiple target maps corresponding to the target image, which can be understood as a set of target maps for the target object. There is no restriction on the generation order of different types of target maps.

[0054] Furthermore, material feature information can also be added in the embodiments of the present disclosure. Specifically, the present disclosure provides a possible implementation method. In response to an instruction to add material feature information, the added material feature information is added to the target image; based on the scanning parameters in the target scanning configuration file, the target image including the added material feature information is scanned and analyzed to obtain target image information of the target image including the added material feature information; and based on the target image information, a target map corresponding to the target image is generated.

[0055] In this way, a target map corresponding to a target image including added material feature information can be generated, providing more options to meet different needs.

[0056] For example, by adding material feature information of cracks, the crack features can be added to the material, and finally a material with crack features can be generated based on the generated target map.

[0057] In the disclosed embodiment, a text to be processed is obtained, and based on an image generation model, a semantic analysis is performed on the text to be processed to obtain text features of the text to be processed, and based on the text features, image features associated with the text features are obtained, a target image is generated based on the image features, and then a target map corresponding to the target image is generated based on the target image and a target scanning configuration file. In this way, based on the image generation model, a target image with more material map features can be generated for the text to be processed, thereby improving efficiency. Moreover, the image generation model is obtained by training based on image samples after material rendering, so the generated target map is more in line with the material map requirements, so that a high-quality target map can be generated based on the target image, and a set of target maps required for generating materials can be generated accordingly, thereby improving accuracy and meeting the requirements for making materials.

[0058] The training process of the image generation model in the embodiment of the present disclosure is described below. In the embodiment of the present disclosure, a possible implementation method is specifically provided for the training method of the image generation model, including:

[0059] S1. Obtain a training sample set.

[0060] The training sample set includes multiple training samples, and each training sample includes an image sample rendered based on a material obtained from a texture sample, and a text label corresponding to the image sample.

[0061] In the embodiment of the present disclosure, the training samples in the training sample set are image samples with text labels and rendered based on materials obtained from the texture samples.

[0062] Specifically for obtaining a training sample set, the present disclosure provides possible implementation methods:

[0063] 1) Acquire multiple texture sample sets, wherein each texture sample set includes multiple texture samples required for generating a material.

[0064] For example, you can obtain multiple authorized sticker samples on the website.

[0065] Furthermore, in order to further improve the quality and processing efficiency, the texture samples may be preprocessed first. Specifically, a possible implementation method is provided: image preprocessing is performed on multiple texture samples in multiple texture sample sets, wherein the image preprocessing includes at least one of the following: denoising processing and brightness equalization processing.

[0066] 2) Obtain corresponding materials according to the multiple texture sample sets respectively, and render the obtained materials to obtain rendered image samples.

[0067] In the disclosed embodiment, the obtained multiple map samples can be subjected to offline batch material generation and rendering, thereby obtaining multiple image samples. The PBR rendering method can be used to improve the authenticity of the image samples, and the obtained image samples are also images with material map features, which are more suitable for map generation tasks.

[0068] 3) Extracting text information of the image samples, obtaining text labels of the image samples, and obtaining a training sample set according to the rendered image samples and the extracted text labels corresponding to the image samples.

[0069] Among them, for extracting text information of image samples and obtaining text labels of image samples, a possible implementation method provided by the present disclosure is, based on a multimodal graphic-text model, respectively inputting image samples into the multimodal graphic-text model, extracting image visual information of the image samples, determining text information corresponding to the image samples, and obtaining text labels of the image samples according to the text information corresponding to the image samples, wherein the multimodal graphic-text model is used to extract text information of images.

[0070] For example, based on the multimodal image-text model, visual question answering and image description information extraction can be used to generate text information based on the image visual information of the image sample, that is, to obtain the text label of the image sample.

[0071] S2. Input the training samples into the image generation model to obtain the text vector representation of the text labels in the training samples, and obtain the image latent space vector representation of the image samples in the training samples. Reconstruct the predicted image based on the image latent space vector representation and the corresponding text vector representation.

[0072] Among them, the image generation model includes a cross-attention layer, which is used to associate the image latent space vector representation and the corresponding text vector representation.

[0073] In the disclosed embodiment, during the training process of the image generation model, the cross-attention layer can be continuously improved and learned based on the training sample set, and the relationship between text information and image information can be better learned, thereby improving the accuracy and performance of the image generation model in generating images from text.

[0074] Specifically, the present disclosure provides a possible implementation method, in which text features are extracted from text labels in training samples through a text encoding module in an image generation model to obtain a text vector representation of the text labels; image features are extracted from image samples in training samples through an image encoding module in the image generation model to obtain an image latent space vector representation of the image samples; and a predicted image is reconstructed based on the image latent space vector representation and the corresponding text vector representation through an image decoding module in the image generation model, wherein the cross-attention layer is located in the image decoding module.

[0075] In the disclosed embodiment, for example, the image generation model adopts a diffusion basic model and a fine-tuning model. For any training sample, the text encoding module can be used to extract text features of the text labels in the training sample to obtain a text vector representation of the text labels. In the diffusion process of the diffusion model, the image encoding module can be used to extract image features, compress the image samples located in the pixel space into the latent space, and obtain the image latent space vector representation of the image samples. Then, the image is reconstructed by the image decoding module, and the image latent space vector representation located in the latent space is reconstructed into a predicted image located in the pixel space. Specifically, the text vector representation and the image latent space vector representation are input into the image decoding module. Through learning and training, the cross-attention layer included in the image decoding module can better learn the text label information and the image sample information. In this way, the image generation model is continuously iterated and trained according to the image samples and the predicted images, so that the predicted image generated by the image generation model is closer to the image sample, and the association relationship with the corresponding text label can also be learned.

[0076] Furthermore, in the disclosed embodiments, feature extraction is performed through the convolution layer in the image generation model, and the convolution layer of the image generation model can also be set to a circular convolution, so that in the image filling process, the opposite edge information can be used to fill new empty pixels, and the opposite edge information can be understood. Therefore, it is set to a circular convolution, which can make it possible to generate a seamless map based on the image generation model, further improving the quality of the target image for making the map, wherein the seamless map means that the top and bottom edges of the image are connected without seams, and the left and right sides are connected without seams.

[0077] S3. According to the image samples and the corresponding predicted images, the image generation model is trained until a preset number of iterations is reached or the target loss function converges, wherein the target loss function at least includes a loss function between the image samples and the corresponding predicted images.

[0078] In this way, in the embodiment of the present disclosure, a texture sample set is obtained, and a training sample set is obtained by rendering the texture sample set, and then an image generation model is obtained by training based on the training sample set. The obtained image generation model is more suitable for the texture generation task scenario. Based on the trained image generation model, a target image with specific material texture features can be generated, thereby improving the effect of subsequent texture generation and improving texture quality and usability.

[0079] The following uses a specific application scenario to illustrate the method for generating a texture in the embodiment of the present disclosure. In the embodiment of the present disclosure, a texture generation tool can also be used to build a pipeline for batch scanning and generating textures. Figure 2 , which is a flow chart of another method for generating a texture in an embodiment of the present disclosure.

[0080] 1) If Figure 2 As shown, based on the image generation model, the text to be processed is input into the image generation model to generate the target image.

[0081] For example, the text to be processed is "a texture image of mahogany", and based on the image generation model, a target image with the texture of mahogany can be generated.

[0082] 2) According to the text to be processed, the material category of the target image is determined, and according to the determined material category, a target scanning configuration file for the target image is determined, and then according to the target scanning configuration file, the target image is scanned and analyzed to generate a target map corresponding to the target image.

[0083] Thus, in the embodiment of the present disclosure, the scanning configuration file can be input into the pipeline, and the scanning parameters more suitable for the target image can be used to perform scanning analysis on the target image. Figure 2 As shown, the target image and the scanning configuration file can be input into the scanning function node to perform scanning analysis on the target image, and finally generate a target map of the target image. For example, the image information of the target image can be obtained through the scanning function node, and the drawing function node can be used to generate a target map corresponding to the target object according to the image information of the target image.

[0084] 3) If Figure 2 As shown, in the process of generating the target map, the functions provided in the map generation tool can be used to efficiently obtain the target map, and the attribute information of the generated target map can be dynamically adjusted according to the needs.

[0085] For example, the roughness, height and other information of the desired target texture can be adjusted through the attribute modification function node in the texture generation tool. In addition, in the embodiment of the present disclosure, the required material feature information can also be added through the feature input function node in the texture generation tool to meet the personalized needs of the material.

[0086] In the disclosed embodiment, an image generation model is trained based on a training sample set of material rendering, and then based on the image generation model, a target image with material map features can be generated, and the target map is generated through the created scanning pipeline. In this way, higher quality material maps can be obtained automatically and more efficiently.

[0087] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0088] Based on the same inventive concept, the embodiment of the present disclosure also provides a sticker generation device corresponding to the sticker generation method. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned sticker generation method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0089] Reference Figure 3 FIG. 1 is a schematic diagram of a map generation device provided by an embodiment of the present disclosure, the device comprising:

[0090] An acquisition module 31 is used to acquire the text to be processed;

[0091] An image generation module 32 is used to perform semantic analysis on the text to be processed based on an image generation model to obtain text features of the text to be processed, and obtain image features associated with the text features based on the text features, and generate a target image based on the image features;

[0092] The map generation module 33 is used to generate a target map corresponding to the target image according to the target image and a target scanning configuration file, wherein the target scanning configuration file includes scanning parameters required for scanning the target image, and the scanning of the target image is used to obtain target image information for generating the target map.

[0093] In an optional implementation, when generating a target map corresponding to the target image according to the target image and the target scanning configuration file, the map generation module 33 is used to:

[0094] Determining the material category of the target image according to the text to be processed;

[0095] According to the material category, determining a target scanning configuration file corresponding to the material category;

[0096] Based on the scanning parameters in the target scanning configuration file, the target image is scanned and analyzed to obtain target image information of the target image, and a target map corresponding to the target image is generated according to the target image information.

[0097] In an optional implementation, when the target image is scanned and analyzed based on the scanning parameters in the target scanning configuration file to obtain target image information of the target image, and a target map corresponding to the target image is generated according to the target image information, the map generation module 33 is used to:

[0098] In response to an instruction to add material feature information, adding the added material feature information to the target image;

[0099] Based on the scanning parameters in the target scanning configuration file, the target image including the added material feature information is scanned and analyzed to obtain target image information of the target image including the added material feature information, and a target map corresponding to the target image is generated according to the target image information.

[0100] In an optional implementation, a training module 34 is further included, and the image generation model is obtained by training the training module 34 through the following steps:

[0101] Acquire a training sample set, wherein the training sample set includes a plurality of training samples, each of the training samples includes an image sample rendered based on a material obtained from a map sample, and a text label corresponding to the image sample;

[0102] Input the training sample into the image generation model, obtain the text vector representation of the text label in the training sample, and obtain the image latent space vector representation of the image sample in the training sample, and reconstruct the predicted image according to the image latent space vector representation and the corresponding text vector representation, wherein the image generation model includes a cross attention layer, and the cross attention layer is used to associate the image latent space vector representation and the corresponding text vector representation;

[0103] The image generation model is trained according to the image sample and the corresponding predicted image until a preset number of iterations is reached or a target loss function converges, wherein the target loss function at least includes a loss function between the image sample and the corresponding predicted image.

[0104] In an optional implementation, a text vector representation of the text label in the training sample is obtained, and an image latent space vector representation of the image sample in the training sample is obtained. When reconstructing a predicted image based on the image latent space vector representation and the corresponding text vector representation, the training module 34 is used to:

[0105] Extracting text features of text labels in the training samples through a text encoding module in the image generation model to obtain a text vector representation of the text labels;

[0106] Extracting image features of the image samples in the training samples through an image encoding module in the image generation model to obtain an image latent space vector representation of the image samples;

[0107] Through the image decoding module in the image generation model, a predicted image is reconstructed according to the image latent space vector representation and the corresponding text vector representation, wherein the cross attention layer is located in the image decoding module.

[0108] In an optional implementation manner, when obtaining the training sample set, the training module 34 is used to:

[0109] Acquire multiple texture sample sets, wherein each of the texture sample sets includes multiple texture samples required for generating a material;

[0110] Obtaining corresponding materials respectively according to the plurality of map sample sets, and rendering the obtained materials to obtain the rendered image samples;

[0111] The text information of the image sample is extracted to obtain the text label of the image sample, and the training sample set is obtained according to the rendered image sample and the text label corresponding to the image sample.

[0112] In an optional implementation, when extracting the text information corresponding to the image sample and obtaining the text label of the image sample, the training module 34 is used to:

[0113] Based on the multimodal graphic-text model, the image samples are respectively input into the multimodal graphic-text model to extract the image visual information of the image samples, and the text information corresponding to the image samples is determined based on the image visual information of the image samples, and the text label of the image sample is obtained based on the text information corresponding to the image samples, wherein the multimodal graphic-text model is used to extract the text information of the image.

[0114] In an optional implementation, the training module 34 is further used to, after acquiring multiple map sample sets, perform image preprocessing on multiple map samples in the multiple map sample sets, wherein the image preprocessing includes at least one of the following: denoising processing and brightness equalization processing.

[0115] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0116] The present disclosure also provides an electronic device, such as Figure 4 FIG. 1 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure, including:

[0117] A processor 41 and a memory 42; the memory 42 stores machine-readable instructions executable by the processor 41, and the processor 41 is used to execute the machine-readable instructions stored in the memory 42. When the machine-readable instructions are executed by the processor 41, the processor 41 is used to perform the following steps:

[0118] Get the text to be processed;

[0119] Based on the image generation model, semantic analysis is performed on the text to be processed to obtain text features of the text to be processed, and based on the text features, image features associated with the text features are obtained, and a target image is generated based on the image features;

[0120] A target map corresponding to the target image is generated according to the target image and a target scanning configuration file, wherein the target scanning configuration file includes scanning parameters required for scanning the target image, and the scanning of the target image is used to obtain target image information for generating the target map.

[0121] In an optional implementation, when generating a target map corresponding to the target image according to the target image and the target scanning configuration file, the processor 41 is configured to:

[0122] Determining the material category of the target image according to the text to be processed;

[0123] According to the material category, determining a target scanning configuration file corresponding to the material category;

[0124] Based on the scanning parameters in the target scanning configuration file, the target image is scanned and analyzed to obtain target image information of the target image, and a target map corresponding to the target image is generated according to the target image information.

[0125] In an optional implementation, when scanning and analyzing the target image based on the scanning parameters in the target scanning configuration file to obtain target image information of the target image, and generating a target map corresponding to the target image according to the target image information, the processor 41 is used to:

[0126] In response to an instruction to add material feature information, adding the added material feature information to the target image;

[0127] Based on the scanning parameters in the target scanning configuration file, the target image including the added material feature information is scanned and analyzed to obtain target image information of the target image including the added material feature information, and a target map corresponding to the target image is generated according to the target image information.

[0128] In an optional implementation manner, the image generation model is obtained by training the processor 41 through the following steps:

[0129] Acquire a training sample set, wherein the training sample set includes a plurality of training samples, each of the training samples includes an image sample rendered based on a material obtained from a map sample, and a text label corresponding to the image sample;

[0130] Input the training sample into the image generation model, obtain the text vector representation of the text label in the training sample, and obtain the image latent space vector representation of the image sample in the training sample, and reconstruct the predicted image according to the image latent space vector representation and the corresponding text vector representation, wherein the image generation model includes a cross attention layer, and the cross attention layer is used to associate the image latent space vector representation and the corresponding text vector representation;

[0131] The image generation model is trained according to the image sample and the corresponding predicted image until a preset number of iterations is reached or a target loss function converges, wherein the target loss function at least includes a loss function between the image sample and the corresponding predicted image.

[0132] In an optional implementation, when obtaining a text vector representation of the text label in the training sample and obtaining an image latent space vector representation of the image sample in the training sample, and reconstructing a predicted image according to the image latent space vector representation and the corresponding text vector representation, the processor 41 is used to:

[0133] Extracting text features of text labels in the training samples through a text encoding module in the image generation model to obtain a text vector representation of the text labels;

[0134] Extracting image features of the image samples in the training samples through an image encoding module in the image generation model to obtain an image latent space vector representation of the image samples;

[0135] Through the image decoding module in the image generation model, a predicted image is reconstructed according to the image latent space vector representation and the corresponding text vector representation, wherein the cross attention layer is located in the image decoding module.

[0136] In an optional implementation manner, when obtaining the training sample set, the processor 41 is used to:

[0137] Acquire multiple texture sample sets, wherein each of the texture sample sets includes multiple texture samples required for generating a material;

[0138] Obtaining corresponding materials respectively according to the plurality of map sample sets, and rendering the obtained materials to obtain the rendered image samples;

[0139] The text information of the image sample is extracted to obtain the text label of the image sample, and the training sample set is obtained according to the rendered image sample and the text label corresponding to the image sample.

[0140] In an optional implementation, when extracting the text information corresponding to the image sample to obtain the text label of the image sample, the processor 41 is used to:

[0141] Based on the multimodal graphic-text model, the image samples are respectively input into the multimodal graphic-text model to extract the image visual information of the image samples, and the text information corresponding to the image samples is determined based on the image visual information of the image samples, and the text label of the image sample is obtained based on the text information corresponding to the image samples, wherein the multimodal graphic-text model is used to extract the text information of the image.

[0142] In an optional implementation, the processor 41 is further used to, after acquiring multiple texture sample sets, perform image preprocessing on multiple texture samples in the multiple texture sample sets, wherein the image preprocessing includes at least one of the following: denoising processing and brightness equalization processing.

[0143] The memory 42 includes a memory 421 and an external memory 422; the memory 421 is also referred to as an internal memory, which is used to temporarily store the operation data in the processor 41 and the data exchanged with the external memory 422 such as a hard disk. The processor 41 exchanges data with the external memory 422 through the memory 421. The specific execution process of the above instructions can refer to the steps of the map generation method described in the embodiment of the present disclosure, and will not be repeated here.

[0144] The embodiment of the present disclosure also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the map generation method described in the above method embodiment are executed. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0145] The embodiments of the present disclosure also provide a computer program product, which carries a program code. The instructions included in the program code can be used to execute the steps of the map generation method described in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.

[0146] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is implemented as a computer storage medium. In another optional embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0147] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0148] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0149] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0150] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0151] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A method for generating a texture, It is characterized in that include: Get the text to be processed; Based on the image generation model, semantic analysis is performed on the text to be processed to obtain text features of the text to be processed, and based on the text features, image features associated with the text features are obtained, and a target image is generated based on the image features; A target map corresponding to the target image is generated according to the target image and a target scanning configuration file, wherein the target scanning configuration file includes scanning parameters required for scanning the target image, and the scanning of the target image is used to obtain target image information for generating the target map.

2. The method according to claim 1, It is characterized in that Generating a target map corresponding to the target image according to the target image and the target scanning configuration file includes: Determining the material category of the target image according to the text to be processed; According to the material category, determining a target scanning configuration file corresponding to the material category; Based on the scanning parameters in the target scanning configuration file, the target image is scanned and analyzed to obtain target image information of the target image, and a target map corresponding to the target image is generated according to the target image information.

3. The method according to claim 2, It is characterized in that The step of scanning and analyzing the target image based on the scanning parameters in the target scanning configuration file to obtain target image information of the target image, and generating a target map corresponding to the target image according to the target image information includes: In response to an instruction to add material feature information, adding the added material feature information to the target image; Based on the scanning parameters in the target scanning configuration file, the target image including the added material feature information is scanned and analyzed to obtain target image information of the target image including the added material feature information, and a target map corresponding to the target image is generated according to the target image information.

4. The method according to any one of claims 1 to 3, It is characterized in that The image generation model is obtained by training through the following steps: Acquire a training sample set, wherein the training sample set includes a plurality of training samples, each of the training samples includes an image sample rendered based on a material obtained from a map sample, and a text label corresponding to the image sample; Input the training sample into the image generation model, obtain the text vector representation of the text label in the training sample, and obtain the image latent space vector representation of the image sample in the training sample, and reconstruct the predicted image according to the image latent space vector representation and the corresponding text vector representation, wherein the image generation model includes a cross attention layer, and the cross attention layer is used to associate the image latent space vector representation and the corresponding text vector representation; The image generation model is trained according to the image sample and the corresponding predicted image until a preset number of iterations is reached or a target loss function converges, wherein the target loss function at least includes a loss function between the image sample and the corresponding predicted image.

5. The method according to claim 4, It is characterized in that Obtaining a text vector representation of the text label in the training sample, and obtaining an image latent space vector representation of the image sample in the training sample, and reconstructing a predicted image according to the image latent space vector representation and the corresponding text vector representation, including: Extracting text features of text labels in the training samples through a text encoding module in the image generation model to obtain a text vector representation of the text labels; Extracting image features of the image samples in the training samples through an image encoding module in the image generation model to obtain an image latent space vector representation of the image samples; Through the image decoding module in the image generation model, a predicted image is reconstructed according to the image latent space vector representation and the corresponding text vector representation, wherein the cross attention layer is located in the image decoding module.

6. The method according to claim 4 or 5, It is characterized in that The obtaining of the training sample set comprises: Acquire multiple texture sample sets, wherein each of the texture sample sets includes multiple texture samples required for generating a material; Obtaining corresponding materials respectively according to the plurality of map sample sets, and rendering the obtained materials to obtain the rendered image samples; The text information of the image sample is extracted to obtain the text label of the image sample, and the training sample set is obtained according to the rendered image sample and the text label corresponding to the image sample.

7. The method according to claim 6, It is characterized in that The extracting text information corresponding to the image sample to obtain a text label of the image sample includes: Based on a multimodal graphic-text model, the image sample is input into the multimodal graphic-text model, image visual information of the image sample is extracted, and text information corresponding to the image sample is determined based on the image visual information of the image sample, and a text label of the image sample is obtained based on the text information corresponding to the image sample, wherein the multimodal graphic-text model is used to extract text information of the image.

8. The method according to claim 6, It is characterized in that After obtaining the plurality of texture sample sets, the method further comprises: Performing image preprocessing on a plurality of the texture samples in a plurality of the texture sample sets, wherein the image preprocessing includes at least one of the following: denoising processing and brightness equalization processing.

9. A map generating device, It is characterized in that include: An acquisition module is used to obtain the text to be processed; An image generation module, configured to perform semantic analysis on the text to be processed based on an image generation model, obtain text features of the text to be processed, obtain image features associated with the text features based on the text features, and generate a target image based on the image features; A map generation module is used to generate a target map corresponding to the target image according to the target image and a target scanning configuration file, wherein the target scanning configuration file includes scanning parameters required for scanning the target image, and the scanning of the target image is used to obtain target image information for generating the target map.

10. An electronic device, It is characterized in that include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is used to execute the machine-readable instructions stored in the memory. When the machine-readable instructions are executed by the processor, the processor performs the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, Features: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images