Large-scale high-quality texture image synthesis method based on hierarchical Bayesian structure

Abstract

The invention discloses a large-scale high-quality texture image synthesis method based on a hierarchical Bayesian structure, and belongs to the technical field of texture image synthesis. According to the method, the texture image is synthesized and described as a double-flow hierarchical Bayesian structure provided by shaping and disturbance on a regular substrate, and the neural network combining the frequency domain and the space domain is designed according to the structure, so that the network can capture global features and local details at the same time, and the overall quality and consistency of the generated image are improved. Through the optimized network structure and parameter setting, the model has high real-time performance when the texture image is generated, and the method is suitable for real-time application scenes. The method does not depend on input texture samples for reasoning, and allows that large-scale high-quality texture images can be generated only by depending on coarse-grained control variables without specific examples. According to the invention, model training is carried out based on adversarial loss, so that the generated texture image is clearer and richer in details.

Description

Technical Field

[0001] The present invention belongs to the technical field of texture image synthesis, and in particular relates to a large-scale high-quality texture image synthesis method based on a hierarchical Bayesian structure. Background Art

[0002] In practical applications such as film post-production and virtual reality, texture synthesis can provide users with realistic scenes. Therefore, texture synthesis has long been a research hotspot in computer vision and computer graphics.

[0003] According to the existing research on texture synthesis at home and abroad, current texture synthesis methods can be divided into: process texture, sample-based texture synthesis and coarse-grained controllable texture synthesis. (1) The process texture method is to synthesize texture as a function of pixel coordinates and a set of adjustable parameters; its main feature is that it can synthesize texture from scratch without raw materials, is independent of image resolution, and can create more accurate textures, especially when the textured object has a large and varying depth. However, this method is difficult to determine the function that can generate a given texture, and the synthesis result is difficult to control. (2) Sample-based texture synthesis refers to the technology of matching or extending certain source images, which is usually proposed to generate large (non-periodic) texture images from small-scale texture images provided by users. It is also a method with important practical application value in many fields at present, but this method has poor flexibility and the network is associated with specific texture images. (3) The coarse-grained controllable texture synthesis method is to generate texture images of arbitrary size out of thin air based on high-level cognitive conditions that can be understood by humans. Previous work in this area obtained fixed-size texture images from label semantics, which cannot be regarded as texture synthesis models in a strict sense. Therefore, there is a lack of current research in this field, but these studies have shown that there is a great demand for coarse-grained controllable texture synthesis in most scenarios.

[0004] In summary, the current difficulties faced by texture synthesis are: (1) the process texture parameters are difficult to adjust and the synthesis results are uncontrollable; (2) sample-based texture synthesis lacks flexibility; (3) there is a lack of in-depth research on coarse-grained controllable texture synthesis. Summary of the invention

[0005] The purpose of the present invention is to provide a large-scale high-quality texture image synthesis method based on a hierarchical Bayesian structure to make up for the shortcomings of the prior art.

[0006] To achieve the above object, the present invention is implemented by adopting the following scheme:

[0007] A large-scale high-quality texture image synthesis method based on a hierarchical Bayesian structure comprises the following steps:

[0008] S1: Construct a texture image synthesis model based on a hierarchical Bayesian structure: The texture image synthesis model includes a generator and a discriminator; the generator integrates a convolutional neural network in the spatial domain and a residual network in the frequency domain, models the regular basis of the texture in the residual network in the frequency domain, and models the shaping and perturbation of the regular basis in the convolutional neural network in the spatial domain; the discriminator has a mirrored network structure with the convolutional neural network in the texture synthesis model;

[0009] S2: preprocessing the data on the collected large texture dataset;

[0010] S3: Train the texture image synthesis model based on the idea of ​​adversarial training;

[0011] S4: Based on the trained texture image synthesis model, coarse-grained control variables are input to finally generate large-scale high-quality texture images.

[0012] Furthermore, the S1 includes:

[0013] S1-1: The frequency domain residual network refers to the design of the frequency domain deep residual network by Transformer and MLP-Mixer, and introduces the quantization operation; the specific structure of the deep residual network is: fully connected layer → 4 identical residual blocks → quantization layer → channel embedding → 4 identical mixing layers → output layer → rearrangement → calculation of correction frequency → MLP layer. Among them, the residual block is composed of Norm layer (layer normalization) → MLP layer → Add layer; the mixing layer is composed of Norm layer → spatial mixing → Norm layer → channel mixing → residual connection. The quantization layer is mainly composed of two steps: query and index. First, the quantization layer divides the input feature vector into four different parts, each of which has the same size; then, the quantization layer calculates the distance between each part and all words in the dictionary, and finds the index of the word closest to it; finally, the quantization layer uses the obtained index to obtain words from the dictionary to replace the corresponding part of the original feature vector to form the output vector.

[0014] After obtaining the features of the quantization layer output, the channel embedding in the deep residual network spreads the feature vector of each instance to different channels of the output signal. To illustrate how channel embedding works, the feature vector output by the quantization layer is expressed as:

[0015] x= <x1,x2,...,x n >,

[0016] Where n represents the dimension of the feature vector. Channel embedding consists of a set of learnable vectors and is expressed as:

[0017]

[0018] Where m represents the number of channel embeddings, e1, e2, …e m Represent different channel embeddings respectively. Add the feature vector output by the quantization layer to the channel embedding to obtain the feature vector of the instance on different channels:

[0019]

[0020] The mixed layer is used to perform multi-level feature transformation and inter-channel interaction on the diffused features; finally, a frequency domain learning network is constructed.

[0021] S1-2: Constructing a convolutional neural network in the spatial domain:

[0022] In the two-stream hierarchical Bayesian structure proposed in the present invention, the conditional random field is placed at the bottom layer to support general texture synthesis. A convolutional neural network with additional input variables (observation variables) is used to simulate the conditional random field. The convolutional neural network is composed of multiple residual modules GB1 and GB2 (Group Block 1 and Group Block 2) alternatingly, where the structure of GB1 is: 1*1 convolution layer → bilinear upsampling → SENoise layer → 3*3 convolution layer → 1*1 convolution layer → residual connection; there is no upsampling layer in the structure of GB2, and the rest of the structure is consistent with GB1.

[0023] S1-21: Bilinear sampling is placed after the 1*1 convolutional layer to improve the resolution of the feature map for subsequent processing.

[0024] S1-22: SENoise layer is an operation that performs random compensation on the feature map. It directly calculates α from the coarse-grained control variable c and the hidden variable z1 i To randomly compensate each channel of the feature map. In addition, use β i and γ i To calibrate the features of each convolutional layer output in the residual module. Assuming X is the feature map obtained after upsampling in step S1-21, then use β i and γ i The calibrated feature map is:

[0025]

[0026] in, and std(X) are the mean and standard deviation of X computed over the instance and spatial axes, β i and γ iIs a vector with the same dimension as the number of channels in X, representing the offset and scaling factors, respectively, and i indexes the convolutional layer to which the offset and scaling factors are applied. The offset and scaling factors are designed to be personalized for each strength. Because the statistics of the feature map have been shown to affect the style of the convolutional neural network output, this method uses this layer to modulate the computation process of the convolutional neural network.

[0027] S1-23: The overall architecture of the convolutional neural network is as follows: after the 1*1 convolutional layer, four modules consisting of alternating GB2 and GB1 are added, followed by two GB2s, and finally a 3*3 convolutional layer is connected for output.

[0028] According to the two-stream hierarchical Bayesian structure proposed in the present invention, the output of the frequency domain residual network is converted into a texture regular basis s by inverse Fourier transform and analog sampling as the observation variable of the conditional random field and input into the convolutional neural network in the spatial domain. The texture regular basis describes the global alignment of texture visual elements in space, and the conditional probability of the conditional random field models the plasticity and perturbation on the texture regular basis.

[0029] Furthermore, the preprocessing in S2 includes: cropping and scaling the original texture image so that the resources required for training are within an affordable range, while avoiding the impact of non-homogeneous content at the edges of some texture images on the model.

[0030] Furthermore, in S3, the model is trained based on the idea of ​​adversarial training:

[0031] S3-1: The texture image synthesis model is trained using the earth moving distance in WGANs, and the entire optimization process is described as:

[0032]

[0033] Where G represents the generator, D represents the discriminator, and P real is the distribution of the real data, P fake is the distribution that generated the data.

[0034] S3-2: Use the Adam optimization algorithm to optimize the generator and discriminator until the model converges.

[0035] Compared with the prior art, the main advantages and positive effects of the present invention are as follows:

[0036] The present invention proposes a two-stream hierarchical Bayesian structure by describing texture image synthesis as shaping and perturbation on a regular basis, and designs a neural network combining frequency domain and spatial domain based on this structure, so that the network can capture global features and local details at the same time, thereby improving the overall quality and consistency of the generated image. The present invention optimizes the network structure and parameter settings, so that the model has high real-time performance when generating texture images, and is suitable for real-time application scenarios.

[0037] Unlike traditional texture synthesis methods, the present invention does not rely on input texture samples for reasoning, allowing large-scale high-quality texture images to be generated without specific examples by relying solely on coarse-grained control variables. The present invention trains the model based on adversarial loss, making the generated texture images clearer and more detailed. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 The flowchart of the large-scale texture synthesis method described in an embodiment of the present invention.

[0039] Figure 2 This is the dual-stream hierarchical Bayesian structure described in the embodiment of the present invention.

[0040] Figure 3 A schematic diagram of a quantization layer dictionary according to an embodiment of the present invention.

[0041] Figure 4 This is the final frequency domain learning network constructed in the embodiment of the present invention.

[0042] Figure 5 This is the overall structure of the texture synthesis model described in the embodiment of the present invention.

[0043] Figure 6 This is the network structure of the discriminator and its residual module described in the embodiment of the present invention.

[0044] Figure 7 These are the experimental results of the embodiments of the present invention on the HRTD-G dataset.

[0045] Figure 8 The following are experimental results on the HRTD dataset described in the embodiments of the present invention.

[0046] Fig. 9 These are the experimental results on the PTD dataset described in the embodiments of the present invention.

[0047] Fig.10 It is a large-scale texture image generated on the HRTD-G dataset described in an embodiment of the present invention.

[0048] Fig.11 This is a large-scale texture image generated on the HRTD dataset described in an embodiment of the present invention.

[0049] Fig.12 These are texture images generated by the combination method described in the embodiment of the present invention on three data sets. DETAILED DESCRIPTION

[0050] The present invention is further explained and illustrated below through specific embodiments in conjunction with the accompanying drawings.

[0051] Embodiment 1:

[0052] A large-scale high-quality texture image synthesis method based on a hierarchical Bayesian structure, such as Figure 1 As shown, the following steps are included:

[0053] Step 1: Construct a texture image synthesis model based on a hierarchical Bayesian structure;

[0054] Step 2: preprocessing the data on the large texture dataset;

[0055] Step 3: Train the model based on the idea of ​​adversarial training;

[0056] Step 4: Input coarse-grained control variables to generate large-scale high-quality texture images.

[0057] This method innovatively describes the formation of texture images as shaping and perturbations on a regular basis, and proposes a two-stream hierarchical Bayesian structure of the model, such as Figure 2 As shown. The specific sampling relationship indicated by this structure is as follows: First, a coarse-grained control variable is sampled from the prior distribution of the coarse-grained control variable, and the sampled coarse-grained control variable is recorded as c, and the prior distribution of the coarse-grained control variable is recorded as π, that is, c~π. Each coarse-grained control variable determines a probability distribution, which is represented by M. Since the coarse-grained control variable can only describe part of the characteristics of the target texture image, each coarse-grained control variable actually corresponds to a series of different types of textures, where each type of texture is the output of a conditional random field under a specific observation variable condition. In other words, M is a joint probability distribution over a conditional random field and an observation variable. Therefore, each sampling from M will obtain a specific conditional random field T and a specific observation variable s. Among them, s corresponds to the regular basis of the texture, which is traced back to the continuous signal for frequency domain modeling, and a random grid is used to obtain a regular basis of the texture with arbitrary shape from the simulated source signal. By inputting the sampled observation variable into the conditional random field T sampled by the left branch, different texture images can be sampled.

[0058] The specific steps include:

[0059] 1. Construct a large-scale texture synthesis model based on hierarchical Bayesian structure:

[0060] Based on the dual-stream hierarchical Bayesian structure, the present invention designs a learning network that integrates the spatial domain and the frequency domain, models the regular basis of texture in the frequency domain, and models the shaping and perturbation of the regular basis in the spatial domain. The core architecture of the network is the residual network in the frequency domain and the convolutional neural network in the spatial domain for simulating conditional random fields.

[0061] (1.1) Residual network in frequency domain:

[0062] In order to effectively predict the frequency domain representation of the texture regular basis, the present invention draws on Transformer and MLP-Mixer to design a residual network in the frequency domain and introduces quantization operations. The specific structure of the deep residual network is: fully connected layer → 4 identical residual blocks → quantization layer → channel embedding → 4 identical mixing layers → output layer → rearrangement → calculation of correction frequency → MLP layer. Among them, the composition of the residual block is Norm layer (layer normalization) → MLP layer → Add layer; the composition of the mixing layer is Norm layer → spatial mixing → Norm layer → channel mixing → residual connection. Among them, the workflow of the quantization layer is as follows Figure 3 As shown, the universal gravitation is simulated to establish a gravitational relationship between the query and the word to solve the word mismatch problem. The quantization layer mainly consists of two steps: query and index. First, the quantization layer divides the input feature vector into four different parts, each of which has the same size; then, the quantization layer calculates the distance between each part and all the words in the dictionary, and finds the index of the word closest to it; finally, the quantization layer uses the obtained index to obtain the word from the dictionary to replace the corresponding part of the original feature vector to form the output vector. The algorithm in Table 1 below gives the forward propagation and backpropagation process of the quantization layer.

[0063] Table 1

[0064]

[0065] After obtaining the features of the quantization layer output, the channel embedding in the network spreads the feature vector of each instance to different channels of the output signal. To illustrate how channel embedding works, the feature vector output by the quantization layer is expressed as:

[0066] x= <x1,x2,...,x n >,

[0067] Where n represents the dimension of the feature vector. Channel embedding consists of a set of learnable vectors and can be expressed as:

[0068]

[0069] Where m represents the number of channel embeddings, e1, e2, …e mRepresent different channel embeddings respectively. We add the feature vector output by the quantization layer to the channel embedding to obtain the feature vector of the instance on different channels:

[0070]

[0071] In this way, the diffusion of feature vectors to different channels is achieved with very few parameters. After feature diffusion, the feature vector on each channel is still a certain distance away from the final frequency domain representation corresponding to the channel. At the same time, considering the possible global dependencies of the image frequency domain representation, the hybrid layer is used to perform multi-level feature transformation and channel interaction on the diffused features. Finally, the following is constructed: Figure 4 The frequency domain learning network shown in the figure. In the figure, f(A) represents the correction frequency in the corresponding discretized box directly calculated according to the amplitude function A. The specific implementation algorithm is shown in Table 2.

[0072] Table 2

[0073]

[0074] (1.2) Construct a convolutional neural network in the spatial domain to simulate conditional random fields:

[0075] In the dual-stream hierarchical Bayesian structure proposed in the present invention (such as Figure 2 ), the conditional random field is placed at the bottom layer to support general texture synthesis. This method uses a convolutional neural network with additional input variables (observation variables) to simulate the conditional random field. The convolutional neural network is composed of multiple residual modules GB1 and GB2 alternately. The structure of GB1 is: 1*1 convolution layer → bilinear upsampling → SENoise layer → 3*3 convolution layer → 1*1 convolution layer → residual connection; there is no upsampling layer in the structure of GB2, and the rest of the structure is consistent with GB1.

[0076] (1.2.1) Bilinear sampling is placed after the 1*1 convolutional layer to improve the resolution of the feature map for subsequent processing.

[0077] (1.2.2) SENoise layer is an operation that performs random compensation on the feature map. It directly calculates α from the coarse-grained control variable c and the hidden variable z1 i To randomly compensate each channel of the feature map. In addition, this method uses β i and γ i To calibrate the features of each convolutional layer output in the residual module. Assuming X is the feature map obtained after upsampling in step 1.2.1, use β i and γ i The calibrated feature map is:

[0078]

[0079] in, and std(X) are the mean and standard deviation of X computed over the instance and spatial axes, β i and γ i Is a vector with the same dimension as the number of channels in X, representing the offset and scaling factors, respectively, and i indexes the convolutional layer to which the offset and scaling factors are applied. The offset and scaling factors are designed to be personalized for each strength. Because the statistics of the feature map have been shown to affect the style of the convolutional neural network output, this method uses this layer to modulate the computation process of the convolutional neural network.

[0080] (1.2.3) The overall architecture of the convolutional neural network is as follows: after the 1*1 convolutional layer, four modules consisting of alternating GB2 and GB1 are added, followed by two GB2s, and finally a 3*3 convolutional layer is connected for output.

[0081] (1.3) The frequency domain learning network is combined with the spatial domain learning network to form a texture synthesis model:

[0082] According to the dual-stream hierarchical Bayesian structure proposed in the present invention, the output of the frequency domain residual network is converted into a texture regular basis s by inverse Fourier transform and analog sampling as the observation variable of the conditional random field and input into the convolutional neural network in the spatial domain. The texture regular basis describes the global alignment of texture visual elements in space, and the conditional probability of the conditional random field models the plasticity and perturbation on the texture regular basis. The frequency domain network and the spatial domain network constructed above are combined to form the final Figure 5 The network shown in the figure takes the coarse-grained control variable c and the hidden variable z1 as input, which are input into the frequency domain network and the spatial domain network; the hidden variable z2 formally constitutes a random tensor and is decomposed into different fragments to supplement the randomness of the final synthesized texture image at different spatial scales.

[0083] 2. Preprocessing of data on large texture datasets:

[0084] (2.1) The original images in the texture dataset have high resolution. If they are used directly for training, the model's requirements for computing resources will exceed our tolerance. Therefore, the present invention crops and scales the original texture images to keep the resources required for training within an acceptable range, while avoiding the impact of non-homogeneous content at the edges of some texture images on the model. When the present invention is implemented, the center part of the image in the texture dataset is cropped and scaled to a size of 224*224 as the final training sample.

[0085] (2.2) The high resolution of the original texture images in the three datasets HRTD-G, HRTD, and PTD allows future research to achieve texture synthesis with higher resolution by using lightweight models or obtaining more computing resources.

[0086] 3. Train the model based on the idea of ​​adversarial training:

[0087] (3.1) In the present invention, the discriminator and the convolutional neural network in the texture synthesis model have a mirrored network structure. The entire synthesis model is trained using the earth-moving distance in WGANs. Therefore, the entire optimization process is described as:

[0088]

[0089] Where G represents the generator, D represents the discriminator, and P real is the distribution of the real data, P fake is the distribution of generated data. The network structure of the discriminator is as follows Figure 6 As shown in (a), 3*3 convolution → 2 consecutive DB2 → 4 DB1 and DB2 cross blocks → 1*1 convolution → aggregation with the feature map obtained by the coarse-grained control variable, where the structure of the DB1 and DB2 modules is mirrored by the convolutional neural network structure in the generator, as shown in Figure 6 (b) and Figure 6 (c) as shown.

[0090] (3.2) The Adam optimization algorithm is used to optimize the generator and discriminator until the model converges.

[0091] 4. Finally, input the coarse-grained control variables to generate large-scale high-quality texture images.

[0092] Embodiment 2:

[0093] The effect of the present invention is further described below in conjunction with simulation experiments:

[0094] The present invention conducts simulation experiments on three texture data sets, and all of them are carried out on the same server. The server is equipped with two NVIDIA TI-TAN RTX graphics cards, two Intel(R) Xeon(R) CPU E5-2678 v3@2.50GHz processors, 64GB memory, Ubuntu 16.04.6LTS operating system, Python 3.8.3, CUDA 11.2, and Pytorch 1.7.0. The three experimental data sets are HRTD-G, HRTD, and PTD. In the HRTD-G dataset, 10 process texture models are used to generate 1000 texture images, and each process texture model generates 10 texture images. The resolution of these texture images is 512*512 and they are stored as eight-bit PNG grayscale images. The texture images in the HRTD dataset are all natural textures collected from the Internet. These texture images have 10 categories, and each category of texture contains 100 texture images, and all are 24-bit RGB color images. The PTD dataset consists of 450 texture images, each of which has a perceptual feature vector consisting of 12 perceptual features.

[0095] Figure 7 , 8 and 9 are the experimental results on HRTD-G, HRTD and PTD datasets respectively. Figure 7 The first to tenth rows in correspond to the first 9 categories in the HRTD-G dataset; Figure 8 The first to tenth rows in the figure correspond to the texture synthesis results of the nine categories of "blanket", "brick", "fabric", "marble", "metal", "water", "sky", "tile" and "wood" respectively; Fig. 9 The bar graph on the left side of each row lists the perceptual feature vectors that generate the texture image in that row. The perceptual feature vectors are arranged in the following order from bottom to top: contrast, repeatability, granularity, randomness, roughness, feature density, directionality, structural complexity, roughness, regularity, local directionality, and uniformity. For ease of observation, the three largest perceptual features in the perceptual feature vectors corresponding to each row of texture images are colored red in the bar graph. In particular, in order to illustrate the spatial expansion capability of the texture model proposed in the present invention, Fig.10 and Fig.11 These are large-scale texture images generated on the HRTD-G and HRTD datasets respectively.

[0096] Because the present invention is the first to propose the synthesis of texture images with arbitrary scales directly from coarse-grained control variables, there is no baseline method in the relevant literature that can be used for performance comparison. In order to make the present invention a benchmark method in this research field, a comparative experiment was conducted to prove the advantages of the present invention over the direct integration of existing technologies. The object of the comparative experiment was the texture image generation method of the present invention and the direct integration of existing technologies. You can first use the image generation method to create a texture image with a fixed size from the coarse-grained control variables, and then apply the texture synthesis method to expand the generated texture image to a larger scale. The first-stage Style-GAN2 method in the combined method is proposed in the article "Analyzing and improving the image quality of stylegan"; the second-stage PSGAN method is proposed in the article "Learning texture manifolds with the periodic spatial GAN". Fig.12 are the texture images generated by the combined method. The texture images in the first, second, and third rows are generated by training the models within the combined method in HRTD, HRTD-G, and PTD, respectively.

[0097] This method uses FID and MSFID as texture quality evaluation indicators. FID assumes that the outputs of the generated samples and the real samples in the middle layer of the Inception-V3 model obey the multivariate Gaussian distribution, and calculates the Fréchet Distance between them by estimating the mean and covariance of the two Gaussian distributions:

[0098]

[0099] where μ g and μ r Represents the mean of the feature vectors output by the generated data and the real data in the middle layer of the Inception-V3 model, ∑ g and∑ r They represent the covariance matrices of the feature vectors output by the generated data and the real data in the middle layer of the Inception-V3 model, and Tr represents the trace of the matrix.

[0100] MSFID is a new texture quality evaluation index created by this method based on FID, which is used to measure the distribution distance between synthetic textures and real textures at multiple scales. Specifically, in order to evaluate the spatial scalability of the texture synthesis model, the FID distance between the synthetic texture and the real texture is calculated at gradually increasing scales, and finally their average value is taken as a comprehensive evaluation of the authenticity, richness and scalability of the texture synthesis model. The calculation is as follows:

[0101]

[0102] Where i represents each scale, and n represents the total number of scales taken. In this experiment, the value of n is 4.

[0103] Table 1 M C The columns represent the evaluation of the combined method, M D The columns represent the evaluation of this method. Each model in the table uses the same hyperparameter settings when training on different datasets. S 、FID M 、FID L and FID H It represents the FID values ​​calculated at four scales of 146, 166, 186 and 206. From Table 3, it is obvious that the method is superior to the combined method.

[0104] Table 3 Quantitative evaluation results

[0105]

[0106]

[0107] The large-scale controllable texture synthesis method based on the dual-stream hierarchical Bayesian structure provided by the present invention completes the construction and training of the texture generation model through the residual network in the frequency domain, the convolutional neural network for simulating conditional random fields in the spatial domain, and the adversarial training method. Of course, the present invention is also applicable to the generation of ordinary images.

[0108] The method proposed in the present invention belongs to coarse-grained controllable texture synthesis, which solves the problem that the process texture synthesis results are difficult to control and the problem that the sample-based texture synthesis method lacks flexibility, and breaks the bottleneck of the fixed scale of the generated results in the early coarse-grained controllable texture synthesis research, becoming the third new paradigm of texture synthesis besides process texture and sample-based texture synthesis. The present invention can directly generate texture images of any scale based on high-level cognitive conditions that can be understood by humans, greatly enriching the application scenarios of texture synthesis, and is of great significance in the fields of artistic creation, image restoration, virtual reality, and film post-production.

[0109] The above description is only a preferred embodiment of the present invention and does not limit the present invention in other forms. Any technician familiar with the profession may use the technical content disclosed above to change or modify it into an equivalent embodiment with equivalent changes and apply it to other fields. However, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A large-scale high-quality texture image synthesis method based on a hierarchical Bayesian structure, characterized in that: The method comprises the following steps: S1: Construct a texture image synthesis model based on a hierarchical Bayesian structure: The texture image synthesis model includes a generator and a discriminator; the generator integrates a convolutional neural network in the spatial domain and a residual network in the frequency domain, models the regular basis of the texture in the residual network in the frequency domain, and models the shaping and perturbation of the regular basis in the convolutional neural network in the spatial domain; the discriminator has a mirrored network structure with the convolutional neural network in the texture synthesis model; S2: preprocessing the data on the collected large texture dataset; S3: Train the texture image synthesis model based on the idea of ​​adversarial training; S4: Based on the trained texture image synthesis model, coarse-grained control variables are input to finally generate large-scale high-quality texture images.

2. The large-scale high-quality texture image synthesis method according to claim 1, characterized in that: In S1, the residual network in the frequency domain draws on Transformer and MLP-Mixer to design a deep residual network in the frequency domain, and introduces quantization operation; the specific structure of the residual network is: fully connected layer → 4 identical residual blocks → quantization layer → channel embedding → 4 identical mixing layers → output layer → rearrangement → calculation of correction frequency → MLP layer; among which, the residual block is composed of Norm layer → MLP layer → Add layer; the mixing layer is composed of Norm layer → spatial mixing → Norm layer → channel mixing → residual connection.

3. The large-scale high-quality texture image synthesis method according to claim 2, characterized in that: The processing process of the residual network in the frequency domain is as follows: first, the quantization layer divides the input feature vector into four different parts, each of which has the same size; then, the quantization layer calculates the distance between each part and all the words in the dictionary, and finds the index of the word closest to it; finally, the quantization layer uses the obtained index to obtain the word from the dictionary to replace the corresponding part of the original feature vector to form an output vector; After obtaining the features of the quantization layer output, the channel embedding in the deep residual network spreads the feature vector of each instance to different channels of the output signal; to illustrate how channel embedding works, the feature vector of the quantization layer output is expressed as: x=<x1,x2,…,x n >, Where n represents the dimension of the feature vector; the channel embedding consists of a set of learnable vectors and is expressed as: Where m represents the number of channel embeddings, e1, e2, …e m Represent different channel embeddings respectively; add the feature vector output by the quantization layer to the channel embedding to obtain the feature vector of the instance on different channels: The mixing layer is used to perform multi-level feature transformation and channel interaction on the diffused features.

4. The large-scale high-quality texture image synthesis method according to claim 1, characterized in that: The convolutional neural network in the spatial domain uses a convolutional neural network with additional input variables to simulate the conditional random field. The convolutional neural network is composed of multiple residual modules GB1 and GB2 alternately, wherein the structure of GB1 is: 1*1 convolution layer→bilinear upsampling→SENoise layer→3*3 convolution layer→1*1 convolution layer→residual connection; there is no upsampling layer in the structure of GB2, and the rest of the structure is consistent with GB1.

5. The large-scale high-quality texture image synthesis method according to claim 4, characterized in that: The processing process of the convolutional neural network in the spatial domain includes: S1-21: Bilinear sampling is placed after the 1*1 convolution layer to improve the resolution of the feature map for subsequent processing; S1-22: SENoise layer is an operation that performs random compensation on the feature map; it directly calculates α from the coarse-grained control variable c and the hidden variable z1 i To randomly compensate each channel of the feature map; in addition, use β i and γ i To calibrate the features of each convolutional layer output in the residual module; assuming that X is the feature map obtained after upsampling, then use β i and γ i The calibrated feature map is: in, and std(X) are the mean and standard deviation of X computed over the instance and spatial axes, β i and γ i is a vector with the same dimension as the number of channels in X, representing the offset and scaling factors respectively, and i indexes the convolutional layer to which the offset and scaling factors are applied; S1-23: After the 1*1 convolutional layer, four modules consisting of alternating GB2 and GB1 are added, followed by two GB2s, and finally a 3*3 convolutional layer is connected for output.

6. The large-scale high-quality texture image synthesis method according to claim 1, characterized in that: The output of the frequency domain residual network is converted into a texture regular basis s by inverse Fourier transform and analog sampling and input into the convolutional neural network in the spatial domain as the observation variable of the conditional random field. The texture regular basis describes the global alignment of texture visual elements in space, while the conditional probability of the conditional random field models the plasticity and perturbation on the texture regular basis.

7. The large-scale high-quality texture image synthesis method according to claim 1, characterized in that: The preprocessing in S2 includes: cropping and scaling the original texture image so that the resources required for training are within an affordable range, while avoiding the impact of non-homogeneous content at the edges of some texture images on the model.

8. The large-scale high-quality texture image synthesis method according to claim 1, characterized in that: In S3, the model is trained based on the idea of ​​adversarial training: S3-1: The texture image synthesis model is trained using the earth moving distance in WGANs, and the entire optimization process is described as: Where G represents the generator, D represents the discriminator, and P real is the distribution of the real data, P fake is the distribution that generated the data; S3-2: Use the Adam optimization algorithm to optimize the generator and discriminator until the model converges.

Images