An image generation method based on generative adversarial network

Abstract

The present invention discloses an image generation method based on a generative adversarial network, comprising the following steps: collecting an image dataset containing different styles, themes, and elements; using NLP technology to train an instruction parsing module to parse user instructions; constructing and training a creative fusion network to fuse features of different dimensions of the image dataset according to user instructions to generate a creative feature representation; constructing a generator and a discriminator based on the generative adversarial network, and using adversarial training to optimize the quality and creativity of the generated images; introducing a creative stimulation mechanism and establishing a user feedback loop; testing the generated images, and optimizing the generation method based on user feedback and evaluation indicators; by combining the generative adversarial network and the creative fusion network, high-quality and creative images can be generated. At the same time, the creative fusion network can fuse multiple features to generate novel and unique images to meet the diverse needs of users.

Description

Technical Field

[0001] The present invention relates to an image generation method, and in particular to an image generation method based on a generative adversarial network. Background Art

[0002] Currently, most models that generate multi-style images rely on real images for style image generation, while the few models that can generate style images from semantic graphs have limitations. They can only use image styles in the same dataset as input and cannot achieve fast migration across datasets or arbitrary styles.

[0003] The Generative Adversarial Network (GAN), proposed by Goodfellow et al. in 2014, is a deep learning-based generative model consisting of two parts: a generator and a discriminator. The generator aims to produce images that are as realistic as possible, while the discriminator attempts to distinguish between real and generated images. Both are continuously optimized through adversarial training, ultimately enabling the generator to produce high-quality images. Compared to other generative models, GANs avoid complex statistical calculations and can generate higher-quality images. They have demonstrated outstanding performance in image generation, particularly in producing realistic images such as handwritten digits, faces, and artwork.

[0004] However, existing image generation methods based on generative adversarial networks still have some shortcomings. For example, they cannot fully understand and integrate the different styles, themes, and elements in user instructions, resulting in a lack of creativity and personalization in the generated images. Furthermore, existing methods have limitations in processing multimodal input data and are unable to effectively extract and utilize key features from this data, resulting in generated images that do not meet user needs.

[0005] Therefore, there is an urgent need for an image generation method based on a generative adversarial network to solve the technical problems existing in the above-mentioned prior art. Summary of the Invention

[0006] The present invention overcomes the shortcomings of the prior art and provides an image generation method based on a generative adversarial network.

[0007] To achieve the above object, the technical solution adopted by the present invention is: an image generation method based on a generative adversarial network, comprising the following steps:

[0008] S1. Collect image datasets containing different styles, themes and elements, and perform preprocessing and annotation;

[0009] S2. Use NLP technology to train the instruction parsing module to parse user instructions;

[0010] S3. Build and train a creative fusion network to fuse features of different dimensions of the image dataset according to user instructions and generate creative feature representations;

[0011] S4. Based on the generative adversarial network, we build a generator and a discriminator, and use adversarial training to optimize the quality and creativity of the generated images.

[0012] S5. Introduce a creative stimulation mechanism to inject random noise into the generation process, perform style transfer and mutation, and establish a user feedback loop; test the generated images and optimize the generation method based on user feedback and evaluation indicators.

[0013] In a preferred embodiment of the present invention, in step S2, parsing the user instruction includes the following steps:

[0014] S201, receiving instructions given by the user in natural language form, and parsing the instructions using NLP technology;

[0015] S202, extracting key information about at least one of style, theme, and element in the instruction;

[0016] S203: Convert the extracted key information into a machine-understandable feature vector or label, and use it as the input of the creative fusion network.

[0017] In a preferred embodiment of the present invention, in step S3, the implementation steps of the creative fusion network include:

[0018] S301. Construct a creative fusion network based on a multi-layer neural network to integrate features of different styles, themes, or elements;

[0019] S302, the creative fusion network contains multiple sub-networks, each responsible for processing different types of input, including image style features, object shape features, and color matching features;

[0020] S303. Through the attention mechanism or feature fusion layer, features of different dimensions are organically combined to form a new creative feature representation.

[0021] In a preferred embodiment of the present invention, in step S4, the generator is used to receive the feature representation output by the creative fusion network, and gradually generate a high-resolution image through upsampling and convolution operations; the discriminator is responsible for judging whether the generated image is real and whether it meets the requirements of the user's instructions.

[0022] In a preferred embodiment of the present invention, in step S5, the steps of implementing the creative stimulation mechanism include:

[0023] S501, during the generation process of the generator, injecting random noise into the input or intermediate layer of the generator to increase the diversity and unpredictability of the generated image;

[0024] S502: Introducing style transfer technology into the creative fusion network to generate an image while maintaining the original style while incorporating new style elements; and fine-tuning the style through mutation operations.

[0025] S503: Allow the user to provide feedback on the generated image, and adjust the generation strategy according to the user feedback to optimize the generated image.

[0026] In a preferred embodiment of the present invention, a minimum loss function is provided in the generator, and the result value of the minimum loss function is optimized by adversarial training to obtain a high-resolution image; wherein the formula expression of the minimum loss function is:

[0027] ;

[0028] Where, Represents the noise distribution Random noise sampled in expected value; Represents the noise distribution Random noise sampled in ; represents the image generated by the generator; Represents the discriminator's judgment result on the generated image;

[0029] The discriminator is provided with a maximum loss function for distinguishing between the image generated by the generator and the real image; wherein the formula expression of the maximum loss function is:

[0030] ;

[0031] Where, Represents the distribution of real data Real images sampled from expected value; Represents the distribution of real data Real image data sampled from ; Indicates that the discriminator hopes to maximize the accuracy of the real image The judgment result of The logarithm of Represents the image generated by the generator The minimized judgment result The logarithm of .

[0032] In a preferred embodiment of the present invention, in step S501, the formula expression of the image generated by the generator after the random noise injection process is: ;in, represents the weight coefficient of random noise; represents random noise;

[0033] In step S502, the style transfer technology is used to integrate the new style features into the generated image processed in step S501; wherein, the formula expression of the generated image after integration is: Where, represents the style transfer function; Indicates new stylistic features; Represents the mutation vector.

[0034] In a preferred embodiment of the present invention, the user provides scoring feedback on the fused generated image, and the feedback value is incorporated into the minimum loss function of the generator; wherein the formula expression of the minimum loss function incorporating the feedback value is: Where, represents a hyperparameter used to control the influence of user feedback on the minimum loss function; Indicates the feedback value, which ranges from [0,1].

[0035] In a preferred embodiment of the present invention, an image generation system based on a generative adversarial network is proposed, which is based on the above-mentioned image generation method and includes: an instruction parsing module, a creative fusion network and a generative adversarial network framework; wherein the instruction parsing module is used to process and implement steps S1-S2 to complete the parsing of user instructions; the creative fusion network receives multimodal input data, preprocesses the input data, and converts it into a feature vector or a feature map, and sets a feature extraction layer in the creative fusion network to extract key features from the input data, and uses a convolution kernel to perform a convolution operation on different key features to generate a fused feature map; the generative adversarial network framework uses a gradient descent algorithm to iteratively optimize the parameters of the creative fusion network, and in each iteration, updates the parameters of the creative fusion network according to the gradient of the minimum loss function to obtain the minimized loss function value.

[0036] In a preferred embodiment of the present invention, the creative fusion network performs enhanced processing such as rotation, scaling, and cropping on the input data before receiving the multimodal input data; the data types of the multimodal input data include images, text descriptions, and audio signals; and the key features include the edges, textures, and color distribution of the images, the semantic information of the text, and the spectral characteristics of the audio.

[0037] The present invention solves the defects existing in the background technology and has the following beneficial effects:

[0038] (1) By combining a generative adversarial network and a creative fusion network, this technical solution can generate high-quality and creative images. The generator continuously optimizes the generation strategy during the training process to improve the realism and detail expression of the image. At the same time, the creative fusion network can fuse multiple features to generate novel and unique images to meet the diverse needs of users.

[0039] (2) Users can interact with the system through natural language to specify image features such as style, theme, and elements. The system generates corresponding images based on user instructions and continuously optimizes the generation strategy through user feedback loops, thereby enhancing user participation and satisfaction, making the generated images more in line with user expectations and needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments described in the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive efforts.

[0041] Figure 1 is a method flow chart of a preferred embodiment of the present invention;

[0042] Figure 2 It is a flowchart of adversarial training optimization generation of images in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0044] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0045] like Figure 1 As shown, an image generation method based on a generative adversarial network includes the following steps:

[0046] S1. Collect image datasets containing different styles, themes, and elements, and perform preprocessing and annotation. The image datasets should cover a wide range of image styles (such as abstract, realistic, cartoon, etc.), themes (such as nature, city, people, etc.), and elements (such as color, shape, texture, etc.) to ensure the diversity and creativity of the generated images.

[0047] Preprocessing steps include image scaling, cropping, and normalization to unify image formats and sizes for easier processing. Images are also annotated with style tags, theme tags, and element tags, providing a foundation for subsequent instruction parsing and creative integration.

[0048] S2. Use NLP technology to train the instruction parsing module to parse user instructions; instructions can be simple phrases or complex sentences, and the system should be able to handle instructions of different complexities.

[0049] Furthermore, in step S2, parsing of the user instruction includes the following steps:

[0050] S201, receiving instructions given by the user in natural language form, and parsing the instructions using NLP technology;

[0051] S202. Extract key information about at least one of the style, theme, and elements in the instruction; when extracting the key information, NLP technologies such as named entity recognition (NER) and dependency parsing may be used;

[0052] The instruction parsing module is trained using NLP technology, which can parse the instructions given by users in natural language, extract key information and convert it into machine-understandable feature vectors or labels, allowing users to customize the style, theme and elements of the generated images according to their needs and preferences, thereby achieving personalized image generation.

[0053] S203: Convert the extracted key information into machine-understandable feature vectors or labels, and use them as input to the creative fusion network. The conversion process involves converting text into vectors or directly mapping text labels into predefined numerical labels.

[0054] S3. Build and train a creative fusion network to fuse features of different dimensions of the image dataset according to user instructions and generate creative feature representations;

[0055] Furthermore, in step S3, the implementation steps of the creative fusion network include:

[0056] S301. Construct a creative fusion network based on a multi-layer neural network to fuse features of different styles, themes, or elements. The network architecture includes convolutional layers, fully connected layers, activation functions, etc., to extract and fuse features.

[0057] S302. The creative fusion network consists of multiple sub-networks, each responsible for processing different types of input, including image style features, object shape features, and color matching features. The sub-networks can focus on different feature dimensions, such as style sub-network, theme sub-network, element sub-network, etc.

[0058] S303: Through the attention mechanism or feature fusion layer, features of different dimensions are organically combined to form a new creative feature representation. The attention mechanism helps the network focus on important features, while the feature fusion layer is responsible for combining features of different dimensions into a new creative feature representation.

[0059] Specifically, the attention mechanism calculates the correlation between different features and assigns a weight to each, allowing the network to focus more on important features. In the creative fusion network, a self-attention module calculates the similarity matrix between features to obtain a weight for each feature, and then fuses the weighted features. This enhances the network's ability to capture key features and improves the effectiveness of feature fusion.

[0060] like Figure 2 As shown in S4, based on the generative adversarial network, a generator and a discriminator are constructed, and adversarial training is used to optimize the quality and creativity of the generated images; the generator gradually converts the low-resolution feature map into a high-resolution image through upsampling (such as deconvolution) and convolution operations.

[0061] Furthermore, in step S4, the generator is used to receive the feature representation output by the creative fusion network, and gradually generate a high-resolution image through upsampling and convolution operations; the discriminator is responsible for judging whether the generated image is real and whether it meets the requirements of the user's instructions.

[0062] S5. Introduce a creative stimulation mechanism to inject random noise into the generation process, perform style transfer and mutation, and establish a user feedback loop; test the generated images and optimize the generation method based on user feedback and evaluation indicators.

[0063] Furthermore, in step S5, the implementation steps of the creative stimulation mechanism include:

[0064] S501. During the generation process of the generator, random noise is injected into the input or intermediate layer of the generator to increase the diversity and unpredictability of the generated images. By injecting random noise, performing style transfer and mutation operations, the diversity and unpredictability of the generated images are increased to further meet the personalized needs of users.

[0065] Specifically, in step S501, the formula expression of the image generated by the generator after the random noise injection process is: ;in, represents the weight coefficient of random noise; represents random noise;

[0066] S502. Introducing style transfer technology into the creative fusion network allows the generated image to incorporate new style elements while maintaining the original style. At the same time, the style is fine-tuned through mutation operations.

[0067] Specifically, in step S502, the style transfer technology is used to integrate the new style features into the generated image processed in step S501; wherein, the formula expression of the generated image after integration is:

[0068] Where, represents the style transfer function; Indicates new stylistic features; Represents the mutation vector.

[0069] S503: Allow the user to provide feedback on the generated image, and adjust the generation strategy according to the user feedback to optimize the generated image.

[0070] Furthermore, a minimum loss function is set in the generator, and the result value of the minimum loss function is optimized through adversarial training to obtain a high-resolution image; wherein the formula expression of the minimum loss function is:

[0071] ;

[0072] Where, Represents the noise distribution Random noise sampled in expected value; Represents the noise distribution Random noise sampled in ; represents the image generated by the generator; Represents the discriminator's judgment result on the generated image.

[0073] Specifically, the following is a further explanation of how to continuously optimize the quality and creativity of generated images through alternating training of the generator and discriminator in a generative adversarial network (GAN).

[0074] (1) Initializing network parameters: The parameters of the generator and discriminator are initialized before training begins, usually using a random initialization method.

[0075] (2) Alternate training of generator and discriminator:

[0076] (1) Training the discriminator: from the real data distribution Sample a batch of real images .

[0077] From the noise distribution Sample a batch of random noise , the corresponding generated image is generated by the generator .

[0078] Use the discriminator to discriminate between this batch of real images and generated images, and calculate the maximum loss function of the discriminator . Use a gradient descent algorithm (such as the Adam optimizer) to update the discriminator parameters to minimize .

[0079] (2) Training the generator: from the noise distribution Sample a new batch of random noise . Generate the corresponding generated image through the generator . Use the discriminator to discriminate this batch of generated images and calculate the minimum loss function of the generator . Then use the gradient descent algorithm to update the parameters of the generator to minimize .

[0080] (3) Introducing creative stimulation mechanisms and user feedback:

[0081] During each generator training process, random noise is injected into the input or intermediate layers of the generator to increase the diversity and unpredictability of the generated images.

[0082] In the creative fusion network, style transfer technology is introduced to enable the generated images to incorporate new style elements while maintaining the original style, and fine-tune the style through mutation operations.

[0083] Allow users to provide feedback on the generated images and incorporate the feedback value into the generator's minimum loss function. Specifically, the feedback value (range [0,1]) and hyperparameters Multiply and then add to the generator's minimum loss function In order to adjust the generation strategy and optimize the generated image.

[0084] What needs to be further explained here is that the introduction of hyperparameters The purpose is to provide user feedback in the form of ratings The form of provides a subjective evaluation of the quality of the generated image. However, this subjective evaluation needs to be combined with the original loss function of the generator to form a comprehensive optimization objective. It plays a role in balancing these two effects. The value of can control the weight of user feedback in the optimization process. For example, when When is large, user feedback has a greater impact on the generator training, and the system will be more inclined to generate images that meet user preferences; on the contrary, when When is smaller, the generator relies more on the original loss function for optimization.

[0085] Different users may have different expectations and requirements for the quality of generated images. The value of can adapt the system to the needs and preferences of different users. For example, for users who value the realism of images more, the value of The value of makes the generator rely more on the original loss function for optimization; for users who value the creativity or personalization of images, the value of The value of makes the system pay more attention to user feedback.

[0086] During the training process, the introduction of user feedback may increase the instability of the optimization process. , the influence of user feedback in the optimization process can be gradually adjusted to control the stability and convergence of the optimization process. For example, in the early stage of training, the The value of , makes the generator perform preliminary optimization based on the original loss function; as the training progresses, it gradually increases The value of , allows the system to gradually adapt to user feedback.

[0087] (4) Adaptive Adversarial Loss: To balance the quality and diversity of generated images, we introduce an adaptive adversarial loss. We use metrics such as the Inception Score and Fréchet Inception Distance (FID) to evaluate the quality of generated images. We also evaluate the diversity of generated images by calculating statistics such as entropy and mutual information of the generated image set.

[0088] Based on the quality and diversity evaluation results of the generated images, the maximum loss function of the discriminator is dynamically adjusted. Specifically, a weighted loss function is set, where the weight coefficients change dynamically according to the quality and diversity of the generated images.

[0089] In each round of iterative training, the quality and diversity indicators of the generated images are first calculated, the maximum loss function of the discriminator is dynamically adjusted according to these indicators, and then the discriminator and generator are alternately trained using the adjusted loss function.

[0090] (5) Iteration termination condition: The training process continues until a predetermined number of training rounds is reached. Alternatively, training can be stopped when the quality and diversity indicators of the generated images reach a certain threshold. During the training process, the model parameters of the generator and discriminator are periodically saved to facilitate model evaluation or deployment after training.

[0091] Preferably, during traditional GAN training, the discriminator's loss function is typically fixed. However, the quality and diversity of generated images vary during training. To balance the quality and diversity of generated images, this method introduces an adaptive adversarial loss to dynamically adjust the discriminator's loss function based on the quality and diversity of generated images. For example, when the quality of generated images is low, the discriminator's error penalty for generated images can be increased; when the diversity of generated images is low, the discriminator's penalty for generated image diversity can be increased. This allows the generator to better balance the quality and diversity of generated images during training.

[0092] Adaptive adversarial loss is a method that dynamically adjusts the discriminator's loss function based on the quality and diversity of generated images. Its core concept is that during GAN training, the quality and diversity of generated images are two interrelated yet conflicting objectives. Traditional GAN training often struggles to optimize both simultaneously, leading to unstable training and even potential problems such as mode collapse. By dynamically adjusting the discriminator's loss function, adaptive adversarial loss enables the generator to better balance the quality and diversity of generated images during training, thereby improving training stability.

[0093] It should be further explained that the adaptive adversarial loss introduced in this method is implemented in the following steps:

[0094] The quality of generated images is evaluated using metrics such as Inception Score and Fréchet Inception Distance (FID). These metrics reflect the distance between the generated image and the real image in the feature space, thereby measuring the realism and clarity of the generated image.

[0095] The diversity of generated images is evaluated by calculating statistics such as entropy and mutual information of the generated image set. These indicators can reflect the degree of difference between different images in the generated image set, thereby measuring the diversity of generated images.

[0096] Based on the quality and diversity evaluation results of the generated images, the discriminator’s loss function is dynamically adjusted. Specifically, a weighted loss function is set, where the weight coefficients change dynamically according to the quality and diversity of the generated images.

[0097] When the quality of the generated image is low, the error penalty weight of the discriminator on the generated image is increased to encourage the generator to improve the quality of the generated image; when the diversity of the generated image is low, the penalty weight of the discriminator on the diversity of the generated image is increased to encourage the generator to increase the diversity of the generated image.

[0098] In each round of iterative training, the quality and diversity indicators of the generated images are first calculated, and the discriminator loss function is dynamically adjusted according to these indicators. The discriminator and generator are trained alternately using the adjusted loss function. The above process is repeated until the predetermined training rounds are reached or other stopping conditions are met.

[0099] Adaptive adversarial loss dynamically adjusts the loss function of the discriminator, enabling the generator to better balance the quality and diversity of generated images during training, thereby helping to avoid problems such as mode collapse during training and improving training stability.

[0100] By increasing the penalty weight of the discriminator for errors in generated images, the generator is encouraged to generate higher quality images, which helps to improve the realism and clarity of the generated images.

[0101] By increasing the penalty weight of the discriminator on the diversity of generated images, the generator is encouraged to generate more diverse images, which helps to avoid the problem of generated images being too single or repetitive.

[0102] Specifically, the user provides scoring feedback on the fused generated image and incorporates the feedback value into the minimum loss function of the generator. The formula for the minimum loss function that incorporates the feedback value is: Where, represents a hyperparameter used to control the influence of user feedback on the minimum loss function; Indicates the feedback value, which ranges from [0,1].

[0103] A maximum loss function is set in the discriminator to distinguish the images generated by the generator from the real images. The formula of the maximum loss function is:

[0104] ;

[0105] Where, Represents the distribution of real data Real images sampled from expected value; Represents the distribution of real data Real image data sampled from ; Indicates that the discriminator hopes to maximize the accuracy of the real image The judgment result of The logarithm of Represents the image generated by the generator The minimized judgment result The logarithm of .

[0106] Based on the above, -E is further explained here. In the loss function of GAN, -E is used to represent the negative value of the expectation of the random variable. By minimizing or maximizing these expectations, the generator and discriminator optimize their own performance respectively, thereby achieving the effect of adversarial training.

[0107] Allowing users to provide feedback on generated images and adjusting the generation strategy based on user feedback. This feedback mechanism can help the system detect and correct errors in a timely manner and optimize the quality of generated images.

[0108] Generative adversarial networks (GANs) continuously optimize the generator's capabilities through the mutual game between the generator and the discriminator, enabling it to generate increasingly realistic images, thereby enabling the generator to capture the complex distribution characteristics in the image data and generate high-quality images.

[0109] By building a creative fusion network based on a multi-layer neural network and integrating the features of different styles, themes or elements, images with unique creativity can be generated, which not only retains the basic elements of the image but also incorporates the user's instructions and preferences, increasing the diversity of the image.

[0110] To protect user privacy, user feedback data needs to be encrypted. Symmetric or asymmetric encryption technologies are used to encrypt, store, and transmit user feedback data. Anonymous feedback can also be used to allow users to provide feedback without revealing their personal information. For example, each user can be given a unique anonymous ID and associated with their feedback data. This allows for the collection of useful user feedback while protecting user privacy.

[0111] Based on the above-mentioned image generation method, an image generation system based on a generative adversarial network is proposed, which includes: an instruction parsing module, a creative fusion network and a generative adversarial network framework; the instruction parsing module:

[0112] Instruction parsing module: Receives instructions given by the user in natural language, such as "generate a future city scene that combines classical architecture and modern technological elements."

[0113] Use natural language processing technology (NLP) to parse instructions and extract key information, such as style (fusion of classical and modern), theme (future city), elements (architecture, technology), etc.

[0114] The parsed instructions are converted into machine-understandable feature vectors or labels as input to the subsequent network.

[0115] Creative Fusion Network: A multi-layered neural network designed to fuse features from different styles, themes, or elements. This network can contain multiple sub-networks, each responsible for processing different types of input (e.g., image style features, object shape features, color matching features, etc.).

[0116] Through the attention mechanism or feature fusion layer, these features of different dimensions are organically combined to form new and creative feature representations.

[0117] Before receiving multimodal input data, the creative fusion network performs enhanced processing such as rotation, scaling, and cropping on the input data; the data types of the multimodal input data include images, text descriptions, and audio signals; the key features include edges, textures, and color distribution of images, semantic information of text, and spectral features of audio.

[0118] Generative Adversarial Network (GAN) Framework: This framework utilizes the classic GAN architecture, consisting of a generator and a discriminator. The generator receives the feature representations output by the creative fusion network and, through operations such as upsampling and convolution, gradually generates high-resolution images. The discriminator is responsible for determining whether the generated images are realistic and meet user requirements. Through adversarial training between the generator and the discriminator, the quality and creativity of the generated images are continuously optimized.

[0119] During the generation process, random noise is injected into the generator's input or intermediate layers to increase the diversity and unpredictability of the generated images. Furthermore, style transfer technology is introduced into the Creative Fusion Network, enabling the generated images to incorporate new stylistic elements while maintaining the original style. Furthermore, through mutation operations, the style is fine-tuned to create unique visual effects.

[0120] User feedback loop: Allows users to provide feedback on generated images, such as selecting preferred styles and elements or making suggestions for improvement. The system adjusts the generation strategy based on user feedback to further optimize the generated images.

[0121] This technical solution can not only be used to generate high-quality and diverse images, but also achieve image style transfer, migrating one artistic style to another image, providing new possibilities for digital art creation.

[0122] By generating realistic images, it can be used for data augmentation, improving the generalization and performance of machine learning models. It can also be applied to fields such as virtual reality, game development, and medical image analysis, providing high-quality image generation and style transfer solutions for these fields.

[0123] The above description is based on the ideal embodiment of the present invention. Based on the above description, relevant personnel can make various changes and modifications without departing from the technical scope of this invention. The technical scope of this invention is not limited to the content of the specification and must be determined according to the scope of the claims.

Claims

1. A method for image generation based on a generative adversarial network, characterized in that: The following steps are involved: S1. Collect image datasets containing different styles, themes, and elements, and perform preprocessing and annotation. S2. Use NLP technology to train the instruction parsing module to parse user instructions; S3. Build and train a creative fusion network to fuse features of different dimensions of the image dataset according to user instructions and generate creative feature representations; S4. Based on the generative adversarial network, we build a generator and a discriminator, and use adversarial training to optimize the quality and creativity of the generated images. S5. Introduce a creative stimulation mechanism to inject random noise into the generation process, perform style transfer and mutation, and establish a user feedback loop; Test the generated images and optimize the generation method based on user feedback and evaluation indicators; In step S3, the implementation steps of the creative fusion network include: S301. Construct a creative fusion network based on a multi-layer neural network to integrate features of different styles, themes, or elements; S302, the creative fusion network contains multiple sub-networks, each responsible for processing different types of input, including image style features, object shape features, and color matching features; S303, through the attention mechanism or feature fusion layer, features of different dimensions are organically combined to form a new creative feature representation; In step S5, the implementation steps of the creative stimulation mechanism include: S501, during the generation process of the generator, injecting random noise into the input or intermediate layer of the generator to increase the diversity and unpredictability of the generated image; S502: Introducing style transfer technology into the creative fusion network to generate an image while maintaining the original style while incorporating new style elements; and fine-tuning the style through mutation operations. S503: Allow the user to provide feedback on the generated image, and adjust the generation strategy according to the user feedback to optimize the generated image.

2. The image generation method based on a generative adversarial network according to claim 1, characterized in that: In step S2, the parsing of the user instruction includes the following steps: S201, receiving instructions given by the user in natural language form, and parsing the instructions using NLP technology; S202, extracting at least one key information about style, theme, and elements in the instruction; S203: Convert the extracted key information into a machine-understandable feature vector or label, and use it as the input of the creative fusion network.

3. The image generation method based on a generative adversarial network according to claim 1, characterized in that: In step S4, the generator is used to receive the feature representation output by the creative fusion network, and gradually generate a high-resolution image through upsampling and convolution operations; the discriminator is responsible for judging whether the generated image is real and whether it meets the requirements of the user's instructions.

4. The image generation method based on a generative adversarial network according to claim 3, characterized in that: The generator is provided with a minimum loss function, and the result value of the minimum loss function is optimized by adversarial training to obtain a high-resolution image; wherein the formula expression of the minimum loss function is: ； Where, Represents the noise distribution Random noise sampled in expected value; Represents the noise distribution Random noise sampled in ; represents the image generated by the generator; Represents the discriminator's judgment result on the generated image; The discriminator is provided with a maximum loss function for distinguishing between the image generated by the generator and the real image; wherein the formula expression of the maximum loss function is: ； Where, Represents the distribution of real data Real images sampled from expected value; Represents the distribution of real data Real image data sampled from ; Indicates that the discriminator hopes to maximize the accuracy of the real image The judgment result of The logarithm of Represents the image generated by the generator The minimized discrimination result The logarithm of .

5. The image generation method based on a generative adversarial network according to claim 1, characterized in that: In step S501, the formula expression of the image generated by the generator after the random noise injection process is: ;in, represents the weight coefficient of random noise; represents random noise; In step S502, the style transfer technology is used to integrate the new style features into the generated image processed in step S501; wherein, the formula expression of the generated image after integration is: Where, represents the style transfer function; Indicates new stylistic features; Represents the mutation vector.

6. The image generation method based on a generative adversarial network according to claim 5, characterized in that: The user provides rating feedback on the fused generated image, and the feedback value is incorporated into the minimum loss function of the generator; wherein the formula expression of the minimum loss function incorporating the feedback value is: Where, represents a hyperparameter used to control the influence of user feedback on the minimum loss function; Indicates the feedback value, which ranges from [0,1].

7. An image generation system based on a generative adversarial network, based on an image generation method according to any one of claims 1 to 6, characterized in that: include: An instruction parsing module, a creative fusion network and a generative adversarial network framework; wherein the instruction parsing module is used to process and implement steps S1-S2 to complete the parsing of user instructions; the creative fusion network receives multimodal input data, preprocesses the input data, and converts it into a feature vector or a feature map, and sets a feature extraction layer in the creative fusion network to extract key features from the input data, and uses convolution kernels to perform convolution operations on different key features to generate a fused feature map; the generative adversarial network framework uses a gradient descent algorithm to iteratively optimize the parameters of the creative fusion network, and in each iteration, updates the parameters of the creative fusion network according to the gradient of the minimum loss function to obtain the minimized loss function value.

8. The image generation system based on a generative adversarial network according to claim 7, characterized in that: Before receiving multimodal input data, the creative fusion network performs enhanced processing such as rotation, scaling, and cropping on the input data; the data types of the multimodal input data include images, text descriptions, and audio signals; the key features include edges, textures, and color distribution of images, semantic information of text, and spectral features of audio.

Images