An image generation method and system based on big data processing

Abstract

The present application relates to the technical field of image generation, and relates to an image generation method and system based on big data processing, wherein the method comprises: obtaining multiple materials input by a user and judging the types of the materials; obtaining embedding vectors of the materials; performing weighted summation on the embedding vectors of all the materials to obtain a multi-modal demand vector; screening out matching materials from all the to-be-matched materials in a preset material library according to the multi-modal demand vector; inputting the matching materials into a generative adversarial network, and determining that the output of the generative adversarial network is a target image; obtaining a problem area indication mask of the target image through an attention mechanism; generating pixel points in the corresponding area of the problem area indication mask in the target image through a generator, and outputting the corrected target image.The present application realizes cross-modal feature fusion, can more comprehensively and accurately evaluate the demand of a user for an image, and then outputs the image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image generation. More specifically, the present application relates to an image generation method and system based on big data processing. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, image generation, as an important branch of artificial intelligence, has made significant progress in recent years. From the initial generation of simple shapes to the creation of complex scenes and realistic textures, image generation models and systems have evolved, bringing revolutionary changes to art creation, design, entertainment and other fields. One of the core technologies of image generation is deep learning, especially generative adversarial networks (GANs) and variational autoencoders (VAEs). These two methods enable machines to "understand" and generate completely new visual content by training models to recognize large amounts of image data and then create works that appear to be created by human artists.

[0003] Generative adversarial networks (GANs) are a landmark technology framework in the field of image generation. The core idea of generative adversarial networks is to generate realistic data through the adversarial training of two networks. The generation network is responsible for generating simulated data, while the discrimination network is responsible for determining whether the input data is real or generated. The generation network continuously optimizes the data it generates, making it difficult for the discrimination network to distinguish, while the discrimination network also continuously optimizes itself to improve the accuracy of its judgment.

[0004] Existing image generation systems (such as models based on GAN, VAE) usually rely on single modal input (such as text or image). For example, text-to-image generation models (such as DALL-E, Stable Diffusion) can generate images based on text descriptions, but cannot effectively integrate multi-modal inputs such as speech and image. Although joint embedding methods (such as CLIP) have been proposed in the field of multi-modal learning, when speech, image, and text appear simultaneously as materials, it is difficult for the system to accurately and effectively generate the image desired by the user. For example, when a user uploads a design sketch (image) and describes "need brighter color tone" (in the form of text or speech), the system cannot automatically increase the weight of visual features to prioritize image requirements. SUMMARY

[0005] To solve the above technical problems of image generation systems that cannot accurately and effectively generate images desired by users when speech, image, and text appear simultaneously as materials, the present application provides solutions in the following aspects.

[0006] In the first aspect, an image generation method based on big data processing includes: obtaining multiple materials input by a user and determining the type of each material, wherein the types of materials include: text type, image type, and speech type; obtaining the embedding vector of each material; wherein, for text type materials, semantic embedding of the material is generated through Transformer, denoted as the embedding vector of the material; for image type materials, visual feature vector of the material is obtained, denoted as the embedding vector of the material; for speech type materials, semantic embedding of the material is obtained after converting the material into text type through speech recognition technology, denoted as the embedding vector of the material; and weighted summing of the embedding vectors of all materials to obtain a multimodal demand vector;

[0007] Based on the multimodal demand vector, matching materials are selected from all materials to be matched in a preset material library, wherein the cosine similarity between the feature vector of the matching material and the multimodal demand vector is greater than a preset threshold; the matching materials are input into a generative adversarial network, and the output of the generative adversarial network is determined to be the target image; a problem region indicator mask of the target image is calculated, wherein the problem region indicator mask is used to determine the region of the target image that needs to be corrected; a generator generates pixels in the region corresponding to the problem region indicator mask in the target image to correct the target image, and the corrected target image is output.

[0008] Preferably, constructing a generative adversarial network includes: constructing an initial generative adversarial network, wherein the initial generative adversarial network includes an initial generator, an initial correction generator, and an initial discriminator; training the initial generative adversarial network according to a preset training set, and obtaining the generative adversarial network after training, wherein the generative adversarial network includes a generator, a correction generator, and a discriminator, wherein the generator is used to generate an image based on the feature vector of the matching material, the correction generator is used to identify the problem region of the image and generate a mask based on the problem region, the correction generator is also used to correct the image based on the mask and the image, and the discriminator is used to identify the problem region of the image and generate a mask based on the position of the problem region.

[0009] Preferably, an image generation method based on big data processing further includes: constructing an adversarial loss function and performing adversarial training on the initial generator and the initial discriminator through the adversarial loss function.

[0010] Preferably, an image generation method based on big data processing further includes: calculating the weights corresponding to each matching material, wherein the weight of the first matching material is... k The weight corresponding to each matching material Satisfying the formula: , For the first k The cosine similarity between the feature vectors of the matched materials and the multimodal demand vector. For the first j The cosine similarity between the feature vectors of the matched materials and the multimodal demand vector, where exp() is a real number. e An exponential function with base 0. k , j All are positive integers. m The number of all matching materials; where, the number of matching materials is calculated. k The formula for the cosine similarity between the feature vector of each matched material and the multimodal demand vector is: , where is the feature vector of the k-th matching material; the matching material and its corresponding weights are input into the generator of the generative adversarial network to obtain the target image.

[0011] Preferably, the formula for inputting the matched materials and their corresponding weights into the generative adversarial network to obtain the target image is: ,in, G The matrix corresponding to the target image. For the first k The feature vectors of the matching materials.

[0012] Preferably, the formula for calculating the problem region indicator mask is as follows: ,in β For dynamic fusion weights, To make the target image G The error detection mask obtained after being input into the discriminator of the generative adversarial network. To make the target image G The correction suggestion mask is obtained after being input into the correction generator of the generative adversarial network.

[0013] Preferably, the formula for calculating the dynamic fusion weight is:

[0014] ;in, σ () is the Sigmoid function. σ The output range of () is [0,1]. MLP It is a multilayer perceptron. MLP () is used to map input data to a one-dimensional numerical value.

[0015] Preferably, the formula for generating pixels in the region corresponding to the problem region indicator mask in the target image by the generator is as follows: ,in This is the matrix corresponding to the corrected target image. Corret () indicates a correction generator. This indicates the problem area indicator mask in the target image generated by the correction generator. HThe matrix corresponding to the pixel point corresponding to the corresponding position.

[0016] Preferably, the formula for calculating the multi-modal demand vector is: , wherein is the embedding vector of the first i material, is the fusion weight corresponding to the first i material, i is a positive integer, n is the number of materials input by the user.

[0017] In the second aspect, an image generation system based on big data processing includes a processor and a memory, and the memory stores a computer program, wherein the processor executes the computer program to implement the image generation method based on big data processing in any one of the summary of the application.

[0018] The present application has the following advantages:

[0019] The present application combines attention mechanism and dynamic weight distribution, unifies the semantic expression of text, image and voice, that is, obtains a vector (multi-modal demand vector) according to different types of materials, realizes cross-modal feature fusion, can more comprehensively and accurately evaluate the demand of the user for the image and then output the image. The present application fuses the recognition results of the image problem area by the correction generator and the discriminator, and obtains the area of the image that needs to be corrected, thereby realizing the cooperation of the generation and correction processes, and improving the quality and compliance of the generated image. BRIEF DESCRIPTION OF DRAWINGS

[0020] The above and other objects, features and advantages of the exemplary embodiments of the present application will be more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0021] Figure 1 is a step flowchart schematically showing an image generation method based on big data processing according to an embodiment of the present application;

[0022] Figure 2 is a logic flowchart schematically showing an image generation method based on big data processing according to an embodiment of the present application;

[0023] Figure 3 is a structural block diagram schematically showing an image generation system based on big data processing according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] With reference to the drawings, the technical solutions in the embodiments of the present application will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments of the present application, all the other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0025] The specific implementation of the present application will be described in detail below with reference to the drawings.

[0026] Figure 1 is a schematic step flow chart of an image generation method based on big data processing according to an embodiment of the present application.

[0027] As shown in Figure 1 , an image generation method based on big data processing includes steps S1 to S7.

[0028] Step S1: Obtain a plurality of materials input by a user and determine the type of each material.

[0029] The type of the material includes: text type, image type, voice type.

[0030] Step S2: Obtain the embedding vector of each material.

[0031] In an embodiment, for the material of text type, the semantic embedding of the material is generated by Transformer, denoted as the embedding vector of the material; for the material of image type, the visual feature vector of the material is obtained, denoted as the embedding vector of the material; for the material of voice type, the semantic embedding of the material is obtained after the material is converted to text type by speech recognition technology, denoted as the embedding vector of the material. In this embodiment, the semantic embedding of the text type material is generated by BERT (Bidirectional Encoder Representations from Transformers). The visual feature vector of the image is extracted by pre-training CNN (Convolutional Neural Network), such as ResNet. The material of voice type is converted to text type by ASR (Automatic Speech Recognition).

[0032] Among them, Transformer is a deep learning model architecture based on the self-attention mechanism. Self-attention dynamically captures long-distance dependencies by calculating the association weights between each element in a sequence and other elements. For example, in a sentence, the meaning of a word may depend on the combination of multiple words in its context. BERT is a pre-trained language model based on the Transformer encoder, which learns the semantic representation of text through bidirectional context training. In other embodiments, the embedding vector of the material is obtained after analyzing the audio spectral features of speech-type materials. CNN is a deep learning model specifically designed for processing grid-structured data (such as images and audio spectra). It extracts local features through convolutional layers, reduces dimensionality through pooling layers, and performs classification or regression through fully connected layers. Pre-trained CNN models (such as ResNet-50) are used to extract visual feature vectors from image-type materials. For example, a product image is input into ResNet, and the output is a high-dimensional vector representing its visual content (such as color and shape). ASR is a technique for converting speech signals into text, which typically includes an acoustic model, a language model, and a decoder.

[0033] Step S3: Weight the embedding vectors of all materials to obtain the multimodal demand vector.

[0034] In one embodiment, the formula for calculating the multimodal demand vector is: ,in For the first i The embedding vector of each material, For the first i The fusion weight corresponding to each material i It is a positive integer. n The number of materials entered by the user.

[0035] In one embodiment, the user inputs the following three materials: text: "Design a summer-themed beverage advertisement"; image: a user-uploaded beverage packaging sketch; voice: "The background needs to include beach elements." Calculating the multimodal demand vector includes: obtaining the embedding vectors of the three materials, denoted as E1, E2, and E3. The relevance score of each embedding vector is calculated using self-attention. The relevance scores of each embedding vector are normalized using the softmax function to obtain the weights of each embedding vector. The weighted sum of these embedding vectors yields the multimodal demand vector. The softmax function is an activation function used in machine learning for multi-class classification problems, converting real vectors into a probability distribution where each element's value is between 0 and 1, and the sum is 1.

[0036] Step S4: Based on the multimodal demand vector, select matching materials from all materials to be matched in the preset material library.

[0037] wherein the cosine similarity between the feature vector of the matching material and the multi-modal demand vector is greater than a preset threshold.

[0038] It should be noted that for the feature vector of any to-be-matched material, the acquisition method is the same as that of the embedding vector of the material in step 2, that is, for a text type to-be-matched material, the semantic embedding of the to-be-matched material is generated by the Transformer, denoted as the feature vector of the material; for an image type to-be-matched material, the visual feature vector of the material is obtained, denoted as the feature vector of the to-be-matched material; for a voice type to-be-matched material, the semantic embedding of the to-be-matched material is obtained after the to-be-matched material is converted into a text type by a speech recognition technology, denoted as the feature vector of the to-be-matched material.

[0039] Step S5: inputting the matching material into the generative adversarial network to determine that the output of the generative adversarial network is a target image.

[0040] In one embodiment, the weight corresponding to each matching material is calculated, wherein the weight corresponding to each matching material is calculated, and the weight corresponding to the i-th matching material satisfies the formula: k wherein is the cosine similarity between the feature vector of the i-th matching material and the multi-modal demand vector, is the cosine similarity between the feature vector of the j-th matching material and the multi-modal demand vector, and exp() is an exponential function with a real number as the base. k j e k j m is a positive integer, and is the number of all matching materials; the matching material and the corresponding weight are input into the generator of the generative adversarial network to obtain a target image. Wherein the formula for inputting the matching material and the corresponding weight into the generative adversarial network to obtain the target image is: G is the matrix corresponding to the target image, and is the feature vector of the i-th matching material. In this embodiment, the formula for calculating the cosine similarity between the feature vector of the i-th matching material and the multi-modal demand vector is: k k k

[0041] Step S6: calculating the problem area indication mask of the target image.

[0042] ​​​​​​​​​​​​​​​The problem area indication mask is used to determine the area of the target image that needs to be corrected.

[0043] In one embodiment, the formula for calculating the problem area indication mask is wherein β is a dynamic fusion weight, is the target image G input into the discriminator of the generative adversarial network to obtain an error detection mask, is the target image G input into the correction generator of the generative adversarial network to obtain a correction suggestion mask.

[0044] In one embodiment, the formula for calculating the dynamic fusion weight is:

[0045] wherein σ () is a Sigmoid function, σ the output range of () is [0, 1], MLP is a multilayer perceptron, MLP () is used to map input data to a one-dimensional numerical value.

[0046] The multilayer perceptron (MLP) is a kind of feedforward artificial neural network, which is composed of an input layer, at least one hidden layer and an output layer. Each layer contains multiple neurons (nodes) connected by learnable weights and nonlinear activation functions, which can fit complex input-output relationships. The Sigmoid function is a nonlinear activation function that maps any real number to the interval [0, 1], and its mathematical expression is: The combination of the Sigmoid function and the multilayer perceptron realizes the automatic allocation of dynamic weights, significantly improving flexibility and accuracy in image generation and correction.

[0047] Step S7: generating pixels in the area corresponding to the problem area indication mask in the target image through the generator to correct the target image, and outputting the corrected target image.

[0048] In one embodiment, the formula for generating pixels in the area corresponding to the problem area indication mask in the target image through the generator is: wherein is the matrix corresponding to the corrected target image, Corret is the correction generator, represents the matrix corresponding to the pixels generated by the correction generator at the position corresponding to the problem area indication mask in the target image. It should be noted that H the matrix corresponding to the pixels generated by the correction generator at the position corresponding to the problem area indication mask in the target image. It should be noted that is a matrix, which represents the mask indicating the problem area based on the target image and the problem area H is a matrix, which represents the mask indicating the problem area based on the target image and the problem area H is a mask indicating the problem area (e.g., missing part, damaged part or area needing repair) in the image. The possible values in the mask are 0 and 1. For the positions marked as problem areas in the mask, the correction generator should generate new pixel values to fill in these areas. For the positions marked as non-problem areas in the mask, the generator should keep the pixel values of the original image.

[0049] The application constructs a generative adversarial network, wherein constructing the generative adversarial network comprises: constructing an initial generative adversarial network, wherein the initial generative adversarial network comprises an initial generator, an initial correction generator and an initial discriminator; training the initial generative adversarial network according to a preset training set, and obtaining the generative adversarial network after the training is completed, wherein the generative adversarial network comprises a generator, a correction generator and a discriminator, the generator is used to generate an image according to a feature vector of matching material, the correction generator is used to judge a problem area of the image and generate a mask according to the problem area, the correction generator is also used to correct the image according to the mask and the image, and the discriminator is used to judge the problem area of the image and generate a mask according to the position of the problem area.

[0050] In one embodiment, the initial generator and the initial discriminator are trained in an adversarial manner through an adversarial loss function.

[0051] In one embodiment, the initial generator adopts a structure such as U-Net or StyleGAN, the input is a feature vector of matching material, and the output is an initial generated image. The initial correction generator is designed as an encoder-decoder structure (such as SegNet), the input is a generated image, and the output is a problem area mask and a corrected image. The initial discriminator is based on a convolutional network (such as PatchGAN), and the output is an image authenticity probability and a problem area positioning mask.

[0052] For the adversarial loss function: the goal of the generator is to generate an image that can deceive the discriminator as much as possible, so that the discriminator judges it as "real"; the goal of the discriminator is to accurately distinguish between generated images and real images, and to locate the problem area. For the joint training of the correction generator: the mask generation loss adopts a cross-entropy loss to optimize the accuracy of problem area positioning.

[0053] Among them, the generator is responsible for preliminary generation, the correction generator is responsible for local correction, and the discriminator combines with the correction generator to provide global authenticity evaluation and problem positioning, and the three form a closed loop optimization. For example, if the generated image has a blurred boundary, the discriminator and the correction generator locate the area and generate a mask, and the correction generator re-renders the boundary pixels according to the mask. The problem area indication mask is generated by the discriminator and the correction generator, and is fused by dynamic weight.

[0054] The present application realizes high coordination of generation and correction through a three-component GAN architecture and an end-to-end joint training strategy. The generator, the correction generator and the discriminator each perform their own functions, combined with dynamic mask fusion and multi-task loss function, which significantly improves the accuracy, compliance and efficiency of image generation, and solves the core pain points of traditional methods in multi-modal processing, dynamic adjustment and inefficient post-processing.

[0055] Figure 2 is a logical flowchart schematically showing an image generation method based on big data processing according to an embodiment of the present application.

[0056] As shown in Figure 2 , after receiving the user input material, the present application first judges the type of the material, and uses different feature extraction methods for different types of materials. After the feature vector extraction of each material is completed, the feature vectors corresponding to each material are fused, and the material library is retrieved according to the fused vectors, and then a plurality of materials matching the fused vectors are obtained in the material library, which are denoted as matching materials. The generative adversarial network generates images according to the matching materials, and then the correction generator re-generates pixels in the areas of the images that need to be corrected and realizes the correction of the images. Finally, the corrected image (i.e. the final image) is output.

[0057] Figure 3 is a structural block diagram schematically showing an image generation system based on big data processing according to an embodiment of the present application.

[0058] The present application also provides an image generation system based on big data processing. As shown in Figure 3 , the system includes a processor and a memory, and the memory stores computer program instructions, which, when executed by the processor, implement the image generation method based on big data processing according to the first aspect of the present application.

[0059] The system also includes a communication interface and a communication bus and other components familiar to those skilled in the art, the settings and functions of which are known in the art, and therefore will not be described here.

[0060] In this description, the terms "communication" and "communicate" are used broadly. For example, a device can communicate information to another device, even though the information need not be received explicitly by the other device. In other words, one device can communicate information to another device by placing the information in a location where the other device is able to retrieve the information, even though one device does not know exactly where or when another device will retrieve the information. The term "communication" can include one or both of these actions, and also can include other actions associated with these actions. For example, the process of placing information in a location where another device is able to retrieve the information can include the actions of encoding the information on a physical medium, transmitting encoded information on a physical medium, or other actions associated with these actions. Similarly, the process of retrieving information can include the actions of receiving the information on a physical medium, decoding encoded information on a physical medium, or other actions associated with these actions. Further, one device can communicate information to another device by causing another device to communicate the information. These provisions are merely examples of what is meant to be a "communication" or "communicate," and these provisions are not intended to limit the scope of the present application to these examples. Additionally, the present application contemplates that devices can communicate information in the form of signals, messages, data, or other information.

[0061] In the description of the present application, the meaning of "a", "an", and "the" is intended to include both singular and plural references unless otherwise indicated. For example, the expressions "a molecule" and "the molecule" can include multiple molecules.

[0062] While the present application has been illustrated and described in detail in the drawings and foregoing description, such illustration and description is to be considered illustrative or exemplary and not restrictive; the present application is not limited to the disclosed embodiments. Various modifications and changes can be made thereto without departing from the spirit and scope of the present application as set forth in the claims below. It is to be understood that the replications of the application described herein can be implemented in various alternatives as described herein.

Claims

1. A big data processing-based image generation method, characterized by, The method comprises the following steps: Obtain a plurality of materials input by a user and determine the type of each material, wherein the type of the material includes: text type, image type, and voice type; Obtain the embedding vector of each material; wherein for the material of the text type, generate the semantic embedding of the material by the Transformer, denoted as the embedding vector of the material; for the material of the image type, obtain the visual feature vector of the material, denoted as the embedding vector of the material; for the material of the voice type, convert the material into the text type by the speech recognition technology, and then obtain the semantic embedding of the material, denoted as the embedding vector of the material; The embedding vectors of all materials are weighted and summed to obtain a multi-modal demand vector, wherein a formula for calculating the multi-modal demand vector is: wherein is an embedding vector of a i th material, is a fusion weight corresponding to the i th material, i is a positive integer, n is a number of materials input by the user; According to the multi-modal demand vector, filter out the matching material from all the to-be-matched materials in the preset material library, wherein the cosine similarity between the feature vector of the matching material and the multi-modal demand vector is greater than a preset threshold; After inputting the matching material into the generative adversarial network, determine the output of the generative adversarial network as a target image, wherein the construction of the generative adversarial network comprises: constructing an initial generative adversarial network, wherein the initial generative adversarial network comprises an initial generator, an initial correction generator, and an initial discriminator; training the initial generative adversarial network according to a preset training set, and obtaining the generative adversarial network after the training is completed, wherein the generative adversarial network comprises a generator, a correction generator, and a discriminator, the generator is used to generate an image according to the feature vector of the matching material, the correction generator is used to determine the problem area of the image and generate a mask according to the problem area, and the correction generator is also used to correct the image according to the mask and the image, and the discriminator is used to determine the problem area of the image and generate a mask according to the position of the problem area. The formula for generating a pixel point corresponding to a region indicated by the problem region indication mask in the target image by the generator is: wherein is a matrix corresponding to the target image after correction, Corret () represents a correction generator, represents a pixel point generated by the correction generator at a position corresponding to the problem region indication mask in the target image H corresponds to a matrix corresponding to the pixel point Calculate the problem area indication mask of the target image, wherein the problem area indication mask is used to determine the area of the target image that needs to be corrected; Generate a pixel point in the area corresponding to the problem area indication mask in the target image by the generator to correct the target image, and output the corrected target image.

2. The image generation method based on big data processing according to claim 1, characterized in that, Further comprising: Construct an adversarial loss function and perform adversarial training on the initial generator and the initial discriminator by the adversarial loss function.

3. The image generation method based on big data processing according to claim 1, characterized in that, Further comprising: Calculate the weight corresponding to each matching material, where the first... k The weight corresponding to each matching material Satisfying the formula: , For the first k The cosine similarity between the feature vectors of the matched materials and the multimodal demand vector. For the first j The cosine similarity between the feature vectors of the matched materials and the multimodal demand vector, where exp() is a real number. e An exponential function with base 0. k , j All are positive integers. m The number of all matching materials; where, the number of matching materials is calculated. k The formula for the cosine similarity between the feature vector of each matched material and the multimodal demand vector is: , where is the feature vector of the kth matched material; Input the matching material and the corresponding weight thereof into the generator of the generative adversarial network to obtain the target image.

4. The image generation method based on big data processing according to claim 3, characterized in that, The formula for inputting the matched materials and their corresponding weights into a generative adversarial network to obtain the target image is as follows: ,in, G The matrix corresponding to the target image. For the first k The feature vectors of the matching materials.

5. The image generation method based on big data processing according to claim 1, characterized in that, The formula of the calculation problem area indication mask is wherein β is a dynamic fusion weight, is an error detection mask obtained after inputting the target image G to a discriminator of the generative adversarial network, is a correction suggestion mask obtained after inputting the target image G to a correction generator of the generative adversarial network.

6. The image generation method based on big data processing according to claim 5, characterized in that, The formula for calculating the dynamic fusion weight is: ; where, σ () is a Sigmoid function, σ the output range of () is [0, 1], MLP is a multi-layer perceptron, MLP () is used to map input data to one-dimensional numerical values.

7. A big data processing based image generation system comprising a processor and a memory, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the image generation method based on big data processing according to any one of claims 1-6.

Images