Image drawing system and method based on big data and artificial intelligence generation system technology

Abstract

The invention relates to the technical field of generative artificial intelligence, and discloses an image drawing system and method based on a big data and artificial intelligence generation system technology, and the image drawing method based on the big data and artificial intelligence generation system technology comprises the steps: collecting the condition constraint information of image drawing of a user; inputting the condition constraint information into an image drawing environment to obtain more than one initial image; obtaining a visual object according to the condition constraint information and the initial image, and constructing an object relationship for the visual object; inputting the visual object and the object relation of the visual object into a knowledge guiding model, and outputting a specialized image by the knowledge guiding model; according to the method, the structured knowledge expressing the short-term tendency of the user can be constructed according to the constraint information input by the user, and the generated image of the universal image generation model is regulated and controlled through learning of the structured knowledge to obtain the image meeting the short-term tendency of the user.

Description

Technical Field

[0001] The present invention relates to the field of generative artificial intelligence technology, and more specifically, to an image drawing system and method based on big data and artificial intelligence generation system technology. Background Art

[0002] The generated text is generally a long-term semantic content. For the same user, their demand for generated text is generally relatively stable and is rarely affected by changes in short-term states such as emotions. Therefore, incremental training during the user's use of the generated model can be used to fine-tune the model to adapt to the user's tendency.

[0003] Unlike text generation, the tendency of image drawing is extremely random and will be affected by changes in short-term states such as user emotions. It does not have medium-term stability, so there is little difference between incremental training and fine-tuning the model and not fine-tuning the model. Therefore, image drawing is extremely dependent on the constraint information input by the user. It is very difficult for beginners to combine and input appropriate constraint information to obtain image drawing results that meet their expectations. The learning cost of the artificial intelligence generation system is very high. Summary of the invention

[0004] The present invention provides an image drawing system and method based on big data and artificial intelligence generation system technology, which solves the technical problems in the related technology that image drawing is extremely dependent on constraint information input by the user, it is very difficult for beginners to combine and input appropriate constraint information to obtain image drawing results that meet their expectations, and the learning cost of the artificial intelligence generation system is very high.

[0005] The present invention provides an image drawing method based on a generation system technology of big data and artificial intelligence, comprising the following steps:

[0006] Step 100, constructing an image drawing environment, the image drawing environment including an image generation model with a general ability to generate images;

[0007] Step 200, obtaining a general knowledge graph in the field specified by the user;

[0008] Step 300, collecting the user's image drawing condition constraint information;

[0009] Step 400, inputting the conditional constraint information in Step 300 into the image drawing environment to obtain one or more initial images;

[0010] Step 500, according to the conditional constraint information in Step 300 and the initial image in Step 400, entities are matched from the knowledge graph in Step 200 through entity matching, and random walks are performed around the matched entities to obtain an entity set, where the entity set includes entities matched from the knowledge graph in Step 200;

[0011] Step 600, abstracting the initial image and entities in the entity set into visualization objects, where one visualization object represents one entity in the initial image or entity set; and visualizing the visualization objects in the visualization environment, and building object relationships for the visualization objects by reading the operations of the user in the visualization environment to establish connections for the visualization objects;

[0012] Step 700, input the visualization objects and the object relationships of the visualization objects in Step 600 into the knowledge guidance model, the knowledge guidance model includes a visualization object knowledge relationship recognition layer, a fusion layer and a specialized image generation layer, wherein the visualization object knowledge relationship recognition layer inputs the visualization objects and the object relationships of the visualization objects, and outputs a knowledge relationship recognition feature matrix, the fusion layer fuses the knowledge relationship recognition feature matrix with the input features of the output layer of the decoder of the image generation model, obtains the fused features to the specialized image generation layer, and the specialized image generation layer outputs a specialized image.

[0013] Further, user-specified areas include: creative design assistance, personalized custom design, and creative material generation.

[0014] Furthermore, the user's conditional constraint information for image drawing includes text description, keywords, and reference images.

[0015] Furthermore, the parameters of the random walk are: the step length is W, the number of steps is m; W is a random number, 3≥W≥1, the value range of m is 5-50, and the default value is 9.

[0016] Furthermore, the content visualized by the visualization object includes an initial image represented by the visualization object or attributes of an entity represented by the visualization object, but the visualization content does not include the relationship between the entities in the knowledge graph.

[0017] Furthermore, the knowledge-guided model is pre-trained with the knowledge relationship restoration layer connected. The calculation formula of the knowledge relationship restoration layer is as follows:

[0018]

[0019] in represents the relational pattern matrix, represents the knowledge relationship recognition feature matrix, Indicates changing a value greater than 0.5 to 1. The element in the i-th row and j-th column of the relationship pattern matrix indicates whether there is an object relationship between the i-th visualization object and the j-th visualization object. If the element value is 1, it means that there is, otherwise it does not exist. Represents transposition; the difference between the output of the knowledge relationship restoration layer and the actual object relationship of the visualized object is used as the pre-training loss.

[0020] Furthermore, the aforementioned image drawing method based on big data and artificial intelligence generation system technology also includes:

[0021] Step 800, add the image output by the image generation model in Step 700 and the specialized image output by the specialized image generation layer to the basis of entity matching in Step 500, and then iterate Step 500-800. The condition for terminating the iteration is receiving an instruction from the user to terminate the image generation.

[0022] Furthermore, a timing module is added to the knowledge-guided model. The timing module inputs a timing sequence. A sequence unit of the timing sequence represents the fusion features generated during one iteration of Step 500-800. The sequence recognition features are then output to the specialized image generation layer, and the specialized image generation layer outputs a specialized image.

[0023] The present invention provides an image drawing system based on big data and artificial intelligence generation system technology, which can be run on a computer and can execute the steps in the aforementioned image drawing method based on big data and artificial intelligence generation system technology.

[0024] The present invention provides a computer-readable storage medium, which stores computer-readable instructions. When the computer-readable instructions are executed by a processor, the processor executes the aforementioned image drawing method based on big data and artificial intelligence generation system technology.

[0025] The beneficial effects of the present invention are that the present invention can construct structured knowledge that expresses the user's short-term tendencies based on the constraint information input by the user, and regulate the generated images of the general image generation model through the learning of structured knowledge to obtain images that meet the user's short-term tendencies, which significantly reduces the requirements on the quality of the user's constraint information and reduces the user's learning and usage costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a flow chart of an image drawing method based on big data and artificial intelligence generation system technology of the present invention. DETAILED DESCRIPTION

[0027] The subject matter described herein will now be discussed with reference to example implementations. It should be understood that the discussion of these implementations is only to enable those skilled in the art to better understand and implement the subject matter described herein, and the functions and arrangements of the elements discussed may be changed without departing from the scope of protection of the present specification. Various examples may omit, replace, or add various processes or components as needed. In addition, the features described in some examples may also be combined in other examples.

[0028] At least one embodiment of the present invention discloses an image drawing method based on big data and artificial intelligence generation system technology, such as Figure 1 As shown, the following steps are included:

[0029] Step 100, constructing an image drawing environment, the image drawing environment including an image generation model with a general ability to generate images;

[0030] In some embodiments of the present invention, the image generation model is trained on a large-scale image dataset.

[0031] In some embodiments of the invention, the image generation model is DALL-E;

[0032] The full name is “Distributed Autoencoder for Language and Image”.

[0033] Step 200, obtaining a general knowledge graph in the field specified by the user;

[0034] User-specified fields include the following:

[0035] Creative design assistance: suitable for creative workers in the fields of graphic design, industrial design, UI design, etc.

[0036] Personalized customization design: Applied to various personalized customization scenarios, such as customized clothing, home furnishings, gifts, etc. Select the knowledge graph of the specific field in different customization fields.

[0037] Creative material generation: In the field of digital content creation, such as advertising, film and television, games, etc., the knowledge graph of a specific field is selected from different digital content creation fields.

[0038] Of course, the present invention does not exclude other fields, and does not limit the level and size of the aforementioned fields. The aforementioned representative examples do not represent all available fields.

[0039] Step 300, collecting the user's image drawing condition constraint information;

[0040] In some embodiments of the present invention, the conditional constraint information of the user for image drawing includes input information in various forms such as text description, keywords, reference images, etc.

[0041] For various forms of input information, this step also includes using multimodal alignment models such as CLIP (Contrastive Language–Image Pre-training) model and ALIGN (A Large-scale Image and Noisy-text embedding) model to map this information into a common semantic space to achieve text-image association.

[0042] Step 400, inputting the conditional constraint information in Step 300 into the image drawing environment to obtain one or more initial images;

[0043] Step 500, according to the conditional constraint information in Step 300 and the initial image in Step 400, entities are matched from the knowledge graph in Step 200 through entity matching, and random walks are performed around the matched entities to obtain an entity set, the entity set including the aforementioned entities matched from the knowledge graph in Step 200;

[0044] In some embodiments of the present invention, the parameters of random walk are: step length is W, number of steps is m; W is a random number, 3≥W≥1, the value range of m is 5-50, and the default value is 9. Random walk is carried out in the knowledge graph of a specific field, so the exploration is not set too strict purpose. In this embodiment, a random step length is used to improve the random jump of the exploration;

[0045] However, it should be noted that the random walk does not repeat itself.

[0046] Step 600, abstracting the initial image and entities in the entity set into visualization objects, where one visualization object represents one entity in the initial image or entity set; and visualizing the visualization objects in the visualization environment, and building object relationships for the visualization objects by reading the operations of the user in the visualization environment to establish connections for the visualization objects;

[0047] In some embodiments of the present invention, the content of the visualization object includes the initial image represented by the visualization object or the attributes of the entity represented by the visualization object, but the visualization content does not include the relationship between the entity in the knowledge graph. This is because the relationship between the entity in the knowledge graph is general or universal knowledge in the field, which violates the purpose of specializing operations for users.

[0048] In some embodiments of the present invention, the visualization environment includes a graphical interactive interface.

[0049] In some embodiments of the present invention, in a visualization environment, a user can indirectly express his or her intentions and preferences by selecting, editing connections, and combining these visualization objects, and by reconstructing the contextual semantics and association relationships between content elements.

[0050] Step 700, input the visualization objects and the object relationships of the visualization objects in Step 600 into the knowledge guidance model, the knowledge guidance model includes a visualization object knowledge relationship recognition layer, a fusion layer and a specialized image generation layer, wherein the visualization object knowledge relationship recognition layer inputs the visualization objects and the object relationships of the visualization objects, and outputs a knowledge relationship recognition feature matrix, the fusion layer fuses the knowledge relationship recognition feature matrix with the input features of the output layer of the decoder of the image generation model, obtains the fused features to the specialized image generation layer, and the specialized image generation layer outputs a specialized image.

[0051] Generally speaking, users can choose the image that meets their needs and preferences from the image output by the image generation model in Step 700 and the specialized image output by the specialized image generation layer.

[0052] In some embodiments of the present invention, the fusion layer first concatenates the knowledge relationship recognition features of all visual objects, and then concatenates them with the input features of the output layer of the decoder of the image generation model to obtain the fused features.

[0053] In some embodiments of the present invention, a row vector of the knowledge relationship recognition feature matrix represents a knowledge relationship recognition feature of a visual object, and the knowledge relationship recognition feature of each visual object is respectively concatenated with the input feature of the output layer of the decoder of the image generation model to update the knowledge relationship recognition feature matrix.

[0054] In some embodiments of the present invention, the expression of the visual object knowledge relationship identification layer is as follows:

[0055]

[0056] in represents the knowledge relationship recognition feature of the vth visualization object, represents the graph weight parameter, and denote the visual object features of the vth and uth visual objects respectively, It is a collection of visualization objects that have object relationships with the visualization object v of the graph structure data. express The total number of visual objects in ;

[0057]

[0058] in represents the relationship factor between the vth and uth visualization objects, tanh represents the hyperbolic tangent function, T represents the transpose, Indicates adjustable parameters; The default value is 1 / N, where N is the dimension of the visualization object features.

[0059] In some embodiments of the present invention, the visual object features are the results of feature engineering processing of the attributes of the visual object, and the attributes of the image are the image itself.

[0060] In some embodiments of the present invention, the knowledge guidance model is connected to the knowledge relationship restoration layer during pre-training, and the calculation formula of the knowledge relationship restoration layer is as follows:

[0061]

[0062] in represents the relational pattern matrix, represents the knowledge relationship recognition feature matrix, Indicates changing a value greater than 0.5 to 1. The element in the i-th row and j-th column of the relationship pattern matrix indicates whether there is an object relationship between the i-th visualization object and the j-th visualization object. If the element value is 1, it means that there is, otherwise it does not exist. represents transpose;

[0063] The difference between the output of the knowledge relationship restoration layer and the actual object relationship of the visualized object is used as the pre-training loss.

[0064] In some embodiments of the present invention, the image generation model in Step 700 does not input the conditional constraint information in Step 300, but takes the initial image and entities in the entity set as new constraint information or adds them to the original conditional constraint information as input to the image generation model.

[0065] In some embodiments of the present invention, the output layer of the decoder and the specialized image generation layer of the image generation model are both multi-layer perceptrons or feed-forward neural networks, which output the pixel values ​​of each pixel of the image one by one at multiple time steps. At each time step, a pixel point is generated and added to the generated pixel points, and this process is repeated until a complete image is generated.

[0066] In some embodiments of the present invention, an image drawing method based on big data and artificial intelligence generation system technology also includes:

[0067] Step 800, add the image output by the image generation model in Step 700 and the specialized image output by the specialized image generation layer to the basis of entity matching in Step 500, and then iterate Step 500-800. The condition for terminating the iteration is receiving an instruction from the user to terminate the image generation.

[0068] Generally speaking, the reason for termination is that images that meet the needs and preferences of users are generated.

[0069] In these embodiments, the knowledge-guided model is improved by adding a timing module to the knowledge-guided model. The timing module inputs a timing sequence. A sequence unit of the timing sequence represents a fusion feature generated during an iteration of Step 500-800. The sequence recognition feature is then output to a specialized image generation layer, and the specialized image generation layer outputs a specialized image.

[0070] The expression of the timing module is as follows:

[0071]

[0072]

[0073]

[0074]

[0075] in, , , represents the first, second, and third weight parameters, , , represents the first, second and third bias parameters, represents the dot product, , and Represent the first, second, and third intermediate states respectively, where represents the fusion features generated during the t-th iteration, and Respectively represent the t-th and t-1-th intermediate representation data, n≥t≥1, n represents the total number of iterations, t=1 , when t=n, is equal to the sequence recognition feature, tanh is the hyperbolic tangent function, Represents the sigmoid function.

[0076] The timing module can preserve the dynamic evolution process during multiple iterations and better identify user tendencies.

[0077] In some embodiments of the present invention, an image drawing system based on big data and artificial intelligence generation system technology is provided, which can be run on a computer and can execute the steps of the aforementioned image drawing method based on big data and artificial intelligence generation system technology.

[0078] In some embodiments of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the processor executes the aforementioned image drawing method based on big data and artificial intelligence generation system technology.

[0079] The above describes an embodiment of the present invention, but this embodiment is not limited to the above-mentioned specific implementation mode. The above-mentioned specific implementation mode is merely illustrative and not restrictive. Under the guidance of this embodiment, ordinary technicians in this field can also make more forms of equivalent embodiments, all of which are within the protection of this embodiment.

Claims

1. An image drawing method based on big data and artificial intelligence generation system technology, characterized in that: The following steps are involved: Step 100, constructing an image drawing environment, the image drawing environment including an image generation model with a general ability of image generation; Step 200, obtaining a general knowledge graph in the field specified by the user; Step 300, collecting the user's image drawing condition constraint information; Step 400, inputting the conditional constraint information in Step 300 into the image drawing environment to obtain one or more initial images; Step 500, according to the conditional constraint information in Step 300 and the initial image in Step 400, entities are matched from the knowledge graph in Step 200 through entity matching, and random walks are performed around the matched entities to obtain an entity set, where the entity set includes entities matched from the knowledge graph in Step 200; Step 600, abstracting the initial image and entities in the entity set into visualization objects, where one visualization object represents one entity in the initial image or entity set; and visualizing the visualization objects in the visualization environment, and building object relationships for the visualization objects by reading the operations of the user in the visualization environment to establish connections for the visualization objects; Step 700, input the visualization objects and the object relationships of the visualization objects in Step 600 into the knowledge guidance model, the knowledge guidance model includes a visualization object knowledge relationship recognition layer, a fusion layer and a specialized image generation layer, wherein the visualization object knowledge relationship recognition layer inputs the visualization objects and the object relationships of the visualization objects, and outputs a knowledge relationship recognition feature matrix, the fusion layer fuses the knowledge relationship recognition feature matrix with the input features of the output layer of the decoder of the image generation model, obtains the fused features to the specialized image generation layer, and the specialized image generation layer outputs a specialized image.

2. The image drawing method based on big data and artificial intelligence generation system technology according to claim 1 is characterized in that: User-specified areas include: creative design assistance, personalized custom design, and creative material generation.

3. The image drawing method based on big data and artificial intelligence generation system technology according to claim 1 is characterized in that: The user's conditional constraint information for image drawing includes text description, keywords, and reference images.

4. The image drawing method based on big data and artificial intelligence generation system technology according to claim 1 is characterized in that: The parameters of the random walk are: step length is W, number of steps is m; W is a random number, 3≥W≥1, the value range of m is 5-50, and the default value is 9.

5. The image drawing method based on big data and artificial intelligence generation system technology according to claim 1 is characterized in that: The content visualized by the visualization object includes the initial image represented by the visualization object or the attributes of the entity represented by the visualization object, but the visualization content does not include the relationship between the entities in the knowledge graph.

6. The image drawing method based on big data and artificial intelligence generation system technology according to claim 1 is characterized in that: When pre-training the knowledge guidance model, the knowledge relationship restoration layer is connected. The calculation formula of the knowledge relationship restoration layer is as follows: ； in represents the relational pattern matrix, represents the knowledge relationship recognition feature matrix, Indicates changing a value greater than 0.5 to 1. The element in the i-th row and j-th column of the relationship pattern matrix indicates whether there is an object relationship between the i-th visualization object and the j-th visualization object. If the element value is 1, it means that there is, otherwise it does not exist. Represents transposition; the difference between the output of the knowledge relationship restoration layer and the actual object relationship of the visualized object is used as the pre-training loss.

7. The image drawing method based on big data and artificial intelligence generation system technology according to claim 1 is characterized in that: Also includes: Step 800, add the image output by the image generation model in Step 700 and the specialized image output by the specialized image generation layer to the basis of entity matching in Step 500, and then iterate Step 500-800. The condition for terminating the iteration is receiving an instruction from the user to terminate the image generation.

8. The image drawing method based on big data and artificial intelligence generation system technology according to claim 7 is characterized in that: A timing module is added to the knowledge-guided model. The timing module inputs a timing sequence. A sequence unit of the timing sequence represents the fusion features generated during an iteration of Step 500-800. The sequence recognition features are then output to the specialized image generation layer, and the specialized image generation layer outputs a specialized image.

9. An image rendering system based on big data and artificial intelligence generation system technology, characterized in that: It can be run on a computer and can execute the steps in the aforementioned image drawing method based on big data and artificial intelligence generation system technology.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes an image drawing method based on big data and artificial intelligence generation system technology as described in any one of claims 1-8.

Images