System and method for image generation

Abstract

A system and a method for image generation are provided. The method includes following steps: a server receives a prompt word from a user device; using a vector corresponding to the prompt word to obtain a vector feature classification result; parsing a generation task corresponding to the vector feature classification result; using the generation task to determine an expert model from an expert model library; executing the expert model to obtain an image generation result; and performing fusion and alignment operation on the image generation result to obtain a target image generation result.

Description

[Technical Field]

[0001] This invention relates to a system and method for image generation. [Previous Technology]

[0002] With the rapid development of generative artificial intelligence (GAI) technology, text-to-image generation technology has been widely applied in various fields, including marketing and advertising, educational technology, and e-commerce. However, existing text-to-image generation technologies face challenges such as low efficiency in model resource allocation, unstable generation results, and difficulty in dynamically adapting to diverse needs. In particular, when processing complex input text, a single generation model struggles to balance diversity and accuracy, resulting in insufficient matching between the quality of the generated image and the required specifications. [Summary of the Invention]

[0003] The purpose of this invention is to integrate multimodal results generated from text, thereby enhancing the consistency of image generation results.

[0004] The image generation system of the present invention includes a user device and a server. The server is communicatively connected to the user device, and the server includes an image generation module and an expert model library. The image generation module includes a dynamic generation strategy planner, an inference generation executor, and a multimodal fusion and alignment element. The dynamic generation strategy planner includes a vector feature classification element, a generation task parsing element, and an expert task allocation network. The server receives prompts from the user device. The vector feature classification element uses vectors corresponding to the prompts to obtain vector feature classification results. The generation task parsing element parses generation tasks corresponding to the vector feature classification results. The expert task allocation network uses generation tasks to determine expert models from the expert model library. The inference generation executor executes the expert models to obtain image generation results. The multimodal fusion and alignment element performs fusion and alignment operations on the image generation results to obtain target image generation results.

[0005] The image generation method of the present invention is suitable for a system including a user device and a server, wherein the server includes an image generation module and an expert model library, wherein the image generation module includes a dynamic generation strategy planner, an inference generation executor, and a multimodal fusion and alignment element, wherein the dynamic generation strategy planner includes a vector feature classification element, a generation task parsing element, and an expert task allocation network, wherein the method includes the following steps: the server receives prompt words from the user device; the vector feature classification element uses vectors corresponding to the prompt words to obtain vector feature classification results; the generation task parsing element parses generation tasks corresponding to the vector feature classification results; the expert task allocation network uses generation tasks to determine expert models from the expert model library; the inference generation executor executes the expert models to obtain image generation results; and the multimodal fusion and alignment element performs fusion and alignment operations on the image generation results to obtain target image generation results.

[0006] Based on the above, the system and method for image generation of the present invention can utilize vectors corresponding to prompt words to obtain vector feature classification results, and parse out the generation task corresponding to the vector feature classification results. Then, the generation task can be used to determine an expert model, and the expert model can be executed to obtain the image generation result. Finally, the present invention can perform fusion and alignment operations on the image generation result to obtain the target image generation result. Based on this, the present invention can utilize prompt words to obtain multiple generation tasks (sub-tasks), and then dynamically allocate suitable expert models to perform corresponding generation, so as to achieve customized selection and processing based on text semantics. In particular, the present invention can fuse multimodal results of text generation (such as: video, narration, audio), thereby enhancing the consistency of image generation results.

Implementation Method

[0007] FIG1 is a schematic diagram illustrating a system 1 for image generation according to an embodiment of the present invention. FIG2 is a further description of the system 1 shown in FIG1. ​​Please refer to FIG1 and FIG2 simultaneously. In this embodiment, system 1 may include a user device 10 and a server 20. The server 20 is communicatively connected to the user device 10. The server 20 may include an image generation module 23 and an expert model library 24, wherein the image generation module 23 may include a dynamic generation strategy planner 231, an inference generation executor 232, and a multimodal fusion and alignment element 233, wherein the dynamic generation strategy planner 231 may include a vector feature classification element 2311, a generation task parsing element 2312, and an expert task allocation network 2313. In one embodiment, the server 20 may include a prompt word parsing module 21 and a prompt word embedding mapping module 22. In one embodiment, the server 20 may include a post-processing and output module 25. In one embodiment, the user device 10 may include a prompt word input guidance module 11, an image output module 12, and a user identity management module 13. In one embodiment, the prompt input guidance module 11, the image output module 12, the user identity management module 13, the prompt parsing module 21, the prompt embedding mapping module 22, the image generation module 23, the expert model library 24, and the post-processing and output module 25 may be software and / or firmware code executed by a processor.

[0008] Figure 3 is an example of the operation of System 1 shown in Figure 1. Please refer to Figures 1, 2, and 3 simultaneously. First, the user can operate the user device 10 to input a prompt. Specifically, after the user identity management module 13 verifies the user's identity, the prompt input guidance module 11 can guide the user to input a prompt. Next, the server 20 can receive the prompt from the user device 10. Then, the server 20 can use the prompt to obtain the target image generation result. Finally, the server 20 can transmit the target image generation result to the user device 10, and the image output module 12 can output the target image generation result to the user.

[0009] The following will continue to explain the implementation method of "server 20 using prompt words to obtain target image generation results". For ease of understanding, the following will use the prompt word S as an example: "Close-up, a young woman is applying skincare products in front of the camera, her expression is pleasant and satisfied, and she happily says: 'The fine lines seem to have decreased!', a bright room, filled with a soft and comfortable atmosphere, accompanied by light and gentle music." It is worth mentioning that the user can also operate the user device 10 to input the parameters of the target image generation result. For example, the parameters of the target image generation result can be a YouTube video (resolution: 1920*1080) or an Instagram video (resolution: 1080*1920).

[0010] Figure 4 is an example of the operation of the prompt word parsing module 21 and the prompt word embedding mapping module 22 shown in Figure 1. Please refer to Figures 1 to 4 simultaneously.

[0011] First, the prompt word parsing module 21 can parse out the features of the prompt words. In one embodiment, the prompt word parsing module 21 can use a sub-word segmentation function to parse out the features of the prompt words. For example, the sub-word segmentation function is WordPiece, but the present invention is not limited thereto. In detail, as shown in Formula 1, the prompt word parsing module 21 can add a start symbol and a stop symbol to the prompt word S. Next, as shown in Formula 2, the prompt word parsing module 21 can use the prompt word S′ after adding the start symbol and the stop symbol to obtain the Token sequence. Next, as shown in Formula 3, the prompt word parsing module 21 can use the Token sequence to obtain the Token ID sequence.

[0012] S′=[CLS]+ WordPiece(S) +[SEP] … (Formula 1)

[0013] T = WordPiece(S′) = {t 1​,t 2​, ..., tn​}, ti​ ∈ V… (Formula 2)

[0014] Token ID sequence={ID(t 1​),ID(t 2​), ..., ID(tn​)}… (Formula 3)

[0015] Where T is the token sequence, ti​ is the token, V is the vocabulary, and ID(ti) is the corresponding token ID retrieved from the vocabulary. In other words, the token ID is the aforementioned feature.

[0016] Next, the prompt word embedding mapping module 22 can utilize features to obtain vectors corresponding to the prompt words. In one embodiment, the prompt word embedding mapping module 22 can utilize a word embedding model and features to obtain vectors corresponding to the prompt words. A word embedding model is, for example, BERT; however, the invention is not limited thereto. Specifically, assume a token ID sequence: X = [101, 791, 4638, 1921, 2523…]. The prompt word embedding mapping module 22 can utilize three embedding layers (word embedding layer, segment embedding layer, and position embedding layer) for vector transformation. This will be further explained below.

[0017] In the word embedding layer operation, the prompt word embedding mapping module 22 can use the Token ID to query the vocabulary embedding matrix (ET) in the language model and convert the Token ID into the corresponding word vector, as shown in Equations 4 and 5. It should be noted that each Token xi corresponds to a d-dimensional vector.

[0018] ET∈RV×d… (Formula 4)

[0019] T i​ = ET​[xi​] xi​ ∈ X… (Formula 5)

[0020] Where V is the size of the vocabulary (e.g., BERT's vocabulary V=21128), and d is the word embedding dimension (e.g., BERT's d=768).

[0021] In the operation of the segmented embedding layer, the prompt word embedding mapping module 22 can use the Segment ID to distinguish the Tokens of different sentences (the Token of the first sentence is Segment ID=0, and so on), and the prompt word embedding mapping module 22 can use the Segment ID to query the corresponding embedding matrix (Es) in the language model to find the corresponding vector value, as shown in Formula 6 and Formula 7.

[0022] E s∈ R c×d… (Formula 6)

[0023] S i​=ES​[si​], si​∈S… (Formula 7)

[0024] Where c is the number of sentences.

[0025] In the operation of the position embedding layer, the prompt word embedding mapping module 22 can mark the position of the Token in the sentence and convert the order of the Token in the sentence into vector values, as shown in Formula 8 and Formula 9.

[0026] EP​∈RL×d… (Formula 8)

[0027] P i​=EP​[i]… (Formula 9)

[0028] Where L is the length of the longest sequence of sentences (e.g., 512), i is the position of the token, and Pi is the position vector, with each position i corresponding to a position vector Pi.

[0029] Finally, the prompt word embedding mapping module 22 can add the three embedding vectors together to calculate the vector corresponding to the prompt word, as shown in Formula 10.

[0030] E i=T i+S i+ P i... (Formula 10)

[0031] That is: E = ET​[X​] + E s​[S​] + E p​[P​].

[0032] Figure 5 is an operational example of the vector feature classification element 2311 and the task generation parsing element 2312 shown in Figure 2. Please refer to Figures 1 to 5 simultaneously. The vector feature classification element 2311 can obtain vector feature classification results using vectors corresponding to prompt words. Then, the task generation parsing element 2312 can parse out the generation task corresponding to the vector feature classification results. In one embodiment, the task generation parsing element 2312 can be implemented using a task classifier. For example, the task generation parsing element 2312 can use a multilayer perceptron (MLP) classifier to determine the generation task. In detail, the task generation parsing element 2312 can perform classification calculations on the input prompt word vector Ep, and the task generation parsing element 2312 can use softmax to calculate the task classification result on the output result o, as shown in Equations 11 and 12.

[0033] o=W 3​σ(W 2​σ(W 1​EP​+b 1​)+b 2​)+b 3​… (Formula 11)

[0034] ​=argmax(softmax(o)) … (Formula 12)

[0035] W1​, W2​, W3​ are learnable weight matrices, b1​, b2​, b3​ are bias terms, σ(x) is a non-linear activation function (such as ReLU), is the classification result (representing the category of the generated task), softmax is used to calculate the probability of each category of the generated task, and argmax is used to obtain the maximum value in the function set.

[0036] Figure 6 is an example of the operation of the expert task assignment network 2313 shown in Figure 2. Please refer to Figures 1 to 6 simultaneously. In this embodiment, the expert task assignment network 2313 can use generated tasks to determine expert models from the expert model library 24. In one embodiment, the expert task assignment network 2313 can be implemented using a multilayer perceptron (MLP). Specifically, the expert task assignment network 2313 can calculate the appropriate expert model probabilities to determine the optimal expert model, as shown in Equations 13 and 14.

[0037] g = softmax(W g​​+bg​) … (Formula 13)

[0038] M* = M argmax(g)… (Formula 14)

[0039] Wg is the weight matrix of the expert selection layer (Wg​ ∈ RK×K), bg is the bias term of the expert selection layer (bg∈ RK), g is the selection weight / probability distribution of the expert model (g ∈ RK), and M* is the most suitable expert model.

[0040] Figure 7 is an example of the operation of the inference generation executor 232 shown in Figure 2. Please refer to Figures 1 to 7 simultaneously. In this embodiment, the inference generation executor 232 can execute an expert model to obtain image generation results. In detail, the inference generation executor 232 can activate the corresponding expert model (determined by the expert task allocation network 2313) to perform generation task calculations.

[0041] Figure 8 is an example of the operation of the multimodal fusion and alignment element 233 shown in Figure 2. Please refer to Figures 1 to 8 simultaneously. In this embodiment, the multimodal fusion and alignment element 233 can perform fusion and alignment operations on the image generation results to obtain the target image generation results.

[0042] In one embodiment, the multimodal fusion and alignment element 233 can be implemented using a self-attention mechanism. Assume the scene's embedding vector is ES, and the character's embedding vector is EC, and that the final output of the multimodal fusion and alignment element 233 after fusing ES and EC is Ecp. Further, ES ∈ RN×d, and EC ∈ RM×d, where N is the number of scene blocks (e.g., ground, buildings, sky, etc.), M is the number of key parts of the character (e.g., head, hands, feet, etc.), and d is the embedding dimension (e.g., 768). The multimodal fusion and alignment element 233 can map ES and EC to the Query, Key, and Value spaces, as shown in Equations 14, 15, and 16.

[0043] Q = WQ​EC​ … (Formula 14)

[0044] K = WK​ES​ … (Formula 15)

[0045] V = WV​ES​ … (Formula 16)

[0046] WQ​, WK​ and WV​ are weight matrices learned by the multimodal fusion and alignment element 233, Q is the information of the role query scene, K is the information in the scene, and V is the actual feature of the scene (used for fusion).

[0047] Next, the multimodal fusion and alignment element 233 can calculate the degree of correlation between key parts of the character and scene blocks to determine which scene blocks the character should focus on. For example, the character's "head" should focus on the "center of the room" rather than the "table". As another example, when parts of the character enter the area illuminated by light, the color or brightness should change accordingly, as shown in Formula 17.

[0048] A = softmaxA ∈ RM×N… (Formula 17)

[0049] Where A is the attention matrix calculated by the multimodal fusion and alignment element 233.

[0050] Finally, the multimodal fusion and alignment element 233 can fuse the character and the scene and output the fusion feature by weighted summation. In this way, the multimodal fusion and alignment element 233 can make the character affect the scene or adjust the character's movements to obtain the target image generation result E SC, as shown in Equation 18.

[0051] E SC​=AV… (Formula 18)

[0052] In another embodiment, the multimodal fusion and alignment element 233 can be implemented using a lip-shape prediction model. Specifically, the multimodal fusion and alignment element 233 can use a neural network-based lip-shape prediction model to map speech features to lip shapes and calculate the lip-shape sequence M, as shown in Equation 19. Table 1 shows an example of the correspondence between speech features and lip shapes.

[0053] M = f mouth(S)… (Formula 19)

[0054] Where S is the input speech feature sequence, f mouth is the mouth shape prediction model, and k is the length of the mouth shape sequence.

[0055] Table 1 speech features mouth shape a Open your mouth b Mouth closed p mouth slightly open m Mouth closed, lips pressed together

[0056] Next, the multimodal fusion and alignment element 233 can use dynamic time planning (DTW) to calculate the optimal alignment path between S and M in the above formula 19 to align the speech signal and mouth movements, thereby ensuring that the speech segment corresponds to the correct mouth shape, as shown in formula 20.

[0057] DTW(S, M) =arg… (Formula 20)

[0058] where π is an aligned index sequence that maps speech time step si to mouth shape time step m π(i), and distance(si,m π(i)) is the distance metric between speech and mouth shape, which can be Euclidean distance or other metric methods.

[0059] Furthermore, when the time steps of speech and lip movements are inconsistent, the multimodal fusion and alignment element 233 can perform interpolation alignment to adjust M, as shown in Formula 21.

[0060] M aligned​=Interp(M, scale factor)… (Formula 21)

[0061] Where, scale factor is a factor that adjusts the length of the mouth shape time step, and Interp is the interpolation method (e.g., linear interpolation or spline interpolation).

[0062] Finally, the multimodal fusion and alignment element 233 can apply the aligned mouth movements to the character model to complete the synchronous alignment of mouth and voice to obtain the target image generation result.

[0063] After the multimodal fusion and alignment element 233 obtains the target image generation result, the post-processing and output module 25 can perform post-processing according to the parameters of the target image generation result and transmit it to the user device 10. Then, the image output module 12 can display the target image generation result to the user.

[0064] FIG9 is a flowchart illustrating a method for image generation according to an embodiment of the present invention, wherein the method can be implemented by the system shown in FIG1. ​​In step S91, the server receives a prompt word from the user device. In step S92, the vector feature classification element uses the vector corresponding to the prompt word to obtain a vector feature classification result. In step S93, the generation task parsing element parses out the generation task corresponding to the vector feature classification result. In step S94, the expert task allocation network uses the generation task to determine an expert model from the expert model library. In step S95, the inference generation executor executes the expert model to obtain an image generation result. In step S96, the multimodal fusion and alignment element performs fusion and alignment operations on the image generation result to obtain a target image generation result.

[0065] In summary, the system and method for image generation of the present invention can utilize vectors corresponding to prompt words to obtain vector feature classification results, and parse out the generation task corresponding to the vector feature classification results. Then, the generation task can be used to determine an expert model, and the expert model can be executed to obtain the image generation result. Finally, the present invention can perform fusion and alignment operations on the image generation result to obtain the target image generation result. Based on this, the present invention can utilize prompt words to obtain multiple generation tasks (sub-tasks), and then dynamically allocate suitable expert models to perform corresponding generation, so as to achieve customized selection and processing based on text semantics. In particular, the present invention can fuse multimodal results of text generation (such as: video, narration, audio), thereby enhancing the consistency of image generation results. [Simplified Explanation of the Diagram]

[0066] Figure 1 is a schematic diagram of a system for image generation according to an embodiment of the present invention. Figure 2 is a further description of the system shown in Figure 1. Figure 3 is an operational example of the system shown in Figure 1. Figure 4 is an operational example of the prompt word parsing module and the prompt word embedding mapping module shown in Figure 1. Figure 5 is an operational example of the vector feature classification element and the generation task parsing element shown in Figure 2. Figure 6 is an operational example of the expert task allocation network shown in Figure 2. Figure 7 is an operational example of the inference generation executor shown in Figure 2. Figure 8 is an operational example of the multimodal fusion and alignment element shown in Figure 2. Figure 9 is a flowchart of a method for image generation according to an embodiment of the present invention.

Claims

1. A system for image generation, comprising: A user device and a server communicatively connected to the user device, wherein the server includes an image generation module and an expert model library, wherein the image generation module includes a dynamic generation strategy planner, an inference generation executor, and a multimodal fusion and alignment element, wherein the dynamic generation strategy planner includes a vector feature classification element, a generation task parsing element, and an expert task allocation network, wherein the server receives prompts from the user device; the vector feature classification element uses vectors corresponding to the prompts to obtain vector feature classification results, wherein the vectors include close-ups, subjects, motion, narration, background, and sound effects, wherein the vector feature classification results include camera movement features corresponding to the close-ups, subject features corresponding to the subjects, motion features corresponding to the motion, narration features corresponding to the narration, background features corresponding to the background, and sound effect features corresponding to the sound effects; the generation task parsing element parses generation tasks corresponding to the vector feature classification results; the expert task allocation network uses the generation tasks to determine expert models from the expert model library; the inference generation executor executes the expert models to obtain image generation results. The multimodal fusion and alignment element performs fusion and alignment operations on the image generation result to obtain the target image generation result.

2. The system as described in claim 1, wherein the server further includes a prompt word parsing module and a prompt word embedding mapping module, wherein the prompt word parsing module parses features of the prompt words; and the prompt word embedding mapping module uses the features to obtain the vector corresponding to the prompt words.

3. The system as described in claim 2, wherein the prompt word parsing module uses a sub-word segmentation function to parse the features of the prompt words.

4. The system as described in claim 2, wherein the prompt word embedding mapping module utilizes a word vector embedding model and the features to obtain the vector corresponding to the prompt word.

5. The system as described in claim 1, wherein the generating task parsing element is implemented as a task classifier.

6. The system as described in claim 1, wherein the expert task assignment network is implemented using a multilayer perceptron (MLP).

7. The system as claimed in claim 1, wherein the multimodal fusion and alignment elements are implemented using a self-attention mechanism.

8. The system as claimed in claim 1, wherein the multimodal fusion and alignment elements are implemented using a mouth shape prediction model.

9. A method for image generation, suitable for a system including a user device and a server, wherein the server includes an image generation module and an expert model library, wherein the image generation module includes a dynamic generation strategy planner, an inferential generation executor, and a multimodal fusion and alignment element, wherein the dynamic generation strategy planner includes a vector feature classification element, a generation task parsing element, and an expert task assignment network, wherein the method The steps include: The server receives a prompt word from the user device; The vector feature classification element uses vectors corresponding to the prompt words to obtain vector feature classification results, wherein the vectors include close-up, subject, motion, narration, background, and sound effects, and wherein the vector feature classification results include camera movement features corresponding to the close-up, subject features corresponding to the subject, motion features corresponding to the motion, narration features corresponding to the narration, background features corresponding to the background, and sound effect features corresponding to the sound effects; the generation task parsing element parses the generation task corresponding to the vector feature classification results; the expert task allocation network uses the generation task to determine an expert model from the expert model library; the inference generation executor executes the expert model to obtain the image generation result; and the multimodal fusion and alignment element performs fusion and alignment operations on the image generation result to obtain the target image generation result.

10. The method of claim 9, wherein the server further includes a prompt parsing module and a prompt embedding mapping module, wherein the method further includes the steps of: the prompt parsing module parsing features of the prompt; and the prompt embedding mapping module using the features to obtain the vector corresponding to the prompt.

11. The method as described in request 10, wherein the step of the prompt word parsing module parsing the features of the prompt word includes: The prompt word parsing module uses a sub-word segmentation function to parse out the features of the prompt words.

12. The method of claim 10, wherein the step of the prompt word embedding mapping module using the features to obtain the vector corresponding to the prompt word includes: The prompt word embedding mapping module uses a word vector embedding model and the features to obtain the vector corresponding to the prompt word.

13. The method of claim 9, wherein the generation of task parsing elements is implemented as a task classifier.

14. The method of claim 9, wherein the expert task assignment network is implemented as a multilayer perceptron (MLP).

15. The method of claim 9, wherein the multimodal fusion and alignment elements are implemented using a self-attention mechanism.

16. The method of claim 9, wherein the multimodal fusion and alignment elements are implemented using a mouth shape prediction model.

Images