Image generation method and image generation system applying the same

Abstract

This invention provides an image generation method and an image generation system using the same. The image generation method includes the following steps: Executing a text-to-image program to generate a base image based on a text prompt. Executing an image expansion drawing program to obtain an expanded image based on the base image. Executing a resolution adjustment program to adjust the resolution of the expanded image to obtain a resolution-adjusted image. Executing a hidden watermark compositing program to composite a hidden watermark onto the resolution-adjusted image to obtain a generated image. The text-to-image program, image expansion drawing program, resolution adjustment program, and hidden watermark compositing program are executed in a pipelined manner through a graphics processor and a neural processor.

Description

Technical Field

[0001] This invention relates to an artificial intelligence method and an artificial intelligence system using the same, and more particularly to an image generation method and an image generation system using the same. Background Technology

[0002] With the rapid development of artificial intelligence technology, various image generation and image modification techniques have emerged. However, currently, if multiple programs need to be executed using artificial intelligence technology, a huge amount of system resources must be consumed to achieve this. For example, an NVIDIA 2080 16GB processor is required to generate a 1280x720 AI image, which makes it difficult for artificial intelligence technology to be widely adopted in various laptops.

[0003] Therefore, effectively reducing system resources and shortening image processing time is the direction that the industry is currently striving to address. Summary of the Invention

[0004] This invention relates to an image generation method and an image generation system using the same, which utilizes a pipeline approach to execute text-to-image conversion, image expansion and rendering, resolution adjustment, and watermark compositing, thereby significantly improving the speed of image generation.

[0005] According to one aspect of the present invention, an image generation method is proposed. The image generation method includes the following steps: Executing a text-to-image program to generate a base image based on a text prompt; Executing an image expansion drawing program to obtain an expanded image based on the base image; Executing a resolution adjustment program to adjust the resolution of the expanded image to obtain a resolution-adjusted image; Executing a hidden watermark synthesis program to synthesize a hidden watermark on the resolution-adjusted image to obtain a generated image. The text-to-image program, the image expansion drawing program, the resolution adjustment program, and the hidden watermark synthesis program are executed in a pipelined manner using a Graphics Processing Unit (GPU) and a Neural Network Processing Unit (NPU).

[0006] According to another aspect of the present invention, an image generation system is proposed. The image generation system includes a text-to-image module, an image expansion and rendering module, a resolution adjustment module, and a hidden watermark synthesis module. The text-to-image module executes a text-to-image program to generate a base image based on a text prompt. The image expansion and rendering module executes an image expansion and rendering program to obtain an expanded image based on the base image. The resolution adjustment module executes a resolution adjustment program to adjust the resolution of the expanded image to obtain a resolution-adjusted image. The hidden watermark synthesis module executes a hidden watermark synthesis program to synthesize a hidden watermark on the resolution-adjusted image to obtain a generated image. The text-to-image program, the image expansion and rendering program, the resolution adjustment program, and the hidden watermark synthesis program are executed in a pipelined manner using a Graphics Processing Unit (GPU) and a Neural Network Processing Unit (NPU).

[0007] According to another aspect of the present invention, an image generation method is proposed. The image generation method includes the following steps: Executing a text-to-image program to generate a base image based on a text prompt; Executing an image expansion drawing program to obtain an expanded image based on the base image; Executing a resolution adjustment program to adjust the resolution of the expanded image to obtain a resolution-adjusted image; Executing a hidden watermark synthesis program to synthesize a hidden watermark on the resolution-adjusted image to obtain a generated image. In the image expansion drawing program, resolution adjustment program, and hidden watermark synthesis program, intermediate computation results are directly obtained from the memory of a Graphics Processing Unit (GPU) or a Neural Network Processing Unit (NPU). Attached Figure Description

[0008] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0009] Figure 1 An example illustrates an image generation method according to an embodiment of the present invention.

[0010] Figure 2 This example illustrates how to execute a text-to-image conversion program, an image expansion drawing program, a resolution adjustment program, and a hidden watermark compositing program in a pipelined manner.

[0011] Figure 3A block diagram of an image generation system according to an embodiment of the present invention is shown.

[0012] Figure 4 A flowchart illustrating an image generation method according to an embodiment of the present invention is shown.

[0013] Figure 5 Example illustration of an image extension drawing procedure.

[0014] Figure 6 The example illustrates the relationship between the preprocessed image and the image to be drawn.

[0015] Figure 7 The example illustrates one way in which the image generation method of the present invention achieves zero copying.

[0016] Figure 8 The example illustrates another way in which the image generation method of the present invention achieves zero copying.

[0017] Figure label:

[0018] 100: Text to Image Module

[0019] 200: Image Extension Drawing Module

[0020] 220: Preprocessing unit

[0021] 230: Cutting unit

[0022] 240: Shielding unit

[0023] 250: Extended Unit

[0024] 260: Noise Reduction Unit

[0025] 280: Splicing unit

[0026] 300: Resolution Adjustment Module

[0027] 400: Hidden watermark compositing module

[0028] 1000: Image Generation System

[0029] B2: Circular Tile

[0030] B21, B23, B25, B27: Corner tiles

[0031] B22, B24, B26, B28: Rectangular connection blocks

[0032] CPU: Central Processing Unit

[0033] GPU: Graphics Processor

[0034] IM1: Base Image

[0035] IM2: Pre-processed image

[0036] IM31, IM32, IM33, IM34: Images to be drawn

[0037] IM51, IM52, IM53, IM54: Images already drawn

[0038] IM8: Expanded image

[0039] IM9: Adjusted resolution image

[0040] IM10, IM101, IM102, IM103: Generated Images

[0041] NPU: Neural Processor

[0042] MK41, MK42, MK43, MK44: Shielded

[0043] PD1: Text-to-Image Program

[0044] PD2: Image Extension Drawing Program

[0045] PD3: Resolution Adjustment Program

[0046] PD4: Hidden Watermark Synthesis Process

[0047] PL1, PL3, PL5, PL7: Corner pixels

[0048] PL2, PL4, PL6, PL8: Side pixels

[0049] PT, PT1, PT2, PT3: Text prompts

[0050] T1, T2, T3, T4, T5, T6, T7: Time points

[0051] WM: Hide Watermark Detailed Implementation

[0052] The technical terms used in this specification are based on common terminology in the field. Where this specification provides further explanation or definition of certain terms, the interpretation of those terms shall be based on the explanation or definition provided in this specification. Each embodiment of the present invention has one or more technical features. Where feasible, those skilled in the art may selectively implement some or all of the technical features in any embodiment, or selectively combine some or all of the technical features in these embodiments.

[0053] Please refer to Figure 1The following example illustrates an image generation method according to an embodiment of the present invention. The image generation method includes, for example, a text-to-image program PD1, an image expansion and drawing program PD2, a resolution adjustment program PD3, and a hidden watermark synthesis program PD4. In the text-to-image program PD1, a base image IM1 is generated based on text prompts PT using artificial intelligence (AI) technology. Next, in the image expansion and drawing program PD2, the base image IM1 is expanded and extended outwards to obtain an expanded image IM8. Then, in the resolution adjustment program PD3, the resolution of the expanded image IM8 is adjusted to obtain a resolution-adjusted image IM9. Next, in the hidden watermark synthesis program PD4, a hidden watermark WM is added to the expanded image IM8 to obtain a generated image IM10. The hidden watermark WM is scattered across several pixels of the generated image, and its presence is imperceptible to the human eye.

[0054] Please refer to Figure 2 The example illustrates a pipelined execution of the text-to-image program PD1, the image expansion and rendering program PD2, the resolution adjustment program PD3, and the hidden watermark compositing program PD4. In this embodiment, the four programs—text-to-image program PD1, image expansion and rendering program PD2, resolution adjustment program PD3, and hidden watermark compositing program PD4—are executed, for example, through a graphics processing unit (GPU) and a neural network processing unit (NPU) in a pipelined manner.

[0055] For example, such as Figure 2 As shown, from time point T1 to time point T2, the text-to-image program PD1 of the text prompt PT1 can be processed by the graphics processing unit (GPU).

[0056] Next, as Figure 2 As shown, from time point T2 to time point T3, the neural processor NPU can process the image extension drawing program PD2 of the text prompt PT1, and the graphics processor GPU can process the text-to-image program PD1 of the text prompt PT2.

[0057] Then, as Figure 2 As shown, from time point T3 to time point T4, the neural processor NPU can process the resolution adjustment program PD3 of text prompt PT1 and the image expansion drawing program PD2 of text prompt PT2, and the graphics processor GPU can process the text-to-image program PD1 of text prompt PT3.

[0058] Next, as Figure 2 As shown, from time point T4 to time point T5, the GPU processes the hidden watermark synthesis program PD4 for text prompt PT1, the image extension rendering program PD2 for text prompt PT3, and the resolution adjustment program PD3 for text prompt PT3. Subsequent time points T6 and T7 follow the same pattern to obtain the generated images IM101, IM102, and IM103 at each time point.

[0059] In other words, in this embodiment, the image generation method is executed in a pipelined manner, which can utilize the graphics processing unit (GPU) and neural processing unit (NPU) to process multiple programs simultaneously, thereby accelerating the processing speed.

[0060] Please refer to Figure 3 The diagram illustrates a block diagram of an image generation system 1000 according to an embodiment of the present invention. The image generation system 1000 includes a text-to-image module 100, an image expansion and rendering module 200, a resolution adjustment module 300, and a hidden watermark compositing module 400. The text-to-image module 100 converts text into an image. The image expansion and rendering module 200 expands the image. The resolution adjustment module 300 adjusts the image resolution. The hidden watermark compositing module 400 adds a watermark to the image.

[0061] like Figure 3As shown, the image expansion drawing module 200 includes a preprocessing unit 220, a cutting unit 230, a masking unit 240, an expansion unit 250, a noise reduction unit 260, and a stitching unit 280. The text-to-image module 100, the image expansion drawing module 200, the preprocessing unit 220, the cutting unit 230, the masking unit 240, the expansion unit 250, the noise reduction unit 260, the stitching unit 280, the resolution adjustment module 300, and the hidden watermark synthesis module 400 are, for example, a circuit board, a storage device storing program code, or a chip. The chip may be, for example, a processor, or other programmable general-purpose or special-purpose microcontroller (MCU), microprocessor, digital signal processor (DSP), programmable controller, application-specific integrated circuit (ASIC), image signal processor (ISP), image processing unit (IPU), arithmetic logic unit (ALU), complex programmable logic device (CPLD), field programmable gate array (FPGA), or other similar components or combinations thereof. The text-to-image module 100, image expansion drawing module 200, resolution adjustment module 300, and hidden watermark synthesis module 400 of the image generation system 1000 respectively execute the text-to-image program PD1, the image expansion drawing program PD2, the resolution adjustment program PD3, and the hidden watermark synthesis program PD4. A flowchart is provided below to illustrate the operation of each component.

[0062] Please refer to the following at the same time Figure 3 and Figure 4 , Figure 4 A flowchart illustrating an image generation method according to an embodiment of the present invention is shown. First, as... Figure 1 As shown, in the text-to-image program PD1, the text-to-image module 100 generates a base image IM1 based on the text prompt PT. For example, when the text prompt PT is received from the user inputting "beautiful sunset", the text-to-image module 100 will use artificial intelligence image generation technology to generate a landscape image with a sunset.

[0063] Next, please refer to Figure 3 and Figure 5 , Figure 5 Example illustration of image expansion drawing procedure PD2. In image expansion drawing procedure PD2, image expansion drawing module 200 obtains expanded image IM8 based on base image IM1. Image expansion drawing module 200 preprocessing unit 220 fills a ring-shaped patch B2 surrounding base image IM1 with several edge pixels PL of base image IM1 to obtain a preprocessed image IM2.

[0064] The ring-shaped block B2 includes several corner blocks B21, B23, B25, and B27, and several rectangular connecting blocks B22, B24, B26, and B28. Rectangular connecting block B22 connects corner blocks B21 and B23; rectangular connecting block B24 connects corner blocks B23 and B25; rectangular connecting block B26 connects corner blocks B25 and B27; and rectangular connecting block B28 connects corner blocks B27 and B21.

[0065] The edge pixel PL includes several corner pixels PL1, PL3, PL5, and PL7, and several side pixels PL2, PL4, PL6, and PL8. Side pixel PL2 is located between corner pixels PL1 and PL3; side pixel PL4 is located between corner pixels PL3 and PL5; side pixel PL6 is located between corner pixels PL5 and PL7; and side pixel PL8 is located between corner pixels PL7 and PL1.

[0066] Corner block B21 is filled with the corner pixel PL1 of the base image IM1; rectangular connection block B22 is filled with the side pixel PL2 of the base image IM1; corner block B23 is filled with the corner pixel PL3 of the base image IM1; rectangular connection block B24 is filled with the side pixel PL4 of the base image IM1; corner block B25 is filled with the corner pixel PL5 of the base image IM1; rectangular connection block B26 is filled with the side pixel PL6 of the base image IM1; corner block B27 is filled with the corner pixel PL7 of the base image IM1; and rectangular connection block B28 is filled with the side pixel PL8 of the base image IM1. This filling action can improve the accuracy and speed of extended drawing. In one embodiment, the filling action can be omitted.

[0067] Next, as Figure 3 and Figure 5As shown, the cutting unit 230 of the image extension drawing module 200 cuts the pre-processed image IM2 into several images to be drawn, IM31, IM32, IM33, and IM34. The number of these images to be drawn, IM31, IM32, IM33, and IM34, is four. Image IM31 includes the upper left corner of the base image IM1 and the upper left corner of the annular block B2; image IM32 includes the upper right corner of the base image IM1 and the upper right corner of the annular block B2; image IM33 includes the lower right corner of the base image IM1 and the lower right corner of the annular block B2; and image IM34 includes the lower left corner of the base image IM1 and the lower left corner of the annular block B2. The dimensions of these images IM31, IM32, IM33, and IM34 are substantially the same.

[0068] In one embodiment, the images to be drawn IM31, IM32, IM33, and IM34 comprise a base image IM1 of the same size. In another embodiment, the images to be drawn IM31, IM32, IM33, and IM34 may comprise a base image IM1 of different sizes.

[0069] Please refer to Figure 6 The example illustrates the relationship between the preprocessed image IM2 and the images to be drawn, IM31, IM32, IM33, and IM34. These images to be drawn, IM31, IM32, IM33, and IM34, partially overlap at the edges to facilitate image stitching.

[0070] Next, as Figure 3 and Figure 5 As shown, the masking unit 240 of the image extension drawing module 200 obtains several masks MK41, MK42, MK43, and MK44 based on the images to be drawn IM31, IM32, IM33, and IM34. In each mask MK41, MK42, MK43, and MK44, pixels corresponding to 0 (diagonal lines) do not require content expansion, while pixels corresponding to 1 (blank areas) do require content expansion. That is, the base image IM1 corresponding to pixels of 0 will be retained, and only the annular patch B2 corresponding to pixels of 1 needs content expansion.

[0071] Then, as Figure 3 and Figure 5As shown, the expansion unit 250 of the image expansion drawing module 200 expands the content of the annular blocks B2 of the portions of the images IM31, IM32, IM33, and IM34 to be drawn, based on the shields MK41, MK42, MK43, and MK44, to obtain several drawn images IM51, IM52, IM53, and IM54. Since the expansion actions of these images IM31, IM32, IM33, and IM34 are performed separately in a pipeline manner, they do not simultaneously occupy computing resources and memory resources.

[0072] In the step of expanding the content of the ring-shaped patch B2 of the portions of the images IM31, IM32, IM33, and IM34 to be drawn, the expansion is performed using positive prompts, without the need for negative prompts. In other words, researchers found that negative prompts are not very helpful for content expansion; therefore, omitting negative prompts and using only positive prompts for content expansion can improve the expansion speed while maintaining a certain level of accuracy.

[0073] Next, as Figure 3 and Figure 5 As shown, the noise reduction unit 260 of the image extension drawing module 200 performs noise reduction on the drawn images IM51, IM52, IM53, and IM54.

[0074] Then, as Figure 3 and Figure 5 As shown, the stitching unit 280 of the image extension drawing module 200 stitches together the drawn images IM51, IM52, IM53, and IM54 to obtain an extended image IM8. Since the images to be drawn IM31, IM32, IM33, and IM34 partially overlap at the edges, the drawn images IM51, IM52, IM53, and IM54 can be successfully stitched together without any abrupt lines.

[0075] Next, as Figure 1 and Figure 3 As shown, in the resolution adjustment program PD3, after the resolution adjustment module 300 adjusts the resolution of the extended image IM8, the image IM9 with the adjusted resolution is obtained. In this program, the resolution of the extended image IM8 can be adjusted according to the resolution of the display.

[0076] Then, as Figure 1 and Figure 3 As shown, in the hidden watermark compositing program PD4, the hidden watermark compositing module 400 composites a hidden watermark WM onto the resized image IM9 to obtain the generated image IM10.

[0077] In the above embodiments, a pipelined approach can be used to execute the text-to-image program PD1, the image expansion and drawing program PD2, the resolution adjustment program PD3, and the hidden watermark synthesis program PD4. Multiple programs can be executed simultaneously, which can significantly speed up the processing speed.

[0078] Please refer to Figure 7 The example illustrates one way in which the image generation method of the present invention achieves zero copy. Figure 7 In this embodiment, the neural processor (NPU) executes a text-to-image program PD1, an image expansion and rendering program PD2, a resolution adjustment program PD3, and a hidden watermark synthesis program PD4. After the text-to-image program PD1 is executed, the intermediate calculation results are stored in the NPU's memory. The image expansion and rendering program PD2 directly retrieves the intermediate calculation results of the text-to-image program PD1 from the NPU's memory. After the image expansion and rendering program PD2 is executed, the intermediate calculation results are stored in the NPU's memory. The resolution adjustment program PD3 directly retrieves the intermediate calculation results of the image expansion and rendering program PD2 from the NPU's memory. After the resolution adjustment program PD3 is executed, the intermediate calculation results are stored in the NPU's memory. The resolution adjustment program PD3 directly retrieves the intermediate calculation results of the hidden watermark synthesis program PD4 from the NPU's memory.

[0079] exist Figure 7 In this embodiment, intermediate computation results do not need to be transmitted to the central processing unit (CPU) first. The neural processing unit (NPU) retrieves intermediate computation results directly from its own memory, which can significantly save data transfer time.

[0080] Please refer to Figure 8 The example illustrates another way in which the image generation method of the present invention achieves zero-copying. Figure 8In this embodiment, the graphics processing unit (GPU) executes a text-to-image program PD1, an image extension and rendering program PD2, a resolution adjustment program PD3, and a hidden watermark compositing program PD4. After the text-to-image program PD1 is executed, the intermediate calculation results are stored in the GPU's memory. The image extension and rendering program PD2 directly retrieves the intermediate calculation results of the text-to-image program PD1 from the GPU's memory. After the image extension and rendering program PD2 is executed, the intermediate calculation results are stored in the GPU's memory. The resolution adjustment program PD3 directly retrieves the intermediate calculation results of the image extension and rendering program PD2 from the GPU's memory. After the resolution adjustment program PD3 is executed, the intermediate calculation results are stored in the GPU's memory. The resolution adjustment program PD3 directly retrieves the intermediate calculation results of the hidden watermark compositing program PD4 from the GPU's memory.

[0081] exist Figure 8 In this embodiment, intermediate computation results do not need to be transferred to the central processing unit (CPU) first. The graphics processing unit (GPU) retrieves the intermediate computation results directly from its own memory, which can significantly save data transfer time.

[0082] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications and improvements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be defined by the claims.

Claims

1. An image generation method, comprising: Run a text-to-image program to generate a base image based on a text prompt; Execute an image expansion drawing program to obtain an expanded image based on the base image; After performing a resolution adjustment procedure to adjust the resolution of the expanded image, a resolution-adjusted image is obtained; and A hidden watermark compositing procedure is executed to composit a hidden watermark onto the resized image, resulting in a generated image. The text-to-image program, the image expansion and drawing program, the resolution adjustment program, and the hidden watermark synthesis program are executed in a pipelined manner through a graphics processing unit (GPU) and a neural network processing unit (NPU).

2. The image generation method as described in claim 1, characterized in that, In the image extension drawing program, the resolution adjustment program, and the hidden watermark synthesis program, an intermediate calculation result is obtained directly from a memory of the graphics processor or a memory of the neural processor.

3. The image generation method as described in claim 1, characterized in that, The image extension drawing program includes: Using multiple edge pixels of the base image, fill in a ring-shaped patch surrounding the base image to obtain a preprocessed image; The preprocessed image is cut into multiple images to be drawn, each image to be drawn including a portion of the base image and a portion of the ring-shaped tile; Based on the multiple images to be drawn, multiple masks are obtained; Based on these multiple masks, the content of the annular block of each portion of the multiple images to be drawn is expanded to obtain multiple drawn images; and The multiple drawn images are stitched together to obtain the expanded image.

4. The image generation method as described in claim 3, characterized in that, The ring-shaped tile includes multiple corner tiles and multiple rectangular connecting tiles. The multiple edge pixels include multiple corner pixels and multiple side pixels. The multiple rectangular connecting tiles connect the multiple corner tiles. The multiple corner tiles are filled with the multiple corner pixels of the base image, and the multiple rectangular connecting tiles are filled with the multiple side pixels of the base image.

5. The image generation method as described in claim 3, characterized in that, The multiple images to be drawn are essentially the same size.

6. The image generation method as described in claim 3, characterized in that, The multiple images to be drawn contain portions of the base image of the same size.

7. The image generation method as described in claim 3, characterized in that, The number of images to be drawn is 4.

8. The image generation method as described in claim 3, characterized in that, The multiple images to be drawn partially overlap.

9. The image generation method as described in claim 3, characterized in that, In each of these masking operations, pixels corresponding to 0 do not require content expansion, while pixels corresponding to 1 do require content expansion.

10. An image generation system, comprising: The text-to-image module is used to execute a text-to-image program to generate a base image based on a text prompt. An image extension drawing module is used to execute an image extension drawing program to obtain an extended image based on the base image; A resolution adjustment module is used to execute a resolution adjustment program to adjust the resolution of the expanded image and obtain an image with adjusted resolution. as well as A hidden watermark compositing module is used to execute a hidden watermark compositing program to composit a hidden watermark on the resized image and obtain a generated image. The text-to-image program, the image expansion and drawing program, the resolution adjustment program, and the hidden watermark synthesis program are executed in a pipelined manner through a graphics processing unit (GPU) and a neural network processing unit (NPU).

11. The image generation system as described in claim 10, characterized in that, In the image extension drawing program, the resolution adjustment program, and the hidden watermark synthesis program, an intermediate calculation result is obtained directly from a memory of the graphics processor or a memory of the neural processor.

12. The image generation system as described in claim 10, characterized in that, The image extension drawing module includes: A preprocessing unit fills a ring-shaped patch surrounding the base image with multiple edge pixels of the base image to obtain a preprocessed image; A cutting unit is used to cut the preprocessed image into multiple images to be drawn, each image to be drawn including a portion of the base image and a portion of the ring-shaped block; A shielding unit is used to obtain multiple shields based on the multiple images to be drawn; An expansion unit is used to expand the content of the annular block of each portion of the plurality of images to be drawn, based on the plurality of shields, to obtain a plurality of drawn images; and A stitching unit is used to stitch together the multiple drawn images to obtain an expanded image.

13. The image generation system as described in claim 12, characterized in that, The ring-shaped tile includes multiple corner tiles and multiple rectangular connecting tiles. The multiple edge pixels include multiple corner pixels and multiple side pixels. The multiple rectangular connecting tiles connect the multiple corner tiles. The multiple corner tiles are filled with the multiple corner pixels of the base image, and the multiple rectangular connecting tiles are filled with the multiple side pixels of the base image.

14. The image generation system as described in claim 12, characterized in that, The multiple images to be drawn are essentially the same size.

15. The image generation system as described in claim 12, characterized in that, The multiple images to be drawn contain portions of the base image of the same size.

16. An image generation method, comprising: Run a text-to-image program to generate a base image based on a text prompt; Execute an image expansion drawing program to obtain an expanded image based on the base image; After performing a resolution adjustment procedure to adjust the resolution of the expanded image, a resolution-adjusted image is obtained; and A hidden watermark compositing procedure is executed to composit a hidden watermark onto the resized image, resulting in a generated image. In the image extension drawing program, the resolution adjustment program, and the hidden watermark synthesis program, an intermediate computation result is obtained directly from the memory of a graphics processing unit (GPU) or a neural network processing unit (NPU).

Images