Training methods for generator network models and the electronic devices that execute them.

Abstract

This invention proposes a training method for a generator network model and an electronic device for executing it. The training method includes the following steps: extracting a first tensor matrix and a second tensor matrix, the first tensor matrix and the second tensor matrix representing a first image and a second image respectively, and each including multiple first parameters and multiple second parameters; generating multiple third images based on multiple differences between the multiple first parameters of the first tensor matrix and the multiple second parameters of the second tensor matrix; performing similarity detection between the multiple original images and the multiple third images; and selecting at least one of the multiple third images with a similarity lower than or equal to a similarity threshold as at least one new sample image.

Description

Technical Field

[0001] This invention relates to a model improvement method and apparatus using neural network computing technology, and particularly to a training method for the generator network model in a Generative Adversarial Network (GAN) model and the electronic device for executing it. Background Technology

[0002] Typical Generative Adversarial Network (GAN) models are used to automatically generate images with patterns that closely resemble real-world objects. The generator and discriminator networks in a GAN model are trained on the original training dataset and then subjected to adversarial training. The patterns in the images generated by the generator become increasingly similar to those of real-world objects, thus enabling the discriminator network to recognize the optimized generator's images as real-world patterns. However, from another perspective, typical GAN ​​models only generate images with patterns almost identical to or highly similar to the original images in the training dataset, failing to produce images rich in pattern feature variations. Therefore, typical GAN ​​models lack the ability to generate creative and novel images. Summary of the Invention

[0003] In view of this, the present invention provides a method for training a generator network model and an electronic device for executing it, which enables the trained generator network model to generate creative and novel images.

[0004] The training method of the generator network model of the present invention includes the following steps: extracting a first tensor matrix and a second tensor matrix, wherein the first tensor matrix and the second tensor matrix represent a first image and a second image respectively and respectively include a plurality of first parameters and a plurality of second parameters; generating a plurality of third images based on a plurality of differences between the plurality of first parameters of the first tensor matrix and the plurality of second parameters of the second tensor matrix; performing similarity detection on the plurality of original images and the plurality of third images; and taking at least one of the plurality of third images with a similarity lower than or equal to a similarity threshold as at least one new sample image.

[0005] The electronic device of the present invention is adapted to execute a generator network model. The electronic device includes a storage device and a processor. The storage device stores the generator network model. The processor is coupled to the storage device. The processor executes the generator network model. The processor extracts a first tensor matrix and a second tensor matrix from the generator network model. The first tensor matrix and the second tensor matrix represent a first image and a second image, respectively, and each includes a plurality of first parameters and a plurality of second parameters. The processor generates a plurality of third images based on a plurality of differences between the plurality of first parameters of the first tensor matrix of the first image and the plurality of second parameters of the second tensor matrix of the second image. The processor performs similarity detection between the plurality of original images and the plurality of third images. The processor uses at least one of the plurality of third images with a similarity lower than or equal to a similarity threshold as at least one new sample image.

[0006] Based on the above, the generator network model training method and the electronic device executing it of the present invention can generate multiple sample images with pattern feature gradient characteristics.

[0007] To make the above features and advantages of the present invention more apparent and understandable, specific embodiments are described below, and detailed descriptions are provided in conjunction with the accompanying drawings. Attached Figure Description

[0008] Figure 1 This is a schematic diagram of an electronic device according to an embodiment of the present invention.

[0009] Figure 2 This is a flowchart of a training method according to an embodiment of the present invention.

[0010] Figure 3A This is a schematic diagram of the original image according to an embodiment of the present invention.

[0011] Figure 3B This is a schematic diagram of a first picture according to an embodiment of the present invention.

[0012] Figure 3C This is a schematic diagram of the third picture according to an embodiment of the present invention.

[0013] Figure 3D This is a schematic diagram of another original image according to an embodiment of the present invention.

[0014] Figure 3E This is a schematic diagram of a second picture according to an embodiment of the present invention.

[0015] Figure 3F This is a schematic diagram of another third picture according to an embodiment of the present invention.

[0016] Figure 4 This is a flowchart of a training method according to another embodiment of the present invention.

[0017] Figure 5 This is a schematic diagram of the process of training a generator network model according to an embodiment of the present invention.

[0018] [List of Labels in the Attached Image]

[0019] 100: Electronic devices

[0020] 110: Processor

[0021] 120: Storage device

[0022] 121: Generative Adversarial Network Model

[0023] 121_1: Generator Network Model

[0024] 121_2: Discriminator Network Model

[0025] 310, 340, 501_1~501_M: Original images

[0026] 311, 321, 331, 341, 351, 361: Patterns

[0027] 320: First picture

[0028] 330, 360, 502_1~502_N: Third image

[0029] 350: Second image

[0030] 503_1~503_P: New sample images

[0031] 504_1~504_K: Images from the new training dataset

[0032] S210, S220, S230, S240, S401~S411: Steps Detailed Implementation

[0033] To make the contents of this invention more readily apparent, the following specific embodiments are provided as examples on which this disclosure can indeed be implemented. Furthermore, wherever possible, elements / components / steps referred to by the same reference numerals in the drawings and embodiments represent the same or similar parts.

[0034] Figure 1 This is a schematic diagram of an electronic device according to an embodiment of the present invention. (See reference) Figure 1The electronic device 100 includes a processor 110 and a storage device 120. The storage device 120 can store a Generative Adversarial Network (GAN) model 121. The processor 110 is coupled to the storage device 120. In this embodiment, the processor 110 can execute the GAN model 121, which may include a generator network model (generating network) and a discriminator network model (discriminator network). The processor 110 can generate new sample images to replace the original sample images during the training and adversarial process of the generator network model and the discriminator network model, so that the trained generator network model can generate new varied images that retain some pattern features of the original sample images but are different from the original sample images in terms of pattern feature combinations.

[0035] In this embodiment, the processor 110 may be, for example, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), or other programmable general-purpose or special-purpose microprocessors, digital signal processors (DSPs), programmable controllers, application-specific integrated circuits (ASICs), programmable logic devices (PLDs), other similar processing devices, or combinations thereof. The storage device 120 may be, for example, dynamic random access memory (DRAM), flash memory, or non-volatile random access memory (NVRAM). The storage device 120 can store the generative adversarial network model 121, related algorithms, and image data, and provides the processor 110 with the ability to read and execute them.

[0036] Figure 2 This is a flowchart of a training method according to an embodiment of the present invention. (See reference) Figure 1 as well as Figure 2The electronic device 100 can execute the following steps S210 to S240 to generate new sample images that can be used to train the generator network model in the generative adversarial network model 121. The processor 110 can execute the generative adversarial network model 121. In step S210, the processor 110 can extract a first tensor matrix and a second tensor matrix from the generator network model. The first tensor matrix and the second tensor matrix represent a first image and a second image, respectively, and each includes a plurality of first parameters and a plurality of second parameters. The aforementioned plurality of first parameters and plurality of second parameters refer to a plurality of vector values ​​in the first tensor matrix and the second tensor matrix, respectively. For example, the tensor matrix can be a three-dimensional space matrix, and the plurality of parameters of the tensor matrix can include a plurality of vector values ​​in the x-direction, a plurality of vector values ​​in the y-direction, and a plurality of vector values ​​in the z-direction of the vector space. In step S220, the processor 110 can generate a plurality of third images based on a plurality of differences between the plurality of first parameters of the first tensor matrix and the plurality of second parameters of the second tensor matrix. In step S230, the processor 110 can perform similarity detection between the plurality of original images and the plurality of third images. In step S240, the processor 110 may use at least one of a plurality of third images with a similarity lower than or equal to a similarity threshold as at least one new sample image.

[0037] For example, please refer to the accompanying text. Figures 3A to 3F , Figures 3A to 3F This is a schematic diagram of multiple images according to an embodiment of the present invention. In this embodiment, under the initial training of a stable generative adversarial network model 121, the processor 110 can randomly extract a first tensor matrix and a second tensor matrix from the latent space of the generator network model to generate multiple, for example, Figure 3A and Figure 3D The original images 310 and 340 shown are similar. Figure 3B First picture 320 and Figure 3E The second image 350. The aforementioned first tensor matrix can represent the pattern content as in the first image 320, and the aforementioned second tensor matrix can represent the pattern content as in the second image 350. The original images 310 and 340 have patterns 311 and 341 respectively, the first image 320 has pattern 321, and the second image 350 has pattern 351. Therefore, the pattern 321 of the first image 320 can have overall similar pattern features to the pattern 311 of the original image 310, and the pattern 351 of the second image 350 can also have overall similar pattern features to the pattern 341 of the original image 340.

[0038] Next, the processor 110 can generate, for example, based on the differences between multiple first parameters of the first tensor matrix of the first image 320 and multiple second parameters of the second tensor matrix of the second image 350. Figure 3C and Figure 3EThe third images 330 and 360 shown are examples where patterns 331 and 361 are novel pattern features observed during the gradual transformation of pattern 321 to pattern 351 (or vice versa). It is noteworthy that when the similarity between the third image 330 and the original images 310 and 340 is lower than or equal to a similarity threshold, the processor 110 uses the third image 330 as a new sample image. Similarly, when the similarity between the third image 360 ​​and the original images 310 and 340 is lower than or equal to a similarity threshold, the processor 110 uses the third image 360 ​​as a new sample image.

[0039] In other words, the processor 110 in this embodiment can generate new sample images based on the pattern features of patterns 311 and 341 in the original images 310 and 340. The patterns of the new sample images can have some degree of difference from the patterns 311 and 341 in the original images 310 and 340, but not be too similar. After multiple recursive training sessions, the processor 110 can gradually replace the images in the original training dataset with the successively added new sample images. Furthermore, after recursively training the generator network model and discriminator network model of the generative adversarial network model 121, the trained generator network model can generate patterns 331 and 361 in the third images 330 and 360 that can be identified by the discriminator network model and have pattern features different from those of the original images 310 and 340, rather than patterns 321 and 351 in the first image 320 and the second image 350 that are similar to those of the original images 310 and 340.

[0040] Figure 4 This is a flowchart of a training method according to another embodiment of the present invention. Figure 5 This is a schematic diagram illustrating the process of training a generative adversarial network model according to an embodiment of the present invention. (See reference) Figure 1 , Figure 4 as well as Figure 5 The electronic device 100 may perform the following steps S401 to S411 to train the generative adversarial network model 121.

[0041] In step S401, the processor 110 can first train the generative adversarial network model 121 to stability using multiple original images 501_1 to 501_M, where M is a positive integer.

[0042] In step S402, processor 110 executes generative adversarial network model 121 to randomly extract a first tensor matrix and a second tensor matrix from the latent space of generator network model 121_1 of generative adversarial network model 121. The first tensor matrix and the second tensor matrix represent the first image and the second image, respectively, for example... Figure 3B The first image 320 and Figure 3D The second image 350. Thus, the processor 110 can find the undisplayed images in the latent space of the generator network model 121_1.

[0043] In step S403, the processor 110 determines whether the blurriness of at least one of the first image and the second image is higher than a first blurriness threshold. If yes, the processor 110 re-executes step S402. If no, the processor 110 executes step S404. In this embodiment, based on the fact that the blurriness of at least one of the first image and the second image is higher than the first blurriness threshold, the processor 110 can re-extract at least one of the new first tensor matrix and the second tensor matrix from the generator network model. In other words, the processor 110 in this embodiment can avoid using images with excessively blurry patterns. In this embodiment, the processor 110 can determine the blurriness of the first image and the second image through at least one of Laplace edge computing and contrast detection.

[0044] In step S404, the processor 110 divides the differences between multiple first parameters of the first tensor matrix (e.g., a 3D spatial matrix) of the first image and multiple second parameters of the second tensor matrix (e.g., a 3D spatial matrix) of the second image by preset parameter values ​​to obtain multiple difference feature values. These difference feature values ​​are then successively accumulated to the corresponding first parameters of the first tensor matrix to generate different third tensor matrices. These different third tensor matrices represent different third images. Furthermore, the aforementioned preset parameter values ​​correspond to the number of times third images are generated, i.e., the number of third images generated in a single training process. For example, if the preset parameter value is 10, it means that the difference feature values ​​are divided by 10 equal parts, so the multiple first parameters of the first tensor matrix can be accumulated 10 times to generate 10 third images (generating 10 different third tensor matrices). Additionally, it is worth noting that the overall value of the tensor matrix of the 10th third image will be equal to the overall value of the second tensor matrix of the second image; therefore, the pattern content of the 10th third image is the same as the pattern content of the second image.

[0045] In step S405, the processor 110 determines whether the blurriness of the third image is higher than the second blurriness threshold. If yes, the processor 110 executes step S406. If no, the processor 110 executes step S407. In this embodiment, based on the fact that the blurriness of the third image is higher than the second blurriness threshold, the processor 110 can discard the third image generated in the current accumulation operation. In other words, the processor 110 in this embodiment can avoid using a third image with an overly blurry pattern. In this embodiment, the processor 110 can determine the blurriness of the third image through at least one of Laplacian edge detection and contrast detection.

[0046] In step S406, the processor 110 determines whether the current accumulation count is equal to the preset count. If yes, the processor 110 executes step S411 to end the execution flow. If no, the processor 110 executes step S404 to perform the next accumulation operation.

[0047] In step S407, the processor 110 determines whether the third image is similar to the multiple original images. If so, the processor 110 executes step S410. If not, the processor 110 executes step S408. In this embodiment, the processor 110 can utilize a fully connected layer (FC layer) in a Convolutional Neural Network (CNN) to perform similarity detection between the third image generated by the current accumulation operation and the original images 501_1 to 501_M in the original training dataset. It is worth noting that when the similarity between the third image generated by the current accumulation operation and each of the original images 501_1 to 501_M is lower than or equal to the similarity threshold, the processor 110 uses the third image 330 as a new sample image. In other words, if the third image generated by the current accumulation operation is too similar to one of the original images 501_1 to 501_M in the original training dataset, the processor 110 discards the third image generated by the current accumulation operation. If the third image generated by the current accumulation operation is not similar to (i.e. different from) one of the original images 501_1 to 501_M in the original training dataset, then the processor 110 executes step S408.

[0048] In step S408, the processor 110 uses the third image generated by the current accumulation operation as a new sample image and performs feature enhancement processing on the new image. In this embodiment, the aforementioned feature enhancement processing may include at least one of gamma filtering and inverse filtering on the new sample image. Therefore, the processor 110 can further process the new sample image into an image with clear patterns and obvious pattern features. In step S409, the processor 110 outputs the new sample image. In step S410, the processor 110 determines whether the current accumulation count is equal to a preset count. If yes, the processor 110 executes step S411 to end the execution flow. If no, the processor 110 executes step S404 to perform the next accumulation operation.

[0049] In detail, such as Figure 5 As shown, processor 110 can discard overly blurry images from the third images 502_1 to 502_N, and then compare them with the original images 501_1 to 501_M of the original training dataset to exclude images that are too similar to the original images 501_1 to 501_M. N is a positive integer. Therefore, processor 110 can use the remaining third images after exclusion as new sample images 503_1 to 503_P, where P is a positive integer. Next, processor 110 can use the new sample images 503_1 to 503_P to replace a portion of the original images 501_1 to 501_M of the original training dataset to generate images 504_1 to 504_K of the new training dataset, where K is a positive integer. In this embodiment, the processor 110 can use new sample images 503_1 to 503_P to train the discriminator network model 121_2 of the generative adversarial network model 121, so that the discriminator network model 121_2 can regard the patterns of the new sample images 503_1 to 503_P as objects that can be retained in the next discrimination operation. Furthermore, the processor 110 can use images 504_1 to 504_K from the new training dataset to retrain the generator network model 121_1 of the generative adversarial network model 121, so that the generator network model 121_1 has a chance to generate images with different pattern features than the original images 501_1 to 501_M in the next image generation (drawing) operation.

[0050] It is worth noting that the images 504_1_504_K in the new training dataset may include new sample images 503_1 to 503_P and a portion of the original images 501_1 to 501_M. In other words, the new sample images 503_1 to 503_P can replace another portion of the original images 501_1 to 501_M. Furthermore, after the generator network model 121_1 and the discriminator network model 121_2 have been trained to a stable state, the processor 110 can execute steps S401 to S411 again. Similarly, after the processor 110 recursively executes these steps multiple times, the original images 501_1 to 501_M may be completely replaced by new sample images. Therefore, the generative adversarial network model 121, after recursively training for multiple epochs, can output images that are completely different from the original images 501_1 to 501_M and have rich pattern feature variations.

[0051] The aforementioned accumulation operation is explained in more detail below. For example, a first tensor matrix T1 (e.g., a three-dimensional spatial matrix) represents a first image, and a second tensor matrix T2 (e.g., a three-dimensional spatial matrix) represents a second image. Taking a preset parameter value of 200 (i.e., N = 200) as an example, subtracting multiple first parameters of the first tensor matrix T1 from multiple second parameters of the second tensor matrix T2 yields a difference matrix D1 (D1 = (T2 - T1)) with multiple differences (the difference matrix D1 is a three-dimensional spatial matrix). Dividing each difference in the difference matrix D1 by 200 yields a difference feature matrix D2 (D2 = D1 / 200) with multiple difference eigenvalues. Therefore, the processor 110 can successively (200 times) accumulate the multiple difference eigenvalues ​​of the difference feature matrix D2 to the corresponding multiple first parameters of the first tensor matrix T1 to successively generate third tensor matrices T3_1 to T3_200 for different third images.

[0052] Therefore, the third tensor matrix T3_1 of the third image generated by the first accumulation can be generated by the matrix operation: T3_1 = T1 + 1 × D2. The third tensor matrix T3_2 of the third image generated by the second accumulation can be generated by the matrix operation: T3_2 = T1 + 2 × D2. The third tensor matrix T3_3 of the third image generated by the third accumulation can be generated by the matrix operation: T3_3 = T1 + 3 × D2. And so on, the third tensor matrix T3_200 of the third image generated by the 200th accumulation can be generated by the matrix operation: T3_200 = T1 + 200 × D2, and the third tensor matrix T3_200 of the third image generated by the 200th accumulation is equal to the second tensor matrix T2 of the second image. In other words, the processor 110 can generate 199 third images 502_1 to 502_199, and the third images 502_1 to 502_199 are the result of the first image being gradually transformed into the second image.

[0053] Next, the processor 110 performs blur and similarity checks on the third images 502_1 to 502_199. When the processor 110 determines that the blur of at least one of the third images 502_1 to 502_199 is higher than a second blur threshold, the processor 110 discards the overly blurry image and performs similarity detection on the remaining third images. Furthermore, after the processor 110 excludes images that are too similar to the original images 501_1 to 501_M, assuming there are 10 images remaining, the processor 110 can, for example, use these 10 images with novel pattern feature changes as 10 new sample images and replace a portion of the original images 501_1 to 501_M. Therefore, the processor 110 can use the images 504_1_504_K of the new training dataset, which includes these 10 new sample images, to retrain the generative adversarial network model 121.

[0054] In summary, the training method and electronic device for the generative adversarial network model of the present invention can effectively generate multiple sample images with gradual changes in pattern features. Furthermore, it can be combined with ambiguity and similarity detection to retain images with clear patterns and novel pattern feature variations as new training data. Therefore, the generative adversarial network model of the present invention, after training, has the function of generating novel images with creative and clear patterns.

[0055] Although the present invention has been disclosed above with reference to embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some changes and modifications without departing from the concept and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the claims.

Claims

1. A method for training a generator network model, comprising: Extract a first tensor matrix and a second tensor matrix, wherein the first tensor matrix and the second tensor matrix represent a first image and a second image respectively, and each includes multiple first parameters and multiple second parameters; Multiple third images are generated based on multiple differences between the multiple first parameters of the first tensor matrix and the multiple second parameters of the second tensor matrix; Perform a similarity detection between multiple original images and these multiple third images; At least one of the plurality of third images with a similarity lower than or equal to a similarity threshold shall be used as at least one new sample image; The steps involved in generating these multiple third images include: Divide each of the multiple differences by a preset parameter value to obtain multiple difference feature values; as well as The multiple difference feature values ​​are successively added to the multiple first parameters of the corresponding first tensor matrix to generate multiple different third tensor matrices, which represent the multiple third images.

2. The training method as described in claim 1, further comprising: Based on the fact that the blur of at least one of the first image and the second image is higher than a first blur threshold, a new first tensor matrix and at least one of the new second tensor matrix are re-extracted.

3. The training method as described in claim 2, further comprising: The blurriness of the first image and the second image is determined by at least one of a Laplacian edge operation and a contrast detection.

4. The training method as described in claim 1, further comprising, before performing the similarity detection: Discard at least one of the plurality of third images whose ambiguity is higher than a second ambiguity threshold in order to perform the similarity detection on the remaining third images.

5. The training method as described in claim 4, further comprising, before performing the similarity detection: The blurriness of the plurality of third images is determined by at least one of a Laplacian edge operation and a contrast detection.

6. The training method as described in claim 1, further comprising: Perform a feature enhancement process on at least one new sample image.

7. The training method of claim 6, wherein the feature enhancement processing includes at least one of performing a gamma filter and an inverse filter on the new sample image.

8. The training method of claim 1, wherein the generator network model is trained to stability via the plurality of original images before extracting the first tensor matrix and the second tensor matrix.

9. An electronic device suitable for executing a generator network model, comprising: A storage device for storing the generator network model; as well as A processor, coupled to the storage device, is used to execute the generator network model. The processor extracts a first tensor matrix and a second tensor matrix from the generator network model. The first and second tensor matrices represent a first image and a second image, respectively, and each includes multiple first parameters and multiple second parameters. The processor generates multiple third images based on multiple differences between the multiple first parameters of the first tensor matrix and the multiple second parameters of the second tensor matrix. The processor divides each of these differences by a preset parameter value to obtain multiple difference feature values. The processor then successively accumulates these difference feature values ​​to the corresponding first parameters of the first tensor matrix to successively generate multiple different third tensor matrices, which represent the multiple third images. The processor performs a similarity detection between multiple original images and multiple third images, and selects at least one of the multiple third images whose similarity is lower than or equal to a similarity threshold as at least one new sample image.

10. The electronic device of claim 9, wherein, based on the fact that the blurriness of at least one of the first image and the second image is higher than a first blurriness threshold, the processor re-extracts a new first tensor matrix and at least one of a new second tensor matrix through the generator network model.

11. The electronic device of claim 10, wherein the processor determines the blurriness of the first image and the second image by at least one of a Laplacian edge calculation and a contrast detection.

12. The electronic device of claim 9, wherein the processor discards at least one of the plurality of third images whose ambiguity is higher than a second ambiguity threshold in order to perform the similarity detection on the remaining third images.

13. The electronic device of claim 12, wherein the processor determines the blurriness of the plurality of third images by at least one of a Laplacian edge calculation and a contrast detection.

14. The electronic device of claim 9, wherein the processor performs a feature enhancement process on the at least one new sample image.

15. The electronic device of claim 14, wherein the feature enhancement process includes at least one of performing a gamma filter and an inverse filter on the least new sample image.

16. The electronic device of claim 9, wherein before extracting the first tensor matrix and the second tensor matrix, the processor trains the generator network model to stability using the plurality of original images.

Images