Image amplification method and apparatus
By using image augmentation equipment and methods, and leveraging image masking augmentation and generative adversarial network models, a large number of labeled sample images are generated, solving the problem of insufficient training data and improving the training efficiency of machine learning models.
Patent Information
- Application Number
- CN202211597130.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2021-12-21
- Filing Date
- 2022-12-12
- Publication Date
- 2026-01-20
- Estimated Expiration
- 2042-12-12
AI Technical Summary
In existing technologies, training machine learning models requires a large amount of manual labeling, especially mask labeling, which is costly and leads to insufficient training data completeness.
Image augmentation devices and methods are used to augment image masks using memory and processors. Combined with a generative adversarial network model, a large number of labeled sample images are generated, including operations such as contour scaling, rotation, translation and overlay. A large number of sample images are generated using a small number of labeled image masks.
It effectively expands the amount of training data, maintains the integrity of the training data, solves the problem of insufficient training samples, and improves the training efficiency of machine learning models.
Smart Images
Figure CN117274285B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present disclosure relates to a data augmentation technique, in particular, to an image augmentation method and device. BACKGROUND
[0002] In the prior art, when training a machine learning model, in addition to the model architecture being trained affecting the recognition and recognition ability, the most critical is the completeness of the training data. However, in the preprocessing stage, a large amount of manpower is often needed to collect a large number of samples to label a large number of samples, especially the label of the mask which has a high labeling cost. Therefore, how to use a small amount of labeled samples to maintain the completeness of the training data is a problem that those skilled in the art are eager to solve. SUMMARY
[0003] One aspect of the present disclosure discloses an image augmentation device, comprising a memory and a processor. The memory is used to store a first image mask and a second image mask. The processor is connected to the memory to perform the following operations: extracting a first object contour from the first image mask, wherein the first object contour corresponds to a first label; superimposing the first object contour in a superimposition area of the second image mask according to an augmentation parameter to generate a third image mask, wherein the augmentation parameter includes a contour scaling parameter, a contour movement distance, a contour rotation angle, and a superimposable range, the third image mask contains the first object contour and a second object contour in the second image mask; and generating a corresponding sample image according to the first object contour and the second object contour in the third image mask through a generative adversarial network model for machine learning.
[0004] In some embodiments, the contour scaling parameter, the contour movement distance, the contour rotation angle, and the superimposable range are related to an object type of the first object contour.
[0005] In some embodiments, the processor is further used to: proportionally scale the first object contour according to the contour scaling parameter, and rotate the first object contour according to the contour rotation angle; move the coordinates of the first object contour in the first image mask horizontally and vertically according to the contour movement distance to generate the coordinates of the first object contour in the second image mask; and determine whether the coordinates of the first object contour in the second image mask are located in the superimposable range to superimpose the first object contour on the position of the coordinates of the first object contour in the second image mask.
[0006] In some embodiments, the processor is further used to: when it is determined that the coordinates of the first object contour in the second image mask are located in the superimposable range, superimpose the first object contour on the position of the coordinates of the first object contour in the second image mask.
[0007] In some embodiments, the first image mask and the second image mask are binary masks, and the processor is further configured to perform a multiplication operation on the first object contour, the second object contour, and the overlapping region according to a plurality of classification values corresponding to the object type of the first object contour, the object type of the second object contour, and the object type of the overlapping region to overlay the first object contour to the overlapping region in the second image mask.
[0008] In some embodiments, the memory is further configured to store a plurality of preset masks and a plurality of preset images corresponding to the plurality of preset masks, and the processor is further configured to train a generative adversarial network model according to the plurality of preset masks and the plurality of preset images to generate a sample image according to a third image mask by the generative adversarial network model, wherein the generative adversarial network model includes a generator model and a discriminator model, and the generator model includes a plurality of encoding layers and a plurality of decoding layers, and the plurality of encoding layers are connected to the plurality of decoding layers by a skip connection method.
[0009] In some embodiments, the processor is further configured to overlay the first object contour to the overlapping region in the first image mask according to the augmentation parameters to generate a fourth image mask, wherein the fourth image mask includes the first object contour and the overlaid first object contour, and generate another sample image according to the first object contour and the overlaid first object contour in the fourth image mask by the generative adversarial network model to perform machine learning.
[0010] Another aspect of the disclosure discloses an image augmentation method, comprising: extracting a first object contour from a first image mask, wherein the first object contour corresponds to a first label; overlaying the first object contour to an overlapping region in a second image mask according to augmentation parameters to generate a third image mask, wherein the augmentation parameters include a contour scaling parameter, a contour movement distance, a contour rotation angle, and an overlayable range, and the third image mask includes the first object contour and a second object contour in the second image mask; and generating a corresponding sample image according to the first object contour and the second object contour in the third image mask by a generative adversarial network model to perform machine learning.
[0011] In some embodiments, the contour scaling parameter, the contour movement distance, the contour rotation angle, and the overlayable range are related to an object type of the first object contour.
[0012] In some embodiments, the image augmentation method further comprises: scaling the first object contour proportionally according to a contour scaling parameter and rotating the first object contour according to a contour rotation angle; moving the coordinates of the first object contour in the first image mask horizontally and vertically according to a contour moving distance to generate the coordinates of the first object contour in the second image mask; and determining whether the coordinates of the first object contour in the second image mask are within the overlapable range to overlap the first object contour at the coordinates of the first object contour in the second image mask.
[0013] In some embodiments, the image augmentation method further comprises: when determining that the coordinates of the first object contour in the second image mask are within the overlapable range, overlapping the first object contour at the coordinates of the first object contour in the second image mask.
[0014] In some embodiments, the first image mask and the second image mask are binary masks, and the image augmentation method further comprises: performing a multiplication operation on the first object contour, the second object contour, and the overlap area according to a plurality of classification values corresponding to the first object contour, the second object contour, and the overlap area to overlap the first object contour to the overlap area in the second image mask, wherein the plurality of classification values are respectively related to the object type of the first object contour, the object type of the second object contour, and the object type of the overlap area.
[0015] In some embodiments, the image augmentation method further comprises: generating a generative adversarial network model according to a plurality of preset masks and a plurality of preset image training corresponding to the plurality of preset masks to generate a sample image according to a third image mask through the generative adversarial network model, wherein the generative adversarial network model comprises a generator model and a discriminator model, and the generator model comprises a plurality of encoding layers and a plurality of decoding layers, and the plurality of encoding layers are connected to the plurality of decoding layers in a skip connection method.
[0016] In some embodiments, the image augmentation method further comprises: overlapping the first object contour to the overlap area in the first image mask according to the augmentation parameter to generate a fourth image mask, wherein the fourth image mask contains the first object contour and the overlapped first object contour; and generating another sample image through the generative adversarial network model according to the first object contour and the overlapped first object contour in the fourth image mask to perform machine learning. BRIEF DESCRIPTION OF DRAWINGS
[0017] Figure 1 is a block diagram of an image augmentation device of the present disclosure.
[0018] Figure 2 is a flowchart of an image augmentation method of the present disclosure.
[0019] Figure 3 is a schematic diagram of a generative adversarial network model in a training phase according to some embodiments of the present disclosure.
[0020] Figure 4 is a schematic diagram of a generative adversarial network model in a usage phase according to some embodiments of the present disclosure.
[0021] Figure 5 is a schematic diagram of an internal architecture of a generator model according to some embodiments of the present disclosure.
[0022] Figure 6 is a schematic diagram of an internal architecture of a discriminator model according to some embodiments of the present disclosure.
[0023] Figure 7 is a schematic diagram of augmenting a multiple DOF mask according to some embodiments of the present disclosure.
[0024] Figure 8 is a schematic diagram of augmenting a multiple DOF mask according to some other embodiments of the present disclosure.
[0025] Figure 9 is a schematic diagram of augmenting a binary mask according to some embodiments of the present disclosure. DETAILED DESCRIPTION
[0026] Referring to Figure 1 , Figure 1 is a block diagram of an image augmentation device 100 according to some embodiments of the present disclosure. In an embodiment, the image augmentation device 100 comprises a memory 110 and a processor 120. The processor 120 is connected to the memory 110.
[0027] In some embodiments, the image augmentation device 100 can be established by a computer, a server or a processing center. In some embodiments, the memory 110 can be implemented by a storage unit, a flash memory, a read-only memory, a hard disk or any equivalent storage component. In some embodiments, the processor 120 can be implemented by a processing circuit, a central processing unit or a computing unit.
[0028] In some embodiments, the image augmentation device 100 is not limited to comprising the memory 110 and the processor 120, and the image augmentation device 100 can further comprise other elements required in operation and application. For example, the image augmentation device 100 can further comprise an output interface (such as a display panel for displaying information), an input interface (such as a touch panel, a keyboard, a microphone, a scanner or a flash memory reader) and a communication circuit (such as a WiFi communication module, a Bluetooth communication module, a wireless telecommunication network communication module, etc.).
[0029] AsFigure 1 As shown, the memory 110 stores a plurality of image masks MSK(1) ~ MSK(N), where N can be any positive integer and is not particularly limited. In an embodiment, the image augmentation device 100 can perform data augmentation and label classification operations based on the image masks MSK(1) ~ MSK(N), which will be described in detail in subsequent paragraphs. Data augmentation is a technique for increasing the amount of training data, also known as data enhancement or data augmentation. In machine learning training, a large amount of training data is often required. Using data augmentation techniques, more augmented training data can be generated based on the original training data to ensure that overfitting does not occur when training a machine learning model.
[0030] In some embodiments, the image masks MSK(1) ~ MSK(N) can be masks (which can be considered as labeled images) generated by performing feature extraction operations on a plurality of images. In some embodiments, the feature extraction operations can be performed based on Mask Region-based Convolutional Neural Networks (Mask R-CNN) or U-net models, etc. In some embodiments, the image masks MSK(1) ~ MSK(N) can include binary masks, semantic ROI masks, or a combination thereof, where the semantic ROI masks can also be 3DOF masks or multiple DOF masks, etc.
[0031] For example, a feature extraction operation can be performed on an image to perform a binary classification of pixels in the image to determine which regions are regions of interest (ROIs) having features and which regions are non-ROIs not having features. In this way, the ROIs can be associated with weights (i.e., labels) having a value of 1, and the non-ROIs can be associated with weights having a value of 0. In this way, the ROIs and the non-ROIs can be set to white and black, respectively, to generate a black and white image based on the weights of the ROIs and the non-ROIs, and this black and white image can be used as the binary mask.
[0032] In another example, feature extraction operations can be performed on the image to classify the pixels in the image into multiple categories, thereby determining which regions have similar features. Regions with different features can then be assigned different numerical weights. In this way, regions with different features can be assigned different RGB colors based on these weights to generate an RGB image, which can then be used as the aforementioned multi-degree-of-freedom mask.
[0033] It is worth noting that although black and white images and RGB images are used as examples of image masks MSK(1) to MSK(N) here, images of other color spaces (e.g., grayscale images) can also be selected as image masks MSK(1) to MSK(N).
[0034] like Figure 1 As shown, processor 120 is coupled to memory 110, and processor 120 runs a Generative Adversarial Network (GAN) model GANM based on corresponding software / firmware instructions. In some embodiments, processor 120 can also run a Masking Augmentation Model (MAM) and a Machine Learning Model (MLM) based on corresponding software / firmware instructions.
[0035] In some embodiments, the Mask Augmentation Model (MAM) can augment the image masks MSK(1) to MSK(N) to generate various new image masks NEW_MSK, the details of which will be described in detail in subsequent paragraphs. It is worth noting that the number of these new image masks NEW_MSK can be much greater than N (for example, N can be 1500, and the number of new image masks NEW_MSK can be 100000).
[0036] In some embodiments, a Generative Adversarial Network (GANM) model can be used to perform data augmentation based on new image masks (NEW_MSK) to generate multiple augmented sample images. For example, in the preprocessing stage, a GANM model can be pre-trained using the pix2pix algorithm based on multiple preset masks and multiple preset images corresponding to these preset masks. Here, a large number of new image masks (NEW_MSK) generated by a mask augmentation model (MAM) can be input into the GANM model to generate a large number of sample image IMGs corresponding to these new image masks (NEW_MSK), each indicating a label for the sample image IMG. The detailed implementation will be explained in subsequent paragraphs. Therefore, a large number of labeled sample images can be generated from a small number of masks via the mask augmentation model (MAM) and the GANM model, which can expand the number of training samples and solve the problem of insufficient training samples.
[0037] In some embodiments, the machine learning model MLM can be any machine learning model (e.g., a masked region convolutional neural network model or a U-net model, etc.), without any particular limitations.
[0038] like Figure 1 As shown, the processor 120 can train the machine learning model MLM based on the aforementioned new image mask NEW_MSK and the corresponding sample image IMG. In other words, the large number of new image masks NEW_MSK generated by the image augmentation device 100 can be applied to any machine learning model.
[0039] See also Figure 2 , Figure 2 This is a flowchart of the image amplification method disclosed herein. Figure 1 The image augmentation device 100 shown can be used to perform Figure 2 All steps in the image amplification method.
[0040] like Figure 2 As shown, firstly, in step S210, the first object contour is extracted from the first image mask, wherein the first object contour corresponds to the first label.
[0041] In some embodiments, the first image mask can be any one of image masks MSK(1) to MSK(N). In some embodiments, the first label is a weight corresponding to the object type of the first object outline. For example, when the first object outline is the outline of a tree, the first label can be the weight of the tree.
[0042] Furthermore, in step S220, the first object contour is superimposed onto the superimposed area in the second image mask according to the amplification parameters to generate a third image mask, wherein the amplification parameters include contour scaling parameters, contour movement distance, contour rotation angle and superimposed range, and the third image mask includes the first object contour and the second object contour in the second image mask.
[0043] In some embodiments, the second image mask can be any one of the image masks MSK(l) ~ MSK(N) different from the first image mask. In some embodiments, the third image mask is a new image mask NEW_MSK generated by overlaying the first object contour into the overlay region of the second image mask according to the augmentation parameters by the mask augmentation model MAM. In some embodiments, the first object contour and the second object contour can be contours of objects of any object type (e.g., contours of trees). In some embodiments, the object type of the second object contour can be different or the same as the object type of the first object contour. For example, the object type of the first object contour is trees, and the object type of the second object contour is people. In another example, the object type of the first object contour and the object type of the second object contour are both vehicles. In other words, the second label corresponding to the second object contour can be the same or different weight from the first label.
[0044] In some embodiments, when the first image mask and the second image mask are both binary masks, the first object contour, the second object contour, and the overlay region in the second image mask can be multiplied by element-wise operation according to the respective classification values of the object types of the first object contour, the second object contour, and the overlay region in the second image mask, respectively, to overlay the first object contour into the overlay region in the second image mask.
[0045] For example, when the first object contour and the overlay region in the second image mask are iris and eyeball, respectively, the classification value of the iris can be 2, and the classification value of the eyeball can be 1.5. It is noted that the classification values can be pre-stored in the memory 110 or pre-set by a user.
[0046] In some embodiments, the contour scaling parameter, the contour moving distance, the contour rotation angle, and the overlayable range are related to the object type of the first object contour. For example, taking the object type of the first object contour as people as an example. At this time, the contour scaling parameter is the scaling ratio of the people, the contour moving distance is the horizontal and vertical moving distance of the people, the contour rotation angle is the angle of the clockwise rotation of the people, and the overlayable range is the range of the region in the second image mask where the people can be overlaid, wherein the height of the scaled people is not greater than the height of the trees in the second image mask, the rotated people cannot be upside-down people, and the moved people cannot be overlaid in the non-pavement position (e.g., in the sky) in the second image mask. In this way, the overlay region of the people in the second image mask can be identified.
[0047] In some embodiments, the first object contour can be scaled proportionally according to contour scaling parameters, and the first object contour can be rotated according to contour rotation angle. Then, the coordinates of the first object contour in the first image mask can be moved horizontally and vertically according to the contour movement distance to generate the coordinates of the first object contour in the second image mask. When it is determined that the coordinates of the first object contour in the second image mask are within the overlay range, the geometrically transformed first object contour can be overlaid onto the position of the first object contour's coordinates in the second image mask.
[0048] Conversely, when it is determined that the coordinates of the first object outline in the second image mask are outside the overlay range, the outline movement distance can be readjusted until the coordinates of the first object outline in the second image mask are within the overlay range, and then the geometrically transformed first object outline is overlaid at the position of the first object outline in the second image mask.
[0049] It is worth noting that the amplification parameters can be adjusted based on the relationships between various object types and the contours of various objects in the image mask. Furthermore, the aforementioned coordinates can be either centroid coordinates or centripetal coordinates.
[0050] Furthermore, in step S230, a sample image IMG is generated by the Generative Adversarial Network (GANM) model based on the first object contour and the second object contour in the third image mask for machine learning.
[0051] In some embodiments, the memory 110 may also store a plurality of preset masks and a plurality of preset images corresponding to the plurality of preset masks, wherein the preset masks and the preset images are different from the image masks MSK(1) to MSK(N) and sample images IMG described above, and the preset masks and the preset images are only used in the pre-training stage and are not used when a large number of sample images IMG need to be generated.
[0052] In some embodiments, a Generative Adversarial Network (GANM) model can be trained based on multiple preset masks and multiple preset images corresponding to the preset masks. The GANM model generates sample images (IMG) based on a third image mask. The GANM model includes a generator model (not shown) and a discriminator model (not shown). The generator model includes multiple encoder layers (not shown) and multiple decoder layers (not shown), with the encoder layers connected to the decoder layers using a skip connection method. It is noteworthy that the training of the GANM model is completed during the pre-training phase. Therefore, during the usage phase, a large number of sample images (IMG) can be generated directly using a small number of image masks (MSK(1) to MSK(N)) via a mask augmentation model (MAM) and the GANM model.
[0053] In some embodiments, the first object contour can be superimposed onto the superimposed region in the first image mask according to amplification parameters to generate a fourth image mask, wherein the fourth image mask includes the first object contour and the superimposed first object contour. Then, a sample image IMG can be generated by a generative adversarial network (GANM) model based on the first object contour in the fourth image mask and the superimposed first object contour for machine learning. In other words, the fourth image mask may contain two first object contours, and the fourth image mask can also be used to generate a sample image IMG.
[0054] In the above embodiments, although the superposition of two image masks is used for illustration, in practical applications, more than two image masks can be superimposed to generate a large number of sample image IMGs. For example, multiple object contours can be extracted from multiple image masks, and the multiple object contours can be superimposed onto any one or more of the multiple image masks.
[0055] Through the above steps, the image augmentation device 100 can perform masking augmentation based on a small number of image masks MSK(1) to MSK(N) to generate a large number of sample images IMG. In this way, the integrity of the training data can be maintained using a small number of labeled samples.
[0056] The following specific examples further illustrate the training of the Generative Adversarial Network (GANM) model during the preprocessing stage and the execution of the GANM model in this disclosure.
[0057] See also Figure 3 , Figure 3 This is a schematic diagram of a Generative Adversarial Network (GANM) model during the training phase, according to some embodiments of this disclosure. Figure 3As shown, the Generative Adversarial Network (GANM) model can include a generator model (GM) and a discriminator model (DM), wherein the generator model (GM) is connected to the discriminator model (DM).
[0058] During the pre-training phase, the generator model GM generates a generated image G_IMG based on a preset mask PRE_MSK and sends the generated image G_IMG to the discriminator model DM. Next, the generator model GM calculates a first loss L1 based on the generated image G_IMG and the preset image PRE_IMG. Then, the discriminator model DM calculates a second loss L2 based on the preset image PRE_IMG, the preset mask PRE_MSK, and the generated image G_IMG generated by the generator model GM, and performs a backpropagation algorithm based on the second loss L2 to adjust the parameters in the discriminator model DM. Finally, the generator model GM performs a backpropagation algorithm based on the first loss L1 and the second loss L2 to adjust the parameters in the generator model GM. This allows for the continuous adjustment of parameters in both the discriminator model (DM) and the generator model (GM) until the discriminator model (DM) can maximize its ability to identify the generated image G_IMG produced by the generator model (GM) as a real image and maximize its ability to identify the preset image PRE_IMG as a real image. This completes the training of the Generative Adversarial Network (GANM) model. It is worth noting that the calculations for the first loss L1 and the second loss L2 can be L1 distance calculation, L2 distance calculation, Charbonnier distance calculation, or a combination thereof.
[0059] See also Figure 4 , Figure 4 This is a schematic diagram of a Generative Adversarial Network (GANM) model in the usage phase according to some embodiments of this disclosure. Figure 4 As shown, during the usage phase, it is only necessary to input the large number of new image masks NEW_MSK generated by the mask augmentation module MAM into the generator module GM in the pre-trained generative adversarial network model GANM to generate a large number of sample images IMG corresponding to the large number of new image masks NEW_MSK.
[0060] See also Figure 5 , Figure 5 This is a schematic diagram of the internal architecture of a generator model GM according to some embodiments of this disclosure. Figure 5As shown, the generator model GM is a U-Net model, and the generator model GM includes 8 encoding layers ENCODER1~ENCODER8, 8 decoding layers DECODER1~DECODER8, and 1 sigmoid layer SIG1. The encoding layers ENCODER2~ENCODER8 are sequentially connected to DECODER8~DECODER2 using a skip connection method.
[0061] It is worth noting that although this example uses 8 encoding layers and 8 decoding layers, the number of encoding and decoding layers can be adjusted according to different needs.
[0062] In detail, the encoding layers ENCODER1 to ENCODER8 are convolutional layers, and the decoding layers DECODER1 to DECODER8 are deconvolutional layers. The convolutional layers ENCODER1 to ENCODER8 perform convolution processing (e.g., Conv2D function), normalization processing (e.g. InstanceNorm function), and LeakyReLU function processing.
[0063] Furthermore, the deconvolutional layers DECODER1 to DECODER8 perform deconvolution processing (e.g., ConvTranspose2D function), normalization processing (e.g., InstanceNorm function), and ReLU function processing.
[0064] In addition, the sigmoid layer SIG1 performs upsampling (e.g., the UpSample2D function), zero-padding (e.g., the ZeroPad function), convolution (e.g., the Conv2D function), and sigmoid processing.
[0065] See also Figure 6 , Figure 6 This is a schematic diagram of the internal architecture of a discriminator model DM according to some embodiments of this disclosure. Figure 6 As shown, the discriminator model DM consists of eight convolutional layers COV1 to COV5 and one sigmoid layer SIG2. It is worth noting that while eight convolutional layers are used here as an example, the number of convolutional layers can be adjusted according to different needs.
[0066] In detail, the convolutional layers in the discriminator model DM perform convolution processing (e.g., Conv2D function), normalization processing (e.g. InstanceNorm function), and LeakyReLU function processing.
[0067] In addition, the sigmoid layer SIG2 performs zero-padding (e.g., the ZeroPad function), convolution (e.g., the Conv2D function), and sigmoid processing.
[0068] By using this generator model (GM) and discriminator model (DM) architecture and skip connections, the clarity of the sample images (IMG) generated by the generative adversarial network (GANM) model can be greatly improved.
[0069] The following examples illustrate the mask augmentation operations performed on different types of masks.
[0070] See also Figure 7 , Figure 7 This is a schematic diagram illustrating the amplification of multi-degree-of-freedom masks MSK1 to MSK2 according to some embodiments of this disclosure. For example... Figure 7 As shown, the object contour CTR1 can be extracted from the multi-degree-of-freedom mask MSK1, where the object contour CTR1 corresponds to a grayscale value (which can be regarded as a label or weight).
[0071] Furthermore, based on the augmentation parameters of object contour CTR1, object contour CTR1 can be reduced by a factor of 0.8, and its centroid coordinates in the multi-DOF mask MSK1 can be shifted horizontally by -30 pixels and vertically by -30 pixels. This allows the shifted centroid coordinates of object contour CTR1 to be found in the multi-DOF mask MSK2.
[0072] For example, when the centroid coordinates of object outline CTR1 in multi-DOF mask MSK1 are (305, 206), these centroid coordinates can be moved to (275, 176). Then, the coordinates (275, 176) can be found in multi-DOF mask MSK2 and used as the centroid coordinates of object outline CTR1 after the movement.
[0073] Furthermore, it can be determined whether the centroid coordinates of the moved object outline CTR1 are within the overlayable range R1 of the multi-degree-of-freedom mask MSK2, where the overlayable range R1 corresponds to the object type of the object outline CTR1 (i.e., a person). In this case, the centroid coordinates of the moved object outline CTR1 are indeed within the overlayable range R1 of the multi-degree-of-freedom mask MSK2. Therefore, the geometrically processed object outline CTR1 can be overlaid on the aforementioned moved centroid coordinates (i.e., the overlay position) in the multi-degree-of-freedom mask MSK2 to generate a new image mask NEW_MSK.
[0074] See also Figure 8 , Figure 8 This is a schematic diagram illustrating the amplification of multi-degree-of-freedom masks MSK1 to MSK2 according to other embodiments of this disclosure. For example... Figure 8 As shown, object contours CTR1 to CTR2 can be extracted from the multi-degree-of-freedom masks MSK1 to MSK2 respectively, where object contours CTR1 to CTR2 can correspond to different grayscale values respectively.
[0075] Furthermore, based on the amplification parameters of object contour CTR1, object contour CTR1 is enlarged by 1.2 times, rotated by 90 degrees, and its centroid coordinates in the multi-DOF mask MSK1 are shifted horizontally by -50 pixels and vertically by -40 pixels. This allows the shifted centroid coordinates of object contour CTR1 to be found in the multi-DOF mask MSK2.
[0076] Furthermore, it can be determined whether the center of gravity coordinates of the object outline CTR1 after movement are located within the overlapable range R1 of the multi-degree-of-freedom mask MSK2, where the overlapable range R1 corresponds to the object type of the object outline CTR1 (i.e., a person can overlap with a road surface). In this case, the center of gravity coordinates of the object outline CTR1 after movement are indeed located within the overlapable range R1 of the multi-degree-of-freedom mask MSK2.
[0077] Furthermore, based on the amplification parameters of the object contour CTR2, the object contour CTR2 is enlarged by 1.8 times, and its centroid coordinates in the multi-DOF mask MSK2 are shifted horizontally by -40 pixels and vertically by -30 pixels. This allows the shifted centroid coordinates of the object contour CTR2 to be found within the multi-DOF mask MSK2.
[0078] Furthermore, it can be determined whether the centroid coordinates of the object contour CTR2 after movement are located within the overlapable range R1 of the multi-degree-of-freedom mask MSK2, where the overlapable range R1 also corresponds to the object type of the object contour CTR2 (i.e., vehicles can overlap with road surfaces). In this case, the centroid coordinates of the object contour CTR2 after movement are indeed located within the overlapable range R1 of the multi-degree-of-freedom mask MSK2.
[0079] Therefore, the geometrically processed object outlines CTR1 to CTR1 can be superimposed on the respective moved centroid coordinates in the multi-degree-of-freedom mask MSK2 to generate a new image mask NEW_MSK.
[0080] See also Figure 9 , Figure 9 This is a schematic diagram illustrating the amplification of binary masks MSK1 to MSK2 according to some embodiments of this disclosure. For example... Figure 9 As shown, the object contour CTR1 can be extracted from the binary mask MSK1, where the object contour CTR1 corresponds to a binary value representing the object (e.g., a black-and-white value of 1 or the higher of a pair of grayscale values). In this embodiment, the object contour CTR1 is the contour of the iris.
[0081] Furthermore, based on the augmented parameters of the object contour CTR1, no geometric changes or movements are made to the object contour CTR1. This allows the centroid coordinates of the object contour CTR1 to be found within the binary mask MSK2.
[0082] Furthermore, it can be determined whether the centroid coordinates of the object outline CTR1 are located within the overlapable range R1 of the binary mask MSK2, where the overlapable range R1 corresponds to the object type of the object outline CTR1 (i.e., iris). In this embodiment, the eye region in the binary mask MSK2 is the overlapable range R1, and this overlapable range R1 corresponds to the same binary value as the aforementioned binary value.
[0083] At this point, the centroid coordinates of the object outline CTR1 are indeed located within the overlayable region R1 of the binary mask MSK2. Since both the object outline CTR1 and the overlayable region R1 have the same binary values, it is necessary to multiply the binary values of all pixels in the object outline CTR1 by the classification value W1 corresponding to the object outline CTR1, and to multiply the binary values of all pixels in the overlayable region R1 of the binary mask MSK2 by the classification value W2 corresponding to the overlayable region R1. In this embodiment, the classification value W1 corresponding to the object outline CTR1 is the iris classification value, and the classification value W2 corresponding to the overlayable region R1 is the eyeball classification value, where these classification values W1 to W2 can be preset by the user. Therefore, the multiplied object outline WEIGHT_CTR can be overlaid on the centroid coordinates of the multiplied binary mask WEIGHT_MSK to generate a new image mask NEW_MSK.
[0084] On the other hand, at the application level, the image augmentation method disclosed in this document can be used in various fields involving machine vision, image classification, data prediction, or data classification. For example, this image augmentation method can be used for labeling medical image classification data, such as generating X-ray images of normal conditions, pneumonia, bronchitis, and heart disease, or generating ultrasound images of normal fetuses and malpositioned fetuses. Furthermore, this image augmentation method can also be used to generate images collected by autonomous driving systems, such as generating images of normal roads, roads with obstacles, and other vehicle traffic. There are other similar machine learning fields that require large amounts of data; for example, the image augmentation method disclosed in this document can also be used in other machine learning-related fields such as spectrogram recognition, spectral recognition, big data analysis, and data feature recognition.
[0085] In summary, the image augmentation device disclosed herein can generate a large number of labeled images from a small number of image masks. Furthermore, it can generate a large number of image masks from a small number of image masks, thereby enabling the generation of a large number of labeled images using a pre-trained generative adversarial network model. In this way, the integrity of the training data can be easily maintained using a small number of labeled samples.
[0086] While specific embodiments of the present disclosure have been disclosed in relation to the above embodiments, these embodiments are not intended to limit the present disclosure. Various alternatives and modifications can be made by those skilled in the art without departing from the principles and spirit of the present disclosure. Therefore, the scope of protection of the present disclosure is determined by the appended claims.
[0087] [Symbol Explanation]
[0088] 100: Image augmentation equipment
[0089] 110: Memory
[0090] 120: Processor
[0091] MSK(1)~MSK(N): Image Mask
[0092] MAM: Masked Augmentation Model
[0093] GANM: Generative Adversarial Network Model
[0094] MLM: Machine Learning Model
[0095] NEW_MSK: New Image Masking
[0096] IMG: Sample Image
[0097] S210~S230: Steps
[0098] GM: Generator Model
[0099] DM: Discriminator Model
[0100] PRE_MSK: Preset mask
[0101] PRE_IMG: Preset Image
[0102] G_MSK: Generate mask
[0103] L1: First Loss
[0104] L2: Second Loss
[0105] ENCODER1~ENCODER8: Encoders
[0106] DECODER~DECODER8: Decoder
[0107] SIG1, SIG2: sigmoid layers
[0108] COV1~COV5: Convolutional layers
[0109] MSK1, MSK2: Masking
[0110] CTR1, CTR2: Object outline
[0111] R1: Stackable range
[0112] W1~W2: Classification values
[0113] WEIGHT_CTR: The outline of the object being processed in the product operation.
[0114] WEIGHT_MSK: A binary mask for product processing.
Claims
1. An image augmentation method, characterized in that, include: A first object contour is extracted from a first image mask, wherein the first object contour corresponds to a first label; The first object contour is superimposed onto the superimposed area of the second image mask according to amplification parameters to generate a third image mask, wherein the amplification parameters include contour scaling parameters, contour movement distance, contour rotation angle, and superimposed range, and the third image mask includes the first object contour and the second object contour in the second image mask; and A generative adversarial network model generates corresponding sample images based on the first object contour and the second object contour in the third image mask for machine learning.
2. The image augmentation method according to claim 1, wherein the contour scaling parameter, the contour movement distance, the contour rotation angle, and the overlay range are related to the object type of the first object contour.
3. The image amplification method according to claim 1, further comprising: The first object contour is scaled proportionally according to the contour scaling parameters, and the first object contour is rotated according to the contour rotation angle. The coordinates of the first object contour in the first image mask are moved horizontally and vertically according to the contour movement distance to generate the coordinates of the first object contour in the second image mask. as well as Determine whether the coordinates of the first object outline in the second image mask are within the overlay range, so as to overlay the first object outline at the position of the first object outline at the coordinates of the second image mask.
4. The image amplification method according to claim 3, further comprising: When it is determined that the coordinates of the first object outline in the second image mask are within the overlay range, the first object outline is overlaid at the position of the coordinates of the first object outline in the second image mask.
5. The image augmentation method according to claim 1, wherein the first image mask and the second image mask are binary masks, and wherein the image augmentation method further comprises: Based on multiple classification values corresponding to the first object contour, the second object contour, and the overlapping region, element-wise multiplication is performed on the first object contour, the second object contour, and the overlapping region to overlap the first object contour onto the overlapping region in the second image mask. The multiple classification values are respectively related to the object type of the first object contour, the object type of the second object contour, and the object type of the overlapping region.
6. The image amplification method according to claim 1, further comprising: The generative adversarial network model is trained based on multiple preset masks and multiple preset images corresponding to the multiple preset masks, so as to generate the sample image based on the third image mask through the generative adversarial network model. The generative adversarial network model includes a generator model and a discriminator model. The generator model includes multiple encoding layers and multiple decoding layers. The multiple encoding layers are connected to the multiple decoding layers by a skip connection method.
7. The image amplification method according to claim 1, further comprising: The first object contour is superimposed onto the superimposed region in the first image mask according to the amplification parameters to generate a fourth image mask, wherein the fourth image mask includes the first object contour and the superimposed first object contour; and The generative adversarial network model generates another sample image based on the first object contour in the fourth image mask and the superimposed first object contour for machine learning.
8. An image augmentation device, characterized in that, include: A memory for storing the first image mask and the second image mask; as well as The processor, connected to the memory, performs the following operations: A first object contour is extracted from the first image mask, wherein the first object contour corresponds to a first label; The first object contour is superimposed onto the superimposed area of the second image mask according to amplification parameters to generate a third image mask, wherein the amplification parameters include contour scaling parameters, contour movement distance, contour rotation angle, and superimposed range, and the third image mask includes the first object contour and the second object contour in the second image mask; and A generative adversarial network model generates corresponding sample images based on the first object contour and the second object contour in the third image mask for machine learning.
9. The image augmentation apparatus of claim 8, wherein the contour scaling parameters, the contour movement distance, the contour rotation angle, and the superimposed range are related to the object type of the first object contour.
10. The image augmentation apparatus according to claim 8, wherein the processor is further configured to: The first object contour is scaled proportionally according to the contour scaling parameters, and the first object contour is rotated according to the contour rotation angle. The coordinates of the first object contour in the first image mask are moved horizontally and vertically according to the contour movement distance to generate the coordinates of the first object contour in the second image mask. as well as Determine whether the coordinates of the first object outline in the second image mask are within the overlay range, so as to overlay the first object outline at the position of the first object outline at the coordinates of the second image mask.
11. The image augmentation apparatus of claim 10, wherein the processor is further configured to: When it is determined that the coordinates of the first object outline in the second image mask are within the overlay range, the first object outline is overlaid at the position of the coordinates of the first object outline in the second image mask.
12. The image augmentation apparatus of claim 8, wherein the first image mask and the second image mask are binary masks, and wherein the processor is further configured to: Based on multiple classification values corresponding to the first object contour, the second object contour, and the overlapping region, element-wise multiplication is performed on the first object contour, the second object contour, and the overlapping region to overlap the first object contour onto the overlapping region in the second image mask. The multiple classification values are respectively related to the object type of the first object contour, the object type of the second object contour, and the object type of the overlapping region.
13. The image augmentation apparatus according to claim 8, wherein the memory is further configured to store a plurality of preset masks and a plurality of preset images corresponding to the plurality of preset masks, wherein the processor is further configured to: The generative adversarial network model is trained based on the multiple preset masks and the multiple preset images to generate the sample image based on the third image mask. The generative adversarial network model includes a generator model and a discriminator model. The generator model includes multiple encoding layers and multiple decoding layers, and the multiple encoding layers are connected to the multiple decoding layers using a skip connection method.
14. The image augmentation apparatus according to claim 8, wherein the processor is further configured to: The first object contour is superimposed onto the superimposed region in the first image mask according to the amplification parameters to generate a fourth image mask, wherein the fourth image mask includes the first object contour and the superimposed first object contour; and The generative adversarial network model generates another sample image based on the first object contour in the fourth image mask and the superimposed first object contour for machine learning.