Data augmentation methods and systems
By generating new object images through cropping, clustering, and generative AI models, and combining image synthesis and style transfer, the problem of insufficient data in new fields is solved, and the training efficiency and accuracy of machine learning models are improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- IND TECH RES INST
- Filing Date
- 2024-12-11
- Publication Date
- 2026-05-26
AI Technical Summary
In machine learning model training, there is a lack of training data in new fields. Existing technologies cannot quickly obtain enough training data in a short period of time, resulting in low model training efficiency and the inability of old data features to be directly applied to new situations.
By utilizing a large amount of old field data, cropping and grouping object images, using a generative AI model to generate new object images, and combining image synthesis and style transfer, the training data of the new field is expanded.
It enables the generation of a large amount of training data that meets the requirements of new fields in a short period of time, improving the efficiency and accuracy of model training and solving the problem of insufficient data.
Smart Images
Figure CN122090193A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a data augmentation method and system, and particularly to a data augmentation method and system based on generative models for object augmentation and image synthesis. Background Technology
[0002] Training a model typically requires a large amount of data to achieve sufficiently accurate classification or prediction results. However, collecting such data is time-consuming and labor-intensive. In situations where there is an urgent need to analyze production lines using artificial intelligence, it is difficult to wait until a sufficient amount of data is collected before starting training and implementing AI in practical applications. Furthermore, even if one wanted to use a large amount of previously collected data for current model training, the features in that old data often do not meet the requirements of the new context, thus rendering it unusable and necessitating the collection of a large amount of new data. Therefore, how to obtain a large amount of training data in a short period of time that can be used to train models for new application scenarios is a crucial issue that needs to be addressed. Summary of the Invention
[0003] This invention provides a data augmentation method and system that can augment new field data based on a large amount of old field data when there is a need to train a machine learning model for a new field but the training data for the new field is insufficient.
[0004] This invention proposes a data augmentation method, comprising: acquiring a first field image captured by a first field and a second field image captured by a second field, wherein the number of first field images is greater than the number of second field images; cropping a first object image and a second object image from the first field image and the second field image respectively; training an object generation model using the first object image and the second object image; generating multiple new object images using the object generation model; merging the multiple new object images into the second field image to form a new object field image; and training an object identification model using the second field image and the new object field image.
[0005] In one embodiment, the data augmentation method further includes: performing a style transformation on the new object field image to create a style-transformed new object field image; and training an object identification model using the second field image, the new object field image, and the style-transformed new object field image.
[0006] In one embodiment, the data augmentation method further includes: grouping the first object image and the second object image into multiple groups; training multiple object generation models corresponding to the multiple groups using the first object image and the second object image; and generating multiple new object images corresponding to the multiple groups using the multiple object generation models.
[0007] In one embodiment, the data augmentation method further includes: obtaining object location information of the second object image in the second field image; and synthesizing multiple new object images into the second field image to form a new object field image based on the object location information.
[0008] In one embodiment, the object generation model is trained using a text command, a first object image, and a second object image.
[0009] The present invention also proposes a data augmentation system, comprising: an image database for storing a first field image captured by a first field and a second field image captured by a second field, wherein the number of first field images is greater than the number of second field images; a first image processing server for obtaining the first field image and the second field image from the image database, the first image processing server comprising: an image cropping module for cropping a first object image and a second object image from the first field image and the second field image, respectively; and a data augmentation server for receiving data from the first image processing server. A data augmentation server includes: a first object image and a second object image; an object generation module for training an object generation model using the first object image and the second object image, and generating multiple new object images using the object generation model; a second image processing server for receiving multiple new object images from the data augmentation server; the second image processing server includes: an image synthesis module for synthesizing the multiple new object images into a second field image to form a new object field image; and an object identification server for training an object identification model using the second field image and the new object field images.
[0010] In one embodiment, the second image processing server further includes: a style conversion module for converting a new object field image into a style-converted new object field image; wherein the object identification server trains an object identification model using the second field image, the new object field image, and the style-converted new object field image.
[0011] In one embodiment, the data augmentation server further includes: an object clustering module for clustering the first object image and the second object into multiple clusters, and an object generation module for training multiple object generation models corresponding to the multiple clusters using the first object image and the second object; and multiple object generation models for generating multiple new object images corresponding to the multiple clusters.
[0012] In one embodiment, the first image processing server further includes: an object positioning module for obtaining object position information of the second object image in the second field image; wherein the image synthesis module synthesizes multiple new object images into the second field image data to form a new object field image based on the object position information.
[0013] In one embodiment, the object generation model is trained using a text command, a first object image, and a second object image.
[0014] Therefore, the data augmentation method and system of the present invention can achieve the effect of augmenting training data in a new field by generating new objects and synthesizing them with images of a new field, and further performing image style conversion. Attached Figure Description
[0015] To gain a more complete understanding of the embodiments and their advantages, reference is now made to the following description taken in conjunction with the accompanying drawings, wherein:
[0016] Figure 1 This is a schematic diagram of a data augmentation system according to an embodiment of the present invention;
[0017] Figure 2 A flowchart of a data amplification method according to an embodiment of the present invention; and
[0018] Figure 3 This is a schematic diagram illustrating image processing using a data augmentation method in one embodiment of the present invention.
[0019] [Explanation of Symbols in the Attached Drawings]
[0020] 100: Data Augmentation System
[0021] 110: First Image Processing Server
[0022] 111: Image Cropping Module
[0023] 112: Object positioning module
[0024] 120: Data Augmentation Server
[0025] 121: Object Grouping Module
[0026] 122: Object Generation Module
[0027] 1221: First Object Generation Model
[0028] 1222: Second Object Generation Model
[0029] 130: Second Image Processing Server
[0030] 131: Image Compositing Module
[0031] 132: Style Transfer Module
[0032] 140: Object Identification Server
[0033] 150: Image Database
[0034] 200: Data Augmentation Methods
[0035] S201~S210: Steps
[0036] P1~P7: objects
[0037] M11: First Field Image
[0038] M21: Second Field Image
[0039] M12: First Object Image
[0040] M22: Second Object Image
[0041] G11: Images of the First Group of Objects
[0042] G21: Images of the Second Group of Objects
[0043] G12: First New Object Image
[0044] G22: Second New Object Image
[0045] M3: New Object Field Image
[0046] M4: Style Transfer New Objects and Field Images Detailed Implementation
[0047] The embodiments of the present invention will now be discussed in detail. However, it will be understood that the embodiments provide many applicable concepts that can be implemented in a wide variety of specific contexts. The discussed and disclosed embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.
[0048] Figure 1 This is a schematic diagram of a data augmentation system 100 according to an embodiment of the present invention. In one embodiment, the data augmentation system 100 includes a first image processing server 110, a data augmentation server 120, a second image processing server 130, an object identification server 140, and an image database 150. The first image processing server 110 is connected to the data augmentation server 120, the second image processing server 130 is connected to the data augmentation server 120, and the object identification server 140 is connected to the second image processing server 130. The first image processing server 110, the data augmentation server 120, the second image processing server 130, and the object identification server 140 are all connected to the image database 150. The first image processing server 110 includes an image cropping module 111 and an object positioning module 112; the data augmentation server 120 includes an object grouping module 121 and an object generation module 122; and the second image processing server 130 includes an image compositing module 131 and a style conversion module 132.
[0049] In one embodiment, the image database 150 stores a first field image and a second field image. The first field image may be a large amount of image data of the first field taken over a long period of time, or a dataset used in the past for training models related to the first field. The second field image is taken or collected in another field that is different from the first field. It may be a completely different field, a field with the same configuration but different lighting color, or a field in the same location but with different decoration configuration, etc. There are no particular restrictions on the difference between the first field and the second field.
[0050] However, the same detection target must be included in both the first and second fields. For example, if the detection target is whether a worker is wearing a helmet, then both the first and second field images must include / not include workers wearing helmets; or if the target is whether a vehicle is equipped with snow chains, then both the first and second field images must include / not include vehicles equipped with snow chains. Furthermore, in this embodiment, the second field is a field that was started or newly established later, therefore the number of images in the second field is less than that in the first field.
[0051] After receiving a first field image and a second field image from the image database 150, the first image processing server 110 uses the image cropping module 111 to crop out the object targets in the first field image and the second field image. The object cropping method can use an object detection model to detect the object targets in the first field image and the second field image and crop them. The object detection model and cropping method used are not particularly limited in this invention as long as they can select and crop specific targets, such as people or objects, from the first field image and the second field image according to instructions.
[0052] In addition to cropping the target object, the first image processing server 110 can also locate the target object in the second field image using the object positioning module 112. The positioning method can use the object detection model or any other method to obtain the object position information such as the X and Y positioning coordinates of the target object in the second field image, which can be used as a position reference for subsequent image synthesis using the image synthesis module 131.
[0053] After cropping the target object, first object images and second object images cropped from the first field image and the second field image are obtained. Since these object images correspond to different object states, the object grouping module 121 can be used to group these objects. Object grouping can be done manually or using common image classification models, such as convolutional neural networks (CNNs). After grouping, the object images are no longer divided into first object images and second object images as originally, but are divided into different groups according to the state of the objects in the object images, such as whether there are workers wearing safety helmets or whether there are vehicles with snow chains. Then, the object generation module 122 can be used to train object generation models corresponding to each group using the object images of each group.
[0054] The object generation model is trained using object images from each group and used to generate new object images corresponding to each group. This object generation model can be a generative AI model that generates images from text, such as the Stable Diffusion Low-Rank Adaptation (Stable Diffusion LoRA) model and the Stable Diffusion XL Low-Rank Adaptation (Stable Diffusion XL LoRA) model based on Stable Diffusion image generation technology. This model generates images that meet the requirements based on images and text descriptions in the training data. Therefore, in this embodiment, the grouped object images are combined with text instructions, such as descriptive text describing the image to be generated, to generate new object images that conform to this group. Since this generative AI model can be trained with both text and images, in another embodiment, it is not necessary to specifically group object images using the object grouping module 121 and manually input related text. Instead, the generative AI model that generates text descriptions from images can be used directly to generate descriptive text corresponding to the object images as text instructions. The object images and text instructions are then used as model inputs to train the object generation model so that the object generation model can generate new object images that meet the requirements.
[0055] After the object generation module 122 generates new object images for each group, the second image processing server 130 composites these new object images with the second field image. When the second field is an environment with changing lighting, style conversion can be performed to generate data of different styles to further increase the data volume. If the second field is a stable lighting environment with low environmental influence, or a controlled environment, style conversion can be omitted to increase the data volume. Image compositing is performed by the image compositing module 131. The image compositing module 131 refers to the object position information obtained from the second field image by the object positioning module 112 to determine the location and distribution area of objects in the second field, and then composites the new object images into the second field image based on the object position information.
[0056] The compositing method allows for the random pasting of numerous new object images from each group into reasonable locations where objects might appear in the second field image. Furthermore, the same second field image can be combined with different new object images to generate multiple new object field images based on the same second field image. For example, in a second field image that already contains one object from group A and one object from group B, pasting in one new object image from group A and one new object image from group B results in a new object field image containing two objects from group A and two objects from group B. Similarly, the same second field image can be further pasted with one new object image from group A and two new object images from group B to generate another new object field image containing two objects from group A and three objects from group B. Furthermore, in one embodiment, the objects can be replaced. For example, in a second field image that already contains one group A object and one group B object, a new group A object image can be pasted onto the group B object to generate a new object field image containing two group A objects. Here, the object images contained in the new object field image are the same as before the pasting, but a new image sample is obtained through replacement. Therefore, even if the number of original second field images is limited, this random and diverse pasting or replacement method can be used to generate a large number of new object field images corresponding to the second field.
[0057] Furthermore, considering that if the second field is an outdoor location or a venue where lighting needs to be adjusted frequently, the data must also include data corresponding to different light tones. Therefore, the style transfer module 132 can be used to further adjust the brightness and tone of the new object field image to increase data corresponding to different weather conditions, morning and evening light, or lighting adjustments. The algorithm used for style transfer can be, for example, a Context-Aware Pyramid Vision Transformer Network (CAP-VSTNet), a Style Shot-based Network (StyleShot), an Adaptive Attention Network (AdaAttN), a Style Identity Network (StyleID), etc., as long as it can be used to adjust the image, the present invention is not limited to these.
[0058] After image processing by the second image processing server 130, a large amount of new object field data corresponding to the second field and additional style-transformed new object field data are obtained as augmented data for the second field image. These data, along with the second field image, are used as training data to train the object identification model by the object identification server 140. The object identification model can be any classification or prediction model. For example, after training, it can be used to instantly identify the grouping status of target objects in the second field or predict upcoming events based on the second field. It can also be combined with an alarm system or event analysis system. The object identification model can be, for example, Faster Region-based Convolutional Neural Network (Faster R-CNN), RetinaNet, You Only LookOnce v4 (YOLOv4), You Only Look Once Version v7 (YOLOv7), CenterNet, also known as Objects as Points, etc. This invention does not impose any particular restrictions on the object identification model.
[0059] Furthermore, the new object images generated by the data augmentation server 120 can be stored in the image database 150. If a third field requires the detection of the same target object in the future, these new object images can be used as training data for the object generation model to produce new object images that meet the requirements of the third field. Alternatively, if the object differences are not significant, they can be directly used for synthesis with the third field images. The new object field data and style-transformed new object field data generated by the second image processing server 130 can also be stored in the image database 150. They can be retrieved at any time when the subsequent object identification server 140 has a new model training requirement corresponding to the second field.
[0060] Figure 2 This is a flowchart of a data amplification method 200 according to an embodiment of the present invention. Figure 3 This is a schematic diagram illustrating image processing using data augmentation method 200 in one embodiment of the present invention. The following refers to... Figure 1 and Figure 2 The data amplification method 200 is explained in detail by taking the detection target as a worker wearing / not wearing protective clothing.
[0061] First, steps S201 and S202 are performed, obtaining a first field image M11 and a second field image M21 from the image database 150. The first field image M11 contains more objects than the second field image M21. The first field image M11 contains objects P1 to P3, and the second field image M21 contains objects P4 to P5. Objects P1, P2, and P4 represent workers not wearing protective clothing, while objects P3 and P5 represent workers wearing protective clothing. Next, steps S203 and S204 are performed by the image cropping module 111, cropping a first object image M12 and a second object image M22 corresponding to objects P1 to P3 and P4 to P5 from the first field image M11 and the second field image M21, respectively.
[0062] Next, in step S205, the first object image M12 and the second object image M22 are grouped using the object grouping module 121. In this embodiment, the grouping is based on whether the worker objects P1~P5 in the first object image M12 and the second object image M22 are wearing protective clothing. These images are then divided into a first group of object images G11 and a second group of object images G21, respectively, corresponding to workers not wearing protective clothing and workers wearing protective clothing. Next, in step S206, the object generation module 122 trains object generation models corresponding to each group. Specifically, the first group of object images G11 is used to train the first object generation model 1221 in the object generation module 122, and the second group of object images G21 is used to train the second object generation model 1222 in the object generation module 122. This establishes a first object generation model 1221 for generating images of workers not wearing protective clothing and a second object generation model 1222 for generating images of workers wearing protective clothing. In addition, if the casual clothes worn by staff who are not wearing protective clothing are too varied or difficult to generate reasonable images, another object generation model can be created for the casual clothes.
[0063] After the establishment of the first object generation model 1221 and the second object generation model 1222 is completed, step S207 is performed to generate the first new object image G12 and the second new object image G22 of each group using the first object generation model 1221 and the second object generation model 1222. That is, the first new object image G12 containing the image of the staff not wearing protective clothing and the second new object image G22 containing the image of the staff wearing protective clothing are generated by these models respectively.
[0064] At this stage, a large number of new object images have been generated by the object generation model and can be stored in the image database 150 for use when new images need to be synthesized later. Alternatively, step S208 can be directly performed, where the image synthesis module 131 synthesizes the new object images into the second field image to form a new object field image. In this embodiment, the image synthesis module 131 uses the object position information of objects P4~P5 obtained from the second field image M21 by the object positioning module 112 to synthesize the first new object image G12 and the second new object image G22 into the second field image M21 to form a new object field image M3. Figure 3 As shown, in the new object field image M3, in addition to objects P4 and P5 that were already in the second field image M21, there are also objects P6 added by pasting in the first new object image G12 and P7 added by pasting in the second new object image G22. After pasting the first new object image G12 and the second new object image G22 into the second field image M21 using various composite combinations, a new object field image M3 with a larger number of objects than the original images can be obtained.
[0065] Since the second field is outdoors in this embodiment, and there are changes in light tone caused by weather and sunlight, step S209 is performed, where the style conversion module 132 performs style conversion on the new object field image M3, such as... Figure 3 The style-transferred new object field image M4 is then processed. Finally, step S210 is performed, in which the object identification server 140 trains the object identification model using the second field image M21, the new object field image M3, and the style-transferred new object field image M4.
[0066] To verify the effectiveness of the data obtained by data augmentation method 200 in training the object recognition model, an experiment was conducted to compare the mean average precision (mAP) of the object recognition model under the augmentation data of object generation and style transfer.
[0067] Table 1
[0068]
[0069] The experimental conditions are shown in Table 1. In this experiment, there are 115,000 first-field images M11, and 10,900 and 3,600 second-field images M21 corresponding to two different fields, respectively. The second-field images M21 of the two different fields are augmented using data augmentation method 200 and used to train the object recognition model. The generation of new object field images M3 is divided into two methods: cropping objects P1~P5 from the first-field images M11 and the second-field images M21 and compositing them with the second-field images M21, and generating a first new object image G12 and a second new object image G22 using the object generation model and compositing them with the second-field images M21, resulting in a total of 5,000 new object field images M3. Furthermore, style-transferred new object field images M4 are generated with and without style transfer, resulting in a total of 5,000 style-transferred new object field images M4.
[0070] Table 2
[0071]
[0072] The experimental results are shown in Table 2. The model trained using data augmentation method 200 helps to improve the average accuracy mAP of the object identification model.
[0073] The embodiments of the present invention have been described above. However, these embodiments are merely for illustrative purposes and are not intended to limit the scope of the present invention. Any person skilled in the art can make various substitutions and modifications without departing from the spirit and scope of the present invention, and all such substitutions and modifications should fall within the scope of the present invention.
Claims
1. A data augmentation method, characterized in that, Include: Acquire a first field image captured by a first field and a second field image captured by a second field, wherein the number of first field images is greater than the number of second field images; A first object image and a second object image are respectively cropped from the first field image and the second field image; An object generation model is trained using the first object image and the second object image; Multiple new object images are generated using this object generation model; The multiple new object images are composited into the second field image to form a new object field image; and An object identification model is trained using the second field image and the new object field image.
2. The data augmentation method according to claim 1, characterized in that, Also includes: The new object field image is style-transformed into a new object field image; and The object identification model is trained using the second field image, the new object field image, and the style-transformed new object field image.
3. The data augmentation method according to claim 1, characterized in that, Also includes: The first object image and the second object image are grouped into multiple groups; Multiple object generation models corresponding to the multi-group are trained using the first object image and the second object image; as well as Multiple object generation models are used to generate multiple new object images corresponding to the multiple groups.
4. The data augmentation method according to claim 1, characterized in that, Also includes: Obtain the position information of an object in the second field image of the second object image; and Based on the object's location information, the images of the multiple new objects are combined into the second field image to form the new object field image.
5. The data augmentation method according to claim 1, characterized in that, The object generation model is trained using a text command, the first object image, and the second object image.
6. A data augmentation system, characterized in that, Include: An image database is used to store images of a first field taken from a first field and images of a second field taken from a second field, wherein the number of images of the first field is greater than the number of images of the second field. A first image processing server obtains the first field image and the second field image from the image database. The first image processing server includes: An image cropping module is used to crop a first object image and a second object image from the first field image and the second field image, respectively. A data augmentation server receives the first object image and the second object image from the first image processing server. The data augmentation server includes: An object generation module is used to train an object generation model with the first object image and the second object image, and to generate multiple new object images with the object generation model. A second image processing server receives the plurality of new object images from the data augmentation server, the second image processing server comprising: An image compositing module is used to composite the multiple new object images onto the second field image to form a new object field image; and An object identification server is used to train an object identification model using the second field image and the new object field image.
7. The data augmentation system according to claim 6, characterized in that, The second image processing server also includes: A style conversion module is used to convert the new object field image into a style-converted new object field image. The object identification server trains the object identification model using the second field image, the new object field image, and the style-transformed new object field image.
8. The data augmentation system according to claim 6, characterized in that, The data augmentation server also includes: An object grouping module is used to group the first object image and the second object into multiple groups. Furthermore, the object generation module trains multiple object generation models corresponding to the multi-group using the first object image and the second object; as well as Multiple object generation models generate multiple new object images corresponding to the multiple groups.
9. The data augmentation system according to claim 6, characterized in that, The first image processing server also includes: An object positioning module is used to obtain the object position information of the second object image in the second field image; The image compositing module combines multiple new object images into the second field image data based on the object's location information to form the new object field image.
10. The data augmentation system according to claim 6, characterized in that, The object generation model is trained using a text command, the first object image, and the second object image.