Target detection data amplification method and system based on large model
Through the large-model-based object detection data amplification method, high-quality and diverse amplified data sets are generated, which solves the problems of insufficient data diversity and information retention in the existing technology, and significantly improves the performance and application of the object detection model. scope.
Patent Information
- Application Number
- CN202411265251.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-10
- Publication Date
- 2024-12-31
AI Technical Summary
The existing target detection data amplification method has shortcomings in data diversity and information retention. Traditional image transformation methods may lead to information loss. The image quality generated by GAN-based methods is difficult to control and is very different from the real scene, which limits the model in actual scenes. performance in.
Using the object detection data amplification method based on the big model, the scene map and target map are generated by training the basic literary and biographical model, and local areas are extracted in combination with the semantic segmentation model, and the difference area is calculated to determine the target annotation data to generate high-quality and diversity amplified dataset.
It significantly improves the performance and application scope of the object detection model, and the generated image quality and diversity are better. It can more accurately identify and locate targets, automatically generate labeled data, reduce segmentation boundaries, and enhance the model's generalization in complex scenarios. ability to transform.
Smart Images
Figure CN119229234A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of target detection and artificial intelligence technology, and in particular to a method and system for amplifying target detection data based on a large model. Background Art
[0002] In object detection tasks, data augmentation is a crucial step to improve model performance. When the number of training samples is limited, the model often faces the risk of overfitting and cannot generalize to new data sets or scenarios. Data augmentation can effectively improve the robustness and accuracy of the model by increasing the diversity of the training set.
[0003] Current data augmentation methods include traditional image transformation methods such as cropping, mirroring, rotation, etc., as well as data augmentation methods based on Generative Adversarial Network (GAN). These methods can increase the diversity of data sets to a certain extent, but they also have some obvious disadvantages. Traditional image transformation methods may cause information loss or increase noise, especially in target detection tasks, which may affect the recognition accuracy and position accuracy of the target. Although the GAN-based method can generate realistic images, the quality of the generated images is difficult to control in practical applications, and the generated images are different from the real target scenes, resulting in poor performance of the model in real scenes.
[0004] In the field of object detection, the application of data augmentation technology is crucial to the performance of the model. When training data is insufficient, data augmentation can improve the generalization ability and robustness of the model by increasing the diversity of the dataset. Traditional data augmentation methods increase the size and diversity of the training set by changing the appearance of the image through basic image transformations such as random cropping, mirroring, rotation, scaling, etc. However, traditional methods have the risk of information loss or introduction of noise, especially in object detection tasks, which may cause the model to lose accuracy in identifying and locating objects.
[0005] In addition, data augmentation methods based on generative adversarial networks (GANs) have also been widely studied and applied. They train generators and discriminators to generate realistic images. In the field of object detection, researchers have tried to use GANs to generate more complex and realistic image samples to improve the performance and generalization ability of the model. Among the existing technical solutions, the GAN-based methods may be difficult to control the quality and diversity of the generated images due to the complex training and generation process of GANs, and the generated target images are often quite different from the real scenes, which may limit the performance of the model in actual scenes. Summary of the invention
[0006] The embodiments of the present application provide a method and system for target detection data augmentation based on a large model. By utilizing the capabilities of the large model, an innovative target detection data augmentation method is proposed to solve the technical problems existing in data augmentation in terms of data diversity and information retention, and significantly improve the performance and application scope of the target detection model.
[0007] The present application embodiment provides a method for augmenting target detection data based on a large model, comprising:
[0008] Using the initial scene graph set to train the basic text graph large model to obtain a scene graph generation large model, and using the scene graph generation large model to generate a scene graph;
[0009] Merge the initial scene atlas and the generated scene graph, and segment the scene graph using a semantic segmentation model to obtain local areas for target generation;
[0010] Using the local area image information in the target atlas and the scene graph, the basic text image large model is trained to obtain the target image generation large model;
[0011] In the local area for target generation, a large model is generated by using the target map to generate the target, so as to obtain an image with added targets;
[0012] Calculating the difference area between the scene graph and the corresponding image with the added target to determine the target annotation data;
[0013] According to the target annotation data and the corresponding images with added targets, an expanded target detection data set is determined.
[0014] Optionally, the basic text-image model includes a text encoder, an image information generator and an image information decoder;
[0015] The text encoder is used to convert the scene semantic text description of the scene graph into a vector representation, wherein the vector representation contains the semantics and context information of the text description;
[0016] The image information generator is used to output a denoised latent space image information matrix from a randomly initialized image information matrix and a text information matrix output by a text encoder;
[0017] The image information decoder is used to receive the latent space representation generated by the image information generator and decode it into an image output.
[0018] Optionally, merging the initial scene atlas and the generated scene graph to obtain a scene atlas, and using a semantic segmentation model to segment the scene graph to obtain a local area for target generation includes: using a semantic segmentation model of a U-Net structure to semantically segment the scene graph to obtain a local area for target generation, wherein a skip-layer connection is provided between the encoder and the decoder of the U-Net structure to directly connect the feature map of the corresponding layer in the encoder to the corresponding layer in the decoder through the skip-layer connection.
[0019] Optionally, using the local area image information in the target atlas and the scene graph to train the basic text graph large model to obtain the target graph generation large model includes: during the training process, using the image information matrix of the local area of the scene graph to replace the randomly initialized image information matrix as input.
[0020] Optionally, in the local area for target generation, generating a large model using the target map to generate the target, so as to obtain an image with added targets includes:
[0021] Based on the local area used for target generation, a large model is generated using the target graph, and a target is generated on the scene graph to obtain an image with added targets.
[0022] Optionally, performing difference area calculation on the scene graph and the image with the target added to determine the target annotation data includes:
[0023] Convert the target-added image and the scene graph into grayscale images;
[0024] Performing difference calculation on two grayscale images to obtain a difference image;
[0025] Based on the OTSU maximum inter-class variance method, the difference image is binarized to obtain a binary image;
[0026] Filling holes in the binary image to obtain a contour image, wherein the foreground area in the contour image is connected and complete;
[0027] For the contour image, removing abnormal contours through threshold screening to obtain a target contour image;
[0028] The minimum circumscribed positive matrix of the target contour map is calculated to obtain the target annotation data.
[0029] Optionally, according to the target annotation data and the corresponding image to which the target is added, determining the expanded target detection dataset includes:
[0030] Abnormal images are eliminated and the corresponding annotation data of abnormal images are deleted to obtain the expanded target detection dataset.
[0031] An embodiment of the present application also proposes a large model-based target detection data augmentation system, including a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the steps of the large model-based target detection data augmentation method as described above are implemented.
[0032] The method of the embodiment of the present application utilizes the capabilities of a large model and proposes an innovative target detection data augmentation method to solve the current technical problems in data diversity and information retention in data augmentation, and significantly improve the performance and application scope of the target detection model.
[0033] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:
[0035] Figure 1 The basic process diagram of the target detection data augmentation method based on a large model in the embodiment of the present application is shown;
[0036] Figure 2 This is a schematic diagram of a scene graph generation model architecture according to an embodiment of the present application;
[0037] Figure 3 This is a schematic diagram of the target graph generation model architecture of an embodiment of the present application;
[0038] Figure 4 This is a schematic diagram of the U-Net scene graph local area extraction model architecture of an embodiment of the present application;
[0039] Figure 5 A schematic diagram of the process of generating annotation data for an embodiment of the present application. DETAILED DESCRIPTION
[0040] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.
[0041] Existing GAN-based data augmentation methods have some significant shortcomings in target detection tasks:
[0042] (1) Insufficient quality and diversity of generated images: Images generated by traditional GANs may have problems with unstable quality and insufficient diversity, which will affect the generalization ability and accuracy of the model in real scenarios.
[0043] (2) Difficulty in generating target scene graphs: Although GAN can generate realistic scene graphs, the generated images often fail to accurately express the target objects, especially in target detection tasks that require clear target location and boundary information.
[0044] (3) Unable to automatically generate labeled data: The target scene graph generated by GAN still needs to be manually labeled to obtain the labeled data of the target, which greatly limits its application in data augmentation in the field of target detection.
[0045] The purpose of the embodiment of the present application is to provide a method for amplifying target detection data based on a large model, such as Figure 1 As shown, the following steps are included:
[0046] In step S101, the basic language graph large model is trained with the initial scene graph set to obtain a scene graph generation large model, and the scene graph generation large model is used to generate a scene graph. In some specific examples, the scene graph can be generated based on the StableDiffusion language graph large model. Based on the collected scene graph data set, the scene graph generation model is obtained by training the basic Stable Diffusion language graph large model through LoRA (Low-Rank Adaptation). For example, in a specific example, it can be implemented through LoRA, which is a low-rank adapter for a large language model. It reduces the number of trainable parameters of the model and implements the training of the large model without losing the model performance as much as possible.
[0047] In step S102, the initial scene atlas and the generated scene graph are merged to obtain a scene atlas, and the scene graph is segmented using a semantic segmentation model to obtain a local area for target generation. In some specific embodiments, the scene graph is semantically segmented using a semantic segmentation model of a U-Net structure to obtain a local area for target graph generation (i.e., a reasonable image area for target generation).
[0048] In step S103, the target atlas and the local area image information in the scene graph are used to train the basic text graph model to obtain the target graph generation model. In the specific example, based on the collected target graph dataset, the target graph generation model is obtained by training the basic Stable Diffusion text graph model through LoRA.
[0049] In step S104, in the local area for target generation, the target graph generation model is used to generate the target to obtain an image with the target added. After the local area is extracted on the scene graph, the target graph generation model is used to generate the target on the scene graph to obtain an image with the target.
[0050] In step S105, the scene graph and the corresponding image of the added object are subjected to difference area calculation to determine the object annotation data. In some examples, after the object is added to the scene graph, the annotation data of the object is generated by calculating the difference area between the image of the added object and the scene graph.
[0051] In step S106, an expanded target detection data set is determined based on the target annotation data and the corresponding image with the added target.
[0052] The method of the embodiment of the present application utilizes the capabilities of a large model and proposes an innovative target detection data augmentation method to solve the current technical problems in data diversity and information retention in data augmentation, and significantly improve the performance and application scope of the target detection model.
[0053] In some embodiments, Figure 2 As shown, the basic text-image model includes a text encoder, an image information generator and an image information decoder;
[0054] The text encoder is used to convert the scene semantic text description of the scene graph (such as vast grasslands, boundless deserts, bustling towns, etc.) into a vector representation, such as a high-dimensional vector representation, which contains the semantics and context information of the text description. Figure 2 The blue text information encoding matrix in the figure. The present invention uses the CLIP (Contrastive Language-Image Pre-Training) multimodal pre-trained neural network model to encode the scene graph text information, which can link the text description with the image semantic space, so that the model can generate related images according to the text description.
[0055] The image information generator is used to output a denoised latent space image information matrix from a randomly initialized image information matrix and a text information matrix output by a text encoder. The image information generator generates image information by repeatedly iterating multiple steps (Steps) through the noise predictor U-Net model. Each iteration removes some noise and adds more information related to the scene graph until a scene graph without noise is finally generated. This is the reverse image generation process of "Diffusion". The Steps value usually defaults to 50 or 100.
[0056] The image information decoder is used to receive the latent space representation generated by the image information generator, such as a low-dimensional latent space representation, and decode it into an image output, such as obtaining a high-quality image output after decoding.
[0057] In some embodiments, the image information decoder is a U-Net structure decoder. A decoder using the U-Net structure combines a deep convolutional network and skip-layer connections, which can effectively retain the structure and details of the image and generate realistic image results.
[0058] In some embodiments, based on the scene atlas and the generated scene graph, segmentation is performed using a semantic segmentation model to obtain a local area for target generation, including: semantic segmentation of the scene graph is performed using a semantic segmentation model of a U-Net structure to obtain a local area for target generation, wherein a skip-layer connection is provided between the encoder and the decoder of the U-Net structure to directly connect the feature map of the corresponding layer in the encoder to the corresponding layer in the decoder through the skip-layer connection.
[0059] Specifically, the embodiment of the present application uses a semantic segmentation model with a U-Net structure to perform semantic segmentation on the scene graph to obtain a local region of the target graph. Figure 4 As shown in Figure 1, it includes two parts: encoder and decoder. The encoder is used to obtain the context information of the image, and the decoder is used to gradually restore the output of the encoder to the size of the original image.
[0060] More specifically, the encoder is used to extract features from the input scene graph, gradually reducing the size of the feature map while increasing the number of feature maps (depth). It consists of a series of convolutional layers, ReLU (Rectified Linear Unit) activation functions, and max pooling layers. The convolutional layers use a 3x3 convolution kernel with a stride of 1 and padding of 1. Each convolutional layer is followed by a ReLU activation function to increase the nonlinearity of the network. The max pooling layer is used to reduce the size of the image while increasing the depth of the network.
[0061] The decoder structure is opposite to the encoder. It restores the feature map size to the size of the original input image through upsampling (such as deconvolution or transposed convolution) and gradually reduces the depth of the feature map. It consists of a series of upsampling layers, convolution layers, and ReLU activation functions. The upsampling layer uses bilinear interpolation to enlarge the output of the encoder to the same size as the input image. After upsampling, the image details are restored through convolution layers and ReLU activation functions.
[0062] like Figure 4As shown, in the embodiment of the present application, a skip-layer connection is introduced between the encoder and the decoder, that is, the feature map of the corresponding layer in the encoder is directly connected to the corresponding layer in the decoder, so that the decoder can utilize high-level semantic information and detailed spatial information from the encoder to improve the accuracy and details of the segmentation.
[0063] In some embodiments, the basic text graph large model is trained using the local area image information in the target atlas and the scene graph to obtain the target graph generation large model, which includes: during the training process, the image information matrix of the local area of the scene graph is used to replace the randomly initialized image information matrix as input.
[0064] The target graph generates the structure of the large model Figure 3 As shown in the figure, it also includes three parts: text encoder, image information generator and image information decoder. Among them, the text encoder uses the CLIP model to convert the semantic text description of the target image into a high-dimensional vector representation. The principle of the image information generator is basically the same as that of the scene graph generation model. The difference is that the target graph generation model uses the image information matrix of the local area in the scene graph instead of the randomly initialized image information matrix as input. In this way, the generated target image can be seamlessly integrated with the scene graph, reducing obvious segmentation boundaries, thereby improving the quality of the generated image. The image information matrix of the local area of the scene graph is calculated by the encoder of the U-Net structure. The image information decoder decodes the low-dimensional latent space representation generated by the image information generator into a high-quality target image.
[0065] In some embodiments, in the local area for target generation, generating a large model using the target map to generate the target, so as to obtain an image with added targets includes:
[0066] Based on the local area for target generation, a large model is generated using the target graph, and targets are generated based on the scene atlas to obtain an image with added targets, wherein multiple positions are selected in the scene graph, for example, multiple positions are randomly selected to generate multiple targets in one scene graph, thereby increasing the diversity of data augmentation.
[0067] In some embodiments, Figure 5 As shown, the scene graph and the image with the added target are subjected to difference area calculation to determine the target annotation data including:
[0068] The target-added image and the scene graph are converted into grayscale images, for example, the target-added image and the scene graph are converted from color images into single-band grayscale images.
[0069] Performing difference calculation on two grayscale images to obtain a difference image;
[0070] Based on the OTSU maximum inter-class variance method, the difference image is binarized and segmented to obtain a binary image.
[0071] Holes are filled in the binary image to obtain a contour image, in which a foreground area is connected and complete. For example, a morphological closing operation can be used to fill holes in the binary image.
[0072] For the contour image, removing abnormal contours through threshold screening to obtain a target contour image;
[0073] The minimum circumscribed positive matrix of the target contour map is calculated to obtain the target annotation data.
[0074] In some embodiments, determining the expanded target detection data set according to the target annotation data and the corresponding target-added image includes:
[0075] Abnormal images are eliminated, and the corresponding annotation data in the abnormal images are deleted to obtain an expanded target detection data set, for example, abnormal images such as those in which the generated targets are obviously deformed or the generated target positions are unreasonable are eliminated.
[0076] The method of the embodiment of the present application can effectively improve the quality and diversity of generated images: using the StableDiffusion large model for image generation can more accurately understand the semantic information in the image and generate new images that retain important features of the target. Compared with traditional GAN, the quality and diversity of images can be better controlled.
[0077] The method of the embodiment of the present application separates the generation of the scene graph and the generation of the target image. The scene graph is first generated using the large model image generation technology, and then the target is further generated on the generated scene graph. The image generated in this way is closer to the target scene graph in the real world. The target in the image can be seamlessly integrated with the scene graph, reducing obvious segmentation boundaries, improving the quality of the generated image, and helping to train the target detection model to more accurately identify and locate targets in various complex scenes.
[0078] The present invention generates a target on a scene graph, and can accurately locate the position of the target, so that the annotation data of the target can be automatically generated, providing greater convenience for the application of augmented data.
[0079] An embodiment of the present application also proposes a large model-based target detection data augmentation system, including a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the steps of the large model-based target detection data augmentation method as described above are implemented.
[0080] It should be noted that in the various embodiments of the present application, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.
[0081] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.
[0082] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.
[0083] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.
Claims
1. A method for target detection data augmentation based on a large model, characterized in that: include: Using the initial scene graph set to train the basic text graph large model to obtain a scene graph generation large model, and using the scene graph generation large model to generate a scene graph; Merge the initial scene atlas and the generated scene graph, and segment the scene graph using a semantic segmentation model to obtain local areas for target generation; Using the local area image information in the target atlas and the scene graph, the basic text image large model is trained to obtain the target image generation large model; In the local area for target generation, a large model is generated by using the target map to generate the target, so as to obtain an image with added targets; Calculating the difference area between the scene graph and the corresponding image with the target added, so as to determine the target annotation data; According to the target annotation data and the corresponding images with added targets, an expanded target detection data set is determined.
2. The method for target detection data augmentation based on a large model according to claim 1, characterized in that: The basic text-image model includes a text encoder, an image information generator and an image information decoder; The text encoder is used to convert the scene semantic text description of the scene graph into a vector representation, wherein the vector representation contains the semantics and context information of the text description; The image information generator is used to output a denoised latent space image information matrix from a randomly initialized image information matrix and a text information matrix output by a text encoder; The image information decoder is used to receive the latent space representation generated by the image information generator and decode it into an image output.
3. The method for target detection data augmentation based on a large model as claimed in claim 2, characterized in that: The initial scene atlas and the generated scene graph are merged to obtain the scene atlas, and the scene graph is segmented using the semantic segmentation model to obtain the local areas for target generation, including: The scene graph is semantically segmented using a semantic segmentation model of a U-Net structure to obtain a local area for target generation, wherein a skip-layer connection is set between the encoder and the decoder of the U-Net structure to directly connect the feature map of the corresponding layer in the encoder to the corresponding layer in the decoder through the skip-layer connection.
4. The method for target detection data augmentation based on a large model as claimed in claim 2, characterized in that: Using the local area image information in the target atlas and the scene graph, the basic text image model is trained to obtain the target image generation model including: During the training process, the image information matrix of the local area of the scene graph is used as input instead of the randomly initialized image information matrix.
5. The method for target detection data augmentation based on a large model as claimed in claim 2, characterized in that: In the local area for target generation, generating a large model using the target map to generate the target, so as to obtain an image with added targets, comprises: Based on the local area used for target generation, a large model is generated using the target graph, and a target is generated on the scene graph to obtain an image with added targets.
6. The method for target detection data augmentation based on a large model as claimed in claim 5, characterized in that: The difference area calculation is performed on the scene graph and the image with the target added to determine the target annotation data including: Convert the target-added image and the scene graph into grayscale images; Performing difference calculation on two grayscale images to obtain a difference image; Based on the OTSU maximum inter-class variance method, the difference image is binarized to obtain a binary image; Filling holes in the binary image to obtain a contour image, wherein the foreground area in the contour image is connected and complete; For the contour image, removing abnormal contours through threshold screening to obtain a target contour image; The minimum circumscribed positive matrix of the target contour map is calculated to obtain the target annotation data.
7. The method for target detection data augmentation based on a large model as claimed in claim 2, characterized in that: According to the target annotation data and the corresponding added target images, it is determined that the expanded target detection dataset includes: Abnormal images are eliminated and the corresponding annotation data of abnormal images are deleted to obtain the expanded target detection dataset.
8. A target detection data augmentation system based on a large model, characterized in that: It comprises a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the steps of the target detection data amplification method based on a large model as described in any one of claims 1 to 7 are implemented.