Sample generation method, device, system and equipment, medium and program product

Abstract

The invention provides a sample generation method, device and system, equipment, a medium and a program product. A specific embodiment of the sample generation method in the application comprises the steps of detecting and segmenting a target object in an original sample image, and generating a foreground image and a mask image of the target object; performing data enhancement on the foreground image and the mask image to generate an enhanced foreground image and an enhanced mask image; fusing the enhanced foreground image and the background image by using the enhanced mask image to generate an augmented sample image; and labeling a category and a position label for the target object in the augmented sample image to generate an augmented sample. According to the embodiment, on the basis of the original sample image, the foreground image and the mask image of the target object are segmented, after data enhancement is performed on the foreground image and the mask image, the foreground image and the mask image are fused with the background image, and the label is labeled at the same time, so that a new sample is augmented, sample collection is easier, and the number of samples is increased.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a sample generation method, device, system, equipment, medium and program product. Background Art

[0002] The machine learning task of object detection relies on a large number of carefully annotated training samples. These training samples not only provide image data, but also annotate the category and location information of the target object. By learning this annotated information, the machine learning model can identify and locate specific targets in the image, thereby realizing the detection of targets in new images. The diversity, quality, and scale of training samples directly affect the performance and generalization ability of the model. Therefore, high-quality training samples are the key to building an efficient object detection model.

[0003] Currently, when training target detection models, images are mainly obtained from public datasets, proprietary datasets or commercial datasets, and target objects are manually labeled to generate training samples. Summary of the invention

[0004] Various aspects of the present application provide a sample generation method, apparatus, system, device, medium and program product to solve the problem that samples are difficult to collect and are in small quantities.

[0005] In one aspect of the present application, a sample generation method is provided, comprising: detecting and segmenting a target object in an original sample image to generate a foreground image and a mask image of the target object; performing data enhancement on the foreground image and the mask image to generate an enhanced foreground image and an enhanced mask image; using the enhanced mask image, fusing the enhanced foreground image with the background image to generate an augmented sample image; and marking the target object in the augmented sample image with a category and a position label to generate an augmented sample.

[0006] On the other hand, the present application provides a sample generation device, including: a segmentation module, configured to detect and segment the target object in the original sample image, and generate a foreground image and a mask image of the target object; an enhancement module, configured to perform data enhancement on the foreground image and the mask image, and generate an enhanced foreground image and an enhanced mask image; a fusion module, configured to use the enhanced mask image to fuse the enhanced foreground image with the background image, and generate an augmented sample image; and a labeling module, configured to label the target object in the augmented sample image with a category and a position label, and generate an augmented sample.

[0007] Another aspect of the present application provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the sample generation method as described above.

[0008] In another aspect of the present application, a computer-readable storage medium is provided, on which computer program instructions are stored. The computer program instructions can be executed by a processor to implement the sample generation method as described above.

[0009] Another aspect of the present application provides a computer program product, including a computer program, wherein the sample generation method as described above is implemented when the computer program is executed by a processor.

[0010] In the solution provided by the embodiment of the present application, based on the original samples, the foreground image and the mask image of the target object are segmented, and after data enhancement of the foreground image and the mask image, they are fused with the background image and labeled at the same time, thereby expanding new samples, making sample collection easier and increasing the number of samples. Moreover, the new samples are generated based on the original samples and are authentic. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0012] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0013] Figure 1 A processing flow chart of an embodiment of the sample generation method provided by the present application;

[0014] Figure 2 A processing flow chart of another embodiment of the sample generation method provided by the present application;

[0015] Figure 3 A processing flow chart of another embodiment of the sample generation method provided by the present application;

[0016] Figure 4 A processing flow chart of a scenario for implementing the sample generation method of this embodiment;

[0017] Figure 5 A schematic diagram of the structure of an embodiment of a sample generation device provided by the present application;

[0018] Figure 6 A schematic diagram of the structure of a device suitable for implementing the solution in the embodiment of the present application;

[0019] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0021] In a typical configuration of the present application, the terminal and the equipment of the service network each include one or more processors (CPU), input / output interface, network interface and memory.

[0022] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0023] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer program instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0024] The embodiment of the present application provides a sample generation method, which first detects and segments the target object in the original sample image to generate a foreground image and a mask image of the target object; then, data enhancement is performed on the foreground image and the mask image to generate an enhanced foreground image and an enhanced mask image; then, the enhanced foreground image is fused with the background image using the enhanced mask image to generate an augmented sample image; finally, the category and position labels are annotated on the target object in the augmented sample image to generate an augmented sample. Based on the original sample, the sample generation method segments the foreground image and the mask image of the target object, performs data enhancement on the foreground image and the mask image, and then fuses them with the background image, annotates the labels, and thereby augments the new sample, making it easier to collect samples and increasing the number of samples. Moreover, the new sample is generated based on the original sample and is authentic.

[0025] In actual scenarios, the execution subject of the above method can be a user device, or a network device, or a device formed by integrating a user device and a network device through a network, or an application running on the above device. The user device includes but is not limited to various terminal devices such as computers, mobile phones, tablets, smart watches, and bracelets. The network device includes but is not limited to network hosts, single network servers, multiple network server sets, or computer sets based on cloud computing. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing (Cloud Computing), where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.

[0026] Figure 1 The processing flow 100 of an embodiment of the sample generation method provided by the present application is shown, and the method at least includes the following processing steps:

[0027] Step S101 , detecting and segmenting the target object in the original sample image to generate a foreground image and a mask image of the target object.

[0028] In this embodiment, the execution subject of the sample generation method can detect and segment the target object in the original sample image to generate a foreground image and a mask image of the target object.

[0029] The sample set can be a training sample set or a test sample set, including samples used to train or test the target detection model. The sample can be a sample image annotated with a label. The sample image is mainly obtained from a public dataset, a self-owned dataset, or a commercial dataset. The label can be used to record the category and location of the target object in the sample image. The target object can be an object of a specific category, such as a vehicle.

[0030] The original sample image may be a sample image selected from a sample set, for example, any sample image randomly selected from a sample set, or another example, a sample image of a specific category selected from a sample set.

[0031] By detecting and segmenting the target object in the original sample image, a foreground image and a mask image of the target object can be generated. The foreground image can be an image of the area marked by the bounding box of the target object. The mask image can be used to specify the area surrounded by the outline of the target object in the image so as to perform certain specific operations on this area without affecting other parts of the image. For example, the label of the original sample image is marked with the position of the target object. Based on the position of the target object, the bounding box of the target object can be determined. By segmenting the bounding box of the target object from the original sample image, the foreground image of the target object can be obtained. By performing target detection on the foreground image, the outline of the target object can be determined. The mask image of the target object can be obtained by setting the pixel value of the pixel points within the outline to 1 and the pixel value of the pixel points outside the outline to 0.

[0032] Step S102: performing data enhancement on the foreground image and the mask image to generate an enhanced foreground image and an enhanced mask image.

[0033] In this embodiment, the execution subject may perform data enhancement on the foreground image and the mask image to generate an enhanced foreground image and an enhanced mask image.

[0034] Data augmentation can be a technique to artificially expand a dataset by generating more equivalent data from limited data. Data augmentation can include, but is not limited to, flipping, random cropping, color jittering, scale transformation, random rotation, noise perturbation, and Gaussian blurring. These methods can be selectively (alone or in any combination) applied to the foreground image and mask image of the target object to create rich visual changes to increase the diversity of samples.

[0035] Flip: You can flip the image horizontally or vertically to increase the perspective.

[0036] Random cropping: Randomly select an area from the image as a new image to simulate different perspectives and focal lengths.

[0037] Color Dithering: Adjust the brightness, contrast, and saturation of an image to simulate different lighting conditions.

[0038] Scaling: scaling the image to simulate objects at different distances.

[0039] Random rotation: Rotate the angle of the image to simulate observing the target object from different angles.

[0040] Noise perturbation: Add random noise to the image to simulate noise interference in the real world.

[0041] Gaussian Blur: Applies a Gaussian filter to the image, simulating a slight blur.

[0042] Step S103: using the enhanced mask image, fuse the enhanced foreground image with the background image to generate an augmented sample image.

[0043] In this embodiment, the execution subject may use the enhanced mask image to fuse the enhanced foreground image with the background image to generate an augmented sample image.

[0044] The background image can be any image to ensure that the augmented new samples are diverse. The augmented sample image can contain the target object in the original sample image and the scene in the background image, so as to simulate the situation where the target object appears in various scenes.

[0045] The enhanced foreground image can be fused with the background image using the enhanced mask image. For example, the fusion position is determined in the background image. First, the enhanced mask image is superimposed on the fusion position, the pixel points in the background image that are the same as the pixel points with a pixel value of 1 in the enhanced mask image are erased, and then the enhanced foreground image is spliced ​​at the fusion position. The fusion position can be a fixed position (such as the middle position) or a random position. In order to ensure that the generated samples are diverse, the fusion position can be randomly selected each time.

[0046] Step S104: label the target object with a category and a position label in the augmented sample image to generate an augmented sample.

[0047] In this embodiment, the above-mentioned execution subject can mark the category and position label of the target object in the augmented sample image to generate an augmented sample. Among them, the category of the target object marked in the augmented sample image is the same as the category of the target object marked in the original sample image. The position of the target object marked in the augmented sample image is the fusion position. The augmented sample can be a COCO (Common Objects in Context) format label file containing category number and position information, so that it can be compatible with the target detection task to facilitate model training.

[0048] The embodiment of the present application provides a sample generation method, which segments the foreground image and mask image of the target object based on the original sample, performs data enhancement on the foreground image and mask image, fuses them with the background image, and annotates them, thereby expanding new samples, making sample collection easier and increasing the number of samples. Moreover, the new samples are generated based on the original samples and are authentic.

[0049] Figure 2 A processing flow 200 of another embodiment of a sample generation method provided by the present application is shown, and the method at least includes the following processing steps:

[0050] Step S201 : inputting the original sample image and the segmentation hint of the target object into any segmentation model to obtain a foreground image and a mask image.

[0051] In this embodiment, the execution subject of the sample generation method can input the original sample image and the segmentation hint of the target object into SAM (Segment Anything Model) to obtain a foreground image and a mask image.

[0052] The sample set can be a training sample set or a test sample set, including samples used to train or test the target detection model. The sample can be a sample image annotated with a label. The sample image is mainly obtained from a public dataset, a self-owned dataset, or a commercial dataset. The label can be used to record the category and location of the target object in the sample image. The target object can be an object of a specific category, such as a vehicle.

[0053] The original sample image may be a sample image selected from a sample set, for example, any sample image randomly selected from a sample set, or another example, a sample image of a specific category selected from a sample set.

[0054] The original sample image and the segmentation hint of the target object are input into SAM. SAM can automatically detect and segment the target object in the original sample image and generate a foreground image and a mask image. Among them, the segmentation hint of the target object can be used to hint the segmentation of the target object. The foreground image can be an image of the area marked by the bounding box of the target object. The mask image can be used to specify the area surrounded by the outline of the target object in the image so as to perform certain specific operations on this area without affecting other parts of the image. SAM is an image segmentation model based on natural language processing, which can generate effective segmentation masks based on any hint. The SAM architecture mainly consists of three parts: image encoder, hint encoder and mask decoder. A powerful image encoder calculates image embedding, a hint encoder embeds hints, and then combines these two information sources into a lightweight mask decoder to predict the segmentation mask.

[0055] Step S202: Perform the same first data enhancement on the foreground image and the mask image at the same time, and perform the second data enhancement on the foreground image to generate an enhanced foreground image and an enhanced mask image.

[0056] In this embodiment, the execution subject may perform the same first data enhancement on the foreground image and the mask image at the same time, and perform the second data enhancement on the foreground image to generate an enhanced foreground image and an enhanced mask image.

[0057] Data augmentation can be a technique to artificially expand a data set by allowing limited data to generate more equivalent data. The first data augmentation may include, but is not limited to, methods such as flipping, random cropping, scale transformation, and random rotation. These methods can be selectively (alone or in any combination) applied to the foreground image and mask image of the target object at the same time to create rich visual changes to increase the diversity of samples. The second data augmentation may include, but is not limited to, methods such as color jittering, noise perturbation, and Gaussian blurring. These methods can be selectively (alone or in any combination) applied to the foreground image of the target object to create rich visual changes to increase the diversity of samples.

[0058] Flip: You can flip the image horizontally or vertically to increase the perspective.

[0059] Random cropping: Randomly select an area from the image as a new image to simulate different perspectives and focal lengths.

[0060] Color Dithering: Adjust the brightness, contrast, and saturation of an image to simulate different lighting conditions.

[0061] Scaling: scaling the image to simulate objects at different distances.

[0062] Random rotation: Rotate the angle of the image to simulate observing the target object from different angles.

[0063] Noise perturbation: Add random noise to the image to simulate noise interference in the real world.

[0064] Gaussian Blur: Applies a Gaussian filter to the image, simulating a slight blur.

[0065] It should be understood that the pixel values ​​of the pixels of the mask image are only 0 and 1, so there is no need to perform the second data enhancement operation such as color dithering, noise disturbance and Gaussian blur.

[0066] Step S203: randomly select a fusion position in the background image.

[0067] In this embodiment, the execution entity may randomly select a fusion position in the background image.

[0068] The background image can be any image, and the fusion position can be a randomly selected position in the background image to ensure that the augmented new samples are diverse.

[0069] Step S204, first use the enhanced mask image to erase the pixel points at the fusion position, and then splice the enhanced foreground image at the fusion position to generate an augmented sample image.

[0070] In this embodiment, the execution subject may first use the enhanced mask image to erase the pixel points at the fusion position, and then splice the enhanced foreground image at the fusion position to generate an augmented sample image.

[0071] First, the enhanced mask image is superimposed on the fusion position, the pixel points in the background image that are the same as the pixel points with a pixel value of 1 in the enhanced mask image are erased, and then the enhanced foreground image is spliced ​​at the fusion position to generate an augmented sample image. The augmented sample image can include the target object in the original sample image and the scene in the background image, so as to simulate the situation where the target object appears in various scenes.

[0072] Step S205 , marking the category and position labels of the target object in the augmented sample image to generate an augmented sample.

[0073] In this embodiment, the above-mentioned execution subject can mark the category and position label of the target object in the augmented sample image to generate an augmented sample. Among them, the category of the target object marked in the augmented sample image is the same as the category of the target object marked in the original sample image. The position of the target object marked in the augmented sample image is the fusion position. The augmented sample can be a COCO format label file containing the category number and the target box position information, so that it can be compatible with the target detection task to facilitate model training.

[0074] from Figure 2 It can be seen that Figure 1 Compared with the corresponding embodiment, the sample generation method in this embodiment highlights the segmentation step, the enhancement step and the fusion step. Therefore, the scheme described in this embodiment can achieve accurate segmentation of the target object by using SAM. Combined with a variety of data enhancement techniques, rich visual changes can be created to increase the diversity of samples. In addition, by using the enhanced mask image to fuse the enhanced foreground image with any background image and randomly generating the fusion position, the appearance of the target object in various scenes can be simulated.

[0075] Figure 3 A processing flow 300 of another embodiment of the sample generation method provided by the present application is shown, and the method at least includes the following processing steps:

[0076] Step S301, determining the proportion of sample images of at least one category in the sample set.

[0077] In this embodiment, the execution subject of the sample generation method may determine the proportion of sample images of at least one category in the sample set.

[0078] The sample set can be a training sample set or a test sample set, including samples used to train or test the target detection model. The sample can be a sample image annotated with a label. The sample image is mainly obtained from a public dataset, a self-owned dataset, or a commercial dataset. The label can be used to record the category and location of the target object in the sample image. The target object can be an object of a specific category, such as a vehicle.

[0079] By counting the proportion of sample images of various categories in the sample set, the balance of the samples in the sample set can be determined. If the proportion of sample images of various categories is not much different, it means that the samples in the sample set are highly balanced; if the proportion of sample images of various categories is greatly different, it means that the samples in the sample set are poorly balanced. In the target detection task, there are usually some categories of sample images that are difficult to collect and the number is small, which will lead to sample imbalance. This sample imbalance will make the model pay less attention to minority categories during training, which will affect the detection accuracy and recall rate of the model on these categories.

[0080] Step S302 : selecting sample images of a category whose proportion is less than a preset proportion threshold as original sample images.

[0081] In this embodiment, the execution entity may select sample images of a category whose proportion is less than a preset proportion threshold as the original sample images.

[0082] The number of sample images of categories whose proportion is less than the preset proportion threshold is relatively small in the sample set. Selecting sample images of these categories as original sample images for sample augmentation can increase the number of sample images of categories with smaller numbers, thereby making the overall distribution of the sample set more balanced.

[0083] Step S303: Detect and segment the target object in the original sample image to generate a foreground image and a mask image of the target object.

[0084] In this embodiment, the execution subject can detect and segment the target object in the original sample image to generate a foreground image and a mask image of the target object.

[0085] By detecting and segmenting the target object in the original sample image, a foreground image and a mask image of the target object can be generated. The foreground image can be an image of the area marked by the bounding box of the target object. The mask image can be used to specify the area surrounded by the outline of the target object in the image so as to perform certain specific operations on this area without affecting other parts of the image. For example, the label of the original sample image is marked with the position of the target object. Based on the position of the target object, the bounding box of the target object can be determined. By segmenting the bounding box of the target object from the original sample image, the foreground image of the target object can be obtained. By performing target detection on the foreground image, the outline of the target object can be determined. The mask image of the target object can be obtained by setting the pixel value of the pixel points within the outline to 1 and the pixel value of the pixel points outside the outline to 0.

[0086] Step S304: perform data enhancement on the foreground image and the mask image to generate an enhanced foreground image and an enhanced mask image.

[0087] In this embodiment, the execution subject may perform data enhancement on the foreground image and the mask image to generate an enhanced foreground image and an enhanced mask image.

[0088] Data augmentation can be a technique to artificially expand a dataset by generating more equivalent data from limited data. Data augmentation can include, but is not limited to, flipping, random cropping, color jittering, scale transformation, random rotation, noise perturbation, and Gaussian blurring. These methods can be selectively (alone or in any combination) applied to the foreground image and mask image of the target object to create rich visual changes to increase the diversity of samples.

[0089] Flip: You can flip the image horizontally or vertically to increase the perspective.

[0090] Random cropping: Randomly select an area from the image as a new image to simulate different perspectives and focal lengths.

[0091] Color Dithering: Adjust the brightness, contrast, and saturation of an image to simulate different lighting conditions.

[0092] Scaling: scaling the image to simulate objects at different distances.

[0093] Random rotation: Rotate the angle of the image to simulate observing the target object from different angles.

[0094] Noise perturbation: Add random noise to the image to simulate noise interference in the real world.

[0095] Gaussian Blur: Applies a Gaussian filter to the image, simulating a slight blur.

[0096] Step S305: determine the distribution of sample images in the sample set.

[0097] In this embodiment, the execution entity may determine the distribution of sample images in the sample set.

[0098] By counting the distribution of sample images in the sample set, the sample distribution characteristics in the sample set can be determined. For example, by counting the distribution of sample images in the sample set, it can be determined that all sample images in the sample set are road images.

[0099] Step S306: Select images with similar distribution as background images.

[0100] In this embodiment, the execution entity may select images with similar distribution as background images.

[0101] Selecting images with similar distribution to the sample images in the sample set can ensure the consistency of the generated augmented samples with the sample set distribution, thereby generating high-quality and high-availability samples. Since the augmented samples are consistent with the sample set distribution, there is no need to perform data cleaning and any manual intervention on the generated augmented samples before they can be used to train the model. For example, for a sample set in which all sample images are road images, the selected background image is also a road image, and the generated augmented sample image is also a road image.

[0102] Step S307: using the enhanced mask image, fuse the enhanced foreground image with the background image to generate an augmented sample image.

[0103] In this embodiment, the execution subject may use the enhanced mask image to fuse the enhanced foreground image with the background image to generate an augmented sample image.

[0104] The background image can be any image to ensure that the augmented new samples are diverse. The augmented sample image can contain the target object in the original sample image and the scene in the background image, so as to simulate the situation where the target object appears in various scenes.

[0105] The enhanced foreground image can be fused with the background image using the enhanced mask image. For example, the fusion position is determined in the background image. First, the enhanced mask image is superimposed on the fusion position, the pixel points in the background image that are the same as the pixel points with a pixel value of 1 in the enhanced mask image are erased, and then the enhanced foreground image is spliced ​​at the fusion position. The fusion position can be a fixed position (such as the middle position) or a random position. In order to ensure that the generated samples are diverse, the fusion position can be randomly selected each time.

[0106] Step S308: annotate the category and position labels of the target object in the augmented sample image to generate an augmented sample.

[0107] In this embodiment, the above-mentioned execution subject can mark the category and position label of the target object in the augmented sample image to generate an augmented sample. Among them, the category of the target object marked in the augmented sample image is the same as the category of the target object marked in the original sample image. The position of the target object marked in the augmented sample image is the fusion position. The augmented sample can be a COCO format label file containing the category number and the target box position information, so that it can be compatible with the target detection task to facilitate model training.

[0108] Step S309: adding the augmented samples to the sample set.

[0109] In this embodiment, the execution entity may add augmented samples to the sample set to expand the number of samples in the sample set.

[0110] Step S310, using the sample set to perform model training to obtain a target detection model.

[0111] In this embodiment, the above-mentioned execution entity can use the sample set to perform model training to obtain a target detection model.

[0112] For the samples in the sample set (original samples and augmented samples), the sample images in the samples can be used as input, and the labels of the sample images can be used as the expected output to perform supervised training on the machine learning model, thereby obtaining a target detection model. Specifically, first, the sample image is input into the machine learning model to predict the category and position of the target object. Then, based on the predicted category and position of the target object, as well as the category and position of the target object recorded by the label, the loss value can be calculated. Among them, the loss value can be used to characterize the difference between the true value and the predicted value. Finally, based on the loss value, the parameters of the machine learning model are continuously adjusted until the loss value meets the preset target, the model converges, and the target detection model is obtained.

[0113] from Figure 3 It can be seen that Figure 1 Compared with the corresponding embodiment, the sample generation method in this embodiment adds the original sample image selection step, the background image selection step and the model training step. Therefore, the scheme described in this embodiment significantly increases the number of samples of minority categories in the target detection task, improves the sample imbalance problem, and thus improves the overall performance of the model to ensure the detection accuracy and recall rate of the model in all categories. Moreover, the generated augmented samples are consistent with the distribution of the sample set, thereby improving the quality and availability of the augmented samples. In addition, the generated samples are diverse and authentic, which can help the model better learn the features of different categories and help the model better generalize to unknown scenes.

[0114] It should be noted that the sample generation method provided in the present application does not require the use of a GPU (Graphics Processing Unit), and the sample generation can be completed only on a CPU (Central Processing Unit), which has low requirements on hardware equipment and is easy to apply.

[0115] Figure 4 A processing flow 400 of a scenario for implementing the sample generation method of this embodiment is shown, which at least includes the following processing steps:

[0116] Step S401: collect road images, and label the categories and position labels of vehicles in the road images to generate a training sample set.

[0117] Step S402 , counting the proportions of road images containing cars, road images containing excavators, and road images containing buses in the training sample set.

[0118] For example, 50% of the road images contain cars, 40% of the road images contain buses, and 10% of the road images contain excavators, wherein the proportion of road images containing excavators is less than the preset proportion threshold of 20%.

[0119] Step S403: selecting a road image including an excavator.

[0120] Step S404: Use SAM to detect and segment the road image containing the excavator to generate an excavator image and its mask image.

[0121] Step S405 , performing scale transformation and random rotation on the excavator image and its mask image at the same time, and performing color dithering and noise disturbance on the excavator image to generate an enhanced excavator image and its enhanced mask image.

[0122] Step S406: Select a road image as a background image.

[0123] Step S407 , randomly selecting a fusion position in the background image, and using the enhanced mask image to fuse the enhanced excavator image with the background image to generate a new road image containing the excavator.

[0124] Step S408: label the excavator in the new road image containing the excavator, generate a new training sample, and add it to the training sample set.

[0125] Step S409: Perform model training using the training sample set to obtain a vehicle detection model.

[0126] Figure 5A schematic diagram of the structure of an embodiment of a sample generation device provided by the present application is shown. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0127] like Figure 5 As shown, the sample generation device 500 of this embodiment may include: a segmentation module 501, an enhancement module 502, a fusion module 503 and a labeling module 504. The segmentation module 501 is configured to detect and segment the target object in the original sample image, and generate a foreground image and a mask image of the target object; the enhancement module 502 is configured to perform data enhancement on the foreground image and the mask image, and generate an enhanced foreground image and an enhanced mask image; the fusion module 503 is configured to use the enhanced mask image to fuse the enhanced foreground image with the background image, and generate an augmented sample image; the labeling module 504 is configured to label the target object in the augmented sample image with a category and a position label, and generate an augmented sample.

[0128] In this embodiment, in the sample generation device 500, the specific processing of the segmentation module 501, the enhancement module 502, the fusion module 503 and the annotation module 504 and the technical effects thereof can be referred to in Figure 1 The relevant descriptions of steps S101 to S104 in the corresponding embodiment are not repeated here.

[0129] In some optional implementations of this embodiment, the segmentation module 501 is further configured to: input the original sample image and the segmentation hint of the target object into any segmentation model to obtain a foreground image and a mask image.

[0130] In some optional implementations of this embodiment, the enhancement module 502 is further configured to: simultaneously perform the same first data enhancement on the foreground image and the mask image, wherein the first data enhancement includes at least one of the following: flipping, random cropping, scale transformation, and random rotation.

[0131] In some optional implementations of this embodiment, the enhancement module 502 is further configured to: perform a second data enhancement on the foreground image, wherein the second data enhancement includes at least one of the following: color jitter, noise disturbance, and Gaussian blur.

[0132] In some optional implementations of this embodiment, the fusion module 503 is further configured to: randomly select a fusion position in the background image; first use the enhanced mask image to erase the pixels at the fusion position, and then splice the enhanced foreground image at the fusion position to generate an augmented sample image.

[0133] In some optional implementations of this embodiment, the sample generation device 500 also includes: a first determination module, configured to determine the proportion of sample images of at least one category in the sample set; a first selection module, configured to select sample images of a category whose proportion is less than a preset proportion threshold as the original sample image.

[0134] In some optional implementations of this embodiment, the sample generation device 500 further includes: a second determination module configured to determine the distribution of sample images in the sample set; and a second selection module configured to select images with similar distribution as background images.

[0135] In some optional implementations of this embodiment, the sample generation device 500 further includes: an adding module configured to add the augmented sample to the sample set; and a training module configured to perform model training using the sample set to obtain a target detection model.

[0136] Based on the same inventive concept, an electronic device is also provided in an embodiment of the present application, and the method corresponding to the electronic device may be the sample generation method in the aforementioned embodiment, and its principle of solving the problem is similar to that of the method. The electronic device provided in an embodiment of the present application includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the methods and / or technical solutions of the aforementioned multiple embodiments of the present application.

[0137] The electronic device may be a user device, or a device formed by integrating a user device and a network device through a network, or may be an application running on the above device. The user device includes but is not limited to various terminal devices such as computers, mobile phones, tablet computers, smart watches, and bracelets. The network device includes but is not limited to network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets, which can be used to implement some processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing, where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.

[0138] Figure 6The structure of a device suitable for implementing the method and / or technical solution in the embodiment of the present application is shown, and the device 600 includes a central processing unit (CPU, Central Processing Unit) 601, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM, Read Only Memory) 602 or the program loaded from the storage part 608 to the random access memory (RAM, Random Access Memory) 603. In RAM 603, various programs and data required for system operation are also stored. CPU 601, ROM 602 and RAM 603 are connected to each other through bus 604. Input / output (I / O, Input / Output) interface 605 is also connected to bus 604.

[0139] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, a touch screen, a microphone, an infrared sensor, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), an LED display, an OLED display, etc., and a speaker, etc.; a storage section 608 including one or more computer-readable media such as a hard disk, an optical disk, a magnetic disk, a semiconductor memory, etc.; and a communication section 609 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet.

[0140] In particular, the methods and / or embodiments in the embodiments of the present application may be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. When the computer program is executed by the central processing unit (CPU) 601, the above functions defined in the method of the present application are executed.

[0141] Another embodiment of the present application further provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of the present application described above.

[0142] Specifically, the present embodiment may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device, or device.

[0143] Computer readable signal media may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than a computer readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0144] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0145] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0146] The flow chart or block diagram in the accompanying drawings shows the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0147] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0148] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or page components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0149] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0150] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0151] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0152] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

[0153] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in a device claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

Claims

1. A sample generation method, comprising: Detecting and segmenting the target object in the original sample image to generate a foreground image and a mask image of the target object; Performing data enhancement on the foreground image and the mask image to generate an enhanced foreground image and an enhanced mask image; Using the enhanced mask image, the enhanced foreground image is fused with the background image to generate an augmented sample image; The target object is labeled with a category and a position label in the augmented sample image to generate an augmented sample.

2. The method according to claim 1, wherein: The detecting and segmenting of the target object in the original sample image to generate a foreground image and a mask image of the target object includes: The original sample image and the segmentation hint of the target object are input into any segmentation model to obtain the foreground image and the mask image.

3. The method according to claim 1, wherein: The performing data enhancement on the foreground image and the mask image comprises: The same first data enhancement is performed on the foreground image and the mask image simultaneously, wherein the first data enhancement includes at least one of the following: flipping, random cropping, scale transformation, and random rotation.

4. The method according to claim 3, wherein: The performing data enhancement on the foreground image and the mask image comprises: Performing second data enhancement on the foreground image, wherein the second data enhancement includes at least one of the following: color jitter, noise disturbance, and Gaussian blur.

5. The method according to claim 1, wherein: The step of using the enhanced mask image to fuse the enhanced foreground image with the background image to generate an augmented sample image includes: Randomly selecting a fusion position in the background image; The pixel points at the fusion position are first erased using the enhanced mask image, and then the enhanced foreground image is spliced ​​at the fusion position to generate the augmented sample image.

6. The method according to any one of claims 1 to 5, wherein: The method further comprises: Determine the proportion of sample images of at least one category in the sample set; A sample image of a category whose proportion is less than a preset proportion threshold is selected as the original sample image.

7. The method according to claim 6, wherein: The method further comprises: Determining the distribution of sample images in the sample set; An image with similar distribution is selected as the background image.

8. The method according to claim 6, wherein: The method further comprises: Adding the augmented sample to the sample set; The sample set is used to perform model training to obtain a target detection model.

9. A sample generating device, comprising: A segmentation module is configured to detect and segment the target object in the original sample image, and generate a foreground image and a mask image of the target object; an enhancement module, configured to perform data enhancement on the foreground image and the mask image to generate an enhanced foreground image and an enhanced mask image; a fusion module configured to fuse the enhanced foreground image with the background image using the enhanced mask image to generate an augmented sample image; The labeling module is configured to label the target object with a category and a position label in the augmented sample image to generate an augmented sample.

10. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

11. A computer readable medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images