Data processing method and device, equipment and storage medium

Abstract

Embodiments of the present disclosure provide a data processing method, device and equipment, and a storage medium. The method comprises: performing fusion processing on a first image set and a second image set to obtain a third image set, wherein the first image set is an initial training image set of a target detection model, and the second image set is used to increase the number of detection targets on images in the first image set; performing target detection and segmentation on images in the third image set to obtain an enhanced source image; and performing image synthesis processing on the enhanced source image and images in the first image set to obtain a fourth image set, wherein the number of detection targets on images in the fourth image set is greater than the number of detection targets on images in the first image set. Thus, other image sets are used to increase the number of detection targets on images in the training image set of the target detection model, i.e. the number of samples in the training data set is increased, and the balance of the number of positive and negative samples in the training image set is improved.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of computers, and more particularly to a data processing method, apparatus, device, and storage medium. Background Technology

[0002] With the development of technologies such as image processing and deep learning, target detection models can be used to achieve multi-target detection in surveillance images in monitoring scenarios.

[0003] Training object detection models requires both positive and negative samples. However, due to limitations in sample collection capabilities and practical considerations, a severe imbalance between positive and negative samples can easily occur. For example, during the training of a flame detection model, the number of samples without flames is usually far greater than the number of samples with flames.

[0004] An extreme imbalance in the number of positive and negative samples will severely affect the generalization ability of object detection models. Therefore, how to effectively improve the balance of the number of positive and negative samples has become an urgent problem to be solved. Summary of the Invention

[0005] Embodiments of this disclosure provide a data processing method, apparatus, device, and storage medium to address the problem of an imbalance in the number of positive and negative samples in a target detection model.

[0006] In a first aspect, embodiments of this disclosure provide a data processing method, comprising:

[0007] The first image set and the second image set are fused to obtain the third image set, wherein the first image set is the initial training image set of the target detection model, and the second image set is used to increase the number of detected targets on the images in the first image set;

[0008] Target detection and segmentation are performed on the images in the third image set to obtain the enhanced source image;

[0009] The enhanced source image and the images in the first image set are combined to obtain a fourth image set, which is the final training image set of the target detection model. The number of detected targets in the images in the fourth image set is greater than the number of detected targets in the images in the first image set.

[0010] In one possible implementation, the fusion process of the first image set and the second image set to obtain the third image set includes:

[0011] The first image set and the second image set are integrated to obtain an integrated image set;

[0012] The images in the integrated image set are filtered to obtain the third image set.

[0013] In one possible implementation, the images in the second image set and the first image set contain bounding boxes and label categories for the detected targets. The step of filtering the images in the integrated image set to obtain the third image set includes:

[0014] In the integrated image set, images with the annotation category being the target category and / or with complete annotation boxes are selected, where the target category is the category of the detected target on the image in the first image set;

[0015] The third image set is obtained based on the selected images.

[0016] In one possible implementation, the step of performing target detection and segmentation on the images in the third image set to obtain the enhanced source image includes:

[0017] In the target category, a first category is determined that appears in the second image set, the target category being the category of the detected target on the images in the first image set;

[0018] On the images in the third image set, the detected targets belonging to the first category are segmented to obtain a first segmented image, wherein the enhanced source image includes the first segmented image.

[0019] In one possible implementation, the step of performing instance segmentation on the detected targets belonging to the first category in the images of the third image set to obtain a first segmented image includes:

[0020] Using an image segmentation model pre-trained on the second image set, instance segmentation is performed on the images in the third image set for the detected targets belonging to the first category, resulting in a first segmented image.

[0021] In one possible implementation, the step of performing target detection and segmentation on the images in the third image set to obtain the enhanced source image includes:

[0022] In the target category, a second category that does not appear in the second image set is determined, where the target category is the category of the detected target on the image in the first image set;

[0023] On the images in the third image set, salient target detection is performed on the detection targets belonging to the second category to obtain a second segmentation image, wherein the enhanced source image includes the second segmentation image.

[0024] In one possible implementation, the image compositing process of the enhanced source image and the images in the first image set includes:

[0025] In the first image set, the target image to be processed is determined;

[0026] In the enhanced source image, the target enhanced source image is determined;

[0027] Determine the target image location corresponding to the target enhancement source image;

[0028] Determine the target image size corresponding to the target enhancement source image;

[0029] According to the target image location and the target image size, the image content in the corresponding area of ​​the target image is replaced with the target enhancement source image.

[0030] In one possible implementation, determining the target enhancement source image in the enhancement source image includes:

[0031] For the target image, the number of enhancement source images is randomly determined;

[0032] The target enhancement source image is randomly determined from the enhancement source images according to the number of enhancement source images.

[0033] In one possible implementation, the target image location includes the target height location and the target width location corresponding to the target enhancement source image, and determining the target image location corresponding to the target enhancement source image includes:

[0034] The target height position is determined based on the normal distribution;

[0035] The target width position is randomly determined.

[0036] In one possible implementation, determining the target height position based on a normal distribution includes:

[0037] Determine the normal distribution function based on the height, width, and / or area of ​​the target image;

[0038] The target height position is randomly determined based on the normal distribution function.

[0039] In one possible implementation, determining the target image size corresponding to the target enhancement source image includes:

[0040] The size of the target image is determined based on the relationship between the height position of the target enhancement source image on the original image to which the target enhancement source image belongs and the height of the target image.

[0041] In one possible implementation, before determining the target image size based on the relationship between the height position of the target enhancement source image on the original image to which the target enhancement source image belongs and the height of the target image, the method further includes:

[0042] It is determined that there are no targets of the same category in the target image, wherein the targets of the same category are of the same category as the targets detected in the target enhancement source image.

[0043] In one possible implementation, determining the target image size corresponding to the target enhancement source image includes:

[0044] It is determined that there are detection targets of the same category on the target image, wherein the detection targets of the same category are of the same category as the detection targets in the target enhancement source image;

[0045] The target image size is determined based on the number of targets of the same category, the image position of the targets of the same category in the target image, and the image position of the target enhancement source image in the original image to which the target enhancement source image belongs.

[0046] In one possible implementation, determining the target image size based on the number of detected targets of the same category, the image positions of the detected targets of the same category in the target image, and the image position of the target enhancement source image in the original image to which the target enhancement source image belongs includes:

[0047] Determine the ratio of the height position of the same category detected target on the target image to the distance from the upper boundary of the same category detected target to the upper boundary of the target image;

[0048] The mean of the ratios is determined based on the number of targets of the same category.

[0049] The size of the target image is determined based on the mean and the distance from the upper boundary of the target enhancement source image to the upper boundary of the original image.

[0050] In one possible implementation, replacing the image content within a corresponding region of the target image with the target enhancement source image according to the target image location and the target image size includes:

[0051] Based on the target image location and the target image size, determine whether the target enhancement source image overlaps with the detected target on the target image;

[0052] If the target enhancement source image overlaps with the detected target on the target image and the overlapping area is greater than a preset threshold, then the image position and image size of the target enhancement source image on the target image are re-determined.

[0053] If the target enhancement source image does not overlap with the detected target on the target image, or if the overlapping area of ​​the target enhancement source image and the detected target on the target image is less than or equal to the preset threshold, then the image content in the corresponding area of ​​the target image is replaced with the target enhancement source image.

[0054] In one possible implementation, after performing image synthesis processing on the enhanced source image and the images in the first image set to obtain the fourth image set, the method further includes:

[0055] In the fourth image set, the images in the fourth image set are classified according to whether there is a detection target on the image and whether different detection targets on the image overlap.

[0056] Secondly, embodiments of this disclosure provide a data processing apparatus, comprising:

[0057] The fusion module is used to fuse the first image set and the second image set to obtain a third image set. The first image set is the initial training image set of the target detection model, and the second image set is used to increase the number of detected targets on the images in the first image set.

[0058] The segmentation module is used to perform target detection and segmentation on the images in the third image set to obtain the enhanced source image;

[0059] The synthesis module is used to perform image synthesis processing on the enhanced source image and the images in the first image set to obtain a fourth image set. The fourth image set is the final training image set of the target detection model. The number of detected targets in the images in the fourth image set is greater than the number of detected targets in the images in the first image set.

[0060] In one possible implementation, the fusion module is specifically used for:

[0061] The first image set and the second image set are integrated to obtain an integrated image set;

[0062] The images in the integrated image set are filtered to obtain the third image set.

[0063] In one possible implementation, bounding boxes and label categories for the detected targets exist on the images in the second image set and the first image set; the fusion module is specifically used for:

[0064] In the integrated image set, images with the annotation category being the target category and / or with complete annotation boxes are selected, where the target category is the category of the detected target on the image in the first image set;

[0065] The third image set is obtained based on the selected images.

[0066] In one possible implementation, the segmentation module is specifically used for:

[0067] In the target category, a first category is determined that appears in the second image set, the target category being the category of the detected target on the images in the first image set;

[0068] On the images in the third image set, the detected targets belonging to the first category are segmented to obtain a first segmented image, wherein the enhanced source image includes the first segmented image.

[0069] In one possible implementation, the segmentation module is specifically used for:

[0070] Using an image segmentation model pre-trained on the second image set, instance segmentation is performed on the images in the third image set for the detected targets belonging to the first category, resulting in a first segmented image.

[0071] In one possible implementation, the segmentation module is specifically used for:

[0072] In the target category, a second category that does not appear in the second image set is determined, where the target category is the category of the detected target on the image in the first image set;

[0073] On the images in the third image set, salient target detection is performed on the detection targets belonging to the second category to obtain a second segmentation image, wherein the enhanced source image includes the second segmentation image.

[0074] In one possible implementation, the synthesis module is specifically used for:

[0075] In the first image set, the target image to be processed is determined;

[0076] In the enhanced source image, the target enhanced source image is determined;

[0077] Determine the target image location corresponding to the target enhancement source image;

[0078] Determine the target image size corresponding to the target enhancement source image;

[0079] According to the target image location and the target image size, the image content in the corresponding area of ​​the target image is replaced with the target enhancement source image.

[0080] In one possible implementation, the synthesis module is specifically used for:

[0081] For the target image, the number of enhancement source images is randomly determined;

[0082] The target enhancement source image is randomly determined from the enhancement source images according to the number of enhancement source images.

[0083] In one possible implementation, the target image location includes the target height location corresponding to the target enhancement source image and the target width location corresponding to the target enhancement source image; the synthesis module is specifically used for:

[0084] The target height position is determined based on the normal distribution;

[0085] The target width position is randomly determined.

[0086] In one possible implementation, the synthesis module is specifically used for:

[0087] Determine the normal distribution function based on the height, width, and / or area of ​​the target image;

[0088] The target height position is randomly determined based on the normal distribution function.

[0089] In one possible implementation, the synthesis module is specifically used for:

[0090] The size of the target image is determined based on the relationship between the height position of the target enhancement source image on the original image to which the target enhancement source image belongs and the height of the target image.

[0091] In one possible implementation, the synthesis module is specifically used for:

[0092] Before determining the size of the target image based on the relationship between the height position of the target enhancement source image on the original image to which the target enhancement source image belongs and the height of the target image, it is determined that there are no detection targets of the same category on the target image, wherein the detection targets of the same category are of the same category as the detection targets in the target enhancement source image.

[0093] In one possible implementation, the synthesis module is specifically used for:

[0094] It is determined that there are detection targets of the same category on the target image, wherein the detection targets of the same category are of the same category as the detection targets in the target enhancement source image;

[0095] The target image size is determined based on the number of targets of the same category, the image position of the targets of the same category in the target image, and the image position of the target enhancement source image in the original image to which the target enhancement source image belongs.

[0096] In one possible implementation, the synthesis module is specifically used for:

[0097] Determine the ratio of the height position of the same category detected target on the target image to the distance from the upper boundary of the same category detected target to the upper boundary of the target image;

[0098] The mean of the ratios is determined based on the number of targets of the same category.

[0099] The size of the target image is determined based on the mean and the distance from the upper boundary of the target enhancement source image to the upper boundary of the original image.

[0100] In one possible implementation, the synthesis module is specifically used for:

[0101] Based on the target image location and the target image size, determine whether the target enhancement source image overlaps with the detected target on the target image;

[0102] If the target enhancement source image overlaps with the detected target on the target image and the overlapping area is greater than a preset threshold, then the image position and image size of the target enhancement source image on the target image are re-determined.

[0103] If the target enhancement source image does not overlap with the detected target on the target image, or if the overlapping area of ​​the target enhancement source image and the detected target on the target image is less than or equal to the preset threshold, then the image content in the corresponding area of ​​the target image is replaced with the target enhancement source image.

[0104] One possible implementation also includes:

[0105] The classification module is used to classify the images in the fourth image set according to whether there is a detection target in the image and whether different detection targets in the image overlap.

[0106] Thirdly, embodiments of this disclosure provide an electronic device, including:

[0107] Memory and processor;

[0108] The memory is used to store program instructions;

[0109] The processor is used to invoke program instructions in the memory to execute the data processing method as described in the first aspect above.

[0110] Fourthly, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the data processing method as described in the first aspect above.

[0111] Fifthly, embodiments of this disclosure provide a computer program product comprising a computer program that, when executed by a processor, implements the data processing method as described in the first aspect.

[0112] The data processing method, apparatus, device, and storage medium provided in the embodiments of this disclosure obtain a third image set by fusing a second image set based on a first image set, and synthesizes an enhanced source image obtained by target detection and segmentation from the images in the third image set with the images in the first image set to obtain a fourth image set. The number of detected targets in the images in the fourth image set is greater than the number of detected targets in the images in the first image set. The first image set is the initial training image set of the target detection model, and the fourth image set is the final training image set of the target detection model.

[0113] As can be seen, the embodiments of this disclosure utilize the method of fusing other image sets on the training image set of the target detection model, then performing target segmentation on the fused image set to obtain enhanced source images, and synthesizing image source images on the images of the training image set. This increases the number of detected targets on the images of the training image set and increases the proportion of images with detected targets in the training image set, thereby achieving the effect of improving the balance of the number of positive and negative samples in the training image set.

[0114] Various feasible embodiments of this disclosure and their technical advantages will be described in detail below. Attached Figure Description

[0115] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0116] Figure 1 Example diagrams illustrating application scenarios provided for embodiments of this disclosure;

[0117] Figure 2 A schematic flowchart of a data method provided in an embodiment of this disclosure;

[0118] Figure 3A flowchart illustrating a data method provided for another embodiment of this disclosure;

[0119] Figure 4 A flowchart illustrating a data method provided for another embodiment of this disclosure;

[0120] Figure 5 A schematic diagram of the structure of a data processing apparatus provided in an embodiment of this disclosure;

[0121] Figure 6 A block diagram of a data processing apparatus provided in an embodiment of this disclosure.

[0122] The accompanying drawings have illustrated specific embodiments of this disclosure, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this disclosure to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0123] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0124] First, the terms used in the embodiments of this disclosure will be explained:

[0125] Salient object detection: Based on salient features in the spatial domain, frequency domain, etc., it quickly and effectively extracts useful regions (e.g., target regions) in a scene for further analysis. Salient object detection models typically detect salient objects in a scene and segment the images of these objects.

[0126] Instance segmentation refers to predicting which category of a detection target each pixel in an image belongs to, while also distinguishing different detection targets of the same category in the image.

[0127] COCO dataset: A large and rich dataset for object detection, segmentation, and captioning. The dataset primarily includes images from complex everyday scenes, and accurately segments the images to label the location and category of detected objects.

[0128] Detection targets: Objects and human figures appearing in an image, such as vehicles and pedestrians. There are various categories of detection targets; for example, the COCO dataset has 80 categories: people, bicycles, airplanes, traffic lights, etc. The detected targets in the image are the samples. For example, in a target detection model for detecting people not wearing helmets, people not wearing helmets are positive samples, and people wearing helmets are negative samples.

[0129] The specific application scenario of the embodiments of this disclosure can be the training scenario of an object detection model. In this scenario, the collected training image set is processed to balance the number of positive and negative samples in the training image set. The processed training image set is used to train the object detection model, thereby improving the performance of the object detection model.

[0130] Figure 1 Example diagrams of application scenarios provided for embodiments of this disclosure, such as... Figure 1 As shown, in the application scenario, the equipment involved includes image processing device 110. Figure 1 Taking image processing device 110 as an example, the server is a device that provides data processing and dataset storage. The server can be a monolithic server or a distributed server spanning multiple computers or a computer data center. The server can also be of various categories, such as, but not limited to, a web server, an application server, a database server, or a proxy server.

[0131] In some embodiments, a server may include hardware, software, or embedded logic components for performing suitable functions supported or implemented by the server, or a combination of two or more such components. For example, a server may be a blade server, a cloud server, or a server group consisting of multiple servers, which may include one or more of the categories of servers described above, etc.

[0132] In some embodiments, the devices involved in the application scenario further include an image acquisition device 120. The image acquisition device 120 is, for example, a terminal device, wherein the terminal device is, for example, a camera, smartphone, laptop, tablet computer, PDA, wearable device, virtual reality device, augmented reality device, etc., or any combination thereof, and is not limited herein. Figure 1 The example is a camera used as the terminal device.

[0133] Image acquisition device 120 can communicate with image processing device 110 via a wireless or wired network. The wireless network can be a 2G, 3G, 4G, or 5G network, or a wireless local area network; no limitation is made here. After acquiring an image, image acquisition device 120 can send the image to image processing device 110. Image processing device 110 stores the image to obtain a training image set for training the object detection model, and can further process the images in the training image set.

[0134] The inventors discovered that the training image set used for object detection models suffers from a severe imbalance between positive and negative samples. For example, in warehouse security management, a vest detection model needs to detect whether workers are wearing vests legally, and an open flame detection model needs to detect potential fire hazards in the warehouse. In the training image set used to train the vest detection model, the number of people wearing vests (positive samples) far exceeds the number of people not wearing vests (negative samples). Similarly, in the training image set used to train the open flame detection model, the number of images without flames (positive samples) far exceeds the number with flames (negative samples). This imbalance between positive and negative samples severely affects the generalization ability of object detection models, and improving the balance between positive and negative samples is a pressing issue that needs to be addressed.

[0135] In related technologies, data augmentation techniques can be used to increase the sample size, as follows:

[0136] Method 1: Utilizing the translation invariance of Convolutional Neural Networks (CNNs), geometric and illumination distortions are used to augment the image data. However, this method affects the detection of small targets in the image (for example, after changing the contrast or adding noise during distortion, the image region containing the small target may be replaced by noise), leading to overfitting of the target detection model.

[0137] Method 2: Utilize a small number of images containing bounding boxes of the detected target from the original image set and / or an open-source dataset, and synthesize the bounding boxes with images from the original image set. However, the bounding boxes contain background information, which can obscure other content in the images from the original image set and introduce noise.

[0138] To address the issue of imbalanced positive and negative sample numbers, embodiments of this disclosure provide a data processing method. In this method, a first image set, serving as the initial training image set for an object detection model, is fused with a second image set to obtain a third image set. Object detection and segmentation are performed on the images in the third image set to obtain enhanced source images. Finally, image synthesis processing is performed between the enhanced source images and the images in the first image set to obtain a fourth image set, which serves as the final training image set for the object detection model.

[0139] As can be seen, the data processing method provided in this disclosure has the following advantages: On the one hand, it increases the number of detected targets in the training image set without increasing the number of images in the training image set, that is, it increases the number of samples, which is conducive to improving the balance of positive and negative samples, and thus improving the model performance of the target detection model; on the other hand, the enhanced source image does not bring background information and does not introduce noise into the images in the training image set, solving the problem of a large amount of background information brought about by directly replacing the bounding boxes onto the image, and improving the quality of the images in the training image set. Thus, this disclosure solves the problem of imbalanced positive and negative samples in the training image set from a data perspective, improves the quantity and quality of samples in the training image set, provides an automatic and stable data synthesis scheme for target detection scenarios (especially multi-target detection scenarios in surveillance scenarios), and solves the sample collection problem.

[0140] The technical solutions of the embodiments of this disclosure and how the technical solutions of this disclosure solve the above-mentioned technical problems will be described in detail below with specific examples. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this disclosure will now be described with reference to the accompanying drawings.

[0141] refer to Figure 2 , Figure 2 This is a schematic flowchart illustrating a data method provided in one embodiment of the present disclosure. Figure 2 As shown, the method includes:

[0142] S201. The first image set and the second image set are fused to obtain the third image set.

[0143] The first image set is the initial training image set for the object detection model, and the second image set is used to increase the number of detected objects on the images in the first image set. The first and second image sets are different image sets. The second image set can be an open-source image set, or it can be an image set used for other tasks or other models. The objects detected by the object detection model appear on multiple images in the second image set.

[0144] For example, if the target detection model detects vehicles, the second image set may contain multiple vehicle images. Of course, other targets may also appear in the images in the second image set, such as vehicles, pedestrians, and animals.

[0145] In this embodiment, after obtaining the first image set for training the target detection model, considering that the number of positive and negative samples in the first image set may be unbalanced due to limitations in sample collection capabilities and actual conditions, especially with a relatively small number of images containing detected targets, a second image set is obtained to balance the number of positive and negative samples in the first image set. This second image set is used to enrich the number of detected targets in the images of the first image set, i.e., to enrich the sample quantity and sample diversity of the first image set. Therefore, after obtaining the second image set, the first image set and the second image set can be fused to obtain a third image set. The first and second image sets can be pre-stored in the current device or other devices.

[0146] S202. Perform target detection and segmentation on the images in the third image set to obtain the enhanced source image.

[0147] In this embodiment, considering that the method of annotating the detected target on the image in object detection is to use a rectangular annotation box, which contains background information other than the detected target, after obtaining the third image set, object detection and segmentation can be performed on the images in the third image set. Compared with traditional object detection, object segmentation is performed on the image along the edge of the detected target, without adding additional background information. After performing object detection and segmentation on the images in the third image set, multiple segmented images of the detected targets can be obtained, that is, multiple enhancement source images. Among them, the enhancement source images contain the detected targets but do not contain additional background information.

[0148] S203. Perform image synthesis processing on the enhanced source image and the images in the first image set to obtain the fourth image set.

[0149] The fourth image set is the final training image set for the object detection model, and the number of detected objects in the images in the fourth image set is greater than the number of detected objects in the images in the first image set.

[0150] In this embodiment, after obtaining multiple enhancement source images, an image to be data-enhanced can be acquired from a first image set. An enhancement source image for data enhancement is then selected from the multiple enhancement source images. This enhancement source image is then combined with the original image to perform image compositing, essentially replacing the image content of a local region in the original image with the enhancement source image, thus making the enhancement source image appear in the original image. This increases the number of detected targets in the image, i.e., increases the number of samples, solving the problem of an imbalance between positive and negative samples caused by the difficulty in acquiring images containing detected targets.

[0151] In this embodiment of the disclosure, other image sets are fused to the initial training image set of the target detection model. Enhanced source images are obtained by target detection and segmentation in the fused image set. Then, image synthesis is performed on the enhanced source images and the images in the initial training image set to increase the number of detected targets in the images in the initial training image set. Without increasing the number of images in the training image set, the number of positive and negative samples is increased, thereby improving the balance of the number of positive and negative samples in the training image set of the target detection model. Moreover, no irrelevant background information is introduced into the images in the training image set.

[0152] In some embodiments, the second image set is an open-source image set. This utilizes publicly available data to increase the number of detectable targets on the images in the training image set of the object detection model, saving data acquisition time.

[0153] In some embodiments, bounding boxes and label categories of the detected targets are present on the images in the second and first image sets, thereby saving data annotation costs and improving the efficiency and accuracy of generating enhanced source images. Furthermore, bounding boxes and label categories of the detected targets may also be present on the images in the first image set, which is beneficial for supervised training of the target detection model.

[0154] In some embodiments, the second image set may be the COCO dataset.

[0155] refer to Figure 3 , Figure 3 A schematic flowchart of a data method provided for another embodiment of this disclosure.

[0156] like Figure 3 As shown, the method includes:

[0157] S301. Integrate the first image set and the second image set to obtain an integrated image set.

[0158] In one example, the first image set and the second image set can be merged to obtain a combined image set.

[0159] In another example, the format of the data in the first image set may differ from that in the second image set. For instance, the format of the images in the first image set may differ from that in the second image set, and the format of the label file (which records the position of the annotation boxes) in the first image set may differ from that in the second image set. Therefore, the formats of the data in the first and second image sets can be unified first, and then the first and second image sets can be merged into an integrated image set.

[0160] S302. Filter the images in the integrated image set to obtain the third image set.

[0161] In this embodiment, since the categories of detected targets in the images of the second image set are not necessarily exactly the same as those in the images of the first image set—for example, when the second image set is the COCO dataset, it contains at least 80 types of detected targets, far more than the number of categories in the first image set—to avoid introducing unnecessary categories, the integrated image set can be filtered based on the categories of detected targets in the images of the first image set after obtaining the integrated image set. Alternatively, the integrated image set can be filtered based on sample quality.

[0162] In one possible implementation, the images in the second image set and the first image set contain bounding boxes and label categories for the detected target. S302 includes: in the integrated image set, filtering out images whose label category is the target category and / or whose bounding boxes are complete; and obtaining a third image set based on the filtered images.

[0163] The target category refers to the category of the detected target in the images of the first image set. For example, if some images in the first image set contain detected targets of the category of pedestrians, and some images contain detected targets of the category of vehicles, then the target category can include people and vehicles.

[0164] In this embodiment, the following filtering operations can be performed on each image in the integrated image set: First, determine if there are bounding boxes in the image whose category is the target category. If not, the image can be deleted from the integrated image set (for example, if the target detection model needs to detect vests, the target category is "human," so images of targets not containing this category can be deleted from the integrated image set); and / or, determine if the bounding boxes in the image are complete (for example, some bounding boxes overlap significantly). If incomplete, the image can be deleted from the integrated image set. Thus, the integrated image set after filtering becomes the third image set. This reduces data redundancy in the third image set and improves the image quality in the third image set.

[0165] S303. Perform target detection and segmentation on the images in the third image set to obtain the enhanced source image.

[0166] S304. Perform image synthesis processing on the enhanced source image and the images in the first image set to obtain the fourth image set.

[0167] S303 and S304 can be referred to in the aforementioned embodiments and will not be described again.

[0168] In one possible implementation, S303 includes: determining a first category appearing in a second image set among the target categories; and performing instance segmentation on the detected targets belonging to the first category in the images of a third image set to obtain a first segmented image.

[0169] In another possible implementation, S303 includes: determining a second category that does not appear in the second image set in the target category; and performing salient target detection on the detected targets belonging to the second category on the images in the third image set to obtain a second segmentation image.

[0170] The enhanced source image includes a first segmented image and / or a second segmented image.

[0171] The target category is the category of the detected target on the images in the first image set, the first category is the target category that appears in the labeled categories included in the second image set, and the second category is the target category that does not appear in the labeled categories included in the second image set.

[0172] In this embodiment, different target detection and segmentation methods can be used for the first and second categories of target categories to improve the accuracy of target detection and segmentation.

[0173] For the first category, an instance segmentation model can be used. On the images in the third image set, the detected targets belonging to the first category are detected and segmented to obtain a first segmented image containing the detected target. The instance segmentation model can only be used to detect and segment targets belonging to specific categories. For the first category, a corresponding instance segmentation model can be pre-trained, thus resulting in higher accuracy for target segmentation using instance segmentation for the first category.

[0174] For the second category, a salient object detection model can be used to detect and segment objects belonging to the second category on the images in the third image set, resulting in a second segmented image containing these objects. The salient object detection model can be used to detect and segment objects of unspecified categories. The second category does not appear in the labeled categories included in the second image set but appears in the target category. The target category is the category of the detected objects on the images in the first image set. Since the first image set is the training model for the object detection model, a suitable object detection model has not yet been trained, making it unsuitable for instance segmentation of objects belonging to the second category. Therefore, using a salient object detection model to detect objects belonging to the second category is more appropriate.

[0175] Optionally, an image segmentation model pre-trained on the second image set is used to perform instance segmentation on the images in the third image set for the detected targets belonging to the first category, resulting in a first segmented image. Since the first category appears in the labeled categories included in the second image set, using the image segmentation model pre-trained on the second image set can effectively improve the accuracy of segmenting the detected targets belonging to the first category. Specifically, the image segmentation model pre-trained on the second image set uses the second image set as the training image set to detect and segment the detected targets corresponding to the labeled categories in the second image set.

[0176] Optionally, when the second image set is the COCO dataset, the COCO pre-trained model can be loaded using the Mask-RCNN model under the Detectron2 object detection platform. This allows for instance segmentation of detected objects belonging to the first category on the images in the third image set, thereby improving the accuracy of instance segmentation. Detectron2 is a computer vision library built on Caffe2 and PyTorch, and the Mask-RCNN model is a deep learning-based instance segmentation algorithm.

[0177] Optionally, the SOD100K model can be used for salient object detection to improve its accuracy and efficiency. The SOD100K model is a highly efficient salient object detection algorithm with extremely low parameter count.

[0178] In this embodiment, by integrating and filtering operations, the image quality of the third image set obtained after fusing the first and second image sets is improved, and data redundancy in the third image set is reduced. This improves the quality and accuracy of the enhanced source images obtained from target detection and segmentation based on the third image set, making the detected targets in the enhanced source images more consistent with the needs of the target detection model. Consequently, the quality of the fourth image set obtained by data synthesis based on the enhanced meta-images and the first image set is effectively improved. Therefore, this embodiment increases the number of samples in the training image set through data synthesis while improving the quality and accuracy of the enhanced source images in data synthesis, thus improving the overall image quality after synthesis.

[0179] Figure 4 A flowchart illustrating a data method provided for another embodiment of this disclosure. (See attached diagram.) Figure 4 As shown, the method includes:

[0180] S401. The first image set and the second image set are fused to obtain the third image set.

[0181] S402. Perform target detection and segmentation on the images in the third image set to obtain the enhanced source image.

[0182] S401 and S402 can be referred to in the aforementioned embodiments and will not be described again.

[0183] S403. In the first image set, determine the target image to be processed.

[0184] The target images to be processed are those in the first image set that are to be used for data synthesis or data enhancement, and the number of target images can be one or more.

[0185] In this embodiment, one or more target images can be sequentially obtained from the first image set, or one or more target images can be randomly obtained from the first image set, or one or more pre-specified target images can be obtained from the first image set.

[0186] S404. In the enhanced source image, determine the target enhanced source image.

[0187] In this embodiment, for the target image, an enhancement source image is determined from the enhancement source images to be used for data enhancement of the target image. For indirect description, the enhancement source image used for data enhancement of the target image (i.e., data synthesis with the target image) is referred to as the target enhancement source image. There can be one or more target enhancement source images; that is, one or more target enhancement source images can be synthesized with a target image to effectively increase the number of detected targets on the target image.

[0188] In one possible implementation, S404 includes: randomly determining the number of enhancement source images for the target image; and randomly determining the target enhancement source image from the enhancement source images according to the number of enhancement source images. Thus, by randomly selecting the number and type of enhancement source images, randomness is increased, which to some extent avoids overfitting of the target detection model caused by training it on the synthesized fourth image set, thereby enhancing the generalization ability of the target detection model.

[0189] In this embodiment, for each target image, it is determined whether to perform data augmentation on the target image. If it is determined that the target image should be augmented, the number of augmentation source images used for data augmentation of the target image is determined, that is, the number of augmentation source images corresponding to the target image is determined, and a number of augmentation source images are randomly selected from the augmentation source images.

[0190] Optionally, for each target image, it can be determined whether to perform data augmentation on the target image based on a preset probability (e.g., 50%).

[0191] Optionally, the number of enhanced source images can be determined within a preset range (e.g., 1-5).

[0192] S405. Determine the location of the target image corresponding to the target enhancement source image.

[0193] S406. Determine the target image size corresponding to the target enhancement source image.

[0194] Among them, the target image position corresponding to the target enhancement source image is the position of the target image after the target enhancement source image is synthesized into the target image, and the target image size corresponding to the target enhancement source image is the size of the target image after the target enhancement source image is synthesized into the target image.

[0195] The target image position corresponding to the target enhancement source includes the target height position and the target width position corresponding to the target enhancement source image. The target height position refers to the height position of the target enhancement source image on the target image after it is synthesized into the target image, and the target width position refers to the width position of the target image on the target image after it is synthesized into the target image.

[0196] In this embodiment, after determining the target enhancement source image for data enhancement of the target image, the target image position and target image size corresponding to the target enhancement source image can be determined, so that the target image and the target enhancement source image can be synthesized according to the target image position and target image size.

[0197] In one possible implementation, the target image position and size corresponding to the target enhancement source image are randomly determined. Specifically, the image regions where the original detected targets are located on the target image can be predetermined. Outside these image regions, the target image position corresponding to the target enhancement source image is randomly determined, and the target image size corresponding to the target enhancement source image is randomly determined based on the size of these image regions. Thus, by using randomness, the diversity of the target enhancement source image's positional distribution on the target image is increased, which is beneficial for improving the generalization ability of the target detection model.

[0198] In one possible implementation, S405 includes: determining the target height position corresponding to the target enhancement source image based on a normal distribution; and randomly determining the target width position corresponding to the target enhancement source image. The inventors, after observing a large amount of camera image information, found that the positional distribution of the detected target on the image basically conforms to a normal distribution. Therefore, by determining the target height position corresponding to the target enhancement source image based on a normal distribution and randomly determining the target width position corresponding to the target enhancement source image, the diversity, accuracy, and rationality of the positional distribution of the target enhancement source image on the target image are improved.

[0199] Besides determining the target height position corresponding to the target enhancement source based on a normal distribution, possible implementations of S405 could also include: determining the target width position corresponding to the target enhancement source image based on a normal distribution; and randomly determining the target height position corresponding to the target enhancement source image. The following description will further illustrate this using the determination of the target height position corresponding to the target enhancement source image based on a normal distribution as an example. The determination of the target width position corresponding to the target enhancement source image based on a normal distribution can be referenced in the description of determining the target height position corresponding to the target enhancement source image based on a normal distribution, and will not be elaborated upon further.

[0200] Optionally, in determining the target height position corresponding to the target enhancement source image based on a normal distribution, a normal distribution function can first be determined based on the height, width, and / or area of ​​the target image, and then the target height position can be randomly determined based on the normal distribution function. Thus, when determining the target height position based on a normal distribution, referencing the height, width, and / or area of ​​the target image improves the rationality and accuracy of the normal distribution, thereby enhancing the rationality and accuracy of the target height position.

[0201] Furthermore, when determining the normal distribution function based on the height of the target image, the expected value of the normal distribution function can be determined based on the height of the target image, and then the standard deviation of the normal distribution function can be determined.

[0202] Optionally, the expected value of the normal distribution function can be determined to be half the height of the target image, and the standard deviation of the normal distribution function can be determined to be 1. In this case, the normal distribution function can be expressed as:

[0203] Where h is the height of the objective function and x represents the height position of the detected target in the target image, the target height position corresponding to the target enhancement source image can be randomly determined based on this normal distribution function.

[0204] Furthermore, when determining the normal distribution function based on the width of the target image, the expected value of the normal distribution function can be determined based on the width of the target image, and then the standard deviation of the normal distribution function can be determined. In this case, the formula for the normal distribution function is similar to the formula for the normal distribution function determined based on the height of the target image, and the h in the above formula can be replaced with the width of the target image.

[0205] Furthermore, when determining the target normal distribution function based on the area of ​​the target image, the expected value of the normal distribution function can be determined based on the area of ​​the target image, and then the standard deviation of the normal distribution function can be determined. In this case, the formula for the normal distribution function is similar to the formula for the normal distribution function determined based on the height of the target image, and the h in the above formula can be replaced with the area of ​​the target image.

[0206] In one possible implementation, considering that objects closer to the camera appear larger in the image, step S406 includes: determining the size of the target image based on the relationship between the height position of the target enhancement source image on the original image to which it belongs and the height of the target image. The relationship between the height position of the target enhancement source image on the original image to which it belongs and the height of the target image can, to some extent, reflect the distance between the target enhancement source image and the camera, improving the realism of the image obtained after data synthesis of the target enhancement source image and the target image.

[0207] The original image to which the target enhancement source image belongs, i.e., the image where the target enhancement source image was located before segmentation, will be referred to as the original image for simplicity.

[0208] In this embodiment, when performing target detection and segmentation on the images in the third image set to obtain the enhanced source image, the positions of the enhanced source image and the enhanced source image on the original image can be saved. For example, a one-to-one correspondence between the enhanced source image, the inference result, and the label can be saved. The inference result is an image with the same size as the original image but only one channel. The pixel values ​​of the pixels in this image are either True (e.g., 1) or False (e.g., 0). If the pixel value is True, it indicates that a detected target exists at that pixel; otherwise, it indicates that a detected target does not exist at that pixel. The label records the position distribution of pixels where a detected target exists. Therefore, based on this one-to-one correspondence, the position of the enhanced source image on the original image can be obtained. From the position of the enhanced source image on the original image, the height position of the target enhanced source image on the original image is obtained. Then, the height position of the target enhanced source image on the original image is compared with the height of the target image. Based on the comparison result, the size of the target image is determined.

[0209] In this embodiment, the detection target located at the top of the image is farther from the camera, while the detection target located at the bottom of the image is closer to the camera. Therefore, based on the relationship between the height position of the target enhancement source image on the original image and the height of the target image, it can be determined whether the target enhancement source image is closer to the camera or farther from the camera. Furthermore, based on whether the target enhancement source image is closer to the camera or farther from the camera, the size of the target image can be determined.

[0210] Optionally, multiple height ranges and corresponding image sizes can be determined based on the height of the target image. The target image size is then determined from among multiple threshold-corresponding image sizes based on a comparison of the height position of the target enhancement source image on the original image with the multiple height ranges. For example, if the height position of the enhancement source image on the original image falls within the first height range, the target image size is determined to be the image size corresponding to the first height range.

[0211] Furthermore, the formula for determining the target image size can be expressed as:

[0212]

[0213] Among them, h a To enhance the height position of the source image on the original image, h′ a Let be the target image size corresponding to the target enhancement source image. Here, the target image size corresponding to the target enhancement source image is the target height corresponding to the target enhancement source image, that is, the height of the target enhancement source image on the target image. The height of the target enhancement source image refers to how tall the target enhancement source image is, and the height position of the target enhancement source image is the coordinate position of the target enhancement source on the two-dimensional coordinate axis of the image.

[0214] Optionally, after determining that there are no similar detection targets in the target image, the target image size is determined based on the relationship between the height position of the target enhancement source image on the original image to which it belongs and the height of the target image. Thus, when there are no reference detection targets in the target image, the image size after the target enhancement source image is synthesized into the target image is reasonably determined based on the height of the target image and the height position of the target enhancement source on the original image.

[0215] Among them, the detected targets of the same category are the same category as the detected targets in the target enhancement source image.

[0216] For example, if the detected target in the target enhancement source image is a pedestrian, but there is no pedestrian in the target image, then the size of the target image is determined based on the relationship between the height position of the target enhancement source image on the original image to which the target enhancement source image belongs and the height of the target image.

[0217] In some embodiments, when there is a detection target of the same category as the detection target in the target enhancement source image (i.e., a detection target of the same category) in the target image, the target image size corresponding to the target enhancement source image is determined by referring to the position of the detection target of the same category in the target image, so as to improve the accuracy and rationality of the target image size.

[0218] At this point, one possible implementation of S406 includes: determining that there are detectable targets of the same category on the target image; and determining the target image size based on the image position of the detectable targets of the same category on the target image and the image position of the target enhancement source image on the original image to which the target enhancement source image belongs.

[0219] In this embodiment, the ratio of the size of a similar detected target to the position of a similar detected target can be calculated, and the product of this ratio and the position of the target enhancement source image in the original image can be calculated to determine the target image size corresponding to the target enhancement source image as the product.

[0220] Considering that there may be multiple detection targets of the same category on the target image, another possible implementation of S405 includes: determining that there are detection targets of the same category on the target image; and determining the target image size based on the number of detection targets of the same category, the image position of the detection targets of the same category on the target image, and the image position of the target enhancement source image on the original image to which the target enhancement source image belongs.

[0221] In this embodiment, when there are multiple similar detection targets in the target image, for each similar detection target in the target image, the ratio of the size of the similar detection target to the position of the similar detection target is calculated. Next, based on the number of similar detection targets in the target image, the average of the ratios of the size of the similar detection targets to the positions of the similar detection targets is calculated. Finally, the product of this average value and the position of the target enhancement source image in the original image is calculated, and the target image size corresponding to the target enhancement source image is determined as this product.

[0222] Optionally, the size of the similar detected targets is the height of the similar detected targets, and the size of the target image corresponding to the target enhancement source is the target height corresponding to the target enhancement source image. Alternatively, the size of the similar detected targets is the width of the similar detected targets, and the size of the target image corresponding to the target enhancement source is the target width corresponding to the target enhancement source image.

[0223] Optionally, taking the size of similar detected targets as the height of similar detected targets and the size of the target image corresponding to the target enhancement source as the target height of the target enhancement source image as an example, for each similar detected target, the ratio of the height position of the similar detected target on the target image to the distance from the upper boundary of the similar detected target to the upper boundary of the target image can be determined. Then, based on the number of similar detected targets, the average value of this ratio can be determined. Based on the average value and the distance from the upper boundary of the target enhancement source image to the upper boundary of the original image, the target image size can be determined, that is, the target height corresponding to the target enhancement source image can be determined. Thus, by referring to the height of similar detected targets, the distance from the upper boundary of similar detected targets to the upper boundary of the target image, and the distance from the upper boundary of the target enhancement source image to the upper boundary of the original image, the accuracy and rationality of the target image size corresponding to the target enhancement source image can be improved.

[0224] Furthermore, the formula for calculating the target image size can be:

[0225] h′ a =avg ob *s a Among them, h ob For the height of similar detection targets, s ob is the distance from the upper boundary of the same type of detected target to the upper boundary of the target image, n is the number of the same type of detected target, and avg is the distance from the upper boundary of the target image to the upper boundary of the target image. ob s is the mean of the ratios of the height position of a detected target of the same class on the target image to the distance from the upper boundary of the detected target of the same class to the upper boundary of the target image. a h′ is the distance from the top boundary of the source image to the top boundary of the original image, used to enhance the target image. a This represents the target image size corresponding to the target enhancement source image, and in this case, it represents the target height corresponding to the target enhancement source image.

[0226] After determining the target height corresponding to the target enhancement source image, the target width corresponding to the target enhancement source image can be determined according to the aspect ratio of the target enhancement source image.

[0227] Optionally, when the size of the same type of detected target is the width of the same type of detected target, and the size of the target image corresponding to the target enhancement source is the target width corresponding to the target enhancement source image, the processing method when the size of the same type of detected target is the height of the same type of detected target, and the size of the target image corresponding to the target enhancement source is the target height corresponding to the target enhancement source image can be referred to. For example, the height in the above formula can be replaced with the width, and the upper boundary can be replaced with the left boundary or the right boundary of the image.

[0228] S407. Replace the image content in the corresponding area of ​​the target image with the target enhancement source image according to the target image location and target image size.

[0229] In this embodiment, after obtaining the target image location and target image size, based on the target image location and target image size, a corresponding image region is determined on the target image by the target enhancement source, and then the image content within the image region is replaced with the target enhancement source image.

[0230] In one possible implementation, S407 includes: determining whether the target enhancement source image overlaps with a detected target on the target image based on the target image position and target image size; if the target enhancement source image overlaps with a detected target on the target image, then redetermining the image position and image size of the target enhancement source image on the target image; if the target enhancement source image does not overlap with a detected target on the target image, then replacing the image content in the corresponding region of the target image with the target enhancement source image. This avoids the target enhancement source image occluding the original detected target on the target image.

[0231] In one possible implementation, considering that objects or people may occlude each other in real-world situations, to increase the number of difficult samples in the training image set of the object detection model, S407 includes: determining whether the target enhancement source image overlaps with the detected target on the target image based on the target image position and target image size; if the target enhancement source image overlaps with the detected target on the target image and the overlapping area is greater than a preset threshold, then redetermining the image position and image size of the target enhancement source image on the target image; if the target enhancement source image does not overlap with the detected target on the target image or the overlapping area of ​​the target enhancement source image and the detected target on the target image is less than or equal to the preset threshold, then replacing the image content in the corresponding area on the target image with the target enhancement source image.

[0232] Therefore, while avoiding severe occlusion of the original detected target in the target image by the target enhancement source image, partial occlusion between the target enhancement source image and the original detected target in the target image is allowed. This increases the number of difficult samples (i.e. occluded samples) in the training image set of the target detection model, improves the diversity of samples in the training sample set, and thus improves the target detection performance of the target detection model.

[0233] This involves re-determining the image position and size of the target enhancement source image on the target image, i.e., re-selecting and determining the target image position and size corresponding to the target enhancement source image.

[0234] In some embodiments, after generating the fourth image set, the images in the fourth image set can be classified based on whether a detection target exists on the images and whether different detection targets overlap. For example, images with overlapping detection targets can be grouped into one category, images without detection targets into another category, and the remaining images into a third category. This classification and organization of the data-augmented training image set facilitates user use, especially by enabling users to select difficult samples for training the target detection model.

[0235] In some embodiments, after generating the fourth image set, a target detection model can be used to train the fourth image set to detect targets on the images in the fourth image set. The target detection model is, for example, the YOLOv5 model, whose input is an RGB three-channel image, and whose output is a visualization image of the detected target and a label file.

[0236] In this embodiment of the disclosure, while increasing the number of positive and negative samples in the training image set and improving the balance of the number of positive and negative samples in the training image set by data synthesis, the diversity, accuracy and rationality of data synthesis are improved by determining the position and size of the enhancement source image, thereby increasing the diversity of samples in the training image set and thus helping to improve the model performance of the target detection model.

[0237] Figure 5 A schematic diagram of the structure of a data processing apparatus provided in an embodiment of this disclosure is shown below. Figure 5 As shown, the device includes:

[0238] The fusion module 501 is used to fuse the first image set and the second image set to obtain a third image set. The first image set is the initial training image set of the target detection model, and the second image set is used to increase the number of detected targets on the images in the first image set.

[0239] The segmentation module 502 is used to perform target detection and segmentation on the images in the third image set to obtain the enhanced source image;

[0240] The synthesis module 503 is used to perform image synthesis processing on the enhanced source image and the images in the first image set to obtain the fourth image set. The fourth image set is the final training image set of the target detection model. The number of detected targets in the images in the fourth image set is greater than the number of detected targets in the images in the first image set.

[0241] In one possible implementation, the fusion module 501 is specifically used to: integrate the first image set and the second image set to obtain an integrated image set; and filter the images in the integrated image set to obtain a third image set.

[0242] In one possible implementation, the images in the second image set and the first image set contain bounding boxes and labeled categories of the detected targets; the fusion module 501 is specifically used to: in the integrated image set, filter out images whose labeled category is the target category and / or whose bounding boxes are complete, and whose target category is the category of the detected targets in the images in the first image set; and obtain a third image set based on the filtered images.

[0243] In one possible implementation, the segmentation module 502 is specifically used to: determine a first category appearing in the second image set in the target category, where the target category is the category of the detected target on the images in the first image set; and perform instance segmentation on the detected target belonging to the first category on the images in the third image set to obtain a first segmented image, wherein the enhancement source image includes the first segmented image.

[0244] In one possible implementation, the segmentation module 502 is specifically used to: use an image segmentation model pre-trained on a second image set to perform instance segmentation on the images in a third image set for the detected targets belonging to the first category, thereby obtaining a first segmented image.

[0245] In one possible implementation, the segmentation module 502 is specifically used to: determine a second category that does not appear in the second image set in the target category, the target category being the category of the detected target on the images in the first image set; and perform salient target detection on the detected target belonging to the second category on the images in the third image set to obtain a second segmented image, wherein the enhancement source image includes the second segmented image.

[0246] In one possible implementation, the synthesis module 503 is specifically used to: determine the target image to be processed in the first image set; determine the target enhancement source image in the enhancement source image; determine the target image position corresponding to the target enhancement source image; determine the target image size corresponding to the target enhancement source image; and replace the image content in the corresponding area of ​​the target image with the target enhancement source image according to the target image position and target image size.

[0247] In one possible implementation, the synthesis module 503 is specifically used to: randomly determine the number of enhancement source images for the target image; and randomly determine the target enhancement source image from the enhancement source images according to the number of enhancement source images.

[0248] In one possible implementation, the target image position includes the target height position corresponding to the target enhancement source image and the target width position corresponding to the target enhancement source image. The synthesis module 503 is specifically used to: determine the target height position based on a normal distribution.

[0249] Randomly determine the target width position.

[0250] In one possible implementation, the synthesis module 503 is specifically used to: determine a normal distribution function based on the height, width, and / or area of ​​the target image; and randomly determine the target height position based on the normal distribution function.

[0251] In one possible implementation, the synthesis module 503 is specifically used to: determine the size of the target image based on the relationship between the height position of the target enhancement source image on the original image to which the target enhancement source image belongs and the height of the target image.

[0252] In one possible implementation, the synthesis module 503 is specifically used to: determine that there are no detection targets of the same category on the target image before determining the target image size based on the relationship between the height position of the target enhancement source image on the original image to which the target enhancement source image belongs and the height of the target image, wherein the detection targets of the same category are of the same category as the detection targets in the target enhancement source image.

[0253] In one possible implementation, the synthesis module 503 is specifically used to: determine that there are detection targets of the same category on the target image, wherein the detection targets of the same category are of the same category as the detection targets in the target enhancement source image; and determine the target image size based on the number of detection targets of the same category, the image position of the detection targets of the same category on the target image, and the image position of the target enhancement source image on the original image to which the target enhancement source image belongs.

[0254] In one possible implementation, the synthesis module 503 is specifically used to: determine the ratio of the height position of the same type of detected target on the target image to the distance from the upper boundary of the same type of detected target to the upper boundary of the target image; determine the mean of the ratio based on the number of same type of detected targets; and determine the target image size based on the mean and the distance from the upper boundary of the target enhancement source image to the upper boundary of the original image.

[0255] In one possible implementation, the synthesis module 503 is specifically used to: determine whether the target enhancement source image overlaps with the detected target on the target image based on the target image position and target image size; if the target enhancement source image overlaps with the detected target on the target image and the overlapping area is greater than a preset threshold, then redetermine the image position and image size of the target enhancement source image on the target image; if the target enhancement source image does not overlap with the detected target on the target image or the overlapping area of ​​the target enhancement source image and the detected target on the target image is less than or equal to the preset threshold, then replace the image content in the corresponding area on the target image with the target enhancement source image.

[0256] In one possible implementation, the data processing device further includes a classification module 504, used to classify the images in the fourth image set according to whether there is a detection target on the image and whether different detection targets on the image overlap.

[0257] Figure 5 The provided data processing device can execute the corresponding method embodiments described above, and its implementation principle and technical effect are similar, so they will not be described again here.

[0258] An embodiment of this disclosure also provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the methods as described in any of the above embodiments.

[0259] An embodiment of this disclosure also provides a program product comprising a computer program stored in a storage medium, wherein at least one processor can read the computer program from the storage medium, and the at least one processor, when executing the computer program, can implement the steps of the method in any of the above embodiments.

[0260] Figure 6 This is a block diagram of a data processing apparatus 600 provided according to an embodiment of the present disclosure. For example, apparatus 600 may be provided as an electronic device, such as a server or terminal. (Refer to...) Figure 6 The apparatus 600 includes a processing component 601, which further includes one or more processors, and memory resources represented by memory 602 for storing instructions, such as application programs, that can be executed by the processing component 601. The application programs stored in memory 602 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 601 is configured to execute instructions to perform the methods of any of the above embodiments.

[0261] The aforementioned processing component 601 can be a processor, which can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc. The aforementioned memory 602 may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device.

[0262] Device 600 may also include a power supply component 603 configured to perform power management of device 600, a wired or wireless network interface 604 configured to connect device 600 to a network, and an input / output (I / O) interface 605. Device 600 may operate on an operating system stored in memory 602, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or similar.

[0263] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. The embodiments of this disclosure are intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0264] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A data processing method, characterized in that, include: The first image set and the second image set are fused to obtain the third image set. The first image set is the initial training image set of the target detection model. The second image set is used to increase the number of detected targets on the images in the first image set and to balance the number of positive and negative samples in the first image set. The images in the third image set are subjected to target detection and segmentation to obtain enhanced source images. The segmentation includes instance segmentation and salient target detection. The enhanced source images contain the detected targets without additional background information. The enhanced source image and the images in the first image set are combined to obtain a fourth image set, which is the final training image set of the target detection model. The number of detected targets in the images in the fourth image set is greater than the number of detected targets in the images in the first image set.

2. The data processing method according to claim 1, characterized in that, The process of fusing the first image set and the second image set to obtain the third image set includes: The first image set and the second image set are integrated to obtain an integrated image set; The images in the integrated image set are filtered to obtain the third image set.

3. The data processing method according to claim 2, characterized in that, The second image set and the images in the first image set contain bounding boxes and label categories for the detected targets. The step of filtering the images in the integrated image set to obtain the third image set includes: In the integrated image set, images with the annotation category being the target category and / or with complete annotation boxes are selected, where the target category is the category of the detected target on the image in the first image set; The third image set is obtained based on the selected images.

4. The data processing method according to claim 1, characterized in that, The step of performing target detection and segmentation on the images in the third image set to obtain the enhanced source image includes: In the target category, a first category is determined that appears in the second image set, the target category being the category of the detected target on the images in the first image set; On the images in the third image set, the detected targets belonging to the first category are segmented to obtain a first segmented image, wherein the enhanced source image includes the first segmented image.

5. The data processing method according to claim 4, characterized in that, The step of performing instance segmentation on the images in the third image set for the detected targets belonging to the first category to obtain a first segmented image includes: Using an image segmentation model pre-trained on the second image set, instance segmentation is performed on the images in the third image set for the detected targets belonging to the first category, resulting in a first segmented image.

6. The data processing method according to claim 1, characterized in that, The step of performing target detection and segmentation on the images in the third image set to obtain the enhanced source image includes: In the target category, a second category that does not appear in the second image set is determined, where the target category is the category of the detected target on the image in the first image set; On the images in the third image set, salient target detection is performed on the detection targets belonging to the second category to obtain a second segmentation image, wherein the enhanced source image includes the second segmentation image.

7. The data processing method according to any one of claims 1-6, characterized in that, The image compositing process of the enhanced source image and the images in the first image set includes: In the first image set, the target image to be processed is determined; In the enhanced source image, the target enhanced source image is determined; Determine the target image location corresponding to the target enhancement source image; Determine the target image size corresponding to the target enhancement source image; According to the target image location and the target image size, the image content in the corresponding area of ​​the target image is replaced with the target enhancement source image.

8. The data processing method according to claim 7, characterized in that, Determining the target enhancement source image in the enhanced source image includes: For the target image, the number of enhancement source images is randomly determined; The target enhancement source image is randomly determined from the enhancement source images according to the number of enhancement source images.

9. The data processing method according to claim 7, characterized in that, The target image location includes the target height location and the target width location corresponding to the target enhancement source image. Determining the target image location corresponding to the target enhancement source image includes: The target height position is determined based on the normal distribution; The target width position is randomly determined.

10. The data processing method according to claim 9, characterized in that, Determining the target height position based on a normal distribution includes: Determine the normal distribution function based on the height, width, and / or area of ​​the target image; The target height position is randomly determined based on the normal distribution function.

11. The data processing method according to claim 7, characterized in that, Determining the target image size corresponding to the target enhancement source image includes: The size of the target image is determined based on the relationship between the height position of the target enhancement source image on the original image to which the target enhancement source image belongs and the height of the target image.

12. The data processing method according to claim 11, characterized in that, Before determining the target image size based on the relationship between the height position of the target enhancement source image on the original image to which the target enhancement source image belongs and the height of the target image, the method further includes: It is determined that there are no detectable targets of the same category on the target image, wherein the detectable targets of the same category are of the same category as the detectable targets in the target enhancement source image.

13. The data processing method according to claim 7, characterized in that, Determining the target image size corresponding to the target enhancement source image includes: It is determined that there are detection targets of the same category on the target image, wherein the detection targets of the same category are of the same category as the detection targets in the target enhancement source image; The target image size is determined based on the number of targets of the same category, the image position of the targets of the same category in the target image, and the image position of the target enhancement source image in the original image to which the target enhancement source image belongs.

14. The data processing method according to claim 13, characterized in that, Determining the target image size based on the number of detected targets of the same category, the image position of the detected targets of the same category in the target image, and the image position of the target enhancement source image in the original image to which the target enhancement source image belongs, includes: Determine the ratio of the height position of the same category detected target on the target image to the distance from the upper boundary of the same category detected target to the upper boundary of the target image; The mean of the ratios is determined based on the number of targets of the same category. The size of the target image is determined based on the mean and the distance from the upper boundary of the target enhancement source image to the upper boundary of the original image.

15. The data processing method according to claim 7, characterized in that, The step of replacing the image content in the corresponding area of ​​the target image with the target enhancement source image according to the target image position and the target image size includes: Based on the target image location and the target image size, determine whether the target enhancement source image overlaps with the detected target on the target image; If the target enhancement source image overlaps with the detected target on the target image and the overlapping area is greater than a preset threshold, then the image position and image size of the target enhancement source image on the target image are re-determined. If the target enhancement source image does not overlap with the detected target on the target image, or if the overlapping area of ​​the target enhancement source image and the detected target on the target image is less than or equal to the preset threshold, then the image content in the corresponding area of ​​the target image is replaced with the target enhancement source image.

16. The data processing method according to any one of claims 1-6, characterized in that, After performing image synthesis processing on the enhanced source image and the images in the first image set to obtain the fourth image set, the process further includes: In the fourth image set, the images in the fourth image set are classified according to whether there is a detection target on the image and whether different detection targets on the image overlap.

17. A data processing apparatus, characterized in that, include: The fusion module is used to fuse the first image set and the second image set to obtain a third image set. The first image set is the initial training image set of the target detection model, the second image set is used to increase the number of detected targets on the images in the first image set, and the second image set is used to balance the number of positive and negative samples in the first image set. The segmentation module is used to perform target detection and segmentation on the images in the third image set to obtain an enhanced source image. The segmentation includes instance segmentation and salient target detection. The enhanced source image contains the detected target without additional background information. The synthesis module is used to perform image synthesis processing on the enhanced source image and the images in the first image set to obtain a fourth image set. The fourth image set is the final training image set of the target detection model. The number of detected targets in the images in the fourth image set is greater than the number of detected targets in the images in the first image set.

18. An electronic device, characterized in that, include: Memory and processor; The memory is used to store program instructions; The processor is used to invoke program instructions in the memory to execute the data processing method as described in any one of claims 1-16.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program; when the computer program is executed, it implements the data processing method as described in any one of claims 1-16.

20. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the data processing method as described in any one of claims 1-16.

Images