Sample generation method, device, equipment and storage medium

Abstract

This application provides a sample generation method, apparatus, device, and storage medium. The method includes: acquiring a target task and corresponding positive example sample data; determining candidate regions corresponding to the target task from the positive example sample data, and determining modification content corresponding to the target task; fusing the modification content and candidate regions to obtain negative example sample data corresponding to the target task. This approach allows for the adaptive generation of negative example sample data that is semantically compatible with the target task, based on a large amount of collected positive example sample data, by combining the semantics of the target task, and obtaining its corresponding sample labeling information.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a sample generation method, device, equipment and storage medium. Background Art

[0002] In many actual classification or detection scenarios, it is often difficult to obtain enough labeled training samples. For example, in some scenarios involving privacy and security, there may only be positive labeled samples, while abnormal sub-classification samples or negative samples are difficult to collect and label accurately. Manually creating these negative samples or abnormal category samples requires a lot of manpower and material resources, and the cost is huge.

[0003] For example, in medical image detection scenarios, due to the involvement of user privacy data, the number of valid labeled samples that can be used for model training is very scarce. In particular, for the detection of a specific disease, such as the detection of a specific tumor, there are often only datasets containing dozens of images with labeled detection areas. With such a small number of labeled samples, it is difficult to train a detection model with better performance. The method of manually generating samples is very inefficient.

[0004] The above problem can be attributed to the problem of small sample classification learning when the number of samples in a certain category is particularly scarce. How to efficiently implement such small sample learning is an urgent problem to be solved. Summary of the invention

[0005] Embodiments of the present invention provide a sample generation method, apparatus, device and storage medium for efficiently and accurately generating negative sample data.

[0006] In a first aspect, an embodiment of the present invention provides a sample generation method, the method comprising:

[0007] Obtaining a target task and positive sample data corresponding to the target task;

[0008] Determining a candidate region corresponding to the target task from the positive sample data, and determining a modified content corresponding to the target task;

[0009] The modified content and the candidate region are fused to obtain negative example sample data corresponding to the target task.

[0010] In a second aspect, an embodiment of the present invention provides a sample generation device, the device comprising:

[0011] An acquisition module, used to acquire a target task and positive sample data corresponding to the target task;

[0012] A determination module, used to determine a candidate region corresponding to the target task from the positive sample data, and determine a modified content corresponding to the target task;

[0013] A fusion module is used to fuse the modified content and the candidate area to obtain negative example sample data corresponding to the target task.

[0014] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor can at least implement the sample generation method described in the first aspect.

[0015] In a fourth aspect, an embodiment of the present invention provides a non-temporary machine-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of an electronic device, the processor can at least implement the sample generation method described in the first aspect.

[0016] In a fifth aspect, an embodiment of the present invention provides a sample generation method, the method comprising:

[0017] Acquire a first medical image corresponding to a medical image detection task, wherein the medical detection task is used to detect whether a set lesion occurs in a target body part, and the first medical image is a medical image indicating that the target body part is in a healthy state;

[0018] Based on the hospital image detection task, determining a candidate image region corresponding to the target body part from the first medical image;

[0019] Acquire a lesion image corresponding to the set lesion;

[0020] The lesion image is superimposed on the candidate image region in the first medical image to obtain a second medical image as a negative example and label information of the second medical image.

[0021] The embodiment of the present invention provides an adaptive sample generation scheme based on the semantics of target tasks. Specifically, there will be different tasks in different application scenarios. To complete a task, it is necessary to pre-train a neural network model or a machine learning model for performing the task. To complete the training of the model, it is necessary to pre-collect enough positive sample data and negative sample data with labeling information (i.e., supervision information). In practical applications, positive sample data is relatively easier to collect, while negative sample data is not easy to collect. Therefore, when facing a certain target task, on the basis of having collected a large amount of positive sample data corresponding to the target task, the candidate area for modification can be determined from the positive sample data in combination with the semantics of the target task (i.e., the description information of the target task), and the modification content corresponding to the target task can be obtained. The modification content actually corresponds to the feature information that should be contained in the negative sample data of the target task. Thus, by fusing the modification content with the candidate area, negative sample data that is compatible with the semantics of the target task can be efficiently and accurately generated, and the corresponding sample labeling information can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 A flow chart of a sample generation method provided by an embodiment of the present invention;

[0024] Figure 2 A flow chart of a sample generation method provided by an embodiment of the present invention;

[0025] Figure 3 A flow chart of a sample generation method provided by an embodiment of the present invention;

[0026] Figure 4 A schematic diagram of an application of a sample generation method provided by an embodiment of the present invention;

[0027] Figure 5 A flow chart of a sample generation method provided by an embodiment of the present invention;

[0028] Figure 6 A schematic diagram of an application of a sample generation method provided by an embodiment of the present invention;

[0029] Figure 7a A flow chart of a sample generation method provided by an embodiment of the present invention;

[0030] Figure 7bA flow chart of a sample generation method provided by an embodiment of the present invention;

[0031] Figure 8 A schematic diagram of an application of a sample generation method provided by an embodiment of the present invention;

[0032] Fig. 9 A schematic diagram of the structure of a sample generation device provided by an embodiment of the present invention;

[0033] Fig.10 For Fig. 9 A schematic diagram of the structure of an electronic device corresponding to the sample generating device provided in the illustrated embodiment. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the case where there is no conflict between the embodiments, the following embodiments and the features in the embodiments can be combined with each other. In addition, the step sequence in the following method embodiments is only an example and not a strict limitation.

[0036] The sample generation method provided in the embodiment of the present invention may be executed by an electronic device, which may be a server or a user terminal, and the server may be a physical server or a virtual server (virtual machine) in the cloud.

[0037] Figure 1 A flow chart of a sample generation method provided by an embodiment of the present invention, such as Figure 1 As shown, the method comprises the following steps:

[0038] 101. Obtain the target task and the positive sample data corresponding to the target task.

[0039] 102. Determine a candidate region corresponding to the target task from the positive sample data, and determine the modified content corresponding to the target task.

[0040] 103. The modified content and the candidate region are integrated to obtain negative sample data corresponding to the target task.

[0041] The target tasks in different application scenarios may be different. For example, in the medical image detection scenario, the target task may be to detect specific tumors; in the forged video detection scenario, the target task is to detect whether there are forged images and / or voices in the video. Therefore, corresponding target tasks can be set for different application scenarios.

[0042] The sample generation method provided by the embodiment of the present invention is generally to automatically generate negative sample data corresponding to the target task, because in practical applications, positive sample data is easier to collect, while negative sample data with labeled information is difficult to collect. The positive sample data and negative sample data of the target task are obtained for supervised training of the neural network model corresponding to the target task.

[0043] As mentioned above, the target tasks in different application scenarios are different, so the models that need to be trained will also be different. Therefore, in order to automatically generate negative sample data for the target tasks in different application scenarios, first, for a certain target task set in the current application scenario, it is necessary to first obtain the positive sample data corresponding to the target task, so as to modify the positive sample data based on the semantic information of the target task to generate negative sample data corresponding to the target task.

[0044] The semantic information of the target task is the description information describing what the target task specifically does. For example, the target task is to detect whether there are fake images and / or voices in the video, or the target task can be to detect whether there is a certain tumor in the image.

[0045] In actual applications, the user (the person responsible for executing the sample generation task) can input the target task and several positive sample data that have been collected corresponding to the target task. After that, the candidate area corresponding to the target task is determined from the positive sample data, as well as the changed content corresponding to the target task. The changed content and the candidate area are then merged to obtain the negative sample data corresponding to the target task and the labeling information of the negative sample data.

[0046] Among them, the candidate area can also be called the candidate change area, which refers to the feature area in the positive sample data that matches the target task, that is, the area that is most concerned when performing the target task. The candidate areas selected for different target tasks will be different.

[0047] The candidate region in the positive sample data can be modified to match the target task, so as to generate negative sample data corresponding to the target task based on the positive sample data. In order to make the modification, it is also necessary to first obtain the modification content corresponding to the target task. The modification content is the content reflecting the "negative" feature, that is, the modification content actually corresponds to the feature information that should be included in the negative sample data of the target task, and is feature information inconsistent with the feature information included in the positive sample data.

[0048] In fact, the prompt information for obtaining candidate regions corresponding to different tasks can be preset, so that the candidate regions corresponding to the current target task can be adaptively determined from the positive sample data corresponding to the current target task based on the prompt information. Similarly, the modified content acquisition method corresponding to different tasks can be preset, and the modified content corresponding to the current target task can be acquired accordingly.

[0049] Finally, by fusing the modified content with the candidate region, negative sample data containing the fusion result can be obtained, and based on the fusion result, the label information corresponding to the target task, i.e., the sample label, can be automatically obtained. In practical applications, this fusion may be reflected in different effects such as superposition, replacement, and scaling. In general, the modified content is integrated into the candidate region.

[0050] In summary, the above scheme can efficiently and accurately generate negative sample data that is semantically compatible with the target task and obtain the corresponding sample labeling information.

[0051] Based on the above process, a number of new negative sample data can be generated, and finally the training of the neural network model corresponding to the target task can be completed based on the positive sample data and the generated negative sample data.

[0052] In addition, in an optional embodiment, in order to further improve the robustness of the model, the following steps may also be included:

[0053] Obtaining the user's attack behavior under the target task, where the attack behavior refers to the behavior that will affect the model performance;

[0054] Editing and processing the positive sample data and / or the negative sample data according to the attack behavior;

[0055] The positive sample data, the negative sample data, and the edited positive sample data and negative sample data are used to train a model corresponding to the target task.

[0056] Among them, the attack behavior under the target task refers to some operation behaviors that the model input data may experience in actual applications when the model corresponding to the target task is used to process the model input data, and the operation behavior will interfere with the processing results of the model. For example, the target task is to detect whether an image contains a certain target object. The attack behavior may include scaling, filtering, adjusting the resolution, forwarding, beautifying, etc. of the input image, which will affect the processing results of the model. Therefore, in order to improve the robustness of the model, in the training stage, these attack behaviors are implemented on the training samples, and the training samples after the implementation of these attack behaviors and the training samples before the implementation of the attack behaviors are used together to train the model, which can improve the robustness of the model.

[0057] For example, attack behaviors include forwarding behaviors, and the impact of forwarding behaviors on training samples is mainly reflected in the compression of training samples, so the collected positive sample data and the generated negative sample data can be compressed. The same is true for other attack behaviors.

[0058] For ease of understanding, the execution process of the above sample generation method will be exemplified below in combination with target tasks in different application scenarios.

[0059] Figure 2 A flow chart of a sample generation method provided by an embodiment of the present invention, such as Figure 1 As shown, the method comprises the following steps:

[0060] 201. Obtain a target task and positive sample data corresponding to the target task, where the target task is to detect whether an image contains a target object, and the positive sample data is a positive sample image that does not contain the target object.

[0061] 202. Determine a candidate image region containing a target object for detecting from the positive sample image.

[0062] 203. Obtain a first negative example sample image that has been collected and corresponds to the target task, and extract a target image region containing a target object according to position marking information in the first negative example sample image, wherein the position marking information is used to mark the position of the target object in the first negative example sample image.

[0063] 204. Adjust the shape of the target image region, and superimpose the adjusted target image region onto the candidate image region in the positive sample image to obtain a second negative sample image and label information of the second negative sample image.

[0064] One application scenario assumed in this embodiment is an image detection scenario, in which a task for detecting whether an input image contains a certain target object can be set as a target task. In fact, the target task needs to detect whether the input image contains the target object, and needs to locate the position area of ​​the target object in the input image to output as a detection result.

[0065] Under this target task, the positive sample image can be an image collected for this application scenario that does not contain the target object. For example, if the target task is to detect whether a certain wine contains impurities, the positive sample image can be obtained by photographing several bottles of wine without impurities, that is, the positive sample image does not contain the target object such as impurities. However, in actual applications, a small number of negative sample images containing the target object can also be obtained, and the location area corresponding to the target object (impurity) can be marked in the negative sample image.

[0066] The purpose of collecting positive sample images that do not contain the target object and negative sample images that contain the target object with labeled information is to train the detection model to complete the target task. In order to ensure the performance of the detection model finally trained, it is necessary to obtain a sufficient number of training samples, especially a sufficient number of negative sample images. Therefore, sufficient training samples with labeled information are an important prerequisite for ensuring model performance.

[0067] Therefore, in order to generate more negative sample images with corresponding marking information, first, a candidate image area containing the target object for detecting the target object is extracted from any positive sample image. Secondly, the first negative sample image corresponding to the target task that has been collected is obtained, and the target image area containing the target object is extracted according to the position marking information in the first negative sample image. The position marking information is used to mark the position of the target object in the first negative sample image.

[0068] The determination of the candidate image region is related to the target task. The target task is to detect the location of the target object in the input image. Then, the candidate image region will correspond to the image region where the target object may be located in the image. For example, in the above-mentioned target task of detecting impurities in wine, when the wine is in a static state, assuming that impurities are more likely to appear near the bottle mouth, the candidate image region can be set to the set bottle mouth region of the wine bottle. At this time, the bottle mouth region needs to be determined as the candidate image region from the positive sample image that may contain the wine bottle and some background objects, that is, the position of the candidate image region needs to be identified and recorded.

[0069] In practical applications, the foreground and background images of the positive sample image can be separated first, and then the candidate image area for detecting the target object can be identified in the foreground image, and the position of the candidate image area can be recorded. Specifically, the recognition process can be completed by obtaining prompt information based on the pre-set candidate image area corresponding to the target task. In the above example of impurity detection in wine, the prompt information is, for example: a rectangular area with a height of 5 cm (or N pixels) from the bottle mouth down in the bottle body is used as the candidate image area.

[0070] Since a small number of negative sample images can be collected, and they contain the position mark information of the target object in the negative sample images, a random sample image is selected from the collected negative sample images as the first negative sample image, and the image area corresponding to the target object is extracted from the first negative sample image according to the position mark information of the target object, which is called the target image area.

[0071] Afterwards, the morphology of the target image area can be adjusted, and the adjusted target image area can be superimposed on the candidate image area in the positive sample image to obtain the second negative sample image and the marking information of the second negative sample image. Among them, the morphology adjustment can include the adjustment of size, shape, etc. A position can be randomly selected in the candidate image area, and the adjusted target image area can be fused to the selected position of the candidate image area, so that the final second negative sample image presents a display effect in which the target object is integrated. The marking information of the second negative sample image is the position mark of the target image area in the second negative sample image.

[0072] It can be understood that, through the above process, by selecting a positive sample image and a negative sample image that has been collected, multiple new negative sample images can be generated. Then, by traversing many positive sample images and negative sample images that have been collected, a large number of new negative sample images can be generated. Finally, all positive sample images and negative sample images can be used to complete the training of the corresponding detection model.

[0073] Figure 2 The image detection scenario in the embodiment shown may be, for example, a medical image detection scenario. Figure 3 To illustrate the execution process of the sample generation method in the medical image detection scenario.

[0074] Figure 3 A flow chart of a sample generation method provided by an embodiment of the present invention, such as Figure 3 As shown, the method comprises the following steps:

[0075] 301. Acquire a first medical image corresponding to a medical image detection task, where the medical detection task is used to detect whether a set lesion occurs in a target body part, and the first medical image is a medical image indicating that the target body part is in a healthy state.

[0076] 302. Based on the hospital image detection task, determine a candidate image region corresponding to the target body part from the first medical image.

[0077] 303. Acquire a lesion image corresponding to the set lesion.

[0078] 304. Superimpose the lesion image onto the candidate image region in the first medical image to obtain a second medical image as a negative example and label information of the second medical image.

[0079] In the medical image detection scenario, due to the involvement of user privacy data, the number of samples with valid labels that can be used for model training is very scarce, especially for the detection of a specific disease, such as the detection of a specific tumor, which often only contains dozens of datasets with labeled detection areas. The size, shape, and position of tumor targets vary greatly, resulting in little difference between tumors and non-tumors. It is difficult for a small amount of labeled data to represent such a complex detection problem. Therefore, it is necessary to expand enough labeled sample data so that a model with good detection performance can be trained.

[0080] In this medical image detection scenario, the target task is to detect whether a target body part has a set lesion. The positive sample data is a medical image showing that the target body part is in a healthy state, which is called the first medical image. For example, the target body part is the lung, and the set lesion is lung cancer, i.e., a lung tumor. A large number of medical images of healthy lungs can be collected as positive samples.

[0081] That is to say, in the medical image detection scenario, using the sample generation method provided by the embodiment of the present invention, it is possible to generate image samples in which the designated marked areas belong to abnormal lesion areas based on normal and healthy image samples, thereby completing the medical image classification or detection task.

[0082] First, taking any first medical image selected from a number of medical images collected as positive samples as an example, the candidate image region corresponding to the target body part is first determined from the first medical image. In practical applications, the first medical image can be various medical images taken by imaging technology. In addition to the image region corresponding to the target body part, the first medical image taken also contains some other pixel regions, from which the image region corresponding to the target body part can be identified as the candidate image region. The image region corresponding to the target body part in the first medical image can be identified based on the contour features of the target body part. In addition, in practical applications, when the first medical image is obtained by photographing the target body part, the pixel brightness or color corresponding to the image region corresponding to the target body part in the first medical image will be clearly distinguished from other pixels (regarded as background). Therefore, the image region corresponding to the target body part in the first medical image, i.e., the candidate image region, can be identified based on the difference in pixel color and brightness.

[0083] Afterwards, a lesion image corresponding to the set lesion is obtained, and the obtained lesion image is superimposed on the candidate image region in the first medical image to obtain a second medical image as a negative example and labeling information of the second medical image.

[0084] In contrast to the healthy state of the target body part in the first medical image mentioned above, in actual applications, a small number of medical images indicating that the target body part has a set lesion can also be collected in advance, which are called third medical images. The third medical image is a negative sample. In order to generate more negative samples based on a large number of positive samples, based on the marking information of the lesion location in the third medical image that has been obtained and indicates that the target body part has a set lesion, the lesion area corresponding to the set lesion is extracted as the above-mentioned lesion image. Afterwards, the morphology of the lesion area can be adjusted, and the adjusted lesion area can be superimposed on the candidate image area identified in the first medical image to obtain a second medical image as a negative sample and the marking information of the second medical image. Among them, the marking information of the second medical image is the position mark of the lesion area in the second medical image.

[0085] It is understandable that each time the lesion area is adjusted in morphology, it can be fused into the candidate image area in the first medical image respectively, so that based on the deformation results of a first medical image and multiple lesion areas, multiple negative sample images can be obtained.

[0086] The generated second medical images as negative sample images and the generated first medical images as positive sample images can be used to train a model for completing the above-mentioned medical detection task.

[0087] In an optional embodiment, to improve the performance of the model, the method may further include:

[0088] Obtaining the user's attack behavior in a medical image detection task, where the attack behavior refers to a behavior that may affect the performance of the model;

[0089] Editing the first medical image and / or the second medical image according to the attack behavior;

[0090] The first medical image, the second medical image, and the edited first medical image and the second medical image are used to train a model corresponding to a medical image detection task.

[0091] In actual applications, the above-mentioned attack behaviors may be photocopying, transferring, compressing and transmitting. For example, when taking a photo of a medical image detected by a user to obtain the corresponding training sample image, the camera resolution will affect the clarity of the image, so the resolution of the medical image can be adjusted to imitate this behavior. For another example, a large number of medical images are stored in the terminal devices of a medical institution. When a server completes the model training task, the terminal device can upload the locally stored medical images to the server. During this process, the operator of the terminal device may compress the local medical images and upload them to the server, so the server can simulate the impact of compressed transmission on medical images (in fact, the medical images collected in the server may not all be obtained through this transmission method. Regardless of the collection method, this simulation effect can be performed on all medical images).

[0092] Figure 4 The above execution process is illustrated by taking the target body part as the lung area and setting the lesion as a tumor as an example.

[0093] Figure 5 A flow chart of a sample generation method provided by an embodiment of the present invention, such as Figure 5 As shown, the method comprises the following steps:

[0094] 501. Obtain a target task and positive sample data corresponding to the target task, wherein the target task is to detect whether video data contains forged speech and images, and the positive sample data is a positive sample video that does not contain forged speech and images.

[0095] 502. Randomly extract candidate speech segments and candidate image frame sequences from the positive sample video.

[0096] 503 . Synthesize a forged speech segment corresponding to the candidate speech segment and synthesize a forged image frame sequence corresponding to the candidate image frame sequence.

[0097] 504. Replace the candidate voice segment with a forged voice segment, and / or replace the candidate image frame sequence with a forged image frame sequence, so as to obtain a negative sample video containing the forged voice segment and / or the forged image frame sequence and labeling information of the negative sample video.

[0098] The sample generation solution provided by the embodiment of the present invention can also be applied to the problem of insufficient labeled training samples in multimodal mixed detection scenarios.

[0099] Multimodal classification scenarios, such as multimodal forged video detection scenarios of public figures. Criminals can use deepfake technology to synthesize fake speech videos of target public figures, triggering a social public opinion crisis and causing damage to individuals and companies. In actual forged video detection scenarios, forged videos may contain a mixture of one or both of the two modalities of forged voice and video frame images, and only certain key segments of the video are forged and synthesized. Most deepfake detection methods only focus on the detection of one of the modalities, and most data sets used to train forged video detection models also have only one modality. If you want to obtain a multimodal forged video detection model with better results, you need to build a data set containing mixed multimodal forged content and train the forged video detection model through multimodal fusion.

[0100] In order to generate more video samples containing multimodal forged information, first, a large number of original videos that do not contain forged information are prepared as positive sample videos. Then, the position of the speech segment and the position of the image frame segment to be synthesized are randomly selected in a positive sample video. The speech segment corresponding to the speech segment position in the positive sample video is called a candidate speech segment, and the multiple frames of images corresponding to the image frame segment position are called candidate image frame sequences. In fact, the candidate speech segment and the candidate image frame sequence do not require time alignment.

[0101] Afterwards, the deep fake technology is used to synthesize the speech content corresponding to the candidate speech segment, which is called a fake speech segment, and synthesize the image frame content corresponding to the candidate image frame sequence, which is called a fake image frame sequence. Afterwards, the synthesized fake speech segment and fake image frame sequence can be used to replace the above candidate speech segment and candidate image frame sequence in the positive sample video, so as to obtain a fake video that integrates the fake speech segment and the fake image frame sequence as a negative sample video. The marking information of the negative sample video can include marking information indicating whether it contains fake speech and fake images, as well as marking information of the corresponding positions of the fake speech and fake images.

[0102] The above execution process is Figure 6 In addition, Figure 6Three multimodal fusion methods are illustrated in the figure: one is a fake video containing original speech and a synthesized fake image frame sequence; the second is a fake video containing a synthesized fake speech clip and a fake image frame sequence; the third is a fake video containing a synthesized fake speech clip and an original image frame sequence.

[0103] The sample generation scheme provided by the embodiment of the present invention can also be applied to some audit scenarios involving security and evidence collection, such as the detection scenario of image theft.

[0104] In some practical applications, businesses can upload images for promotion and introduction to a service platform, which can then be output to users through the service platform. For example, businesses can upload store images and related introduction information to a map application platform, so that when users use the map application to search for nearby food, supermarkets, etc., they can see the introduction information of the relevant businesses and their uploaded images.

[0105] When merchants upload store and product images, the service platform (such as the above-mentioned map application platform) needs to review the uploaded images, and only images that pass the review can be published to the corresponding service program. An example of an audit task is to review whether the uploaded image is a stolen image, because in actual applications, the sample labels that are easier to obtain usually only have two categories: whether it is stolen or not. Therefore, a deep learning model can be trained based on the collected sample images marked with whether it is stolen or not. However, the service platform may also want to obtain more detailed reasons for the failure of the audit, such as the image to be tested may contain watermarks from other platforms, the image to be tested is a screenshot or a computer-generated effect, or the image to be tested is a copy of another user's image. These are the sub-category labels that the service platform wants to obtain for images that fail the audit, and the number of samples of these sub-category labels that can be obtained is scarce and it is difficult to have correct manual labels.

[0106] When the service platform needs to add a new detection category, the sample generation solution provided by the embodiment of the present invention can automatically generate a large number of labeled samples corresponding to the new needs of the service platform to retrain or enhance the training of the model.

[0107] That is to say, after a deep learning model is deployed, it is often possible to encounter the need for adding new sub-classification detection tasks. For example, in the platform image theft detection scenario, after the model is deployed, it may be necessary to obtain more detailed sub-classification detection items of the reasons for failure. These sub-classifications include that the image to be tested may contain other platform watermarks, the image to be tested is a screen capture image or a stolen splicing synthesis effect image, the image to be tested is a re-photographed stolen image, and so on. These newly added sub-classification requirements often account for a small proportion, with only a very small number of labeled samples, which cannot meet the requirements of model fine-tuning training. At this time, the solution provided by the embodiment of the present invention can be applied to perform sample data generation and processing on the newly added sub-classification tasks, so as to meet the actual landing needs and improve the actual landing performance of the deep model. The following is an exemplary description in conjunction with the following embodiments.

[0108] Figure 7a A flow chart of a sample generation method provided by an embodiment of the present invention, such as Figure 7a As shown, the method comprises the following steps:

[0109] 701a. Obtain a target task and positive sample data corresponding to the target task. The target task is a sub-classification task added under the set image classification task. The positive sample data is a positive sample image corresponding to the set image classification task. The sub-classification task includes identifying whether the image contains a set watermark.

[0110] 702a. Determine the background image region of the positive sample image.

[0111] 703a. Select a target watermark from a set of set watermarks, where the watermark stored in the watermark set is different from the set watermark.

[0112] 704a. Fusing the target watermark into the background image region of the positive sample image to obtain a negative sample image corresponding to the sub-classification task and labeling information of the negative sample image.

[0113] The set image classification task in this embodiment is, for example, the binary classification task of whether the image is stolen as mentioned above. When you want to know the specific theft category for the stolen image, you can add a corresponding sub-classification task. In this embodiment, it is assumed that the added sub-classification task includes a watermark theft classification task, that is, determining whether the watermark contained in the input image is the watermark of a certain set service platform.

[0114] When initially training the model to perform the above binary classification task (i.e., corresponding to the above-set image classification task), a large number of training sample images are collected. In this embodiment, the positive sample images therein are used as the positive sample images shared by each newly added sub-classification task.

[0115] For the subtask of watermark classification, assuming that the service platform currently used is service platform A, if the image uploaded by the merchant carries the watermark of other service platforms such as service platform B, then the image is considered to be a stolen image and cannot pass the review, and the reason for failure is: using watermarks from other service platforms.

[0116] In order to generate negative sample images corresponding to this subtask, first, the background image area of ​​the positive sample image is identified, that is, the image area that does not contain the foreground object. Figure 8 In the figure, the foreground object is the human body in the image, then the pixel area in the image except the pixel area corresponding to the human body is the background image area. In addition, some watermark images of other service platforms can be collected in advance to form a watermark set, and a watermark can be randomly selected from the watermark set as the current target watermark. After that, the target watermark is integrated into the background image area of ​​the positive sample image, which is equivalent to randomly inserting the target watermark into a certain position in the background image area, thereby obtaining a negative sample image. At the same time, the labeling information of the negative sample image is generated based on the semantics of the subtask. For example, if the semantic representation subtask needs to identify whether the input image contains watermarks of other service platforms, the labeling information can be: containing watermarks of other service platforms; if the semantic representation subtask needs to detect whether it contains watermarks of other service platforms and the watermark position, the labeling information can be: containing watermarks of other service platforms and the corresponding position information of the watermark in the negative sample image. The above processing process refers to Figure 8 .

[0117] Figure 7b A flow chart of a sample generation method provided by an embodiment of the present invention, such as Figure 7b As shown, the method comprises the following steps:

[0118] 701b. Obtain the target task and the positive sample data corresponding to the target task. The target task is a sub-classification task added under the set image classification task. The positive sample data is the positive sample image corresponding to the set image classification task. The sub-classification task includes identifying whether there is any screenshot theft in the image.

[0119] 702b. Extract foreground image regions of different sizes from the positive sample image.

[0120] 703b. Get the external border.

[0121] 704b. Adaptively merge the added borders to the edges of foreground image regions of different sizes to obtain negative example sample images corresponding to the sub-classification task and labeling information of the negative example sample images.

[0122] In this embodiment, it is assumed that the additional sub-classification task includes identifying whether there is a screenshot theft situation in the image, that is, determining whether the input image is a screenshot image.

[0123] For the screenshot theft identification subtask, it is necessary to reflect the impact of the "screenshot" operation on the image display effect in the generated negative sample images. Therefore, in order to generate negative sample images corresponding to this subtask, first, foreground image regions of different sizes are extracted from the positive sample images, because screenshots are often taken of foreground objects, such as Figure 8 As shown in , the foreground object is the human body in the figure, then different rectangular areas surrounding the human body can be captured as different foreground image areas. In addition, it is found in actual applications that users are often not so precise when taking screenshots, and many screenshots will have black and white or white borders. Therefore, some common border thicknesses can be obtained by counting a large number of screenshots. Based on this, a border thickness can be randomly selected to generate an additional border that matches the size of each foreground image area. Finally, each additional border is placed correspondingly to the edge of the corresponding foreground image area, that is, the foreground image area is built into the corresponding additional border to synthesize an image as the negative sample image under this subtask. At this time, the marking information of the negative sample image can be: there is a case of screenshot theft in the negative sample image. The above processing process refers to Figure 8 .

[0124] In summary, in order to solve the problem of insufficient training samples with label information in multiple types of scenarios such as user privacy scenarios, security audit scenarios, and multimodal mixed detection scenarios, an embodiment of the present invention proposes a small sample learning framework for semantic generation of corresponding negative sample data based on adaptive target tasks, which can facilitate the adaptive transformation of a large number of positive samples obtained to generate sample data of the target category, and can obtain sufficient sample data of the target category.

[0125] As mentioned above, the sample generation method provided by the present invention can be executed in the cloud, and several computing nodes can be deployed in the cloud, each of which has computing, storage and other processing resources. In the cloud, multiple computing nodes can be organized to provide a certain service. Of course, a computing node can also provide one or more services. The way to provide the service in the cloud can be to provide a service interface to the outside, and the user calls the service interface to use the corresponding service. The service interface includes software development kits (Software Development Kit, referred to as SDK), application programming interfaces (Application Programming Interface, referred to as API), and other forms.

[0126] For the solution provided in the embodiment of the present invention, the cloud can provide a service interface for sample generation service, and the user can call the service interface through the user device to trigger a call request to the cloud. The cloud determines the computing node that responds to the request and uses the processing resources in the computing node to execute the various steps provided in the above embodiment. For example Figure 4 Schematically, the user terminal triggers a request including a positive sample image and a target task to the cloud server cluster, and the cloud server responds to the request and executes a process of generating a negative sample image corresponding to the target task.

[0127] The sample generation device of one or more embodiments of the present invention will be described in detail below. Those skilled in the art will appreciate that these devices can be configured using commercially available hardware components through the steps taught in this solution.

[0128] Fig. 9 A schematic diagram of the structure of a sample generation device provided by an embodiment of the present invention is shown in FIG. Fig. 9 As shown, the device includes: an acquisition module 11, a determination module 12, and a fusion module 13.

[0129] The acquisition module 11 is used to acquire the target task and the positive sample data corresponding to the target task.

[0130] The determination module 12 is used to determine the candidate area corresponding to the target task from the positive sample data, and determine the modified content corresponding to the target task.

[0131] The fusion module 13 is used to fuse the modified content and the candidate region to obtain negative example sample data corresponding to the target task.

[0132] Optionally, the device also includes: a training module, used to obtain the user's attack behavior under the target task, the attack behavior refers to the behavior that will affect the model performance; edit the positive sample data and / or the negative sample data according to the attack behavior; use the positive sample data, the negative sample data, and the positive sample data and negative sample data after the editing process to train the model corresponding to the target task.

[0133] Optionally, the target task is to detect whether the image contains a target object, and the positive sample data is a positive sample image that does not contain the target object. Based on this, the determination module 12 is specifically used to: determine from the positive sample image a candidate image region for detecting the target object; obtain the first negative sample image that has been collected and corresponds to the target task; extract the target image region containing the target object according to the position marking information in the first negative sample image, and the position marking information is used to mark the position of the target object in the first negative sample image. The fusion module 13 is specifically used to: adjust the morphology of the target image region, and superimpose the adjusted target image region onto the candidate image region in the positive sample image to obtain a second negative sample image and marking information of the second negative sample image.

[0134] Optionally, the target task is to detect whether a set lesion occurs in a target body part, and the positive sample data is a first medical image indicating that the target body part is in a healthy state. Based on this, the determination module 12 is specifically used to: determine the candidate image area corresponding to the target body part from the first medical image; obtain the collected second medical image indicating that the target body part has the set lesion; and extract the lesion area corresponding to the set lesion according to the position mark information in the second medical image. The fusion module 13 is specifically used to: adjust the morphology of the lesion area, and superimpose the adjusted lesion area onto the candidate image area in the first medical image to obtain a third medical image as a negative sample and the mark information of the third medical image.

[0135] Optionally, the target task is to detect whether the video data contains forged speech and images; the positive sample data is a positive sample video that does not contain forged speech and images. Based on this, the determination module 12 is specifically used to: randomly extract candidate speech segments and candidate image frame sequences from the positive sample video; synthesize forged speech segments corresponding to the candidate speech segments and synthesize forged image frame sequences corresponding to the candidate image frame sequences. The fusion module 13 is specifically used to: replace the candidate speech segment with the forged speech segment, and / or replace the candidate image frame sequence with the forged image frame sequence, so as to obtain a negative sample video containing the forged speech segment and / or the forged image frame sequence and the labeling information of the negative sample video.

[0136] Optionally, the target task is a sub-classification task added under a set image classification task, and the positive sample data is a positive sample image corresponding to the set image classification task.

[0137] Wherein, optionally, the sub-classification task includes identifying whether the image contains a set watermark. Based on this, the determination module 12 is specifically used to: determine the background image area of ​​the positive sample image; select a target watermark from a set of watermarks, and the watermark stored in the watermark set is different from the set watermark. The fusion module 13 is specifically used to: fuse the target watermark into the background image area of ​​the positive sample image to obtain a negative sample image corresponding to the sub-classification task and the labeling information of the negative sample image.

[0138] Wherein, optionally, the sub-classification task includes identifying whether there is a screenshot theft situation in the image. Based on this, the determination module 12 is specifically used to: extract foreground image areas of different sizes from the positive sample image; obtain an external border. The fusion module 13 is specifically used to: adaptively fuse the external border to the edge of the foreground image area of ​​different sizes to obtain a negative sample image corresponding to the sub-classification task and the labeling information of the negative sample image.

[0139] Fig. 9 The device shown can execute the steps provided in the aforementioned embodiments. For detailed execution process and technical effects, please refer to the description in the aforementioned embodiments, which will not be repeated here.

[0140] In one possible design, the above Fig. 9 The structure of the sample generating device shown can be implemented as an electronic device. Fig.10 As shown, the electronic device may include: a processor 21, a memory 22, and a communication interface 23. The memory 22 stores executable codes, and when the executable codes are executed by the processor 21, the processor 21 can at least implement the sample generation method provided in the above embodiments.

[0141] In addition, an embodiment of the present invention provides a non-temporary machine-readable storage medium, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor can at least implement the sample generation method provided in the aforementioned embodiment.

[0142] The device embodiments described above are merely illustrative, wherein the network elements described as separate components may or may not be physically separated. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0143] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by adding a necessary general hardware platform, and of course can also be implemented by combining hardware and software. Based on such an understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a computer product, and the present invention can be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A sample generation method, characterized in that: include: Obtaining a target task and positive sample data corresponding to the target task; Determining a candidate region corresponding to the target task from the positive sample data, and determining a modified content corresponding to the target task; Fusion of the modified content and the candidate region to obtain negative sample data corresponding to the target task; The target task includes: detecting whether the image contains a target object, the positive sample data is a positive sample image that does not contain the target object; determining a candidate region corresponding to the target task from the positive sample data, and determining a modified content corresponding to the target task; fusing the modified content and the candidate region to obtain negative sample data corresponding to the target task, including: According to the preset candidate region acquisition prompt information corresponding to the target task, a candidate image region for detecting the target object is determined from the positive sample image, wherein the candidate region acquisition prompt information is used to describe the image region corresponding to the candidate image region in the positive sample image; Acquire a first negative example sample image that has been collected and corresponds to the target task; extracting a target image region containing the target object according to position marking information in the first negative example sample image, wherein the position marking information is used to mark a position of the target object in the first negative example sample image; The shape of the target image area is adjusted, and the adjusted target image area is superimposed on a randomly selected position in the candidate image area in the positive sample image to obtain a second negative sample image and labeling information of the second negative sample image.

2. The method according to claim 1, characterized in that The method further comprises: Obtaining the user's attack behavior under the target task, where the attack behavior refers to a behavior that may affect the model performance; Editing the positive sample data and / or the negative sample data according to the attack behavior; The positive sample data, the negative sample data, and the edited positive sample data and negative sample data are used to train a model corresponding to the target task.

3. The method according to claim 1, characterized in that The target task is to detect whether a set lesion occurs in a target body part, and the positive sample data is a first medical image indicating that the target body part is in a healthy state; The determining of the candidate region corresponding to the target task from the positive sample data and the determining of the modified content corresponding to the target task comprises: Determining a candidate image region corresponding to the target body part from the first medical image; Acquire a second medical image that has been collected and shows that the target body part has the set lesion; extracting a lesion area corresponding to the set lesion according to the position mark information in the second medical image; The fusing the modified content and the candidate region to obtain negative sample data corresponding to the target task includes: The morphology of the lesion area is adjusted, and the adjusted lesion area is superimposed on the candidate image area in the first medical image to obtain a third medical image as a negative example and labeling information of the third medical image.

4. The method according to claim 1, characterized in that: The target task is to detect whether the video data contains forged speech and images; the positive sample data is a positive sample video that does not contain forged speech and images; The determining of the candidate region corresponding to the target task from the positive sample data and the determination of the modified content corresponding to the target task include: Randomly extracting candidate speech segments and candidate image frame sequences from the positive sample video; synthesizing a forged speech segment corresponding to the candidate speech segment and synthesizing a forged image frame sequence corresponding to the candidate image frame sequence; The fusing the modified content and the candidate region to obtain negative sample data corresponding to the target task includes: The candidate voice segment is replaced by the forged voice segment, and / or the candidate image frame sequence is replaced by the forged image frame sequence to obtain a negative sample video containing the forged voice segment and / or the forged image frame sequence and labeling information of the negative sample video.

5. The method according to claim 1, characterized in that The target task is a sub-classification task added under the set image classification task, and the positive sample data is a positive sample image corresponding to the set image classification task.

6. The method according to claim 5, characterized in that The sub-classification task includes identifying whether the image contains a set watermark; The determining of the candidate region corresponding to the target task from the positive sample data and the determination of the modified content corresponding to the target task include: Determining a background image region of the positive sample image; Selecting a target watermark from a set of watermarks, wherein the watermark stored in the watermark set is different from the set watermark; The fusing the modified content and the candidate region to obtain negative sample data corresponding to the target task includes: The target watermark is merged into the background image region of the positive sample image to obtain a negative sample image corresponding to the sub-classification task and labeling information of the negative sample image.

7. The method according to claim 5, characterized in that The sub-classification task includes identifying whether there is a screenshot theft situation in the image; The determining of the candidate region corresponding to the target task from the positive sample data and the determination of the modified content corresponding to the target task include: Extracting foreground image regions of different sizes from the positive sample image; Get the external border; The fusing the modified content and the candidate region to obtain negative sample data corresponding to the target task includes: The added borders are adaptively fused to the edges of the foreground image regions of different sizes respectively, so as to obtain negative example sample images corresponding to the sub-classification task and labeling information of the negative example sample images.

8. A sample generating device, characterized in that: include: An acquisition module, used to acquire a target task and positive sample data corresponding to the target task; A determination module, used to determine a candidate region corresponding to the target task from the positive sample data, and determine a modified content corresponding to the target task; A fusion module, used to fuse the modified content and the candidate region to obtain negative sample data corresponding to the target task; Wherein, the target task includes: detecting whether the image contains the target object, the positive sample data is a positive sample image that does not contain the target object; the determination module is specifically used to obtain prompt information according to a preset candidate area corresponding to the target task, and determine the candidate image area containing the target object for detecting the target object from the positive sample image, and the candidate area acquisition prompt information is used to describe the image area corresponding to the candidate image area in the positive sample image; obtaining the first negative sample image corresponding to the target task that has been collected; extracting the target image area containing the target object according to the position mark information in the first negative sample image, and the position mark information is used to mark the position of the target object in the first negative sample image; The fusion module is specifically used to adjust the shape of the target image area, and superimpose the adjusted target image area on a randomly selected position in the candidate image area in the positive sample image to obtain a second negative sample image and marking information of the second negative sample image.

9. An electronic device, characterized in that: include: A memory, a processor, and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor executes the sample generation method according to any one of claims 1 to 7.

10. A non-transitory machine-readable storage medium, characterized in that: The non-transitory machine-readable storage medium stores executable code, and when the executable code is executed by a processor of an electronic device, the processor is caused to execute the sample generation method according to any one of claims 1 to 7.

11. A sample generation method, characterized in that: include: Acquire a first medical image corresponding to a medical image detection task, wherein the medical image detection task is used to detect whether a set lesion occurs in a target body part, and the first medical image is a medical image indicating that the target body part is in a healthy state; Based on preset candidate region acquisition prompt information corresponding to the medical image detection task, determining a candidate image region corresponding to the target body part from the first medical image, the candidate region acquisition prompt information being used to describe an image region corresponding to the candidate image region in the first medical image; Acquire a third medical image that has been collected and shows that the target body part has the set lesion; extracting a lesion area corresponding to the set lesion from the third medical image according to the position mark information in the third medical image; Adjusting the morphology of the lesion area, and superimposing the adjusted lesion area on a randomly selected position in the candidate image area in the first medical image, so as to obtain a second medical image as a negative sample and labeling information of the second medical image; Obtaining the user's attack behavior in the medical image detection task, where the attack behavior refers to a behavior that may affect the performance of the model; Editing the first medical image and / or the second medical image according to the attack behavior; The first medical image, the second medical image, and the first medical image and the second medical image after the editing process are used to train a model corresponding to the medical image detection task.

Images