Difficult sample mining method, device, equipment and computer-readable storage medium

Abstract

This application discloses a difficult sample mining method, apparatus, device, and computer-readable storage medium, belonging to the field of computer vision technology. The difficult sample mining method comprises: inputting the image to be mined multiple times into a stable diffusion model to obtain sets of first mask images; inputting each set of first mask images into a target diffusion model to obtain first predicted mask images corresponding to each set of first mask images; and determining that the image to be mined is a difficult sample when the overall similarity of each first predicted mask image is less than a preset threshold. This application provides a difficult sample mining strategy that improves the efficiency of difficult sample mining.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a difficult sample mining method, apparatus, device, and computer-readable storage medium. Background Art

[0002] With the continuous development of technology, machine learning models have been widely used in various industries.

[0003] At present, in order to improve the performance of machine learning models for image processing, more attention needs to be paid to difficult samples when training machine learning models to help them better learn difficult-to-predict situations. However, the traditional way of mining difficult samples is usually that after the machine learning model has been trained on large-scale data (that is, after learning complex feature representations), if the prediction result output by the machine learning model based on the current input sample is inaccurate during the model training process, the current sample will be regarded as a difficult sample, resulting in low efficiency in mining difficult samples.

[0004] In summary, how to provide a difficult sample mining strategy to improve the efficiency of difficult sample mining has become a technical problem that needs to be solved urgently in the field of computer vision technology. Summary of the Invention

[0005] The main purpose of this application is to provide a difficult sample mining method, apparatus, device, and computer-readable storage medium, and to provide a difficult sample mining strategy to improve the efficiency of difficult sample mining.

[0006] To achieve the above objectives, the present application provides a difficult sample mining method, which comprises the following steps:

[0007] Inputting the image to be mined multiple times into the stable diffusion model to obtain each first mask image set, wherein the first mask image set includes multiple mask images obtained after inputting the image to be mined multiple times into the stable diffusion model;

[0008] Inputting each of the first mask image sets into a target diffusion model to obtain a first predicted mask image corresponding to each of the first mask image sets, wherein the target diffusion model is trained using a second mask image set corresponding to the training sample image as model input data and using the first target mask image with the highest quality score in the second mask image set as the model training label, and the quality score of the mask image is obtained based on the intersection-over-union ratio between the mask image and the true mask image of the training sample image;

[0009] When the overall similarity of each of the first predicted mask images is less than a preset threshold, the image to be mined is determined to be a difficult sample.

[0010] Optionally, the method further includes:

[0011] Acquire a training sample set, wherein the training sample set includes a plurality of training sample data, and each training sample data includes a second mask image set corresponding to a training sample image, and a first target mask image with a highest quality score in the second mask image set;

[0012] The second mask image set in each piece of training sample data is used as model input data, and the first target mask image in each piece of training sample data is used as a model training label. The preset initial diffusion model is trained to obtain the target diffusion model.

[0013] Optionally, the step of obtaining a training sample set includes:

[0014] Inputting the training sample image into the stable diffusion model to obtain the second mask image set;

[0015] calculating a quality score of each mask image in the second mask image set, and determining the mask image with the highest quality score in the second mask image set as the first target mask image;

[0016] A piece of the training sample data is determined based on the second mask image set and the first target mask image, and the training sample set is constructed based on multiple pieces of the training sample data.

[0017] Optionally, the step of calculating the quality score of each mask image in the second mask image set includes:

[0018] Obtaining a true mask image of the training sample image;

[0019] Calculating a first intersection-over-union (IoU) between the true mask image and a first sample mask image, wherein the first sample mask image is any one mask image in the second mask image set;

[0020] The first intersection-over-union ratio is determined as a quality score of the first sample mask image.

[0021] Optionally, the step of training a preset initial diffusion model to obtain the target diffusion model includes:

[0022] Inputting the model input data into a preset initial diffusion model to obtain a second predicted mask image;

[0023] Calculating a loss between the second predicted mask image and the model training label;

[0024] The model parameters of the initial diffusion model are optimized based on the loss to obtain a target diffusion model.

[0025] Optionally, before the step of determining that the image to be mined is a difficult sample when the overall similarity of each of the first predicted mask images is less than a preset threshold, the method further includes:

[0026] respectively calculating a second intersection-over-union ratio between any two mask images in each of the first predicted mask images;

[0027] An average value of the second intersection-over-union ratios is determined as the overall similarity of the first prediction mask images.

[0028] Optionally, the method further includes:

[0029] Acquire a verification sample set, wherein the verification sample set includes multiple verification sample data, each verification sample data includes a third mask image set corresponding to a verification sample image, and a second target mask image with a highest quality score in the third mask image set;

[0030] Inputting the third mask image sets in each piece of the verification sample data into the target diffusion model respectively to obtain each third predicted mask image;

[0031] respectively calculating a third intersection-over-union ratio between the second target mask image and the third predicted mask image corresponding to each of the verification sample data;

[0032] The preset threshold is determined based on an average value of the third intersection-over-union ratios.

[0033] In addition, to achieve the above-mentioned purpose, the present application also provides a difficult sample mining device, which comprises the following steps:

[0034] a first model calculation module, configured to input the image to be mined multiple times into the stable diffusion model to obtain first mask image sets, wherein the first mask image set includes multiple mask images obtained by inputting the image to be mined multiple times into the stable diffusion model;

[0035] a second model calculation module, configured to input each of the first mask image sets into a target diffusion model to obtain a first predicted mask image corresponding to each of the first mask image sets, wherein the target diffusion model is trained using the second mask image set corresponding to the training sample image as model input data and the first target mask image with the highest quality score in the second mask image set as the model training label, and the quality score of the mask image is obtained based on the intersection-over-union ratio between the mask image and the true mask image of the training sample image;

[0036] The sample determination module is configured to determine that the image to be mined is a difficult sample when the overall similarity of each of the first predicted mask images is less than a preset threshold.

[0037] In addition, to achieve the above-mentioned purpose, the present application also provides a difficult sample mining device, which includes: a memory, a processor, and a difficult sample mining program stored on the memory and runnable on the processor. When the difficult sample mining program of the difficult sample mining device is executed by the processor, the steps of the difficult sample mining method described above are implemented.

[0038] In addition, to achieve the above objectives, the present application also provides a computer-readable storage medium, on which a difficult sample mining program is stored. When the difficult sample mining program is executed by a processor, the steps of the difficult sample mining method described above are implemented.

[0039] In an embodiment of the present application, a plurality of first mask image sets are obtained by inputting an image to be mined into a stable diffusion model multiple times, wherein the first mask image sets include a plurality of mask images obtained after inputting the image to be mined into the stable diffusion model, and each first mask image set is input into a target diffusion model respectively to obtain a mask image in each first mask image set predicted by the target diffusion model with the highest overall similarity to the true mask image of the image to be mined, i.e., a first predicted mask image. The target diffusion model is trained using a second mask image set corresponding to the training sample image as model input data and a mask image with the highest quality score in the second mask image set (i.e., the first target mask image) as a model training label. The quality score of the mask image is obtained based on the intersection-and-union ratio between the mask image and the true mask image of the training sample image. Then, when the overall similarity of each first predicted mask image is less than a preset threshold, the image to be mined is determined to be a difficult sample.

[0040] In this way, the present application utilizes the stable diffusion model to randomly generate multiple mask images based on the characteristics of the input image, thereby obtaining multiple mask images of the image to be mined. The image to be mined is then input into the stable diffusion model multiple times to obtain multiple sets of mask images. These sets of mask images are used as the basis for determining whether the image to be mined is a difficult sample, thereby avoiding the influence of the randomness of the mask images on the determination result of the image to be mined. The multiple mask images are then processed simultaneously based on the pre-trained target diffusion model to obtain the predicted mask image in each set of mask images predicted by the target diffusion model that is closest to the true value. Since the stable diffusion model is known to accurately identify pixel features in simple samples, the generated mask images are relatively accurate, that is, the overall similarity of the mask images is high. Therefore, when the overall similarity of each predicted mask image is less than a preset threshold, the overall similarity of each set of first mask images is determined to be low, thereby determining that the image to be mined is a difficult sample, i.e., mining of difficult samples is achieved. Therefore, compared to traditional mining methods that can only discover difficult samples during model training, the present application uses the stable diffusion model and the target diffusion model to mine difficult samples, thereby improving the efficiency of difficult sample mining. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 Schematic diagram of the device structure of the hardware operating environment of the difficult sample mining device involved in the embodiment of the present application;

[0042] Figure 2 This is a flowchart of the steps of the first embodiment of the difficult sample mining method of this application;

[0043] Figure 3 This is a schematic diagram of the combined model training process involved in one embodiment of the difficult sample mining method of this application;

[0044] Figure 4 This is a schematic diagram of the combined model application process involved in one embodiment of the difficult sample mining method of this application;

[0045] Figure 5 This is a schematic diagram of the functional modules of an embodiment of the difficult sample mining device of the present application.

[0046] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0047] It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application.

[0048] Reference Figure 1 , Figure 1 This is a schematic diagram of the device structure of the hardware operating environment of the difficult sample mining device involved in the embodiment of the present application.

[0049] It should be noted that the embodiments of the present application relate to a difficult sample mining device in the field of computer vision technology. Specifically, the difficult sample mining device can be a smartphone, a PC (personal computer), a tablet computer, a portable computer, etc.

[0050] like Figure 1 As shown, the difficult sample mining device may include: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may optionally be a storage device independent of the aforementioned processor 1001. For the convenience of description below, the execution body of the method steps in each embodiment is omitted.

[0051] Those skilled in the art will understand that Figure 1 The structure of the difficult sample mining device shown in the figure does not constitute a limitation to the difficult sample mining device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0052] like Figure 1 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a difficult sample mining program.

[0053] exist Figure 1 In the terminal shown, the network interface 1004 is mainly used to connect to the backend server and communicate data with the backend server; the user interface 1003 is mainly used to connect to the client and communicate data with the client; and the processor 1001 can be used to call the difficult sample mining program stored in the memory 1005 and perform the following operations:

[0054] Inputting the image to be mined multiple times into the stable diffusion model to obtain each first mask image set, wherein the first mask image set includes multiple mask images obtained after inputting the image to be mined multiple times into the stable diffusion model;

[0055] Inputting each of the first mask image sets into a target diffusion model to obtain a first predicted mask image corresponding to each of the first mask image sets, wherein the target diffusion model is trained using a second mask image set corresponding to the training sample image as model input data and using the first target mask image with the highest quality score in the second mask image set as the model training label, and the quality score of the mask image is obtained based on the intersection-over-union ratio between the mask image and the true mask image of the training sample image;

[0056] When the overall similarity of each of the first predicted mask images is less than a preset threshold, the image to be mined is determined to be a difficult sample.

[0057] Furthermore, the processor 1001 may also be configured to call a difficult sample mining program stored in the memory 1005 and perform the following operations:

[0058] Acquire a training sample set, wherein the training sample set includes a plurality of training sample data, and each training sample data includes a second mask image set corresponding to a training sample image, and a first target mask image with a highest quality score in the second mask image set;

[0059] The second mask image set in each piece of training sample data is used as model input data, and the first target mask image in each piece of training sample data is used as a model training label. The preset initial diffusion model is trained to obtain the target diffusion model.

[0060] Furthermore, the operation of obtaining a training sample set includes:

[0061] Inputting the training sample image into the stable diffusion model to obtain the second mask image set;

[0062] calculating a quality score of each mask image in the second mask image set, and determining the mask image with the highest quality score in the second mask image set as the first target mask image;

[0063] A piece of the training sample data is determined based on the second mask image set and the first target mask image, and the training sample set is constructed based on multiple pieces of the training sample data.

[0064] Furthermore, the operation of calculating the quality score of each mask image in the second mask image set includes:

[0065] Obtaining a true mask image of the training sample image;

[0066] Calculating a first intersection-over-union (IoU) between the true mask image and a first sample mask image, wherein the first sample mask image is any one mask image in the second mask image set;

[0067] The first intersection-over-union ratio is determined as a quality score of the first sample mask image.

[0068] Furthermore, the operation of training the preset initial diffusion model to obtain the target diffusion model includes:

[0069] Inputting the model input data into a preset initial diffusion model to obtain a second predicted mask image;

[0070] Calculating a loss between the second predicted mask image and the model training label;

[0071] The model parameters of the initial diffusion model are optimized based on the loss to obtain a target diffusion model.

[0072] Furthermore, before the step of determining that the image to be mined is a difficult sample when the overall similarity of each of the first predicted mask images is less than a preset threshold, the processor 1001 may also be configured to call a difficult sample mining program stored in the memory 1005 and perform the following operations:

[0073] respectively calculating a second intersection-over-union ratio between any two mask images in each of the first predicted mask images;

[0074] An average value of the second intersection-over-union ratios is determined as the overall similarity of the first prediction mask images.

[0075] Furthermore, the processor 1001 may also be configured to call a difficult sample mining program stored in the memory 1005 and perform the following operations:

[0076] Acquire a verification sample set, wherein the verification sample set includes multiple verification sample data, each verification sample data includes a third mask image set corresponding to a verification sample image, and a second target mask image with a highest quality score in the third mask image set;

[0077] Inputting the third mask image sets in each piece of the verification sample data into the target diffusion model respectively to obtain each third predicted mask image;

[0078] respectively calculating a third intersection-over-union ratio between the second target mask image and the third predicted mask image corresponding to each of the verification sample data;

[0079] The preset threshold is determined based on an average value of the third intersection-over-union ratios.

[0080] Based on the above structure, various embodiments of the difficult sample mining method are proposed.

[0081] Please refer to Figure 2 , Figure 2 This is a flow chart of the first embodiment of the difficult sample mining method of the present application. It should be noted that although the logical order is shown in the flow chart, in some cases, the difficult sample mining method of the present application can of course also perform the steps shown or described in a different order than here. In this embodiment, the execution subject of the difficult sample mining method can be a personal computer, a smart phone or other device, which is not limited in this embodiment. For the convenience of description below, the execution subject of the method steps in each embodiment is omitted. The difficult sample mining method includes steps S10 to S30:

[0082] Step S10 : Inputting the image to be mined multiple times into the stable diffusion model to obtain first mask image sets, wherein the first mask image set includes multiple mask images obtained after inputting the image to be mined multiple times into the stable diffusion model.

[0083] Currently, several approaches are available for hard sample mining based on large models: Deep neural network-based methods use deep neural networks to learn feature representations for image, video, or text data and use these features to identify hard samples. This approach typically combines deep neural networks with other techniques, such as data augmentation. Generative adversarial network (GAN)-based methods are deep learning models that can generate realistic samples or identify real samples by training two neural networks. In hard sample mining, GANs can be used to generate samples similar to hard samples and use them for model training. Transfer learning-based methods apply an already trained model to a new task or dataset. In hard sample mining, useful features can be extracted from an already trained model and used to identify hard samples. Data augmentation-based methods generate more data by modifying existing data. Active learning-based methods use a small number of labeled samples to train a model and then use the model to select the most valuable samples for labeling. In difficult sample mining, active learning can be used to select the most valuable difficult samples for labeling, thereby improving model performance. However, the aforementioned methods suffer from the drawback of consuming significant computational resources and time. Furthermore, the most commonly used difficult sample mining method currently identifies a sample as difficult when, during model training, the prediction result for the current sample is found to be inaccurate—that is, when the model cannot classify the current sample. This results in low mining efficiency.

[0084] The Stable Diffusion (stable diffusion model) algorithm can divide an image into multiple regions based on the similarity between pixels and identify those regions with similar pixel features, that is, the algorithm has the function of image segmentation. It should be noted that the stable diffusion model in this application is used for image segmentation, and the number of generated mask images can be controlled by setting the iteration parameters of the stable diffusion model. The number of mask images generated by the stable diffusion model is the currently set number of iterations, wherein the mask image is the semantic segmentation image. Specifically, the n_iter (iteration, number of iterations) parameter value in the stable diffusion model is set according to actual needs to randomly generate intermediate states of different degrees, wherein the n_iter parameter value can be set to 100, 200 or 400, etc. When the n_iter parameter value is set to 100, the stable diffusion model can generate 100 mask images based on an input image, that is, a mask image (mask) of the intermediate state.

[0085] The image to be mined represents any image that needs to be judged as a "difficult sample." Each time the image to be mined is fed into the stable diffusion model, it generates multiple mask images based on the image to be mined. These multiple mask images generated each time are considered a mask image set (hereinafter referred to as the first mask image set for clarity). Therefore, by feeding the image to be mined multiple times into the stable diffusion model, multiple first mask image sets can be obtained.

[0086] In a specific implementation, the n_iter parameter value is 10. Then, each time the image to be mined is input into the stable diffusion model, 10 intermediate state mask images can be obtained. These 10 mask images are used as a first mask image set. When the image to be mined is input into the stable diffusion model 100 times, 100 first mask image sets containing 10 mask images can be obtained.

[0087] It should be noted that since the mask images generated by the stable diffusion model for semantic segmentation of images are random, for relatively simple samples, the difference between the mask images generated by the stable diffusion model for semantic segmentation of images in each iteration will be small. However, for difficult samples, the mask images generated by the stable diffusion model in each iteration will differ greatly. This is mainly because when faced with difficult samples, the stable diffusion model cannot correctly identify the pixel range with the same semantics in the image.

[0088] Step S20: Input each of the first mask image sets into a target diffusion model to obtain a first predicted mask image corresponding to each of the first mask image sets. The target diffusion model is trained using a second mask image set corresponding to the training sample image as model input data and the first target mask image with the highest quality score in the second mask image set as the model training label. The quality score of the mask image is obtained based on the intersection-over-union ratio between the mask image and the true mask image of the training sample image.

[0089] The initial diffusion model is pre-trained to obtain a target diffusion model. Specifically, a training sample set is constructed based on multiple training sample images and the true mask image corresponding to each training sample image. This training sample set serves as the training sample set for the combined model, where the combined model is composed of a stable diffusion model and a diffusion model. The training sample images in the second training sample set are then input into the stable diffusion model in the combined model. Multiple mask images are generated using the stable diffusion model to obtain a set of mask images corresponding to the training sample images (hereinafter referred to as the second mask image set for distinction). The second mask image set corresponding to each training sample image serves as the model input data for the initial diffusion model. The intersection over union (IOU) is calculated between each mask image in the second mask image set and the true mask image to obtain the IOU (intersection over union) corresponding to each mask image. The IOU of the mask image is used as the quality score of the mask image. The mask image with the highest quality score in the second mask image set (hereinafter referred to as the first target mask image for distinction) serves as the model training label for the initial diffusion model. The initial diffusion model is trained based on the model input data and the model training labels to obtain the target diffusion model.

[0090] Each first mask image set output by the stable diffusion model is input into the trained target diffusion model to obtain the mask image closest to the true mask image in the first mask image set predicted by the target diffusion model (hereinafter referred to as the first predicted mask image for distinction).

[0091] For example, Figure 3The figure shows a schematic diagram of the combined model training process, in which the training sample image is input into the stable diffusion model, wherein the stable diffusion model includes an encoder (Encoder) and a decoder (Decoder). The stable diffusion model generates multiple mask outputs based on the training sample image, and the multiple mask outputs serve as inputs to the diffusion model (i.e., the initial diffusion model). At the same time, after obtaining the multiple mask outputs, the quality score of each mask is manually calculated. It should be noted that the quality score calculation step can be performed manually or by an execution entity integrated with a code program for calculating the quality score. The diffusion model then predicts the quality score of each mask input and outputs the mask with the highest quality score among the masks. It should be noted that after obtaining the mask output (i.e., the predicted quality score), the mask output and the actual quality score can be used to optimize and correct the diffusion model, or the mask with the highest quality score output by the diffusion model and the actual highest quality score can be directly used to optimize and correct the diffusion model. This application does not impose any restrictions on this.

[0092] After obtaining the target diffusion model, during the inference phase, for a new stable diffusion intermediate state sequence, the image mask output through stable diffusion is fed into the trained diffusion model. The diffusion model predicts and outputs the quality score for the intermediate output mask. The intermediate state mask with the highest score is the final result.

[0093] Step S30: When the overall similarity of each of the first predicted mask images is less than a preset threshold, determining that the image to be mined is a difficult sample.

[0094] A similarity threshold, i.e., the aforementioned preset threshold, is determined in advance based on actual needs. The overall similarity between each first predicted mask image is calculated. If the overall similarity between each first predicted mask image is less than the preset threshold, it is determined that the overall similarity of the set of first mask images is not within the required range, and the current image to be mined can be determined as a difficult sample.

[0095] In a feasible implementation, other parameters for calculating the similarity between two semantically segmented images may be introduced, such as Jaccard similarity, average Hamming distance, and structural similarity index, etc. This application does not limit the specific calculation method of the overall similarity.

[0096] In one feasible implementation, before step S30, the difficult sample mining method further includes steps A10 to A20:

[0097] Step A10 : calculating the second intersection-over-union ratios between any two mask images in each of the first predicted mask images.

[0098] The intersection-over-union (IoU) between each pair of mask images in each first prediction mask image (hereinafter referred to as the second IoU for distinction) is calculated respectively.

[0099] Step A20: Determine the average value of each of the second intersection-over-union ratios as the overall similarity of each of the first predicted mask images.

[0100] The average value of each second intersection-over-union ratio is calculated and used as the overall similarity between each first prediction mask image. The average value represents the overall similarity of each first mask image set.

[0101] In one feasible implementation, the intersection-and-union (IoU) between any two mask images in each first predicted mask image is calculated respectively, where the IoU value ranges from 0 to 1, where 0 indicates that the two mask images have no overlapping pixel ranges, and 1 indicates that the pixel ranges of the two mask images completely overlap. Then, the average value of each IoU is calculated. When the average value is less than a preset threshold, the current image to be mined is determined to be a difficult sample.

[0102] Exemplarily, after inputting the image to be mined into the stable diffusion model 100 times, 100 predicted mask images, i.e., 100 first predicted mask images, are obtained. Then, the IOU values ​​between all mask image pairs in the 100 first predicted mask images are calculated, and the average of each IOU value is calculated. When the average value is less than a preset threshold, it is determined that the current image to be mined is a difficult sample.

[0103] In one feasible implementation, the difficult sample mining method further includes steps B10 to B40:

[0104] Step B10: Obtain a verification sample set, wherein the verification sample set includes multiple verification sample data, and one verification sample data includes a third mask image set corresponding to a verification sample image, and a second target mask image with the highest quality score in the third mask image set.

[0105] The above-mentioned verification sample set includes multiple verification sample data, and one verification sample data includes a mask image set corresponding to the same verification sample image (hereinafter referred to as the third mask image set for distinction), and the target mask image with the highest quality score in the third mask image set (hereinafter referred to as the second target mask image for distinction). It should be noted that when the stable diffusion model is regarded as a module for preprocessing the input data input to the diffusion model (the output data here is the training sample image, the verification sample image or the image to be mined), the above-mentioned verification sample set containing the third mask image set and the second target mask image is regarded as the verification sample set of the diffusion model. The verification sample set of the diffusion model (i.e., the target diffusion model) in this case is referred to as the first verification sample set for distinction; when the stable diffusion model and the diffusion model are regarded as a combined model as a whole, the verification sample set of the combined model (hereinafter referred to as the second verification sample set for distinction) includes the verification sample image and the real semantic segmentation image of the verification sample image.

[0106] In a specific embodiment, the verification sample image is input into the stable diffusion model to obtain a third mask image set, and then based on the true semantic segmentation image corresponding to the verification sample image, the intersection-over-union ratio between each mask image in the third mask image set and the true semantic segmentation image is calculated, and the intersection-over-union ratio is determined as the quality score of the mask image. Then, the mask image with the highest quality score among the mask images is determined as the second target mask image. Finally, the first verification sample set is constructed based on the third mask image set and the second target mask image.

[0107] Step B20: Input the third mask image sets in each piece of the verification sample data into the target diffusion model to obtain each third predicted mask image.

[0108] The third mask image sets in each verification sample data in the verification sample set are respectively input into the target diffusion model, and the target diffusion model is used to generate the corresponding predicted mask images of each third mask image set (hereinafter referred to as third predicted mask images for distinction).

[0109] Step B30 , respectively calculating a third intersection-over-union (IoU) between the second target mask image and the third predicted mask image corresponding to each piece of the verification sample data.

[0110] The intersection-over-union (IoU) between the second target mask image and the third predicted mask image corresponding to each verification sample data is calculated respectively (hereinafter referred to as the third IoU for distinction), that is, the similarity between each prediction result output by the target diffusion model and the true value is calculated.

[0111] Step B40: Determine the preset threshold based on the average value of each of the third intersection-over-union ratios.

[0112] Calculate the average value of each third intersection-over-union ratio, and subtract the preset value 0.1 from the average value to obtain the above-mentioned preset threshold value, wherein the present application does not limit the size of the above-mentioned preset value. It should be noted that the average value of each third intersection-over-union ratio represents the overall similarity of each third mask image set, and can also characterize the degree of mastery of complex features by the current target diffusion model, that is, the ability to recognize complex features. Therefore, when the overall similarity of each first mask image set corresponding to the image to be mined is less than the preset threshold, it means that the recognition ability of the target diffusion model cannot recognize the complex features in the current image to be mined, so the current image to be mined can be determined to be a difficult sample. In other words, when the average IOU of 100 times is 0.1 lower than the average IOU of the verification sample set, it indicates that the model as a whole has difficulty in locating the specific boundaries of the target in the image well, and it is difficult to fit the target well. There are certain differences in the mask results output multiple times. This image is a difficult sample.

[0113] For example, after obtaining the target diffusion model, in the actual application of the combined model, such as Figure 4 The schematic diagram of the combined model application process shown in the figure first inputs the image to be mined into the stable diffusion model multiple times to obtain multiple mask image sets, and then uses each mask image set as the input of the target diffusion model. The target diffusion model outputs the predicted mask images corresponding to each mask image set based on the input, calculates the IOU values ​​between all mask image pairs in the multiple predicted mask images, and calculates the average of each IOU value to determine whether the average value is less than the preset threshold. If so, the current image to be mined is determined to be a difficult sample.

[0114] The present application obtains multiple first mask image sets by inputting the image to be mined into a stable diffusion model multiple times, wherein the first mask image set includes multiple mask images obtained after inputting the image to be mined into the stable diffusion model, and each first mask image set is input into a target diffusion model respectively to obtain the mask image in each first mask image set predicted by the target diffusion model with the highest overall similarity to the true mask image of the image to be mined, i.e., the first predicted mask image, wherein the target diffusion model is trained using the second mask image set corresponding to the training sample image as model input data and the mask image with the highest quality score in the second mask image set (i.e., the first target mask image) as the model training label, and the quality score of the mask image is obtained based on the intersection-and-union ratio between the mask image and the true mask image of the training sample image, and then when the overall similarity of each first predicted mask image is less than a preset threshold, the image to be mined is determined to be a difficult sample.

[0115] In this way, the present application utilizes the stable diffusion model to randomly generate multiple mask images based on the characteristics of the input image, obtains multiple mask images of the image to be mined, and uses the multiple mask images as the basis for determining whether the image to be mined is a difficult sample, thereby avoiding the influence of the randomness of the mask image on the determination result of the image to be mined. Inputting the image to be mined into the stable diffusion model multiple times to obtain multiple groups of mask images further reduces the influence of the randomness of the mask image on the determination result of the image to be mined, and then processing the multiple mask images at the same time based on the pre-trained target diffusion model to obtain the target diffusion model predicted in each group of mask images. The predicted mask image closest to the true value, since the known stable diffusion model can accurately identify the pixel features in simple samples, the generated mask images are relatively accurate, that is, the overall similarity of the mask images is high, so when the overall similarity of each predicted mask image is less than the preset threshold, it is determined that the overall similarity of each first mask image set is low, so that the image to be mined is determined to be a difficult sample, that is, the mining of difficult samples is realized. Therefore, compared with the traditional mining method that can only discover difficult samples during the model training process, this application uses a stable diffusion model and a target diffusion model to realize the mining of difficult samples, thereby improving the efficiency of mining difficult samples.

[0116] Furthermore, based on the first embodiment of the difficult sample mining method of the present application, a second embodiment of the difficult sample mining method of the present application is proposed.

[0117] In this embodiment, the difficult sample mining method further includes steps C10 to C20:

[0118] Step C10: Acquire a training sample set, wherein the training sample set includes multiple training sample data, and one training sample data includes a second mask image set corresponding to a training sample image, and a first target mask image with the highest quality score in the second mask image set.

[0119] The above-mentioned training sample set includes multiple training sample data, and one training sample data includes a mask image set corresponding to the same training sample image (hereinafter referred to as the second mask image set for distinction), and the target mask image with the highest quality score in the second mask image set (hereinafter referred to as the first target mask image for distinction). It should be noted that when the stable diffusion model is regarded as a module for preprocessing the input data input to the diffusion model (the output data here is the training sample image, verification sample image or image to be mined), the above-mentioned training sample set including the second mask image set and the first target mask image is regarded as the training sample set of the diffusion model. The training sample set of the diffusion model (i.e., the target diffusion model) in this case is hereinafter referred to as the first training sample set for distinction; when the stable diffusion model and the diffusion model are regarded as a combined model as a whole, the training sample set of the combined model (hereinafter referred to as the second training sample set for distinction) includes the training sample image and the true mask image of the training sample image.

[0120] In step C20 , the second mask image set in each piece of training sample data is used as model input data, and the first target mask image in each piece of training sample data is used as a model training label to train the preset initial diffusion model to obtain the target diffusion model.

[0121] The second mask image set in each training sample data is used as the model input data, and the first target mask image in each training sample data is used as the model training label. Training is performed based on the model input data and the model training label to obtain a trained target diffusion model.

[0122] In one feasible implementation, step C10 includes steps C101 to C102:

[0123] Step C101: input the training sample image into the stable diffusion model to obtain the second mask image set.

[0124] The training sample images in the training sample set are input into the stable diffusion model, and the second mask image set is output through the stable diffusion model.

[0125] Step C102 : calculating the quality score of each mask image in the second mask image set, and determining the mask image with the highest quality score in the second mask image set as the first target mask image.

[0126] The quality score represents the overall similarity between the mask image output by the stable diffusion model and the true mask image. When constructing the training sample set of the initial diffusion model, the training sample image needs to be input into the stable diffusion model, and the quality score of each mask image in the second mask image set output by the stable diffusion model is calculated. It should be noted that the present application does not limit the method of calculating the quality score. For example, in one feasible implementation, other parameters for calculating the overall similarity of two semantic segmentation images are introduced, such as Jaccard similarity, average Hamming distance, and structural similarity index, etc., so as to calculate the similarity between the mask image and the true mask image, and determine the similarity as the quality score of the mask image. The mask image with the highest quality score in the second mask image set is determined as the first target mask image. It should be understood that the function of the diffusion model is to calculate the quality scores of each of the multiple mask images input, and output the mask image with the highest quality score among the multiple mask images.

[0127] In another feasible implementation manner, the step C102 includes steps C1021 to C1023:

[0128] Step C1021: Acquire a true mask image of the training sample image.

[0129] The training sample image is segmented in advance through image segmentation methods such as manual labeling, threshold segmentation or edge detection to obtain the real mask image corresponding to the training sample image.

[0130] Step C1022 : calculating a first intersection-over-union (IoU) between the true mask image and a first sample mask image, wherein the first sample mask image is any mask image in the second mask image set.

[0131] Any mask image in the second mask image set is called a first sample mask image, and the intersection-and-union ratio (IoU) between the true mask image and the first sample mask image is calculated (hereinafter referred to as the first IoU for distinction). That is, the first IoU corresponding to each mask image in the second mask image set is calculated.

[0132] Step C1023: Determine the first intersection-over-union ratio as the quality score of the first sample mask image.

[0133] The first intersection-over-union ratio of the first sample mask image is determined as the quality score of the first sample mask image. Similarly, the quality score of each mask image in the second mask image set is obtained.

[0134] Step C1022: Determine the mask image with the highest quality score in the second mask image set as the first target mask image.

[0135] The mask image with the highest quality score in the second mask image set is determined as the first target mask image. It should be understood that the function of the diffusion model is to calculate the quality scores of each of the multiple input mask images and output the mask image with the highest quality score among the multiple mask images.

[0136] Step C103 : determining one piece of the training sample data based on the second mask image set and the first target mask image, and constructing the training sample set based on multiple pieces of the training sample data.

[0137] A piece of training sample data is determined based on the second mask image set and the first target mask image of a training sample image, thereby determining the training sample data corresponding to each of all training sample images in the training sample set, and then constructing a training sample set of an initial diffusion model based on all the training sample data.

[0138] In a specific embodiment, the training sample images are input into a stable diffusion model to obtain a second mask image set. Then, based on the true mask images corresponding to the training sample images, the intersection-over-union ratio between each mask image in the second mask image set and the true mask image is calculated, and the intersection-over-union ratio is determined as the quality score of the mask image. Then, the mask image with the highest quality score among the mask images is determined as the first target mask image. Finally, a first verification sample set is constructed based on the second mask image set and the first target mask image.

[0139] In this embodiment, step C20 includes steps C201 to C203:

[0140] Step C201 : inputting the model input data into a preset initial diffusion model to obtain a second predicted mask image.

[0141] The model input data is input into the pre-built initial diffusion model to obtain the prediction result of the initial model output (hereinafter referred to as the second prediction mask image for distinction).

[0142] Step C202 , calculating the loss between the second predicted mask image and the model training label.

[0143] Determine the loss function of the initial diffusion model, and calculate the loss between the second predicted mask image and the model training label based on the loss function. Specifically, this application does not limit the specific type of the loss function. For example, in one feasible implementation, the loss function of the initial diffusion model can be mean squared error (MSE), Dice loss, etc.

[0144] Step C203 : Optimizing the model parameters of the initial diffusion model based on the loss to obtain a target diffusion model.

[0145] The model parameters of the initial diffusion model are optimized based on the loss between the second predicted mask image and the model training labels to obtain the target diffusion model.

[0146] In a feasible implementation, the Dice loss function is used to measure the overlap between the segmentation result of the model and the true segmentation, thereby optimizing the model parameters of the initial diffusion model to obtain the target diffusion model.

[0147] In this way, the present application obtains a combined model for difficult sample mining by combining the stable diffusion model and the diffusion model, wherein the stable diffusion model can output multiple intermediate state mask images based on an input image, thereby expanding the training sample capacity of the diffusion model, and the performance of the diffusion model can be improved after a large amount of training and learning; moreover, the diffusion model can process multiple mask images at the same time and output the mask image closest to the true value in each mask image, thereby improving the efficiency of difficult sample mining.

[0148] In addition, an embodiment of the present application also provides a difficult sample mining device.

[0149] Please refer to Figure 5 , Figure 5 This is a functional module diagram of an embodiment of the difficult sample mining device of this application, as shown in FIG. Figure 5 As shown, the difficult sample mining device of this application includes:

[0150] A first model calculation module 10 is configured to input the image to be mined multiple times into a stable diffusion model to obtain first mask image sets, wherein the first mask image set includes multiple mask images obtained by inputting the image to be mined multiple times into the stable diffusion model;

[0151] a second model calculation module 20, configured to input each of the first mask image sets into a target diffusion model to obtain a first predicted mask image corresponding to each of the first mask image sets, wherein the target diffusion model is trained using the second mask image set corresponding to the training sample image as model input data and the first target mask image with the highest quality score in the second mask image set as the model training label, wherein the quality score of the mask image is obtained based on the intersection-over-union ratio between the mask image and the true mask image of the training sample image;

[0152] The sample determination module 30 is configured to determine that the image to be mined is a difficult sample when the overall similarity of each of the first predicted mask images is less than a preset threshold.

[0153] Furthermore, the difficult sample mining device of the present application further includes:

[0154] A first acquisition module is configured to acquire a training sample set, wherein the training sample set includes a plurality of training sample data, each training sample data including a second mask image set corresponding to a training sample image, and a first target mask image with a highest quality score in the second mask image set;

[0155] The training module is used to use the second mask image set in each training sample data as model input data and the first target mask image in each training sample data as model training label to train the preset initial diffusion model to obtain the target diffusion model.

[0156] Furthermore, the first acquisition module includes:

[0157] a model calculation unit, configured to input the training sample image into the stable diffusion model to obtain the second mask image set;

[0158] a quality score calculation unit, configured to calculate a quality score of each mask image in the second mask image set, and determine the mask image with the highest quality score in the second mask image set as the first target mask image;

[0159] A construction unit is configured to determine a piece of the training sample data based on the second mask image set and the first target mask image, and to construct the training sample set based on a plurality of pieces of the training sample data.

[0160] Furthermore, the quality score calculation unit is also used to obtain a true mask image of the training sample image; calculate a first intersection-and-union ratio between the true mask image and a first sample mask image, wherein the first sample mask image is any one mask image in the second mask image set; and determine the first intersection-and-union ratio as the quality score of the first sample mask image.

[0161] Furthermore, the training module is also used to input the model input data into a preset initial diffusion model to obtain a second predicted mask image; calculate the loss between the second predicted mask image and the model training label; and optimize the model parameters of the initial diffusion model based on the loss to obtain a target diffusion model.

[0162] Furthermore, the difficult sample mining device of the present application further includes:

[0163] a first calculation module, configured to respectively calculate a second intersection-over-union ratio between any two mask images in each of the first predicted mask images;

[0164] The overall similarity calculation module is configured to determine an average value of the second intersection-over-union ratios as the overall similarity of the first predicted mask images.

[0165] Furthermore, the difficult sample mining device of the present application further includes:

[0166] a second acquisition module, configured to acquire a verification sample set, wherein the verification sample set includes a plurality of verification sample data pieces, each verification sample data piece including a third mask image set corresponding to a verification sample image, and a second target mask image with a highest quality score in the third mask image set;

[0167] a third model calculation module, configured to input the third mask image sets in each piece of the verification sample data into the target diffusion model to obtain third predicted mask images;

[0168] a second calculation module, configured to respectively calculate a third intersection-over-union ratio between the second target mask image and the third prediction mask image corresponding to each piece of the verification sample data;

[0169] A threshold determination module is used to determine the preset threshold based on an average value of each of the third intersection-over-union ratios.

[0170] The present application also provides a computer storage medium storing a difficult sample mining program. When the difficult sample mining program is executed by a processor, the steps of the difficult sample mining program method described in any of the above embodiments are implemented.

[0171] The specific embodiments of the computer storage medium of the present application are basically the same as the above-mentioned embodiments of the difficult sample mining program method of the present application, and will not be repeated here.

[0172] The present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the steps of the difficult sample mining method of the present application as described in any of the above embodiments, which will not be repeated here.

[0173] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0174] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0175] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD) as described above, and includes several instructions for enabling a difficult sample mining device (which can be a TWS headset, etc.) to execute the methods described in the various embodiments of the present application.

[0176] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A difficult sample mining method, characterized in that: The difficult sample mining method comprises the following steps: Inputting the image to be mined multiple times into the stable diffusion model to obtain each first mask image set, wherein the first mask image set includes multiple mask images obtained after inputting the image to be mined multiple times into the stable diffusion model; Inputting each of the first mask image sets into a target diffusion model to obtain a first predicted mask image corresponding to each of the first mask image sets, wherein the target diffusion model is trained using a second mask image set corresponding to the training sample image as model input data and using the first target mask image with the highest quality score in the second mask image set as the model training label, and the quality score of the mask image is obtained based on the intersection-over-union ratio between the mask image and the true mask image of the training sample image; When the overall similarity of each of the first predicted mask images is less than a preset threshold, determining that the image to be mined is a difficult sample; Before the step of determining that the image to be mined is a difficult sample when the overall similarity of each of the first predicted mask images is less than a preset threshold, the method further includes: respectively calculating a second intersection-over-union ratio between any two mask images in each of the first predicted mask images; An average value of the second intersection-over-union ratios is determined as the overall similarity of the first prediction mask images.

2. The difficult sample mining method according to claim 1, characterized in that: The method further comprises: Acquire a training sample set, wherein the training sample set includes a plurality of training sample data, and each training sample data includes a second mask image set corresponding to a training sample image, and a first target mask image with a highest quality score in the second mask image set; The second mask image set in each piece of training sample data is used as model input data, and the first target mask image in each piece of training sample data is used as a model training label. The preset initial diffusion model is trained to obtain the target diffusion model.

3. The difficult sample mining method according to claim 2, characterized in that: The step of obtaining a training sample set includes: Inputting the training sample image into the stable diffusion model to obtain the second mask image set; calculating a quality score of each mask image in the second mask image set, and determining the mask image with the highest quality score in the second mask image set as the first target mask image; A piece of the training sample data is determined based on the second mask image set and the first target mask image, and the training sample set is constructed based on multiple pieces of the training sample data.

4. The difficult sample mining method according to claim 3, characterized in that: The step of calculating the quality score of each mask image in the second mask image set includes: Obtaining a true mask image of the training sample image; Calculating a first intersection-over-union (IoU) between the true mask image and a first sample mask image, wherein the first sample mask image is any one mask image in the second mask image set; The first intersection-over-union ratio is determined as a quality score of the first sample mask image.

5. The difficult sample mining method according to claim 2, wherein: The step of training the preset initial diffusion model to obtain the target diffusion model includes: Inputting the model input data into a preset initial diffusion model to obtain a second predicted mask image; Calculating a loss between the second predicted mask image and the model training label; The model parameters of the initial diffusion model are optimized based on the loss to obtain a target diffusion model.

6. The difficult sample mining method according to any one of claims 1 to 5, characterized in that: The method further comprises: Acquire a verification sample set, wherein the verification sample set includes multiple verification sample data, each verification sample data includes a third mask image set corresponding to a verification sample image, and a second target mask image with a highest quality score in the third mask image set; Inputting the third mask image sets in each piece of the verification sample data into the target diffusion model respectively to obtain each third predicted mask image; respectively calculating a third intersection-over-union ratio between the second target mask image and the third predicted mask image corresponding to each of the verification sample data; The preset threshold is determined based on an average value of the third intersection-over-union ratios.

7. A difficult sample mining device, characterized in that: The difficult sample mining device comprises the following steps: a first model calculation module, configured to input the image to be mined multiple times into the stable diffusion model to obtain first mask image sets, wherein the first mask image set includes multiple mask images obtained by inputting the image to be mined multiple times into the stable diffusion model; a second model calculation module, configured to input each of the first mask image sets into a target diffusion model to obtain a first predicted mask image corresponding to each of the first mask image sets, wherein the target diffusion model is trained using the second mask image set corresponding to the training sample image as model input data and the first target mask image with the highest quality score in the second mask image set as the model training label, and the quality score of the mask image is obtained based on the intersection-over-union ratio between the mask image and the true mask image of the training sample image; a sample determination module, configured to determine that the image to be mined is a difficult sample when the overall similarity of each of the first predicted mask images is less than a preset threshold; The difficult sample mining device further includes: a first calculation module, configured to respectively calculate a second intersection-over-union ratio between any two mask images in each of the first predicted mask images; The overall similarity calculation module is configured to determine an average value of the second intersection-over-union ratios as the overall similarity of the first predicted mask images.

8. A difficult sample mining device, characterized in that: The difficult sample mining device includes: a memory and a processor, wherein the memory stores a difficult sample mining program that can be run on the processor, and when the difficult sample mining program is executed by the processor, the steps of the difficult sample mining method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a difficult sample mining program, which, when executed by a processor, implements the steps of the difficult sample mining method according to any one of claims 1 to 6.

Images