Sampling method and device of sample image and electronic equipment

Abstract

The invention discloses a sampling method and device of a sample image and electronic equipment, and belongs to the technical field of image processing and the technical field of industrial detection. The method comprises the following steps: clustering batch sample images according to at least one of a model type of a target deep learning model, a sample type of each sample image, a real label mask image and a predicted label mask image to obtain a plurality of first clustering categories; the model type is an image classification type or an image segmentation type; for each first clustering category, collecting a target number of target sample images from the sample images contained in the first clustering category; the target number is smaller than the number of sample images contained in the corresponding first clustering category; and determining the target sample image as the training sample of the target deep learning model, so that the redundancy of the training sample is greatly reduced, the iteration delay is reduced, and meanwhile, the diversity of the training sample can be kept.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology and industrial detection technology, and in particular relates to a method, device and electronic equipment for sampling a sample image. Background Art

[0002] Deep learning models are widely used in industrial production processes, such as detecting whether industrial products contain defects and detecting the types of defects in industrial products. Due to business needs or business rectification, it is inevitable to adjust the deep learning model. In the batch production process of industrial products, the number of new sample images may reach tens of thousands, and there is a large amount of redundancy between sample images. Too many invalid or repeated sample images may not bring performance improvements to the deep learning model. Instead, it may cause serious time consumption due to tens of thousands of iterations of sample images. Summary of the invention

[0003] The present application aims to solve at least one of the technical problems existing in the related art. To this end, the present application proposes a method, device and electronic device for sampling sample images, which greatly reduces the redundancy of training samples, reduces the delay of iteration, and can also maintain the diversity of training samples.

[0004] In a first aspect, the present application provides a method for sampling a sample image, the method comprising:

[0005] Clustering a batch of sample images according to at least one of a model type of a target deep learning model, a sample type of each sample image, a true label mask image, and a predicted label mask image to obtain a plurality of first clustering categories; the model type is an image classification type or an image segmentation type; each first clustering category includes at least one sample image;

[0006] For each first cluster category, a target number of target sample images are collected from the sample images included in the first cluster category; the target number is less than the number of sample images included in the corresponding first cluster category;

[0007] Determine the target sample image as a training sample for the target deep learning model.

[0008] According to the sampling method of sample images of the present application, batches of sample images are clustered according to the model type of the target deep learning model, the sample type of each sample image, the real label mask image and the predicted label mask image, to obtain multiple first clustering categories. For each first clustering category, a target number of target sample images are collected from the sample images included in the first clustering category; the target number is less than the number of sample images included in the corresponding first clustering category; the target sample images are determined to be training samples of the target deep learning model, which greatly reduces the redundancy of training samples and reduces the iteration delay. At the same time, it can also maintain the diversity of training samples, thereby improving the efficiency of deep learning model iteration in industrial production lines and the feasibility of deep learning models in the industrial quality inspection industry.

[0009] According to an embodiment of the present application, the sample types include a first type and a second type; the sample images of the first type are images that have not been detected by the target deep learning model; the sample images of the second type are images of target objects that are missed by the target deep learning model; clustering batches of sample images according to at least one of the model type of the target deep learning model, the sample type of each sample image, a true label mask image, and a predicted label mask image includes:

[0010] When the model type is an image segmentation type and the sample type is the first type or the second type, determining a first bounding box where the target object is located from the true label mask image of each sample image;

[0011] Cut and scale a first sub-image of an area corresponding to a first bounding box from each sample image;

[0012] Creating a mapping relationship between each sample image and the corresponding first sub-image;

[0013] Clustering the first sub-images of each sample image according to a target clustering algorithm to obtain a plurality of second cluster categories; each second cluster category includes at least one first sub-image;

[0014] According to the mapping relationship between the first sub-image and the corresponding sample image, the first cluster category corresponding to each second cluster category is obtained.

[0015] According to one embodiment of the present application, the second type of sample image is determined based on the following method:

[0016] For a sample image detected by a target deep learning model, a first bounding box of a target object is determined from a true label mask image of the sample image, and a second bounding box of the target object is determined from a predicted label mask image of the sample image;

[0017] When it is determined that there is no intersection area between the area corresponding to the first circumscribed frame in the sample image and the area corresponding to the second circumscribed frame in the sample image, the sample image is determined to be a sample image of the second type.

[0018] According to an embodiment of the present application, the model type includes an image segmentation model; the sample type includes a third type; the sample image of the third type is an image of a target object that is misdetected by the target deep learning model; clustering a batch of sample images according to at least one of the model type of the target deep learning model, the sample type of each sample image, a true label mask image, and a predicted label mask image, includes:

[0019] When the model type is an image segmentation type and the sample type is the third type, determining a second bounding box where the target object is located from the predicted label mask image of each sample image;

[0020] Cut and scale a second sub-image of the area corresponding to the second bounding box from each sample image;

[0021] Creating a mapping relationship between each sample image and the corresponding second sub-image;

[0022] Clustering the second sub-images of each sample image according to a target clustering algorithm to obtain a plurality of third cluster categories; each third cluster category includes at least one second sub-image;

[0023] According to the mapping relationship between the second sub-image and the corresponding sample image, the first cluster category corresponding to each third cluster category is obtained.

[0024] According to an embodiment of the present application, clustering a batch of sample images according to at least one of a model type of a target deep learning model, a sample type of each sample image, a true label mask image, and a predicted label mask image includes:

[0025] When the model type of the target deep learning model is an image classification type, scaling and / or cropping each sample image to the same size;

[0026] Each sample image is clustered according to a target clustering algorithm to obtain a plurality of first cluster categories.

[0027] According to one embodiment of the present application, before collecting a target number of target sample images from the sample images included in the first cluster category, the method further includes:

[0028] The target quantity is determined based on:

[0029] Obtaining a target number according to a target sampling ratio and the number of sample images included in the first clustering category; or

[0030] Get the preset target quantity.

[0031] According to an embodiment of the present application, clustering a batch of sample images according to at least one of a model type of a target deep learning model, a sample type of each sample image, a true label mask image, and a predicted label mask image includes:

[0032] Determine the image to be loaded according to the model type and / or the sample type of each sample image; the image to be loaded is at least one of the sample image, the real label mask image of the sample image, and the predicted label mask image;

[0033] Load the image to be loaded according to the image path of the image to be loaded.

[0034] In a second aspect, the present application provides a sampling device for a sample image, the device comprising:

[0035] A first processing module, configured to cluster a batch of sample images according to at least one of a model type of a target deep learning model, a sample type of each sample image, a true label mask image, and a predicted label mask image, to obtain a plurality of first cluster categories; the model type is an image classification type or an image segmentation type; and each first cluster category includes at least one sample image;

[0036] A second processing module is used to collect a target number of target sample images from the sample images included in the first cluster category for each first cluster category; the target number is less than the number of sample images included in the corresponding first cluster category;

[0037] The third processing module is used to determine that the target sample image is a training sample of the target deep learning model.

[0038] According to the sampling device for sample images of the present application, batches of sample images are clustered according to the model type of the target deep learning model, the sample type of each sample image, the real label mask image and the predicted label mask image to obtain multiple first clustering categories. For each first clustering category, a target number of target sample images are collected from the sample images included in the first clustering category; the target number is less than the number of sample images included in the corresponding first clustering category; the target sample images are determined to be training samples of the target deep learning model, which greatly reduces the redundancy of training samples and reduces the iteration delay. At the same time, it can also maintain the diversity of training samples, thereby improving the efficiency of deep learning model iteration in industrial production lines and the feasibility of deep learning models in the industrial quality inspection industry.

[0039] In a third aspect, the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the sample image sampling method provided in the first aspect is implemented.

[0040] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the sample image sampling method provided in the first aspect above.

[0041] In a fifth aspect, the present application provides a chip, the chip comprising a processor and a communication interface, the communication interface and the processor are coupled, the processor is used to run a program or instruction to implement the sample image sampling method provided in the first aspect.

[0042] In a sixth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the sample image sampling method provided in the first aspect above.

[0043] The above one or more technical solutions in the embodiments of the present application have at least one of the following technical effects:

[0044] By clustering batches of sample images according to the model type of the target deep learning model, the sample type of each sample image, the real label mask image and at least one of the predicted label mask images, multiple first clustering categories are obtained. For each first clustering category, a target number of target sample images are collected from the sample images included in the first clustering category; the target number is less than the number of sample images included in the corresponding first clustering category; the target sample images are determined as training samples of the target deep learning model, which greatly reduces the redundancy of training samples and reduces the iteration delay. At the same time, it can also maintain the diversity of training samples, thereby improving the efficiency of deep learning model iteration in industrial production lines and the feasibility of deep learning models in industrial quality inspection industries.

[0045] In addition, since a large number of missed detection sample images (second type of sample images) will appear in the iteration process of the target deep learning model, the missed detection probability of the target deep learning model is reduced by cropping the first sub-image of the missed detection area where the missed target object is located from the missed detection sample image, performing cluster sampling, and then adding it to the target deep learning model for iterative reinforcement.

[0046] In addition, since a large number of over-inspected sample images (sample images of the third type) will appear in the iteration process of the target deep learning model, and there is no target object in the real label mask image of the over-inspected sample image, the second sub-image corresponding to the over-inspected area in the over-inspected sample image (the area corresponding to the target object in the predicted label mask image) is cropped according to the predicted label mask image, and after cluster sampling, it is added to the target deep learning model for iterative reinforcement, thereby reducing the over-inspection probability of the target deep learning model.

[0047] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0049] Figure 1 This is one of the flowcharts of the sample image sampling method provided in the embodiment of the present application;

[0050] Figure 2 It is a flowchart of a method for sampling sample images of a target deep learning model of an image classification type provided in an embodiment of the present application;

[0051] Figure 3 It is a flowchart of a method for sampling sample images of a target deep learning model of the image segmentation type provided in an embodiment of the present application;

[0052] Figure 4 is a schematic diagram of the structure of a sampling device for sample images provided in an embodiment of the present application;

[0053] Figure 5 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0055] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0056] In conjunction with the accompanying drawings, the sample image sampling method, sample image sampling device, electronic device and readable storage medium provided in the embodiments of the present application are described in detail below through specific embodiments and their application scenarios.

[0057] The sampling method of the sample image may be applied to a terminal, and may be specifically executed by hardware or software in the terminal.

[0058] The sampling method for sample images provided in the embodiments of the present application may be executed by an electronic device or a functional module or functional entity in the electronic device that can implement the sampling method for sample images. The electronic devices mentioned in the embodiments of the present application include but are not limited to mobile phones, tablet computers, computers, cameras, and wearable devices. The sampling method for sample images provided in the embodiments of the present application is described below using an electronic device as an example of the execution subject.

[0059] like Figure 1 As shown, the sampling method of the sample image includes: step 110, step 120 and step 130.

[0060] Step 110: Cluster the batches of sample images according to the model type of the target deep learning model, the sample type of each sample image, at least one of the true label mask image and the predicted label mask image to obtain multiple first cluster categories.

[0061] The model type is an image classification type or an image segmentation type; each first clustering category includes at least one sample image.

[0062] A deep learning model is a machine learning model based on artificial neural networks (ANN). It automatically learns the features and representations of data through a multi-level neural network structure, thereby achieving tasks such as pattern recognition, classification, and regression. The "depth" of deep learning refers to the hierarchical structure of multiple layers (usually more than 2 layers) of neurons in the network, allowing the model to learn increasingly complex features and abstractions in the data.

[0063] According to their functions, deep learning models can be divided into classification models, regression models, generation models, sequence generation and processing models, target detection and segmentation models, reinforcement learning models, autoencoder models, and graph neural networks. Each model plays an important role in different application scenarios, and choosing the right model is crucial to solving specific problems.

[0064] The above-mentioned various types of models can be further divided into fine-grained categories according to their functions. For example, classification models include image classification models, text classification models, etc., and target detection and segmentation models include target detection models and image segmentation models, etc.

[0065] The target deep learning model of the embodiment of the present application is a model for processing images, and the model types of the target deep learning model include the above-mentioned image classification model and image segmentation type.

[0066] Among them, the image classification model aims to automatically identify and classify the input image into predefined categories. Simply put, the task of the image classification model is to analyze the image and assign one or more labels to the image based on its content. The label here is the image category, such as cat, dog, person, etc.

[0067] The purpose of the image segmentation model is to divide the image into several semantically meaningful regions or "segments", so that each region corresponds to a specific object, background or other part of interest to some extent.

[0068] In the embodiment of the present application, the sample type is used to indicate the type of data source of the sample image; the sample type includes but is not limited to the first type, the second type and the third type.

[0069] Among them, the first type of sample images are images that have not been detected by the target deep learning model, and the first type of sample images are also called "initial training images".

[0070] The second type of sample image is an image of a target object that is missed by the target deep learning model. The target object may be any object, such as a person, an animal, an object, etc., without limitation. When performing defect detection, the target object may refer to a defect. The target object exists in the second type of sample image, but the target object is not fully detected. Therefore, the second type of sample image is also called a "missed detection image".

[0071] The third type of sample image is an image in which the target object is mistakenly detected by the target deep learning model. For example, there is no target object in a sample image, but after detection by the target deep model, the target object is mistakenly detected. In this case, the sample image can be determined to be a sample image of the third type. The third type of sample image is also called an "over-detected image."

[0072] In the production process of industrial products, due to the high consistency of industrial products, the collected batches of image samples contain a lot of redundant data. Too many invalid or repeated sample images may not bring performance improvements to the deep learning model, but may cause serious time consumption due to tens of thousands of iterations of sample images. Manual selection of training samples is more cumbersome and time-consuming, and it also wastes human resources.

[0073] To avoid the above situation, the embodiment of the present application clusters batches of sample images. Since the sample images in each clustering category have high similarity and even duplicate images appear, a part of the target sample images can be selected from each clustering category for training the target deep learning model. Both the image classification model and the image segmentation model have corresponding clustering methods.

[0074] It is understandable that since the image classification model is designed to classify images, regardless of the sample type of the sample image, its training label is the category of the image. Therefore, when the target deep learning model is trained or optimized, sample images of various sample types can be clustered based on the same clustering method, that is, various types of sample images can be directly clustered according to the target clustering algorithm (such as distance-based clustering algorithm---Kmeans algorithm, density-based clustering algorithm---dbscan algorithm).

[0075] In actual application, training can be performed sequentially according to categories. In the training stage of a certain category, the image path and corresponding category label of sample images of the same category can be input, so that the corresponding image can be obtained according to the corresponding image path.

[0076] It can be understood that since the image segmentation model is intended to identify each target object in the image, a label mask image (Label Mask Image) of the sample image is required.

[0077] The pixel value of each pixel in the label mask image represents the category or label to which the position belongs. The pixel value in the label mask image is also called the label value or label value, which is widely used in image segmentation tasks, where the value (or color) of each pixel corresponds to a category label, which is used to represent the classification information of different areas in the image. For example, in defect detection, the label mask image can be used to mark the areas of various defects, and the pixel values ​​represent categories such as "cracks" or "stains".

[0078] Label mask images are divided into true label mask images and predicted label mask images. The true label mask images are usually manually annotated or provided by experts, indicating the true category of each pixel in the image; the predicted label mask images are generated by the trained model and indicate the model's predicted category for each pixel in the image.

[0079] When training the image segmentation model, you can also train it in sequence according to categories:

[0080] For the first type of sample images (initial training samples), it is necessary to input the image path of the sample image of the same category and the image path of the corresponding true label mask image, so that the corresponding sample image and the true label mask image can be obtained based on the image path of the sample image and the image path of the true label mask image, and the sample images can be clustered according to the true label mask image.

[0081] For the second type of sample images, it is necessary to input the image path of the sample image of the same category, the image path of the corresponding true label mask image, and the image path of the corresponding predicted label mask image, so that the corresponding sample image, true label mask image and predicted label mask can be obtained based on the image path of the sample image, the image path of the true label mask image and the image path of the predicted label mask image, and the sample images can be clustered based on the true label mask image and the predicted label mask.

[0082] For the third type of sample images, it is necessary to input the image path of the sample image of the same category and the image path of the corresponding predicted label mask, so that the corresponding image can be obtained according to the corresponding image path, and the corresponding sample image and the predicted label mask can be obtained according to the image path of the sample image and the image path of the predicted label mask image, and the sample images are clustered according to the predicted label mask.

[0083] The embodiments of the present application can pre-process (scaling, cropping) and cluster batches of sample images based on a target clustering algorithm to obtain multiple first clustering categories; the target clustering algorithm is, for example, a Kmeans algorithm, a dbscan algorithm, etc., each first clustering category includes at least one sample image, and the number of sample images contained in each first clustering category may be the same or different. Usually, the number of sample images contained in each first clustering category is different.

[0084] In addition, the number of the first clustering categories, that is, the number of clusters, can be a custom preset value, or it can be determined based on the total number of sample images and the target ratio. For example, if the target ratio is 20%, the product of the total number of sample images and the target ratio can be rounded up to obtain the number of clusters. For example, if the total number of sample images is 300, then the number of clusters is 60.

[0085] Step 120 : For each first cluster category, collect a target number of target sample images from the sample images included in the first cluster category; the target number is less than the number of sample images included in the corresponding first cluster category.

[0086] Since the sample images in each first clustering category are relatively similar, and some are even repeated, repeated sample images usually cannot bring performance improvements to the deep learning model, but will increase the iteration time. In order to reduce the redundancy of training data while maintaining data diversity, the embodiment of the present application collects a target number of target sample images from the sample images included in the first clustering category for each first clustering category; the target number is less than the number of sample images included in the corresponding first clustering category.

[0087] The target number can be a custom preset value, or it can be determined based on the number of sample images in the first clustering category and the sampling ratio. Specifically, the product of the number of sample images in the first clustering category and the sampling ratio can be rounded up to obtain the target number of the first clustering category. The specific values ​​of the sampling ratio and the aforementioned target ratio can be the same or different, and there is no limitation on this.

[0088] For example, if the sampling ratio is 20%, and the number of sample images in a first cluster category is 20, then the target number of the first cluster category is 4 of the total number of samples.

[0089] In addition, when collecting a target number of target sample images from the first cluster category, a target number of target sample images can be randomly collected from the first cluster category. In addition, from the perspective of enriching sample diversity, sampling can be performed based on image features (such as color, texture, shape, etc.), and images that are highly representative and have obvious boundaries in the feature space can be selected, which usually relies on the feature vector representation of the image; distance-based sampling can also be performed by calculating the distance between the image and the cluster center or other images of the category, and selecting images that are farther or closer to the center for sampling, etc. Other sampling methods can also be selected, and there is no restriction on this.

[0090] Step 130: Determine that the target sample image is a training sample for the target deep learning model.

[0091] After determining the target sample images of each first clustering category, the embodiment of the present application can use each target sample image as a training sample for the target deep learning model. Specifically, the image path and label of the target sample image can be returned for subsequent training or optimization adjustment of the target deep learning model.

[0092] The embodiment of the present application obtains multiple first clustering categories by clustering batches of sample images according to the model type of the target deep learning model, the sample type of each sample image, the real label mask image and the predicted label mask image. For each first clustering category, a target number of target sample images are collected from the sample images included in the first clustering category; the target number is less than the number of sample images included in the corresponding first clustering category; the target sample images are determined to be training samples of the target deep learning model, which greatly reduces the redundancy of training samples and reduces the iteration delay. At the same time, it can also maintain the diversity of training samples, thereby improving the efficiency of deep learning model iteration in industrial production lines and the feasibility of deep learning models in the industrial quality inspection industry.

[0093] In some embodiments, after step 130, the method further includes:

[0094] Based on each target sample image and the label of the target sample image, the target deep learning model is adjusted.

[0095] Iteration or optimization adjustment of the target deep learning model usually refers to the optimization of the model parameters of the target deep learning model, characterized by the weight parameters and bias parameters of the target deep learning network model, which determine how the target deep learning model processes the input sample data.

[0096] The purpose of adjusting or optimizing the target deep learning model is to adjust these parameters so that the prediction of the target deep learning model is as close to the actual value as possible. It is usually optimized by minimizing the loss function (such as mean square error, cross entropy loss, etc.). The detailed process can be found in the prior art and will not be repeated here.

[0097] In some embodiments, clustering a batch of sample images according to at least one of a model type of a target deep learning model, a sample type of each sample image, a true label mask image, and a predicted label mask image includes:

[0098] When the model type of the target deep learning model is an image classification type, scaling and / or cropping each sample image to the same size;

[0099] Each sample image is clustered according to a target clustering algorithm to obtain a plurality of first cluster categories.

[0100] When training a target deep learning model of an image classification type, the embodiment of the present application can unify the size of each sample image, scale and / or crop each sample image to the same size, such as a unified size of 128*128, or other sizes set according to actual needs. Subsequently, each sample image is clustered through a target clustering algorithm to obtain each first clustering category.

[0101] In some embodiments, the sample types include a first type and a second type; the sample images of the first type are images that have not been detected by the target deep learning model; the sample images of the second type are images of target objects that are missed by the target deep learning model; clustering batches of sample images according to at least one of the model type of the target deep learning model, the sample type of each sample image, the real label mask image, and the predicted label mask image includes:

[0102] When the model type is an image segmentation type and the sample type is the first type or the second type, determining a first bounding box where the target object is located from the true label mask image of each sample image;

[0103] Cut and scale a first sub-image of an area corresponding to a first bounding box from each sample image;

[0104] Creating a mapping relationship between each sample image and the corresponding first sub-image;

[0105] Clustering the first sub-images of each sample image according to a target clustering algorithm to obtain a plurality of second cluster categories; each second cluster category includes at least one first sub-image;

[0106] According to the mapping relationship between the first sub-image and the corresponding sample image, the first cluster category corresponding to each second cluster category is obtained.

[0107] For the target deep learning model of image segmentation type, the sample images belong to the images to be segmented, and the backgrounds of the images to be segmented are basically the same. The background areas without target objects (such as defects) are basically invalid images. In addition, when the background area is large and the target object area is small, these sample images may be relatively similar. Directly clustering these sample images may result in invalid clustering (one cluster category appears). Since image segmentation aims to segment target objects, in order to increase the differences between sample images, the sample images also need to be segmented and cropped during sampling.

[0108] Since the sample image of the first type is an initial training sample that has not been detected by the target deep learning model, when training the target deep learning model of the image segmentation type, segmentation and cropping can be directly performed based on the real label mask image of the sample image.

[0109] For a sample image of the second type, the sample image is an image that has been detected by the target deep learning model and in which the target object is missed. Specifically, the sample image of the second type is determined based on the following method:

[0110] For a sample image detected by a target deep learning model, a first bounding box of a target object is determined from a true label mask image of the sample image, and a second bounding box of the target object is determined from a predicted label mask image of the sample image;

[0111] When it is determined that there is no intersection area between the area corresponding to the first circumscribed frame in the sample image and the area corresponding to the second circumscribed frame in the sample image, the sample image is determined to be a sample image of the second type.

[0112] For the sample image detected by the target deep learning model, there is not only a real label mask image, but also a predicted label mask image. According to the target label value (such as 1) corresponding to the target object (such as a defect) in the real label mask image, the first bounding box of the blob area corresponding to the target object is determined, and according to the target label value corresponding to the target object (such as a defect) in the predicted label mask image, the second bounding box of the blob area corresponding to the target object is determined.

[0113] The bounding box, also known as the circumscribed rectangular box, is used to describe the positioning information of the target object. It is the smallest rectangle that can cover the target object, with its sides parallel to the coordinate axes of the image. The four sides of the rectangular box are aligned with the horizontal and vertical directions of the image, and the area of ​​the box is minimized to cover the object.

[0114] An area corresponding to the same position as the first circumscribed frame and an area corresponding to the same position as the second circumscribed frame can be determined in the sample image. If there is no intersection between the two areas, the sample image can be determined to be a sample image of the second type. If there is an intersection between the two areas, it can be determined that the sample image is not a sample image of the second type.

[0115] Whether it is the first type of sample image that has not been detected, or the second type of sample image with missed target objects, adding all of them to iterative training will lead to low training efficiency. At the same time, missed detection only means missing part of the target objects on the partial image. Therefore, it is necessary to crop the small image of the missed area, that is, to cut the first sub-image of the area corresponding to the first external bounding box of the target object from the first type and second type sample images, obtain the first sub-image where the target object is located, and create a mapping relationship (or corresponding relationship) between the sample image and the corresponding first sub-image.

[0116] And each sample image is scaled to the same size, which may be 128*128 or other sizes, without limitation.

[0117] After scaling each first sub-image of the first type or the second type to the same size, the first sub-image of each sample image can be clustered according to the target clustering algorithm to obtain multiple second clustering categories; each second clustering category includes at least one first sub-image, and since there is a mapping relationship between the sample image and the corresponding first sub-image, the first clustering category corresponding to each second clustering category can be obtained according to the mapping relationship between the first sub-image and the corresponding sample image.

[0118] In addition, in some embodiments, a target number of first sub-images may be directly selected from the second cluster category, and sample images that have a mapping relationship with the selected target number of first sub-images are sampled target sample images.

[0119] The embodiment of the present application reduces the probability of missed detection of the target deep learning model by cropping the first sub-image of the area where the missed target object is located from the sample image, performing cluster sampling, and then adding the target deep learning model for iterative reinforcement.

[0120] In some embodiments, the model type includes an image segmentation model; the sample type includes a third type; the sample image of the third type is an image of a target object that is misdetected by the target deep learning model; clustering a batch of sample images according to at least one of the model type of the target deep learning model, the sample type of each sample image, the real label mask image, and the predicted label mask image includes:

[0121] When the model type is an image segmentation type and the sample type is the third type, determining a second bounding box where the target object is located from the predicted label mask image of each sample image;

[0122] Cut and scale a second sub-image of the area corresponding to the second bounding box from each sample image;

[0123] Creating a mapping relationship between each sample image and the corresponding second sub-image;

[0124] Clustering the second sub-images of each sample image according to a target clustering algorithm to obtain a plurality of third cluster categories; each third cluster category includes at least one second sub-image;

[0125] According to the mapping relationship between the second sub-image and the corresponding sample image, the first cluster category corresponding to each third cluster category is obtained.

[0126] Since a large number of sample images of the third type will pass the inspection during the iteration process of the target deep learning model, and there is no target object in the real label mask image of the passed sample image, it is necessary to crop the second sub-image corresponding to the passed area in the sample image (the area corresponding to the target object in the predicted label mask image) according to the predicted label mask image, and then add it to the target deep learning model for iterative reinforcement after cluster sampling, so as to reduce the over-inspection probability of the target deep learning model.

[0127] Specifically, when the model type is an image segmentation type and the sample type is a third type, a second bounding box of the blob area where the target object is located is determined from the predicted label mask image of each sample image; a second sub-image of the area corresponding to the second bounding box is cut from each sample image; a mapping relationship between the sample image of the third type and the corresponding second sub-image is created, the second sub-image is scaled to a uniform size, and a plurality of third cluster categories are obtained based on clustering the second sub-images of each sample image; each third cluster category includes at least one second sub-image; and a first cluster category corresponding to each third cluster category can be obtained based on the mapping relationship between the second sub-image and the corresponding sample image.

[0128] In addition, in some embodiments, a target number of second sub-images may be directly selected from the third cluster category, and sample images that have a mapping relationship with the selected target number of second sub-images are sampled target sample images.

[0129] In some embodiments, before collecting a target number of target sample images from the sample images included in the first cluster category, the method further includes:

[0130] The target quantity is determined based on:

[0131] Obtaining the target number according to the target sampling ratio and the number of sample images included in the first clustering category; or

[0132] Get the preset target quantity.

[0133] This has been explained in the above-mentioned embodiment and will not be described again here.

[0134] In some embodiments, clustering a batch of sample images according to at least one of a model type of a target deep learning model, a sample type of each sample image, a true label mask image, and a predicted label mask image includes:

[0135] Determine the image to be loaded according to the model type and / or the sample type of each sample image; the image to be loaded is at least one of the sample image, the real label mask image of the sample image, and the predicted label mask image;

[0136] Load the image to be loaded according to the image path of the image to be loaded.

[0137] It can be found from the above embodiments that when the model type is an image classification model, no matter the sample type is the first type, the second type or the third type, the image to be loaded includes the sample image.

[0138] When the model type is an image segmentation model and the sample type is the first type, the images to be loaded are the sample image and the true label mask image of the sample image.

[0139] When the model type is an image segmentation model and the sample type is the second type, the images to be loaded include a sample image, a true label mask image of the sample image, and a predicted label mask image.

[0140] When the model type is an image segmentation model and the sample type is the third type, the image to be loaded includes a sample image and a predicted label mask image.

[0141] In actual applications, a "one-click intelligent training" button can be set. On the page, the image path of the image to be loaded is entered according to the model type and / or the sample type of each sample image. After the input is completed, the "one-click intelligent training" button is triggered to filter the target sample images in the background, and train the target deep learning model according to the target sample images and corresponding training labels.

[0142] like Figure 2 As shown, the embodiment of the present application provides a flowchart of a method for sampling sample images of a target deep learning model of an image classification type, including:

[0143] Loading a first type of sample image, a second type of sample image, or a third type of sample image, and a sample image of the same category;

[0144] Scale and / or crop each sample image to the same size;

[0145] Clustering each sample image according to a target clustering algorithm to obtain a plurality of first cluster categories;

[0146] For each first cluster category, a target number of target sample images are collected from the sample images included in the first cluster category; the target number is less than the number of sample images included in the corresponding first cluster category;

[0147] Determine the target sample image as a training sample of the target deep learning model;

[0148] like Figure 3 As shown, the embodiment of the present application provides a flowchart of a method for sampling sample images of a target deep learning model of an image segmentation type, including:

[0149] When the sample type is the first type or the second type, determine the first bounding box where the target object is located from the true label mask image of each sample image of the first type or the second type (the first bounding box in the sample image of the second type is the location of the missed target object); cut the first sub-image of the area corresponding to the first bounding box from each sample image, and create a mapping relationship between each sample image and the corresponding first sub-image; scale each first sub-image to the same size; cluster the first sub-image of each sample image according to the target clustering algorithm to obtain multiple second cluster categories; each second cluster category includes at least one first sub-image; according to the mapping relationship between the first sub-image and the corresponding sample image, obtain the first cluster category corresponding to each second cluster category; for each first cluster category, collect a target number of target sample images from the sample images included in the first cluster category; the target number is less than the number of sample images included in the corresponding first cluster category; determine that the target sample image is a training sample of the target deep learning model.

[0150] When the sample type is the third type, determine the second bounding box where the target object is located from the predicted label mask image of each sample image of the third type; cut the second sub-image of the area corresponding to the second bounding box from each sample image; create a mapping relationship between each sample image and the corresponding second sub-image; scale each second sub-image to the same size; cluster the second sub-image of each sample image according to the target clustering algorithm to obtain multiple third clustering categories; each third clustering category includes at least one second sub-image; according to the mapping relationship between the second sub-image and the corresponding sample image, obtain the first clustering category corresponding to each third clustering category; for each first clustering category, collect a target number of target sample images from the sample images included in the first clustering category; the target number is less than the number of sample images included in the corresponding first clustering category; determine that the target sample image is a training sample of the target deep learning model.

[0151] Figures 2 to 3 The detailed implementation process of each step in the above embodiment is shown in the above embodiment and will not be described again here.

[0152] The deep learning detection model used in the field of industrial quality inspection will collect new sample images and sample images that have been missed or over-detected during the training process during the online process. After multiple iterative training, the amount of data will become larger and more time-consuming, and may even affect the normal production of the production line. The data sampling scheme for intelligent training based on the above process can eliminate the redundancy of image sample data, maintain diversity, greatly improve the learning of defective samples that have been passed and missed, improve the timeliness of production line model updates and iterations, and improve the feasibility of deep learning detection models in the field of industrial quality inspection.

[0153] In addition, by inputting all sample images as a whole, the solution provided by the embodiment of the present application can be more flexibly embedded into the training process of various deep learning models, maintaining the flexibility of algorithm design; the image size of clustering input can also be set to a smaller size in application scenarios with better efficiency, further optimizing and improving the efficiency of clustering. When using other clustering algorithms such as dbscan, there is no need to set the number of clustering subclasses, reducing the sensitivity of parameter settings to clustering effects, and being more robust to noise data.

[0154] The sample image sampling method provided in the embodiment of the present application can be executed by a sample image sampling device. In the embodiment of the present application, the sample image sampling device executing the sample image sampling method is taken as an example to illustrate the sample image sampling device provided in the embodiment of the present application.

[0155] The embodiment of the present application also provides a sampling device for sample images.

[0156] like Figure 4 As shown, the sampling device for the sample image includes: a first processing module 410 , a second processing module 420 and a third processing module 430 .

[0157] A first processing module 410 is used to cluster a batch of sample images according to at least one of a model type of a target deep learning model, a sample type of each sample image, a true label mask image, and a predicted label mask image to obtain a plurality of first cluster categories; the model type is an image classification type or an image segmentation type; each first cluster category includes at least one sample image;

[0158] The second processing module 420 is used to collect a target number of target sample images from the sample images included in the first cluster category for each first cluster category; the target number is less than the number of sample images included in the corresponding first cluster category;

[0159] The third processing module 430 is used to determine that the target sample image is a training sample of the target deep learning model.

[0160] According to the sampling device of sample images provided in the embodiment of the present application, a batch of sample images are clustered according to the model type of the target deep learning model, the sample type of each sample image, the real label mask image and the predicted label mask image to obtain multiple first cluster categories. For each first cluster category, a target number of target sample images are collected from the sample images included in the first cluster category; the target number is less than the number of sample images included in the corresponding first cluster category; the target sample images are determined to be training samples of the target deep learning model, which greatly reduces the redundancy of training samples and reduces the delay of iteration. At the same time, the diversity of training samples can be maintained, thereby improving the efficiency of deep learning model iteration in industrial production lines and the feasibility of deep learning models in industrial quality inspection industries.

[0161] In some embodiments, the sample type includes a first type and a second type; the first type of sample image is an image that has not been detected by the target deep learning model; the second type of sample image is an image of a target object that is missed by the target deep learning model; the first processing module 410 includes:

[0162] The first processing submodule is used for:

[0163] When the model type is an image segmentation type and the sample type is the first type or the second type, determining a first bounding box where the target object is located from the true label mask image of each sample image;

[0164] Cut and scale a first sub-image of an area corresponding to a first bounding box from each sample image;

[0165] Creating a mapping relationship between each sample image and the corresponding first sub-image;

[0166] Clustering the first sub-images of each sample image according to a target clustering algorithm to obtain a plurality of second cluster categories; each second cluster category includes at least one first sub-image;

[0167] According to the mapping relationship between the first sub-image and the corresponding sample image, the first cluster category corresponding to each second cluster category is obtained.

[0168] In some embodiments, the first processing submodule is further used to: for a sample image detected by the target deep learning model, determine a first bounding box of the target object from a true label mask image of the sample image, and determine a second bounding box of the target object from a predicted label mask image of the sample image;

[0169] When it is determined that there is no intersection area between the area corresponding to the first circumscribed frame in the sample image and the area corresponding to the second circumscribed frame in the sample image, the sample image is determined to be a sample image of the second type.

[0170] In some embodiments, the model type includes an image segmentation model; the sample type includes a third type; the sample image of the third type is an image of a target object that is misdetected by the target deep learning model; the first processing module 410 further includes:

[0171] A second processing submodule, for determining a second bounding box where the target object is located from the predicted label mask image of each sample image when the model type is an image segmentation type and the sample type is a third type;

[0172] Cutting and scaling a second sub-image of an area corresponding to the second bounding box from each sample image;

[0173] Creating a mapping relationship between each sample image and the corresponding second sub-image;

[0174] Clustering the second sub-images of each sample image according to a target clustering algorithm to obtain a plurality of third cluster categories; each third cluster category includes at least one second sub-image;

[0175] According to the mapping relationship between the second sub-image and the corresponding sample image, the first cluster category corresponding to each third cluster category is obtained.

[0176] In some embodiments, the first processing module 410 further includes:

[0177] The third processing submodule is used for:

[0178] When the model type of the target deep learning model is an image classification type, scaling and / or cropping each sample image to the same size;

[0179] Each sample image is clustered according to a target clustering algorithm to obtain a plurality of first cluster categories.

[0180] In some embodiments, the first processing module 410 is further configured to determine the target quantity based on the following method:

[0181] Obtaining the target number according to the target sampling ratio and the number of sample images included in the first clustering category; or

[0182] Get the preset target quantity.

[0183] In some embodiments, the first processing module 410 is further configured to:

[0184] Determine the image to be loaded according to the model type and / or the sample type of each sample image; the image to be loaded is at least one of the sample image, the real label mask image of the sample image, and the predicted label mask image;

[0185] Load the image to be loaded according to the image path of the image to be loaded.

[0186] The sampling device of the sample image in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0187] The sampling device of the sample image in the embodiment of the present application may be a device having an operating system. The operating system may be a Microsoft (Windows) operating system, an Android (Android) operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0188] The sample image sampling device provided in the embodiment of the present application can achieve Figures 1 to 3 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0189] In some embodiments, Figure 5 As shown, the embodiment of the present application also provides an electronic device 400, including a processor 501, a memory 502, and a computer program stored in the memory 502 and executable on the processor 501. When the program is executed by the processor 501, each process of the above-mentioned sample image sampling method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0190] It should be noted that the electronic devices in the embodiments of the present application include mobile electronic devices and non-mobile electronic devices.

[0191] An embodiment of the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned sample image sampling method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0192] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0193] An embodiment of the present application further provides a computer program product, including a computer program, which implements the above-mentioned sample image sampling method when executed by a processor.

[0194] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0195] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned sample image sampling method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0196] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0197] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0198] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0199] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0200] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0201] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

Claims

1. A method for sampling a sample image, characterized in that: include: Clustering a batch of sample images according to at least one of a model type of a target deep learning model, a sample type of each sample image, a true label mask image, and a predicted label mask image to obtain a plurality of first cluster categories; the model type is an image classification type or an image segmentation type; each of the first cluster categories includes at least one sample image; For each first cluster category, a target number of target sample images are collected from the sample images included in the first cluster category; the target number is less than the number of sample images included in the corresponding first cluster category; Determine that the target sample image is a training sample for the target deep learning model.

2. The method for sampling sample images according to claim 1, characterized in that: The sample types include a first type and a second type; the sample images of the first type are images that have not been detected by the target deep learning model; the sample images of the second type are images of target objects that are missed by the target deep learning model; clustering batches of sample images according to at least one of the model type of the target deep learning model, the sample type of each sample image, a true label mask image, and a predicted label mask image includes: When the model type is the image segmentation type and the sample type is the first type or the second type, determining a first bounding box where the target object is located from the true label mask image of each sample image; Cutting and scaling a first sub-image of an area corresponding to the first external frame from each sample image; Creating a mapping relationship between each sample image and the corresponding first sub-image; Clustering the first sub-images of each sample image according to a target clustering algorithm to obtain a plurality of second cluster categories; each second cluster category includes at least one first sub-image; According to the mapping relationship between the first sub-image and the corresponding sample image, the first cluster category corresponding to each second cluster category is obtained.

3. The method for sampling sample images according to claim 2, characterized in that: The second type of sample image is determined based on the following method: For a sample image detected by the target deep learning model, determine a first bounding box of a target object from a true label mask image of the sample image, and determine a second bounding box of the target object from a predicted label mask image of the sample image; When it is determined that there is no intersection area between the area corresponding to the first circumscribed frame in the sample image and the area corresponding to the second circumscribed frame in the sample image, the sample image is determined to be the sample image of the second type.

4. The method for sampling sample images according to claim 2, characterized in that: The model type includes an image segmentation model; the sample type includes a third type; the sample image of the third type is an image of a target object that is misdetected by the target deep learning model; clustering batches of sample images according to at least one of the model type of the target deep learning model, the sample type of each sample image, a true label mask image, and a predicted label mask image, includes: When the model type is the image segmentation type and the sample type is the third type, determining a second bounding box where the target object is located from the predicted label mask image of each sample image; Cutting and scaling a second sub-image of the area corresponding to the second external frame from each sample image; Creating a mapping relationship between each sample image and the corresponding second sub-image; Clustering the second sub-images of each sample image according to a target clustering algorithm to obtain a plurality of third cluster categories; each third cluster category includes at least one second sub-image; According to the mapping relationship between the second sub-image and the corresponding sample image, the first cluster category corresponding to each third cluster category is obtained.

5. The method for sampling sample images according to claim 1, characterized in that: The clustering of batches of sample images according to at least one of the model type of the target deep learning model, the sample type of each sample image, the real label mask image, and the predicted label mask image includes: When the model type of the target deep learning model is the image classification type, scaling and / or cropping each sample image to the same size; Each sample image is clustered according to a target clustering algorithm to obtain a plurality of first cluster categories.

6. The method for sampling sample images according to claim 1, characterized in that: Before collecting a target number of target sample images from the sample images included in the first cluster category, the method further includes: The target quantity is determined based on: Obtaining the target number according to the target sampling ratio and the number of sample images included in the first clustering category; or Obtain the preset target quantity.

7. The method for sampling a sample image according to any one of claims 1 to 6, characterized in that: The clustering of batches of sample images according to at least one of the model type of the target deep learning model, the sample type of each sample image, the real label mask image, and the predicted label mask image includes: Determine an image to be loaded according to the model type and / or the sample type of each sample image; the image to be loaded is at least one of a sample image, a real label mask image of the sample image, and a predicted label mask image; The image to be loaded is loaded according to the image path of the image to be loaded.

8. A sampling device for a sample image, characterized in that: include: A first processing module, configured to cluster a batch of sample images according to at least one of a model type of a target deep learning model, a sample type of each sample image, a true label mask image, and a predicted label mask image, to obtain a plurality of first cluster categories; the model type is an image classification type or an image segmentation type; each of the first cluster categories includes at least one sample image; A second processing module is used to collect a target number of target sample images from the sample images included in the first cluster category for each first cluster category; the target number is less than the number of sample images included in the corresponding first cluster category; The third processing module is used to determine that the target sample image is a training sample of the target deep learning model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the sample image sampling method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the sample image sampling method according to any one of claims 1 to 7 is implemented.

Images