Model training method, image feature extraction method, target detection method and device
By clustering and adjusting the loss value of the sample images extracted during the model training process, self-supervised learning is achieved, and the high cost and low accuracy problems caused by manual annotation are solved, which improves the training efficiency and result accuracy of the model.
Patent Information
- Application Number
- CN202011035233.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-09-27
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2040-09-27
AI Technical Summary
In the prior art, there is a problem that due to manual labeling of samples during model training, the problem of high labor costs and inaccurate model output results.
By extracting part of the sample images from the first sample set as the target sample image, input it into the initial model, obtaining feature information, clustering, determining the positive and negative sample images, and adjusting the model parameters based on the loss value, realizing the image feature extraction model Training, adopt self-supervised learning method to reduce the dependence of manual annotation.
It reduces labor costs, avoids subjective differences caused by manual annotation, and improves the accuracy of model output results.
Smart Images

Figure CN112232384B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of computer technology, and specifically to a model training method, an image feature extraction method, and a target detection method and device. Background Art
[0002] With the development of artificial intelligence, model training tasks are increasing. For example, when training models such as image feature extraction models and object detection models, a large number of labeled samples are usually required to enable the model to perform fully supervised learning.
[0003] In the prior art, samples are usually labeled manually, and the labeled samples are used for model training. This manual labeling method is labor-intensive. At the same time, the subjective differences in manual labeling will lead to inaccurate labeling, which will lead to inaccurate output results of the model. Summary of the invention
[0004] The embodiments of the present application propose a model training method, an image feature extraction method, a target detection method and a device to solve the technical problems in the prior art of high labor costs due to manual labeling of samples during the model training process and low accuracy of the model's output results.
[0005] In a first aspect, an embodiment of the present application provides an image feature extraction model training method, the method comprising: obtaining a first sample set, the first sample set including sample images; extracting some sample images from the first sample set as target sample images, and performing the following training steps: inputting each target sample image into an initial model to obtain feature information of each target sample image; clustering the obtained feature information, and determining a negative sample image corresponding to each target sample image based on the clustering result; determining a positive sample image corresponding to each target sample image; determining a loss value based on the positive sample images and negative sample images corresponding to each target sample image, and adjusting the parameters of the initial model based on the loss value; in response to detecting that the training of the initial model is completed, determining the initial model after adjusting the parameters as the image feature extraction model.
[0006] In a second aspect, an embodiment of the present application provides an image feature extraction method, the method comprising: acquiring a target image; inputting the target image into an image feature extraction model trained using the method in the first aspect to obtain feature information of the target image.
[0007] In a third aspect, an embodiment of the present application provides a target detection method, comprising: acquiring a target image; inputting the target image into a pre-trained target detection model to obtain a target detection result of the target image, wherein the target detection model includes an image feature extraction model, and the image feature extraction model is trained by the method in the first aspect.
[0008] In a fourth aspect, an embodiment of the present application provides an image feature extraction model training device, the device comprising: an acquisition unit, configured to acquire a first sample set, the first sample set including sample images; a first training unit, configured to extract some sample images from the first sample set as target sample images, and perform the following training steps: input each target sample image into an initial model to obtain feature information of each target sample image; cluster the obtained feature information, and determine the negative sample image corresponding to each target sample image based on the clustering result; determine the positive sample image corresponding to each target sample image; determine a loss value based on the positive sample image and the negative sample image corresponding to each target sample image, and adjust the parameters of the initial model based on the loss value; in response to detecting that the training of the initial model is completed, determine the initial model after adjusting the parameters as the image feature extraction model.
[0009] In the fifth aspect, an embodiment of the present application provides an image feature extraction device, which includes: an acquisition unit, configured to acquire a target image; an input unit, configured to input the target image into an image feature extraction model trained using the method in the first aspect, to obtain feature information of the target image.
[0010] In the sixth aspect, an embodiment of the present application provides a target detection device, which includes: an acquisition unit, configured to acquire a target image; an input unit, configured to input the target image into a pre-trained target detection model to obtain a target detection result of the target image, wherein the target detection model includes an image feature extraction model, and the image feature extraction model is trained by the method in the first aspect.
[0011] In the seventh aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in the first aspect.
[0012] In an eighth aspect, an embodiment of the present application provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.
[0013] The model training method, image feature extraction method, target detection method and device provided by the embodiments of the present application extract part of the sample images from the first sample set as the target sample images, input each target sample image into the initial model, and obtain the feature information of each target sample image; then cluster the obtained feature information, and determine the negative sample image corresponding to each target sample image based on the clustering result; determine the positive sample image corresponding to each target sample image; then determine the loss value based on the positive sample image and the negative sample image corresponding to each target sample image, and adjust the parameters of the initial model based on the loss value; finally, when the initial model training is completed, the initial model after adjusting the parameters is determined as the image feature extraction model. This implementation method clusters the feature information to automatically determine the positive and negative samples, thereby adopting a self-supervised learning training method. The samples used in this training method do not need to be manually labeled, thereby greatly reducing the labor cost. Since it does not involve the subjective difference problem in the sample labeling process, the accuracy of the model output result is improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:
[0015] Figure 1 is a flow chart of an embodiment of a model training method according to the present application;
[0016] Figure 2 is a schematic diagram of the model training process according to the present application;
[0017] Figure 3 is a flow chart of an embodiment of the image feature extraction method according to the present application;
[0018] Figure 4 is a flow chart of an embodiment of a target detection method according to the present application;
[0019] Figure 5 is a structural schematic diagram of an embodiment of a model training device according to the present application;
[0020] Figure 6 is a structural schematic diagram of an embodiment of an image feature extraction device according to the present application;
[0021] Figure 7 is a schematic structural diagram of an embodiment of a target detection device according to the present application;
[0022] Figure 8 It is a schematic diagram of the structure of a computer system for implementing an electronic device in an embodiment of the present application. DETAILED DESCRIPTION
[0023] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.
[0024] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0025] Please refer to Figure 1 , which shows a process 100 according to an embodiment of a model training method of the present application. The model training method comprises the following steps:
[0026] Step 101: Obtain a first sample set.
[0027] In this embodiment, the execution subject of the model training method (such as an electronic device such as a server) can obtain the first sample set in a variety of ways. For example, the execution subject can obtain the existing first sample set stored in another server (such as a database server) for storing samples through a wired connection or a wireless connection. For another example, a user can collect samples through a terminal device, so that the above-mentioned execution subject can receive the samples collected by the terminal and store these samples locally, thereby generating a first sample set. It should be pointed out that the above-mentioned wireless connection method may include but is not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX (World Interoperability for Microwave Access, Global Microwave Access Interoperability) connection, Zigbee (Purple Bee Protocol) connection, UWB (ultra wideband, ultra-wideband) connection, and other wireless connection methods currently known or to be developed in the future.
[0028] In this embodiment, the first sample set may include a large number of sample images, where the sample images may be unlabeled images.
[0029] It is understandable that many image processing models (such as target detection models, target recognition models, image classification models, etc.) need to extract image features when processing images, and therefore all include image feature extraction networks. Conventional training methods usually use labeled image samples to supervise the entire model, and therefore need to rely on a large number of accurately labeled samples. However, the role of the image feature extraction network in various image processing models is to extract the feature information of the image, and it does not need to output accurate results (such as classification results, recognition results, etc.). Therefore, the structure and parameters of the image feature extraction network in the image processing model of different application scenarios are often the same or similar, and the difference mainly lies in the subsequent network structure (such as classification network) connected to the image feature extraction network. In view of this, the embodiment of the present application uses the image feature extraction network in these image processing models as a training object, and uses a large number of unlabeled image samples to train the image feature extraction network to obtain an image feature extraction model that can accurately extract image features. On this basis, a small amount of labeled image samples can be further used to train the entire image processing model so that the image processing model accurately outputs the desired results. Since unlabeled images are easy to obtain, this embodiment uses unlabeled images to train the image feature extraction model, which can greatly save the manpower cost of the labeling process and avoid the problem of inaccurate model output caused by subjective differences in manual labeling.
[0030] In some optional implementations of this embodiment, the first sample set can be obtained by performing data enhancement on the second sample set. At this time, the first sample set may include the original sample images in the second sample set and the enhanced sample images after the original sample images are enhanced. Specifically, an unlabeled second sample set may be first obtained, and the second sample set includes the original sample images. The second sample set here may be various existing unlabeled image sets. Afterwards, at least one of the following operations is performed on the original sample image to obtain an enhanced sample image corresponding to the original sample image: random cropping, horizontal flipping, chromaticity adjustment, brightness adjustment, saturation adjustment, and Gaussian noise addition. Finally, the original sample images in the second sample set and the obtained enhanced sample images are aggregated to obtain the first sample set. In practice, the data enhancement operation can be performed in a GPU (Graphics Processing Unit) to shorten the image processing time.
[0031] Step 102: extract some sample images from the first sample set as target sample images.
[0032] In this embodiment, the execution subject may select samples from the first sample set in step 1, and perform the training steps from step 103 to step 106. The method and number of samples to be extracted are not limited in this application. For example, at least one sample may be randomly extracted, or a sample with better clarity of the sample image may be extracted.
[0033] Step 103: input each target sample image into the initial model to obtain feature information of each target sample image.
[0034] In this embodiment, the image feature extraction network in an untrained image processing model (such as a target detection model, a target recognition model, an image classification model, etc.) can be used as an initial model, and each target sample image extracted in the previous step can be input into the initial model to obtain the feature information of each target sample image.
[0035] As an example, the initial model can be the backbone network in the target detection model. After each target sample image is input into the initial model, feature maps of multiple scales can be output, such as feature maps of 5 scales, which are respectively denoted as P3, P4, P5, P6, and P7. Among them, P3, P4, and P5 can be obtained by convolution of 1×1 size with three feature maps (which can be denoted as C3, C4, and C5, respectively) generated by a deep convolutional neural network, P6 can be obtained by convolution operation of P5 with a step size of 2 (which can be regarded as downsampling), and P7 can be obtained by convolution operation of P6 with a step size of 2. This embodiment does not limit the number and scale of feature maps. Here, the backbone network can be connected to a fully connected layer (FC) to obtain the feature maps of multiple scales. Figure 1 The data are input into the fully connected layer to obtain feature information in vector form.
[0036] Step 104 , clustering the obtained feature information, and determining the negative sample image corresponding to each target sample image based on the clustering result; and determining the positive sample image corresponding to each target sample image.
[0037] In this embodiment, the execution subject may use various clustering algorithms, such as k-means clustering algorithm (k-means), to cluster the obtained feature information and obtain clustering results. As an example, if the preset batch_size (the number of target sample images selected for one training) is 256, and 10 clusters are set, then after clustering the batch_size, the 256 target sample images may be divided into 10 clusters. Among them, the similarity of the feature information of the target sample images in the same cluster is relatively high, and the similarity of the feature information of the target sample images in different clusters is relatively low.
[0038] As an example, the execution subject can cluster the obtained feature information through the following steps: Step 1, obtain a preset number (e.g., 10) of cluster centers. The cluster centers can be in the same form as the feature information, such as 128-dimensional vectors. When the training step is executed for the first time, the cluster centers can be randomly selected from the obtained feature information, and when the training step is executed subsequently, the cluster centers can be updated based on the clustering results.
[0039] The second step is to detect the distance from each feature information to each cluster center. For example, the distance to the cluster center can be represented by the inner product calculation result of the vector.
[0040] In the third step, for each feature information obtained, the cluster corresponding to the cluster center with the smallest distance to the feature information is taken as the cluster to which the feature information belongs.
[0041] It should be noted that after clustering the obtained feature information, for each cluster, the above-mentioned execution subject can select one feature information from the cluster as the target feature information one by one, and weighted sum the cluster center of the cluster and the target feature information to obtain the weighted sum result, and finally replace the cluster center of the cluster with the weighted sum result. For details, please refer to the following formula:
[0042] C'=a×C+(1-a)×current_feature
[0043] Among them, a is the preset weight, C is the original cluster center, current_feature is the currently selected target feature information. C' is the updated cluster center. Therefore, during the training process, as the feature information is updated, the clustering results and cluster centers are also updated in real time.
[0044] In this embodiment, after obtaining the clustering results of the feature information, the above-mentioned execution entity can determine the positive sample images and negative sample images corresponding to each target sample image based on the clustering results; it can also determine the negative sample images corresponding to each target sample image based on the clustering results, and determine the positive sample images corresponding to each target sample image by other means.
[0045] As an example, for each target sample image, the execution entity may use other target sample images belonging to the same cluster as the target sample image as positive sample images, and use other target sample images belonging to different clusters as negative sample images.
[0046] As another example, for each target sample image, the above-mentioned execution entity may select some sample images (such as target sample images that meet preset conditions) from other target sample images belonging to the same cluster as the target sample image as positive sample images, and use other target sample images that belong to different clusters from the target sample image as negative sample images.
[0047] As another example, for each target sample image, the above-mentioned execution entity may select some sample images (such as target sample images that meet preset conditions) from other target sample images belonging to the same cluster as the target sample image as positive sample images, and use the remaining sample images as negative sample images.
[0048] In some optional implementations of the present embodiment, the sample images in the first sample set may include an original sample image (such as an image A of a dog) and an enhanced sample image corresponding to the original sample image (for example, including image A' obtained by horizontally flipping image A and image A" obtained by chromatically adjusting image A). At this time, for each target sample image, other samples belonging to the same cluster as the target sample image usually include the original sample image and / or enhanced sample image of the target sample image. Specifically, the following two cases are included: if the target sample image is the original sample image, then other samples belonging to the same cluster as the target sample image include the enhanced sample image of the target sample image. If the target sample image is the enhanced sample image, then other samples belonging to the same cluster as the target sample image include the original sample image of the target sample image and other enhanced sample images of the original sample image.
[0049] At this time, based on the clustering results, the above execution entity can determine the positive sample image and the negative sample image corresponding to each target sample image through the following steps:
[0050] The first step is to set a cluster label for each target sample image based on the clustering results.
[0051] Among them, target sample images whose feature information belongs to the same cluster have the same cluster label, and target sample images whose feature information does not belong to the same cluster have different cluster labels.
[0052] Cluster labels are used to indicate and distinguish different clusters, and cluster labels are different from category labels in conventional labeled sample images. For example, a cluster may include an original sample image A of a poodle, enhanced sample images A' and A" of the poodle, and an original sample image B of a Samoyed and enhanced sample images B' and B" of the Samoyed. If A, A', A", B, B' and B" are clustered into one category, then A, A', A", B, B' and B" may have the same cluster label.
[0053] In the second step, for each target sample image, an enhanced sample image and / or an original sample image corresponding to the target sample image is selected from the remaining sample images having the same clustering label as the target sample image as the positive sample image corresponding to the target sample image; and each sample image having a different clustering label from the target sample image is selected as the negative sample image corresponding to the target sample image.
[0054] Continuing with the above example, if a target sample image is an original sample image A of a poodle in a cluster, its corresponding positive sample image may include enhanced sample images A' and A". of the poodle, and its corresponding negative sample image may include target sample images in other clusters, such as an original sample image C of a British shorthair cat, enhanced sample images C' and C". of the British shorthair cat, and an original sample image D of a ginger cat, enhanced sample images D' and D". of the British shorthair cat, etc.
[0055] It should be noted that the original sample image B of a Samoyed belonging to the same cluster as the target sample image and the enhanced sample images B' and B'' of the Samoyed may not be considered in the process of training the initial model, and may be specifically processed according to the task scenario in the subsequent training of the image processing model (such as the target detection model, the target recognition model, the image classification model, etc.). For example, if the training task is to classify dog breeds, dogs of different breeds (such as the above-mentioned poodles and Samoyeds) may be used as different types of samples when training the dog breed classification model. If the training task is species classification (such as distinguishing different species such as dogs and cats), dogs of different breeds (such as the above-mentioned poodles and Samoyeds) may be used as samples of the same type when training the species classification model. For another example, if the training task is dog identification (such as distinguishing different dogs), different dogs (such as the above-mentioned poodles and Samoyeds, or a poodle and another poodle) may be used as different types of samples when training the dog identification model.
[0056] In an optional implementation, the enhanced sample image and the original sample image already have a certain association relationship, for example, when the enhanced sample image is generated from the original sample image, the enhanced sample image is named with the same prefix as the original sample image, or when the enhanced sample image is generated, the association between the enhanced sample image and the original sample image is stored. In this case, the enhanced sample image corresponding to the original sample image can be directly searched and used as the positive sample image corresponding to the target sample image, and the execution subject determines the negative sample image corresponding to each target sample image based on the clustering result according to the above steps.
[0057] By determining the positive sample image and negative sample image corresponding to each target sample image based on the clustering results, the distance between the positive sample image and the target sample image can be small enough, while the distance between the negative sample image and the target sample image can be large enough to achieve the purpose of classification. At the same time, by determining the positive sample image and negative sample image corresponding to each target sample image based on the clustering results, the model can autonomously learn the similarity between sample images and realize automatic labeling based on clustering results, avoiding the problem of inaccurate labels caused by human subjective factors, and can greatly reduce the labeling cost.
[0058] Step 105: determine a loss value based on the positive sample images and the negative sample images corresponding to each target sample image, and adjust the parameters of the initial model based on the loss value.
[0059] In this embodiment, the above-mentioned execution subject can determine the loss value based on the positive sample image and the negative sample image corresponding to each target sample image, and adjust the parameters of the initial model based on the loss value. Among them, the above-mentioned loss value is the value of the loss function, and the loss function is a non-negative real-valued function that can be used to characterize the difference between the detection result and the true result. In general, the smaller the loss value, the better the robustness of the model. The loss function can be set according to actual needs. Here, the distance between the feature information of each target sample image and the feature information of the positive sample image corresponding to it, and the distance between the feature information of each target sample image and the feature information of the negative sample image corresponding to it are first calculated. Then, the calculated distance is input into the preset loss function to obtain the loss value. After that, the above-mentioned execution subject can use the loss value to update the parameters of the initial model. Here, the gradient of the loss value relative to the model parameters can be obtained by the back propagation algorithm, and then the model parameters are updated based on the gradient by the gradient descent algorithm. Thus, each time a sample image is input, the parameters of the convolutional neural network can be updated once based on the loss value corresponding to the sample image until the training is completed.
[0060] In practice, whether training is completed can be determined in a variety of ways. As an example, when the loss value converges to a certain value, it can be determined that the training is completed. As another example, if the number of training times of the initial model is equal to a preset number of times, it can be determined that the training is completed. This embodiment does not specifically limit the determination conditions for the completion of training.
[0061] It should be noted that if it is determined that the initial model has been trained, the following step 106 can be continued. If it is determined that the initial model has not been trained, some sample images can be extracted from the first sample set as target sample images, and the initial model with adjusted parameters and the new target sample images can be used to continue the above training steps. It should be pointed out that the extraction method here is not limited in this application. For example, when there are a large number of sample images in the first sample set, the execution subject can extract sample images that have not been extracted from them.
[0062] In some optional implementations of this embodiment, the above execution entity may calculate the loss value by the following steps:
[0063] In the first step, for each target sample image, the sum of the distances between the feature information of the target sample image and the feature information of each positive sample image corresponding to the target sample image is detected (for ease of distinction, it can be called the sum of the first distances), and the sum of the second distances between the feature information of the target sample image and the feature information of each negative sample image corresponding to the target sample image is detected (can be called the sum of the second distances).
[0064] In the second step, the ratio of the sum of the first distances and the sum of the second distances corresponding to each target sample image is used as the loss value corresponding to the target sample image.
[0065] The third step is to sum the loss values corresponding to each target sample image to obtain the loss value of the above initial model.
[0066] Therefore, for each target sample image, the smaller the difference between the feature information of the target sample image and the feature information of the corresponding positive sample images, and the larger the difference between the feature information of the corresponding negative sample images, the smaller the loss value. The ultimate goal is to make the difference between the feature information of the target sample image and the feature information of the corresponding positive sample images as small as possible, and the difference between the feature information of the corresponding negative sample images as large as possible, so as to improve the accuracy of feature extraction by the model.
[0067] Step 106, in response to detecting that the initial model training is completed, determining the initial model after adjusting parameters as the image feature extraction model.
[0068] In this embodiment, in response to detecting that the initial model training is completed, the execution entity may determine the initial model after adjusting parameters as the image feature extraction model.
[0069] As an example, Figure 2 is a schematic diagram of the model training process according to this application. Figure 2 As shown, first obtain a first sample set, which may contain a large number of sample images, specifically including original sample images and enhanced sample images corresponding to the original sample images. Then, select a group of target sample images from the first sample set and input them into the initial model, such as a backbone network, to obtain a multi-scale feature map. Then, input the multi-scale feature map into the fully connected layer to obtain feature information in vector form. Then, cluster the feature information, and determine the positive sample image and negative sample image corresponding to the target sample image based on the clustering result. Then determine the loss value based on the positive sample image and the negative sample image, and finally update the parameters of the initial model based on the loss value. Thus, a training process is completed. The above training process can be iteratively performed multiple times to obtain an image feature extraction model.
[0070] In some optional implementations of this embodiment, the execution entity may store the generated image feature extraction model locally, or send it to a terminal device or a database server for storing data.
[0071] In some optional implementations of this embodiment, the execution subject may use the image feature extraction model as the feature extraction network in the target detection model to establish an initial target detection model. Among them, in addition to the feature extraction network, the initial target detection model may also include a classification network, a position detection network, etc. Then, the above-mentioned execution subject may obtain a third sample set that has been labeled, and the third sample set includes sample images with category annotations. Finally, the sample images in the third sample set are used as input, and based on the category annotations of the input sample images, the initial target detection model is trained using a machine learning method to obtain a trained target detection model. Since the feature extraction network in the target detection model has been trained, the training of the initial target detection model can be completed using a small number of labeled image samples. As a result, the manpower cost of manual labeling can be greatly reduced.
[0072] The method provided by the above-mentioned embodiment of the present application extracts part of the sample images from the first sample set as the target sample images, inputs each target sample image into the initial model, and obtains the feature information of each target sample image; then clusters the obtained feature information, and determines the negative sample image corresponding to each target sample image based on the clustering result; determines the positive sample image corresponding to each target sample image; then determines the loss value based on the positive sample image and the negative sample image corresponding to each target sample image, and adjusts the parameters of the initial model based on the loss value; finally, when the initial model training is completed, the initial model after adjusting the parameters is determined as the image feature extraction model. This implementation method clusters the feature information to automatically determine the positive and negative samples, thereby adopting a self-supervised learning training method. The samples used in this training method do not need to be manually labeled, thereby greatly reducing the labor cost. Since it does not involve the subjective difference problem in the sample labeling process, it improves the accuracy of the model output results.
[0073] Further references Figure 3 , which shows a process 300 of another embodiment of an image feature extraction method. The process 300 of the image feature extraction method includes the following steps:
[0074] Step 301, acquiring a target image.
[0075] In this embodiment, the execution subject of the image feature extraction method may obtain a target image, wherein the target image may be an image whose image features are to be extracted.
[0076] Step 302: input the target image into a pre-trained image feature extraction model to obtain feature information of the target image.
[0077] In this embodiment, the execution subject can input the target image into a pre-trained image feature extraction model to obtain feature information of the target image. The image feature extraction model in this embodiment can be a model such as the one described above. Figure 1 The specific generation process can be found in Figure 1 The relevant description of the embodiments will not be repeated here.
[0078] It should be noted that the image feature extraction method of this embodiment can be used to test the image feature extraction model generated by the above embodiment. Then, the image feature extraction model can be continuously optimized according to the test results. This method can also be a practical application method of the image feature extraction model generated by the above embodiment. Using the image feature extraction model generated by the above embodiment to perform image feature extraction helps to improve the accuracy of the extracted image features.
[0079] Further references Figure 4, which shows a process 400 of another embodiment of a target detection method. The process 400 of the target detection method includes the following steps:
[0080] Step 401, acquiring a target image.
[0081] In this embodiment, the execution subject of the target detection method may obtain a target image, wherein the target image may be an image whose image features are to be extracted.
[0082] Step 402, input the target image into a pre-trained target detection model to obtain a target detection result of the target image, wherein the target detection model includes an image feature extraction model.
[0083] In this embodiment, the execution subject may input the target image into a pre-trained target detection model to obtain feature information of the target image. The target detection model in this embodiment may include an image feature extraction model to extract image features. The image feature extraction model is a model that uses the above-mentioned Figure 1 The specific generation process can be found in Figure 1 The relevant description of the embodiments will not be repeated here.
[0084] It should be noted that the target detection method of this embodiment can be used to test the image feature extraction model generated by the above embodiment. Then, the image feature extraction model can be continuously optimized according to the test results. This method can also be a practical application method of the image feature extraction model generated by the above embodiment. Using the image feature extraction model generated by the above embodiment to extract image features in the target detection process helps to improve the accuracy of the extracted image features and improve the accuracy of target detection.
[0085] Further references Figure 5 As an implementation of the methods shown in the above figures, the present application provides an embodiment of an image feature extraction model training device, and the device embodiment is similar to Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.
[0086] like Figure 5As shown, the model training device 500 of this embodiment includes: an acquisition unit 501, configured to acquire a first sample set, wherein the first sample set includes sample images; a first training unit 502, configured to extract part of the sample images from the first sample set as target sample images, and perform the following training steps: input each target sample image into the initial model to obtain feature information of each target sample image; cluster the obtained feature information, and determine the negative sample image corresponding to each target sample image based on the clustering result; determine the positive sample image corresponding to each target sample image; determine the loss value based on the positive sample image and the negative sample image corresponding to each target sample image, and adjust the parameters of the initial model based on the loss value; in response to detecting that the training of the initial model is completed, determine the initial model after adjusting the parameters as the image feature extraction model.
[0087] In some optional implementations of the present embodiment, the acquisition unit 501 is further configured to: acquire an unlabeled second sample set, wherein the second sample set includes an original sample image; perform at least one of the following operations on the original sample image to obtain an enhanced sample image corresponding to the original sample image: random cropping, horizontal flipping, chroma adjustment, brightness adjustment, saturation adjustment, and Gaussian noise addition; and aggregate the original sample image in the second sample set and the obtained enhanced sample image to obtain a first sample set.
[0088] In some optional implementations of the present embodiment, the sample images in the first sample set include original sample images and enhanced sample images corresponding to the original sample images; and the first training unit 502 is further configured to: set a clustering label for each target sample image based on the clustering result, wherein target sample images whose feature information belongs to the same cluster have the same clustering label, and target sample images whose feature information does not belong to the same cluster have different clustering labels; for each target sample image, select the enhanced sample image and / or the original sample image corresponding to the target sample image from the remaining sample images having the same clustering label as the target sample image as the positive sample image corresponding to the target sample image; and use each sample image having a different clustering label from the target sample image as the negative sample image corresponding to the target sample image.
[0089] In some optional implementations of the present embodiment, the first training unit 502 is further configured to: for each target sample image, detect the sum of the first distances between the feature information of the target sample image and the feature information of each positive sample image corresponding to the target sample image, and detect the sum of the second distances between the feature information of the target sample image and the feature information of each negative sample image corresponding to the target sample image; use the ratio of the sum of the first distances to the sum of the second distances as the loss value corresponding to the target sample image; sum the loss values corresponding to each target sample image to obtain the loss value of the initial model.
[0090] In some optional implementations of this embodiment, the above-mentioned first training unit 502 is further configured to: obtain a preset number of cluster centers; detect the distance from each obtained feature information to each cluster center; for each obtained feature information, take the cluster corresponding to the cluster center with the smallest distance to the feature information as the cluster to which the feature information belongs.
[0091] In some optional implementations of this embodiment, the above-mentioned training step also includes: for each cluster, selecting one feature information from the cluster as the target feature information, weightedly summing the cluster center of the cluster with the above-mentioned target feature information to obtain a weighted summation result, and replacing the cluster center of the cluster with the above-mentioned weighted summation result.
[0092] In some optional implementations of this embodiment, the above-mentioned device also includes: an execution unit, which is configured to: in response to detecting that the above-mentioned initial model has not been trained, re-extract some sample images from the above-mentioned first sample set as target sample images, and use the initial model with adjusted parameters and the new target sample images to continue to execute the above-mentioned training steps.
[0093] In some optional implementations of the present embodiment, the above-mentioned device also includes: a second training unit, configured to: use the above-mentioned image feature extraction model as the feature extraction network in the target detection model to establish an initial target detection model; obtain a labeled third sample set, wherein the above-mentioned third sample set includes sample images with category annotations; use the sample images in the above-mentioned third sample set as input, and based on the category annotations of the input sample images, use a machine learning method to train the above-mentioned initial target detection model to obtain a trained target detection model.
[0094] The device provided by the above-mentioned embodiment of the present application extracts part of the sample images from the first sample set as the target sample images, inputs each target sample image into the initial model, and obtains the feature information of each target sample image; then clusters the obtained feature information, and determines the negative sample image corresponding to each target sample image based on the clustering result; determines the positive sample image corresponding to each target sample image; then determines the loss value based on the positive sample image and the negative sample image corresponding to each target sample image, and adjusts the parameters of the initial model based on the loss value; finally, when the initial model training is completed, the initial model after adjusting the parameters is determined as the image feature extraction model. This implementation method clusters the feature information to automatically determine the positive and negative samples, thereby adopting a self-supervised learning training method. The samples used in this training method do not need to be manually labeled, thereby greatly reducing the labor cost. Since it does not involve the subjective difference problem in the sample labeling process, the accuracy of the model output result is improved.
[0095] Further references Figure 6 As an implementation of the methods shown in the above figures, the present application provides an embodiment of an image feature extraction device, which is similar to Figure 3 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.
[0096] like Figure 6 As shown, the image feature extraction device 600 in this embodiment includes: an acquisition unit 601, configured to acquire a target image; an input unit 602, configured to input the target image into an image feature extraction model to obtain feature information of the target image.
[0097] It is understood that the units described in the device 600 are similar to those described in the reference Figure 3 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the device 600 and the units included therein, and will not be described in detail here.
[0098] Further references Figure 7 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a target detection device, which is similar to Figure 4 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.
[0099] like Figure 7As shown, the target detection device 700 of this embodiment includes: an acquisition unit 701, configured to acquire a target image; an input unit 702, configured to input the target image into a pre-trained target detection model to obtain a target detection result of the target image, wherein the target detection model includes an image feature extraction model, and the image feature extraction model is Figure 4 The method in the corresponding embodiment is trained.
[0100] It is understood that the units described in the device 700 are similar to those described in the reference Figure 3 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the device 700 and the units included therein, and will not be described in detail here.
[0101] Reference below Figure 8 , which shows a schematic diagram of the structure of a computer system 800 suitable for implementing an electronic device of an embodiment of the present application. Figure 8 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0102] like Figure 8 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the system 800 are also stored. The CPU 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.
[0103] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that a computer program read therefrom is installed into the storage section 808 as needed.
[0104] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the central processing unit (CPU) 801, the above functions defined in the method of the present application are executed. It should be noted that the computer-readable medium described in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, an apparatus or a device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, an apparatus or a device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.
[0105] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0106] The units involved in the embodiments described in this application may be implemented by software or hardware. The units described may also be arranged in a processor, wherein the names of these units do not constitute limitations on the units themselves in certain circumstances.
[0107] As another aspect, the present application also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by the device, the device: obtains a first sample set, which includes sample images; extracts part of the sample images from the first sample set as target sample images, and performs the following training steps: inputs each target sample image into the initial model to obtain feature information of each target sample image; clusters the obtained feature information, and determines the negative sample image corresponding to each target sample image based on the clustering result; determines the positive sample image corresponding to each target sample image; determines the loss value based on the positive sample image and the negative sample image corresponding to each target sample image, and adjusts the parameters of the initial model based on the loss value; in response to detecting that the initial model training is completed, the initial model after adjusting the parameters is determined as the image feature extraction model.
[0108] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.
Claims
1. A method for training an image feature extraction model, characterized in that: The method comprises: Acquire a first sample set, the first sample set including sample images, wherein the sample images in the first sample set include original sample images and enhanced sample images corresponding to the original sample images; Extracting some sample images from the first sample set as target sample images, and performing the following training steps: inputting each target sample image into the initial model to obtain feature information of each target sample image; clustering the obtained feature information, and determining the negative sample image corresponding to each target sample image based on the clustering result; determining the positive sample image corresponding to each target sample image; determining the loss value based on the positive sample image and the negative sample image corresponding to each target sample image, and adjusting the parameters of the initial model based on the loss value; in response to detecting that the training of the initial model is completed, determining the initial model after adjusting the parameters as the image feature extraction model; The determining of the negative sample images corresponding to each target sample image based on the clustering result includes: A cluster label is set for each target sample image based on the clustering result, wherein target sample images whose feature information belongs to the same cluster have the same cluster label, and target sample images whose feature information does not belong to the same cluster have different cluster labels; Using each sample image having a different clustering label from the target sample image as a negative sample image corresponding to the target sample image; The determining of the positive sample images corresponding to each target sample image includes: For each target sample image, an enhanced sample image and / or an original sample image corresponding to the target sample image is selected from the remaining sample images having the same clustering label as the target sample image as a positive sample image corresponding to the target sample image; Alternatively, for each target sample image, an enhanced sample image and / or an original sample image corresponding to the target sample image is selected as a positive sample image corresponding to the target sample image.
2. The method according to claim 1, characterized in that: The obtaining of the first sample set comprises: Acquire an unlabeled second sample set, where the second sample set includes original sample images; Perform at least one of the following operations on the original sample image to obtain an enhanced sample image corresponding to the original sample image: random cropping, horizontal flipping, chroma adjustment, brightness adjustment, saturation adjustment, and Gaussian noise addition; The original sample images and the obtained enhanced sample images in the second sample set are aggregated to obtain a first sample set.
3. The method according to any one of claims 1-2, characterized in that: The determining of the loss value based on the positive sample images and the negative sample images corresponding to each target sample image includes: For each target sample image, detecting the sum of first distances between feature information of the target sample image and feature information of each positive sample image corresponding to the target sample image, and detecting the sum of second distances between feature information of the target sample image and feature information of each negative sample image corresponding to the target sample image; taking the ratio of the sum of the first distances to the sum of the second distances as the loss value corresponding to the target sample image; The loss values corresponding to each target sample image are summed to obtain the loss value of the initial model.
4. The method according to claim 3, characterized in that The clustering of the obtained feature information includes: Get a preset number of cluster centers; Detect the distance between each feature information obtained and each cluster center; For each piece of feature information obtained, the cluster corresponding to the cluster center with the smallest distance from the feature information is taken as the cluster to which the feature information belongs.
5. The method according to claim 4, characterized in that After clustering the obtained feature information, the training step further includes: For each cluster, one feature information is selected from the cluster one by one as the target feature information, the cluster center of the cluster and the target feature information are weighted summed to obtain a weighted summation result, and the cluster center of the cluster is replaced by the weighted summation result.
6. The method according to claim 5, characterized in that The method further comprises: In response to detecting that the initial model has not been trained, some sample images are re-extracted from the first sample set as target sample images, and the training step is continued using the initial model with adjusted parameters and the new target sample images.
7. The method according to claim 6, characterized in that The method further comprises: Using the image feature extraction model as a feature extraction network in a target detection model to establish an initial target detection model; Acquire a third labeled sample set, wherein the third sample set includes sample images with category labels; The sample images in the third sample set are taken as input, and based on the category annotations of the input sample images, the initial target detection model is trained using a machine learning method to obtain a trained target detection model.
8. A method for extracting image features, characterized in that: The method comprises: Get the target image; The target image is input into an image feature extraction model trained by the method of any one of claims 1 to 7 to obtain feature information of the target image.
9. A target detection method, characterized in that: The method comprises: Get the target image; The target image is input into a pre-trained target detection model to obtain a target detection result of the target image, wherein the target detection model includes an image feature extraction model, and the image feature extraction model is trained by one of the methods of claims 1-7.
10. An image feature extraction model training device, characterized in that: The device comprises: An acquisition unit is configured to acquire a first sample set, wherein the first sample set includes sample images, wherein the sample images in the first sample set include original sample images and enhanced sample images corresponding to the original sample images; The first training unit is configured to extract some sample images from the first sample set as target sample images, and perform the following training steps: input each target sample image into the initial model to obtain feature information of each target sample image; cluster the obtained feature information, and determine the negative sample image corresponding to each target sample image based on the clustering result; determine the positive sample image corresponding to each target sample image; determine the loss value based on the positive sample image and the negative sample image corresponding to each target sample image, and adjust the parameters of the initial model based on the loss value; in response to detecting that the training of the initial model is completed, determine the initial model after adjusting the parameters as the image feature extraction model; The first training unit is further configured to: set a cluster label for each target sample image based on the clustering result, wherein target sample images whose feature information belongs to the same cluster have the same cluster label, and target sample images whose feature information does not belong to the same cluster have different cluster labels; for each target sample image, select an enhanced sample image and / or an original sample image corresponding to the target sample image from the remaining sample images having the same cluster label as the target sample image as a positive sample image corresponding to the target sample image; or, for each target sample image, select an enhanced sample image and / or an original sample image corresponding to the target sample image as a positive sample image corresponding to the target sample image; and use each sample image having a different cluster label from the target sample image as a negative sample image corresponding to the target sample image.
11. An image feature extraction device, characterized in that: The device comprises: an acquisition unit, configured to acquire a target image; An input unit is configured to input the target image into an image feature extraction model trained by the method of one of claims 1-7 to obtain feature information of the target image.
12. A target detection device, characterized in that: The device comprises: an acquisition unit, configured to acquire a target image; An input unit is configured to input the target image into a pre-trained target detection model to obtain a target detection result of the target image, wherein the target detection model includes an image feature extraction model, and the image feature extraction model is trained by one of the methods of claims 1-7.
13. An electronic device, characterized in that: include: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 9.
14. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.
Citation Information
Patent Citations
Image clustering result evaluation method and apparatus
CN108564102A
Pedestrian re-identification method and device based on unsupervised learning and medium
CN110263697A
Attribute recognition model training method and device, electronic equipment and storage medium
CN111178403A
Video classification model training method, device and equipment and storage medium
CN111401474A
Method and device for identifying image
CN111582185A
