Image processing model training method and device

Abstract

The present disclosure relates to a training method and device for an image processing model, the training method for the image processing model comprising: obtaining a training data set, the training data set comprising a plurality of image data; performing feature extraction on each image data in the training data set to obtain a plurality of image features; clustering the plurality of image features, extracting a first sample from the plurality of clustered image features and determining a second sample based on the first sample, constructing the first sample and the second sample as a positive sample pair, wherein the second sample is a positive sample of the first sample; determining an image feature of the plurality of clustered image features that is different from the second sample category as a negative sample; constructing a target loss function through the positive sample pair and the negative sample; and updating the parameters of the image processing model based on the target loss function to train the image processing model.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and more specifically, to a training method and device for an image processing model. Background Art

[0002] Contrastive learning is very representative in the current training methods of image processing models. It learns the characteristics of given samples through the target loss function. Therefore, the quality of the target loss function directly affects the performance of the trained image processing model.

[0003] In the current mainstream contrastive learning method, due to the characteristics of the target loss function itself, the effect of contrastive learning is not ideal. For example, the use of contrastive loss will cause the network to fall into a local optimization point, making it difficult to fit the overall data; while the use of triple loss will only see a few samples each time, resulting in insufficient recognition of features by the image processing model. It can be seen that the current training method of image processing models still needs to be improved. Summary of the invention

[0004] The present disclosure provides a method and device for training an image processing model to at least solve the problems in the above-mentioned related technologies, or may not solve any of the above-mentioned problems.

[0005] According to a first aspect of an embodiment of the present disclosure, a method for training an image processing model is provided, comprising: obtaining a training data set, the training data set comprising a plurality of image data; performing feature extraction on each image data in the training data set to obtain a plurality of image features; clustering the plurality of image features; extracting a first sample from the plurality of image features after clustering and determining a second sample based on the first sample, constructing the first sample and the second sample as a positive sample pair, wherein the second sample is a positive sample of the first sample; determining an image feature of the plurality of image features after clustering that is different from the second sample category as a negative sample; constructing a target loss function through the positive sample pair and the negative sample; and updating the parameters of the image processing model based on the target loss function to train the image processing model.

[0006] Optionally, extracting a first sample from the multiple image features after clustering and determining a second sample based on the first sample includes: randomly extracting a first sample from the multiple image features after clustering; determining a cluster center corresponding to the first sample; and determining an image feature closest to the cluster center as the second sample.

[0007] Optionally, determining the image features that are different from the second sample category among the multiple image features after clustering as negative samples includes: determining the image features that are different from the second sample category among the image features after clustering stored in a pre-established memory bank as negative samples, wherein the memory bank is used to store a predetermined number of image features selected from the remaining image features of the multiple image features after clustering excluding the positive sample pairs.

[0008] Optionally, constructing a target loss function through the positive sample pairs and the negative samples, and updating the parameters of the image processing model based on the target loss function to train the image processing model, includes: calculating the value of the target loss function through the positive sample pairs and all negative samples in the memory bank; and updating the parameters of the image processing model based on the value of the target loss function to train the image processing model.

[0009] Optionally, after the parameters of the image processing model are updated, the stored clustered image features are removed from the memory bank, and then a new predetermined number of image features are selected from the remaining image features of the multiple clustered image features except the positive sample pairs and the used image features and stored in the memory bank for the next update of the parameters of the image processing model.

[0010] According to a second aspect of an embodiment of the present disclosure, a training device for an image processing model is provided, comprising: a training data acquisition unit, configured to acquire a training data set, wherein the training data set comprises a plurality of image data; a feature extraction unit, configured to perform feature extraction on each training data in the training data set to obtain a plurality of image features; a feature clustering unit, configured to cluster the plurality of image features; a positive sample pair determination unit, configured to extract a first sample from the plurality of image features after clustering and determine a second sample based on the first sample, and construct the first sample and the second sample into a positive sample pair, wherein the second sample is a positive sample of the first sample; a negative sample determination unit, configured to determine an image feature in the plurality of image features after clustering that is different from the second sample category as a negative sample; a target loss function construction unit, configured to construct a target loss function through the positive sample pair and the negative sample; and a model parameter updating unit, configured to update the parameters of the image processing model based on the target loss function to train the image processing model.

[0011] Optionally, the positive sample pair determination unit may be configured to: randomly extract a first sample from the plurality of clustered image features; determine a cluster center corresponding to the first sample; and determine an image feature closest to the cluster center as a second sample.

[0012] Optionally, the negative sample determination unit may be configured to determine image features that are different from the second sample category from the clustered image features stored in a pre-established memory library as negative samples, wherein the memory library is used to store a predetermined number of image features selected from the remaining image features of the multiple image features after clustering excluding the positive sample pairs.

[0013] Optionally, the target loss function construction unit may be configured to: calculate the value of the target loss function through the positive sample pairs and all negative samples in the memory bank; the model parameter updating unit may be configured to: update the parameters of the image processing model based on the value of the target loss function to train the image processing model.

[0014] Optionally, the training device of the image processing model also includes a memory update unit, which can be configured to: after updating the parameters of the image processing model once, remove the stored clustered image features from the memory, and then select a new predetermined number of image features from the remaining image features of the multiple clustered image features except the positive sample pairs and the used image features, and store them in the memory for the next update of the parameters of the image processing model.

[0015] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, comprising: at least one processor; and at least one memory storing computer executable instructions, wherein the computer executable instructions, when executed by the at least one processor, cause the at least one processor to execute a training method for an image processing model according to the present disclosure.

[0016] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium storing instructions is provided, which, when executed by at least one processor, prompts the at least one processor to execute a training method for an image processing model according to the present disclosure.

[0017] According to a fifth aspect of an embodiment of the present disclosure, there is provided a computer program product, comprising computer instructions, which, when executed by at least one processor, implement the training method of the image processing model according to the present disclosure.

[0018] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0019] According to the training method and device of the image processing model disclosed in the present invention, after extracting multiple image features of image data in the training data set, these image features will be preprocessed first, that is, the acquired image features will be clustered first, so that each image feature in the feature space is identifiable, and after determining the positive sample pair, the image features of different categories from the positive sample pair can be used as negative samples, without having to use all other image features as negative samples, thereby reducing the feature lag phenomenon caused by training all image features except the positive sample pair as negative samples; on the other hand, although the present invention reduces the number of negative samples used in training, compared with the two loss functions of contrastive loss and triplet loss, the number of negative samples is still significantly more, so it can improve the problem of image processing training falling into local solutions, thereby improving the performance of the image processing model.

[0020] In addition, the present disclosure is configured to extract a first sample from multiple image features after clustering and determine a second sample based on the first sample, and construct the first sample and the second sample into a positive sample pair, wherein the second sample is a positive sample of the first sample. This can avoid the problem of feature loss caused by constructing positive samples through representation transformation, thereby improving feature recognition in the training process and further broadening the application scenarios of image processing models.

[0021] In addition, the present disclosure is also provided with a memory bank for storing a predetermined number of image features selected from the remaining image features except the determined positive sample pairs from the multiple image features after clustering. The parameters of the image processing model can be updated once after all the image features in a memory bank are trained. Therefore, the present disclosure can reduce the number of times the parameters of the image processing model are updated, thereby improving the feature deviation (feature lag) that occurs during the training process of the image processing model, and further improving the performance of the image processing model.

[0022] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0024] Figure 1 is a flowchart illustrating a training method of an image processing model according to an exemplary embodiment of the present disclosure.

[0025] Figure 2 is a schematic diagram illustrating a training process of an image processing model according to an exemplary embodiment of the present disclosure.

[0026] Figure 3 is a block diagram illustrating a training apparatus for an image processing model according to an exemplary embodiment of the present disclosure.

[0027] Figure 4 is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0030] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0031] With the development of deep learning, contrastive learning has been applied to the training of image processing models. The quality of the contrastive loss function used in the contrastive learning process has an important impact on the performance of the trained image processing model.

[0032] There are two main contrast loss functions in the prior art, Contrastive loss and Triplet loss. Among them, the contrast loss function first introduced the contrast idea into the loss function. Its main idea is to compare the distance between two features, so as to bring them closer or farther apart, and achieve "making two similar samples continue to be similar, and dissimilar samples become more distant". The expression of this loss function is as follows:

[0033]

[0034] Where N is the number of sample features, d = || a n -b n ||2 represents the Euclidean distance between the features of the two samples, y is the label of whether the two samples match, y=1 means that the two samples are similar or matched, y=0 means they do not match, and margin is the set threshold.

[0035] The triplet loss function is an advanced version of the contrast loss function. It introduces three variables in the contrast process. It selects a central data (anchor) from the sample set, and then selects a positive sample and a negative sample. The optimization goal of the function is to pull the positive sample closer to the center and push the negative sample away from the center. A triplet relationship is formed. Its expression is as follows:

[0036]

[0037] in, Represents the Euclidean distance between the central data and the positive sample, Represents the Euclidean distance between the central data and the negative samples, and m is the set threshold.

[0038] The purpose of the triplet loss function is to make a sample similar to the positive sample and make it distant from the negative sample. In later practice, people found that there are often simple and complex positive and negative samples in the sample set. When calculating the triplet loss, complex samples should be used. Therefore, the current triplet loss defaults to including the complex sample selection step.

[0039] Both contrast loss functions are local optimization functions, that is, the features of a sample are only compared with 1 or 2 samples to change the distance between the contrast features. This local optimization will make the network fall into a local point and it is difficult to fit the overall data. Therefore, the cross entropy function is usually added as a global guide during training to help the learning of the contrast loss function.

[0040] In the unsupervised model learning scenario, contrastive learning treats each sample as a class, and then treats the result of the change in the representation of this class as a positive sample, and all other samples as negative samples. The expression of its contrast loss function is as follows:

[0041]

[0042] Among them, f θ (x t ) represents a sample, represents the positive sample, represents negative samples, and τ is the temperature coefficient. When calculating, this function will perform an inner product between a sample and all other samples to learn the characteristics of this sample. For example, there are 3 pictures of American shorthair cats, Persian cats, and dogs. Taking the Persian cat as an example, first perform image enhancement (such as rotation, color change, random erasure), and then use the result as a positive sample, and use the remaining American shorthair cats and dogs as negative samples for comparative learning. In the feature space, these three samples will be separated.

[0043] However, this technical solution has at least the following disadvantages: First, this contrastive learning method constructs positive samples through representation transformation, but this representation transformation is not applicable to fine-grained recognition problems. This is because some scenes are very sensitive to representation, such as bird recommendations and antique recommendations in business. After the representation transformation, some key features may be lost, resulting in unsatisfactory learning results and limiting the application scenarios of contrastive learning. Second, this contrastive learning will treat all samples except positive samples as negative samples each time, which will lead to feature lag. Because when learning, the batch size is taken in small batches, and the neural network model will be updated after each batch. Therefore, after all samples are extracted by the neural network model, the extracted negative samples are no longer extracted by a model, so there will be feature deviations. When there are many sample categories, the performance of the model will decline.

[0044] In order to obtain an image processing model with better performance, the present disclosure proposes a training method and device for an image processing model based on contrastive learning. Specifically, after extracting multiple image features of a training data set, the present disclosure first performs some preprocessing on these image features, that is, first clustering the acquired image features, so that each image feature in the feature space is identifiable. After determining the positive sample pairs, image features of different categories from the positive sample pairs can be used as negative samples in training, without having to use all other image features as negative samples, thereby reducing the feature lag phenomenon caused by training all image features except the positive sample pairs as negative samples, so as to improve the performance of the trained image processing model; on the other hand, although the present disclosure reduces the number of negative samples used in the training process, compared with the two loss functions of contrastive loss and triplet loss, the number of negative samples is still significantly larger, so the problem of image model training falling into local solutions can be improved, thereby improving the performance of the trained image processing model.

[0045] In addition, the present disclosure extracts a first sample from the clustered multiple image features and determines a second sample based on the first sample, constructs the first sample and the second sample into a positive sample pair, wherein the second sample is a positive sample of the first sample, which can avoid the problem of image feature loss caused by constructing positive samples through representation transformation, thereby improving the feature recognition of the image processing model in the application process, and further broadening the application scenarios of the image processing model. Figures 1 to 4 A training method for image processing and a training device for image processing according to exemplary embodiments of the present disclosure are described in detail.

[0046] Figure 1 is a flow chart showing a training method of an image processing model according to an exemplary embodiment of the present disclosure. Here, the image processing model can be applied to, but not limited to, image recognition tasks, image classification tasks, image segmentation tasks, etc. In addition, the training method according to the present disclosure can also be applied to text processing models, for example, text recognition tasks, etc.

[0047] Reference Figure 1 In step 101, a training data set is obtained, and the training data set includes multiple image data. Here, the image data can be single image data or video frame data in video data. In addition, when the training method of the present disclosure is applied to a text processing model, the training data set can include multiple text data.

[0048] Here, the training data set can be a public database, such as ImageNet, PASCAL VOC, etc., or it can be a database established based on a specific goal, and the present disclosure does not impose any restrictions on this.

[0049] In step 102, image feature extraction is performed on each training data in the training data set to obtain multiple image features.

[0050] According to an exemplary embodiment of the present disclosure, a neural network can be used to extract image features of each training data in a training data set. In some embodiments, the neural network can be a convolutional neural network, which performs multiple convolutions on the training data to continuously extract and compress the image features of the training data, thereby obtaining reliable features of the training data. Of course, the neural network can also be any of the currently commonly used neural networks, as long as it can extract the image features of the training data, and the present disclosure does not limit this. The extracted image features can be features that characterize the original image and are significantly different from images of other categories. For example, after feature extraction of animal images in the training data set, the image features can be feature vectors that characterize the animal's fur color, pattern, limb shape, and facial features.

[0051] In step 103, the acquired multiple image features are clustered.

[0052] According to an exemplary embodiment of the present disclosure, a currently common clustering method can be used to cluster the acquired multiple image features. For example, a density-based DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm can be used. The algorithm defines a cluster as the maximum set of points connected by density, can divide areas with sufficient density into clusters, and can find clusters of any shape in a noisy spatial data set. It has two important parameters: Eps and MmPtS, where Eps is the neighborhood radius when defining density, and MmPts is the threshold when defining core points. When processing data, the algorithm first selects a data object point p from the data set at random. If the selected data object point p is a core point for the parameters Eps and MinPts, then find all data object points that are density-reachable from p to form a cluster; if the selected data object point p is an edge point, then select another data object point, and repeat this process until all points are processed. In step 103, after the multiple image features are clustered by the DBSCAN algorithm, multiple clusters can be formed. In some embodiments, a partition-based K-means algorithm, a hierarchy-based BIRCH algorithm, a density-based OPTIC algorithm, etc. may also be used, and the present disclosure does not limit this.

[0053] According to an exemplary embodiment of the present disclosure, pseudo labels can be assigned to multiple image features after clustering. Among them, pseudo labels refer to labels with certain uncertainty, which are different from labels with definite meanings that are manually marked. This is because the essence of clustering is to divide a data set into different classes or clusters according to a certain standard (such as a distance criterion), so that the similarity of data objects in the same cluster is as large as possible, and the difference of data objects not in the same cluster is as large as possible. In other words, clustered objects are clustered together because of similarity, which itself has uncertainty to a certain extent. In some embodiments, the assigned pseudo labels can be numbers. For example, after clustering the image features, four clusters are formed, namely class A, class B, class C and noise. The assigned pseudo labels can be "class A" → "0", "class B" → "1", "class C" → "2" and "noise" → "-1". Of course, pseudo labels can also be other forms of expression, which are not limited by the present disclosure. After being assigned pseudo-labels, image features become recognizable. Therefore, during the training of image processing models, negative samples can be determined based on the pseudo-labels of image features, without having to take all other samples except positive samples as negative samples. This can improve the feature lag phenomenon in the training process and thus improve the performance of the trained image processing model.

[0054] In step 104, a first sample is extracted from the clustered plurality of image features and a second sample is determined based on the first sample, and the first sample and the second sample are constructed as a positive sample pair, wherein the second sample is a positive sample of the first sample.

[0055] According to an exemplary embodiment of the present disclosure, a first sample can be randomly extracted from a plurality of image features after clustering, and the cluster center corresponding to the first sample can be determined according to the cluster to which the first sample belongs, and the image feature closest to the cluster center can be determined as the second sample. Specifically, the image features are divided into a plurality of clusters after clustering, and each cluster corresponds to a cluster center to characterize the average performance of the plurality of image features in the cluster. In some embodiments, the Mahalanobis distance can be used to determine the feature closest to the cluster center. The Mahalanobis distance is used to represent the covariance distance between data, is not affected by the dimension, and can eliminate the interference of the correlation between image features, thereby effectively calculating the distance between two image features. In some embodiments, other commonly used distance indicators such as Euclidean distance, Manhattan distance or Hamming distance can also be used to determine the second sample, and the present disclosure does not limit this.

[0056] After the second sample is determined, the first sample and the second sample can be constructed as a positive sample pair, wherein the second sample is the positive sample of the first sample. In the present disclosure, since the features are clustered and the Mahalanobis distance is used to determine the positive sample (second sample) of the randomly selected first sample, compared with the method of determining the positive sample by feature transformation, the image features of the sample can be avoided from being lost, so that it can be applied to solve problems such as fine-grained recognition, and broaden the use scenarios of the trained image processing model.

[0057] In step 105, image features that are different from the second sample category among the plurality of clustered image features are determined as negative samples.

[0058] According to an exemplary embodiment of the present disclosure, image features that are different from the category of the second sample can be determined from the clustered image features stored in a pre-established memory library, and the image features that are different from the category of the second sample can be used as negative samples, wherein the memory library is used to store a predetermined number of image features selected from the remaining image features of the multiple image features after clustering excluding the determined positive sample pairs.

[0059] Specifically, since the present disclosure clusters the image features and assigns pseudo labels to the clustered image features, each image feature is identifiable. After the positive sample pair is determined in step 104, the image features of different categories from the positive sample pair can be used as negative samples, reducing the number of negative samples, thereby avoiding using all image features other than the positive samples as negative samples, thereby alleviating the phenomenon of image feature lag; on the other hand, compared with the traditional contrast loss function and triple loss function, the number of negative samples in the present disclosure is still significantly more, which can avoid falling into a local solution. Furthermore, in the training method of the image processing model using generalized contrast learning, the batch size is taken according to small batches, and the parameters of the image processing model are updated after each batch. Therefore, after all samples are extracted using the image processing model, the extracted negative samples are no longer extracted by the same image processing model, resulting in deviations in image features. In view of this, the present disclosure pre-sets a memory bank for storing a predetermined number of image features selected from the remaining image features except the determined positive sample pairs from the multiple image features after clustering (for example, a predetermined number (for example, 10) batches of features can be selected, each batch including 5 features). Therefore, after all the image features in a memory bank are compared and learned, the parameters of the image processing model can be updated once, that is, it can be updated once after a predetermined number of batches. Therefore, compared with the training method of the image processing model using generalized contrast learning, the present disclosure can reduce the number of parameter updates of the image processing model, thereby improving the image feature deviation (feature lag) that occurs during the training process. In some embodiments, more or fewer image features can be stored in the memory bank according to actual conditions, and the present disclosure does not limit the number of features stored in the memory bank.

[0060] In step 106, a target loss function is constructed using the positive sample pairs and the negative samples.

[0061] Specifically, after the operations from step 101 to step 105, positive sample pairs that have not undergone feature transformation and negative samples with a certain number scale but less than the number of negative samples trained using the generalized contrastive learning method for the image processing model can be determined, and the constructed positive sample pairs and multiple negative samples are applied to the target loss function, which can make the calculated value of the target loss function better. For example, the target loss function can be expressed as, but not limited to, the following target loss function:

[0062]

[0063] Among them, f θ (x t ) represents a sample, represents the positive sample, represents negative samples, and τ is the temperature coefficient.

[0064] In step 107, the parameters of the image processing model are updated based on the target loss function to train the image processing model.

[0065] According to an exemplary embodiment of the present disclosure, the value of the target loss function may be calculated by using positive sample pairs and all negative samples in the memory bank, and the parameters of the image processing model may be updated based on the value of the target loss function.

[0066] According to an exemplary embodiment of the present disclosure, after the update, the stored clustered image features are removed from the memory bank, and then a new predetermined number of image features are selected from the remaining image features of the clustered multiple image features except the positive sample pairs and the used image features and stored in the memory bank for the next update of the parameters of the image processing model.

[0067] Figure 2 is a schematic diagram illustrating a training process of an image processing model according to an exemplary embodiment of the present disclosure.

[0068] exist Figure 2 In the figure, 201 represents a plurality of image features after clustering, 202 represents a memory bank, 203 represents a predetermined number of image features stored in the memory bank, and 204 represents a determined positive sample pair. A predetermined number of clustered image features are stored in the memory bank 202. For example, 10 batches of clustered image features can be stored, each batch including 5 image features. Of course, more or fewer image features can be stored according to the actual situation. The present disclosure does not limit the number of image features stored in the memory bank. After the positive sample pair 204 is determined from the clustered image features using the method of step 104, the positive sample pair 204 can be compared with the image features stored in the memory bank 202 for learning, wherein, when an image feature of the same category as the second sample in the positive sample pair is encountered, the image feature of the same category can be recorded as 0, and compared with the image feature of a different category from the second sample (negative sample). After the positive sample pair 204 is compared with all the negative samples stored in the memory 202, the parameters of the image processing model can be updated once. After the update, the stored clustered image features are removed from the memory 202, and then new features of the same number other than the positive sample pair and the used image features are selected from the clustered multiple image features 201 and stored in the memory 202 for the next update of the parameters of the image processing model.

[0069] According to an exemplary embodiment of the present disclosure, the contrast loss function used in the training method of the image processing model of the present disclosure is compared with the current main contrast loss function in two application scenarios (such as classification and detection), and the test results are shown in Table 1 below. In Table 1, Trip is a triple loss function, N-pair is a contrast loss function, Cross-Entropy is a cross entropy function, MC is a loss function used in the contrast learning method of the present disclosure, Market-to-Duke and Duke-to-Market are large data sets, and mAP and rank-1 are two evaluation indicators. The test results in Table 1 show that the contrast loss function MC (Memory-based Contrastive Loss) provided by the present disclosure exceeds the triple contrast function and also exceeds the globally optimized cross entropy function, verifying the effectiveness of the present disclosure.

[0070] Table 1

[0071]

[0072] According to an exemplary embodiment of the present disclosure, the training method of the image processing model provided by the present disclosure is compared with the training method of the current main image processing model in two application scenarios (such as classification and detection), and the test results are shown in Table 2. In Table 2, InstDisc and MoCo are the training methods of the current main image processing models, and MB is the training method of the image processing model provided by the present disclosure. The test results show that the training method of the image processing model of the present disclosure has a better classification effect.

[0073] Table 2

[0074]

[0075] The image processing model training method provided in the present disclosure can train an image processing model with better performance. Under the same application conditions, it has better discrimination of image features and can be applied to scenarios such as classification, segmentation and detection.

[0076] Figure 3 is a block diagram illustrating a training apparatus for an image processing model according to an exemplary embodiment of the present disclosure.

[0077] Reference Figure 3 According to an exemplary embodiment of the present disclosure, the training device 300 of the image processing model may include a training data acquisition unit 301, a feature extraction unit 302, a feature clustering unit 303, a positive sample pair determination unit 304, a negative sample pair determination unit 305, a target loss function construction unit 306 and a model parameter updating unit 307.

[0078] The training data acquisition unit 301 can acquire a training data set, which includes a plurality of image data. Here, the training data set can be a public database, such as ImageNet, PASCAL VOC, etc., or a database established based on a specific goal, which is not limited in the present disclosure.

[0079] The feature extraction unit 302 can perform image feature extraction on each training data in the training data set to obtain multiple image features. Specifically, the feature extraction unit 302 can be a neural network. In some embodiments, the neural network can be a convolutional neural network, which performs multiple convolutions on the training data to continuously extract and compress the features of the training data, thereby obtaining reliable features of the training data. Of course, the neural network can also be any commonly used neural network at present, as long as it can extract the features of the training data, and the present disclosure does not limit this.

[0080] According to an exemplary embodiment of the present disclosure, the feature extraction unit 302 may be a neural network. In some embodiments, the neural network may be a convolutional neural network, which performs multiple convolutions on the training data to continuously extract and compress the image features of the training data, thereby obtaining reliable features of the training data. Of course, the neural network may also be any commonly used neural network, as long as it can extract the image features of the training data, and the present disclosure does not impose any restrictions on this. The extracted image features may be features that characterize the original image and are significantly different from images of other categories. For example, after feature extraction of animal images in the training data set, the image features may be feature vectors that characterize the animal's fur color, pattern, limb shape, and facial features.

[0081] The feature clustering unit 303 may cluster the extracted multiple image features.

[0082] According to an exemplary embodiment of the present disclosure, the feature clustering unit 303 can cluster the acquired multiple features using the current common clustering method. For example, the density-based DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm can be used. The algorithm defines a cluster as the maximum set of density-connected points, can divide areas with sufficient density into clusters, and can find clusters of any shape in a noisy spatial data set. It has two important parameters: Eps and MmPtS, where Eps is the neighborhood radius when defining density, and MmPts is the threshold when defining core points. When processing data, the algorithm first selects a data object point p from the data set at random. If the selected data object point p is a core point for the parameters Eps and MinPts, then find all data object points that are density-reachable from p to form a cluster; if the selected data object point p is an edge point, then select another data object point. Repeat this process until all points are processed. In this step, multiple features can form multiple clusters after being clustered by the DBSCAN algorithm. In some embodiments, a partition-based K-means algorithm, a hierarchy-based BIRCH algorithm, a density-based OPTIC algorithm, etc. may also be used, and the present disclosure does not limit this.

[0083] The positive sample pair determination unit 304 may extract a first sample from the clustered plurality of image features and determine a second sample based on the first sample, and construct the first sample and the second sample into a positive sample pair, wherein the second sample is a positive sample of the first sample.

[0084] According to an exemplary embodiment of the present disclosure, the positive sample pair determination unit 304 can randomly extract a first sample from the multiple image features after clustering, and determine the cluster center corresponding to the first sample according to the cluster where the first sample is located, and determine the image feature closest to the cluster center as the second sample. Specifically, the image features are divided into multiple clusters after clustering, and each cluster corresponds to a cluster center to characterize the average performance of multiple image features in the cluster. In some embodiments, the Mahalanobis distance can be used to determine the image feature closest to the cluster center. The Mahalanobis distance is used to represent the covariance distance between data, which is not affected by the dimension and can eliminate the interference of the correlation between image features, thereby effectively calculating the distance between two image features. In some embodiments, other commonly used distance indicators such as Euclidean distance, Manhattan distance or Hamming distance can also be used to determine the second sample, and the present disclosure does not limit this.

[0085] The negative sample determination unit 305 may determine the image features that are different from the second sample category among the clustered multiple image features as negative samples. According to an exemplary embodiment of the present disclosure, the negative sample determination unit 305 may determine the features that are different from the category of the second sample from the clustered features stored in a pre-established memory bank based on the second sample, and use the features that are different from the category of the second sample as negative samples. The memory bank is used to store a predetermined number of features selected from the remaining features of the clustered multiple features except the positive sample pair.

[0086] The target loss function construction unit 306 may construct a target loss function using the positive sample pairs and the negative samples.

[0087] The model parameter updating unit 307 can update the parameters of the image processing model based on the target loss function to train the image processing model.

[0088] According to an exemplary embodiment of the present disclosure, the value of the target loss function may be calculated by using positive sample pairs and all negative samples in the memory bank, and the parameters of the image processing model may be updated based on the value of the target loss function.

[0089] According to an exemplary embodiment of the present disclosure, the contrastive learning device 300 also includes a memory update unit 308. After the parameters of the image processing model are updated, the memory update unit 308 can remove the stored clustered image features from the memory, and then select a predetermined number of new image features from the remaining image features except the positive sample pairs and the used image features from the clustered multiple image features and store them in the memory for the next update of the parameters of the image processing model.

[0090] Figure 4 is a block diagram illustrating an electronic device 400 according to an exemplary embodiment of the present disclosure.

[0091] Reference Figure 4 The electronic device 400 includes at least one memory 401 and at least one processor 402, wherein the at least one memory 401 stores a set of computer executable instructions, and when the computer executable instruction set is executed by the at least one processor 402, a training method for an image processing model according to an exemplary embodiment of the present disclosure is executed.

[0092] As an example, the electronic device 400 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above-mentioned instruction set. Here, the electronic device 400 is not necessarily a single electronic device, but may also be any device or circuit collection capable of executing the above-mentioned instructions (or instruction sets) individually or in combination. The electronic device 400 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0093] In the electronic device 400, the processor 402 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0094] The processor 402 may execute instructions or codes stored in the memory 401, which may also store data. Instructions and data may also be sent and received over a network via a network interface device, which may employ any known transmission protocol.

[0095] The memory 401 may be integrated with the processor 402, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 401 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 401 and the processor 402 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 402 can read files stored in the memory.

[0096] In addition, the electronic device 400 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 400 may be connected to each other via a bus and / or a network.

[0097] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by at least one processor, the at least one processor is prompted to perform the training method of the image processing model according to the present disclosure. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R, BD-RW, , BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0098] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided, and instructions in the computer program product may be executed by a processor of a computer device to complete the training method of the image processing model according to the exemplary embodiment of the present disclosure.

[0099] According to the training method and device of the image processing model disclosed in the present invention, after extracting multiple image features of image data in the training data set, these image features will be preprocessed first, that is, the acquired image features will be clustered first, so that each image feature in the feature space is identifiable, and after determining the positive sample pair, the image features of different categories from the positive sample pair can be used as negative samples, without having to use all other image features as negative samples, thereby reducing the feature lag phenomenon caused by using all image features except the positive sample pair as negative samples for contrast training; on the other hand, although the present invention reduces the number of negative samples used in training, compared with the two loss functions of contrastiveloss and triplet loss, the number of negative samples is still significantly more, so it can improve the problem of image processing training falling into local solutions, thereby improving the performance of the image processing model.

[0100] In addition, the present disclosure is configured to extract a first sample from multiple image features after clustering and determine a second sample based on the first sample, and construct the first sample and the second sample into a positive sample pair, wherein the second sample is a positive sample of the first sample. This can avoid the problem of feature loss caused by constructing positive samples through representation transformation, thereby improving feature recognition in the training process and further broadening the application scenarios of image processing models.

[0101] In addition, the present disclosure is also provided with a memory bank for storing a predetermined number of image features selected from the remaining image features except the determined positive sample pairs from the multiple image features after clustering. The parameters of the image processing model can be updated once after all the image features in a memory bank are trained. Therefore, the present disclosure can reduce the number of times the parameters of the image processing model are updated, thereby improving the feature deviation (feature lag) that occurs during the training process of the image processing model, and further improving the performance of the image processing model.

[0102] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0103] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A training method for an image processing model, characterized in that: include: Acquire a training data set, wherein the training data set includes a plurality of image data; Performing feature extraction on each image data in the training data set to obtain a plurality of image features corresponding one-to-one to the plurality of image data; Clustering the multiple image features; Extracting a first sample from the plurality of clustered image features and determining a second sample based on the first sample, constructing the first sample and the second sample into a positive sample pair, wherein the second sample is a positive sample of the first sample; Determine the image features that are different from the second sample category among the plurality of clustered image features as negative samples; Constructing a target loss function through the positive sample pairs and the negative samples; updating the parameters of the image processing model based on the target loss function to train the image processing model; Wherein, determining the second sample based on the first sample includes: Determine a cluster center corresponding to the first sample; An image feature closest to the cluster center is determined as a second sample.

2. The image processing model training method according to claim 1, characterized in that: The extracting a first sample from the plurality of clustered image features comprises: A first sample is randomly extracted from the plurality of clustered image features.

3. The image processing model training method according to claim 1, characterized in that: The step of determining the image features different from the second sample category among the clustered plurality of image features as negative samples comprises: An image feature different from a second sample category from clustered image features stored in a pre-established memory bank is determined as a negative sample, wherein the memory bank is used to store a predetermined number of image features selected from the remaining image features of the plurality of clustered image features except the positive sample pairs.

4. The training method of the image processing model according to claim 3, characterized in that: The constructing a target loss function by using the positive sample pairs and the negative samples, and updating the parameters of the image processing model based on the target loss function to train the image processing model includes: Calculate the value of the target loss function through the positive sample pairs and all negative samples in the memory bank; The image processing model is trained by updating the parameters of the image processing model based on the value of the target loss function.

5. The method for training an image processing model according to claim 4, wherein: Also includes: After the parameters of the image processing model are updated, the stored clustered image features are removed from the memory bank, and a predetermined number of new image features are selected from the remaining image features of the clustered multiple image features except the positive sample pairs and the used image features and stored in the memory bank for the next update of the parameters of the image processing model.

6. A training device for an image processing model, characterized in that: include: A training data acquisition unit is configured to: acquire a training data set, wherein the training data set includes a plurality of image data; A feature extraction unit is configured to: perform feature extraction on each training data in the training data set to obtain a plurality of image features corresponding to the plurality of image data one by one; A feature clustering unit is configured to: cluster the multiple image features; A positive sample pair determination unit is configured to: extract a first sample from the plurality of clustered image features and determine a second sample based on the first sample, and construct the first sample and the second sample into a positive sample pair, wherein the second sample is a positive sample of the first sample; A negative sample determination unit is configured to: determine image features different from the second sample category among the plurality of clustered image features as negative samples; The target loss function construction unit is configured to: construct a target loss function through the positive sample pair and the negative sample; A model parameter updating unit, configured to: update the parameters of the image processing model based on the target loss function to train the image processing model; Wherein, the positive sample pair determination unit is configured as: Determine a cluster center corresponding to the first sample; An image feature closest to the cluster center is determined as a second sample.

7. The image processing model training device according to claim 6, characterized in that: The positive sample pair determination unit is configured as follows: A first sample is randomly extracted from the plurality of clustered image features.

8. The image processing model training device according to claim 6, characterized in that: The negative sample determination unit is configured as follows: An image feature different from a second sample category from clustered image features stored in a pre-established memory bank is determined as a negative sample, wherein the memory bank is used to store a predetermined number of image features selected from the remaining image features of the plurality of clustered image features except the positive sample pairs.

9. The image processing model training device according to claim 8, characterized in that: The target loss function construction unit is configured to: calculate the value of the target loss function through the positive sample pair and all negative samples in the memory bank; The model parameter updating unit is configured to update the parameters of the image processing model based on the value of the target loss function to train the image processing model.

10. The image processing model training device according to claim 9, characterized in that: Also includes: The memory update unit is configured as follows: After the parameters of the image processing model are updated once, the stored clustered image features are removed from the memory bank, and then a new predetermined number of image features are selected from the remaining image features of the clustered multiple image features except the positive sample pairs and the used image features and stored in the memory bank for the next update of the parameters of the image processing model.

11. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer executable instructions, Wherein, when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the training method of the image processing model as described in any one of claims 1 to 5.

12. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to perform the training method of the image processing model as described in any one of claims 1 to 5.

13. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the training method of the image processing model as claimed in any one of claims 1 to 5 is implemented.

Images