Clustering-based transfer feature learning method and device for image processing

Abstract

The present disclosure provides a clustering-based transfer feature learning method and device for image processing, an image processing device, and an image processing method. The transfer feature learning method includes: clustering the target sample to generate K clusters, where K is an integer and K≥1; calculating the inherent cluster distribution of each target sample based on the generated K clusters; performing transfer feature learning on each target sample according to the source sample by using a transfer feature learning algorithm, and obtaining the intermediate layer features of the target sample; calculating the cluster allocation distribution of each target sample using the obtained intermediate layer features; and adjusting the transfer feature learning algorithm based on the inherent cluster distribution and / or the cluster allocation distribution to ensure that the feature representation obtained by the transfer feature learning retains the basic data structure of the target sample.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and in particular, to a clustering-based migration feature learning method and device for image processing. Background Art

[0002] Existing transfer feature learning techniques are often applicable to closed scenarios, that is, it is necessary to assume that the source domain and the target domain have exactly the same and known categories. However, in real life, the target domain often has categories that the source domain has never seen, making the closed scenario no longer closed. In this case, if the existing transfer feature learning techniques in such closed scenarios are directly applied to real-life scenarios, it often brings many problems. For example, if the data of the source domain and the target domain are simply matched, the new categories in the target domain will be forced to be consistent with the known categories in the source domain, which is unreasonable.

[0003] Therefore, a new transfer feature learning method is needed, which can be applied not only to closed scenes but also to real scenes to obtain feature representations that are more suitable for target samples. Summary of the invention

[0004] The object of the present disclosure is to solve at least part or all of the above problems.

[0005] One aspect of the present disclosure provides a clustering-based transfer feature learning method for image processing, including: clustering target samples to generate K clusters, where K is an integer and K≥1; calculating the inherent cluster distribution of each target sample based on the generated K clusters; performing transfer feature learning on each target sample based on the source sample by using a transfer feature learning algorithm, and obtaining the intermediate layer features of the target sample; calculating the cluster assignment distribution of each target sample using the obtained intermediate layer features; and adjusting the transfer feature learning algorithm based on the inherent cluster distribution and / or the cluster assignment distribution to ensure that the feature representation obtained by the transfer feature learning retains the underlying data structure of the target sample.

[0006] In one example, adjusting the transfer feature learning algorithm based on the inherent cluster distribution and / or the cluster assignment distribution may include: minimizing the KL divergence between the inherent cluster distribution and the cluster assignment distribution.

[0007] In another example, adjusting the transfer feature learning algorithm based on the inherent cluster distribution and / or the cluster assignment distribution may include: minimizing the KL divergence between the inherent cluster distribution and the cluster assignment distribution under the premise of constraining the relationship between clusters.

[0008] In yet another example, adjusting the transfer feature learning algorithm based on the inherent cluster distribution and / or the cluster assignment distribution may include maximizing the mutual information between the intermediate layer features, the predicted classification steps output by the transfer feature learning algorithm, and the cluster assignment distribution.

[0009] In another example, the target samples may be clustered by using a k-means algorithm.

[0010] In another example, the transfer feature learning algorithm may be any one of a self-training algorithm, a multi-view training algorithm, and a self-ensembling algorithm.

[0011] In another example, the mutual information includes at least one of global mutual information and local mutual information between the intermediate layer features, the predicted classification step, and the cluster assignment distribution.

[0012] According to another aspect of the present disclosure, a clustering-based transfer feature learning device for image processing is provided, comprising: a clustering engine, configured to cluster target samples to generate K clusters, where K is an integer and K≥1; a first cluster distribution calculation module, configured to calculate the inherent cluster distribution of each target sample based on the generated K clusters; a transfer feature learning module, configured to perform transfer feature learning on each target sample according to the source sample by using a transfer feature learning algorithm, and obtain the intermediate layer features of the target sample; a second cluster distribution calculation module, configured to calculate the cluster assignment distribution of each target sample using the obtained intermediate layer features; and an algorithm adjustment module, configured to adjust the transfer feature learning algorithm based on the inherent cluster distribution and / or the cluster assignment distribution to ensure that the feature representation obtained through the transfer feature learning retains the underlying data structure of the target sample.

[0013] According to another aspect of the present disclosure, there is provided an image processing method, comprising: receiving an image sample to be processed, processing the image sample by using a transfer feature learning method as described in any one of the aforementioned example embodiments to obtain a feature representation for the image sample, and outputting the feature representation.

[0014] According to another aspect of the present disclosure, there is provided an image processing device, comprising: a communicator, configured to receive an image sample to be processed; and a transfer feature learning device as described in the aforementioned example embodiment, configured to process the image sample to obtain a feature representation for the image sample, wherein the communicator is further configured to output the feature representation.

[0015] According to another aspect of the present disclosure, there is provided an image processing device, comprising: a memory configured to store one or more executable instructions; and a processor connected to the memory and used to execute the executable instructions stored in the memory to execute the migration feature learning method as described in any one of the aforementioned example embodiments or the image processing method as described in the aforementioned example embodiments.

[0016] According to another aspect of the present disclosure, a computer-readable storage medium is provided, characterized in that the computer-readable storage medium stores computer instructions, and when the instructions are executed by a processor, the migration feature learning method as described in any one of the aforementioned example embodiments or the image processing method as described in the aforementioned example embodiments is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 (a) and (b) schematically show the domain adaptation results of applying the prior art transfer feature learning method to the closed set and the open set respectively;

[0018] Figure 2 Schematically shows a flow chart of a clustering-based transfer feature learning method for image processing according to an exemplary embodiment of the present disclosure;

[0019] Figure 3 Schematically showing the domain adaptation result of the clustering-based transfer feature learning method for image processing according to an exemplary embodiment of the present disclosure;

[0020] Figure 4 A schematic diagram of the total loss of a clustering-based transfer feature learning method for image processing according to an example embodiment of the present disclosure is shown.

[0021] Figure 5 The schematic framework of the self-ensembling with category-agnostic cluster (SE-CC) feature learning method is schematically shown;

[0022] Figure 6 Schematically shows a block diagram of a clustering-based transfer feature learning apparatus for image processing according to an exemplary embodiment of the present disclosure;

[0023] Figure 7 A schematic diagram schematically illustrates a method for implementing a clustering-based transfer feature learning device for image processing according to an exemplary embodiment of the present disclosure; and

[0024] Figure 8 A flowchart schematically illustrates an image processing method according to an exemplary embodiment of the present disclosure; and

[0025] Fig. 9 A block diagram of an image processing apparatus according to an exemplary embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0026] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0027] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0028] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0029] When using expressions such as "at least one of A, B, and C, etc.", it should generally be interpreted as the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.). When using expressions such as "at least one of A, B, or C, etc.", it should generally be interpreted as the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, or C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0030] In the drawings, the same or similar reference numerals are used to represent the same or similar structures.

[0031] Existing transfer feature learning techniques are often applicable to closed scenarios, that is, they need to assume that the source domain and the target domain have exactly the same and known categories. Figure 1(a) in the figure shows the domain adaptation result obtained by applying the existing transfer feature learning technology to an ideal closed scene. In an ideal closed scene, the categories of all samples in the source domain and the target domain are samples of known categories. However, in real life, the target domain often has categories that the source domain has never seen, or the source domain itself has samples of unknown categories, so the scene is no longer closed but open. In this case, if the existing transfer feature learning technology in this closed scene is directly applied to an open scene, it often brings many problems. Figure 1 (b) shows the domain adaptation results obtained by applying the existing transfer feature learning technology to the open scene. Figure 1 In (b), since the samples of the source domain and the target domain are simply matched, the samples of unknown categories in the target domain are forced to be consistent with the samples of known categories in the source domain, resulting in incorrect division, which is unreasonable.

[0032] To this end, a clustering-based transfer feature learning method is proposed, which can be applied not only to closed scenes but also to real scenes to obtain feature representations that are more suitable for target samples.

[0033] Figure 2 The flowchart of the clustering-based transfer feature learning method for image processing according to an exemplary embodiment of the present disclosure is schematically shown, and Figure 3 The domain adaptation result of the clustering-based transfer feature learning method for image processing according to an example embodiment of the present disclosure is schematically shown.

[0034] like Figure 2 As shown, a clustering-based transfer feature learning method 200 for image processing according to an exemplary embodiment of the present disclosure may include: clustering the target samples to generate K clusters, where K is an integer and K≥1; calculating the inherent cluster distribution of each target sample based on the generated K clusters; performing transfer feature learning on the target samples based on the source samples by using a transfer feature learning algorithm, and obtaining intermediate-layer features of the target samples; calculating the cluster assignment distribution of each target sample using the obtained intermediate-layer features; and adjusting the transfer feature learning algorithm based on the inherent cluster distribution and / or the cluster assignment distribution to ensure that the feature representation obtained through the transfer feature learning retains the underlying data structure of the target sample.

[0035] Specifically, in step S210, samples from the target domain (ie, target samples) are clustered to generate K clusters, for example, Cluster 1, Cluster 2, Cluster 3, ..., Cluster K, where K is an integer and K ≥ 1. Optionally, the samples may be image samples.

[0036] Clustering is an important data analysis method for grouping data in an unsupervised manner. The operation of clustering target samples is independent of categories, that is, category-agnostic. After the target samples are clustered in a category-agnostic manner, the resulting clusters can convey discriminative knowledge of known and unknown categories that are specific to the target domain. In the example embodiments of the present disclosure, various known clustering algorithms or clustering algorithms that may be known in the future can be used to cluster the target samples, for example, the k-means algorithm.

[0037] In step S220, the inherent cluster distribution of each target sample is calculated based on the generated K clusters, Cluster 1, Cluster 2, Cluster 3, ..., Cluster K. In the exemplary embodiment of the present disclosure, the basic data structure of the target sample is represented as the joint relationship between the target sample and all class-independent clusters, that is, the inherent cluster distribution of the sample relative to all clusters.

[0038] For example, in one instance, for each target sample x t , by applying the Softmax function with respect to the cosine similarity between the sample and each cluster center to measure its inherent cluster distribution Its kth element (i.e., used to represent x t and the center μ of the kth cluster k The cosine similarity between ) is expressed as follows:

[0039]

[0040] where coS() is the cosine similarity function, ρ is the temperature parameter of the softmax used for scaling, and the center μ of each cluster is k is defined as the average of all samples belonging to this cluster.

[0041] Next, in step S230, a transfer feature learning algorithm is used to perform transfer feature learning on the target sample according to the source sample, and the intermediate layer features of the target sample are obtained. The clustering-based transfer feature learning method according to the exemplary embodiment of the present disclosure can be applied to various transfer feature learning algorithms for image processing, such as self-training algorithms, multi-view training algorithms, and self-ensembling algorithms. Here, the description of various transfer feature learning algorithms will be omitted. If the sample is an image sample, the intermediate layer features obtained in this step are the intermediate layer features of the image.

[0042] Then, in step S240, the obtained intermediate layer features are used to calculate the clustering distribution of each sample. After the intermediate layer features of the sample are obtained by the transfer feature learning algorithm, the learning sample represented by the intermediate layer features can be calculated by using the same calculation method as step S220. Distribution of cluster assignments relative to all clusters

[0043]

[0044] in YesP clu The kth element in indicates that the learning sample The probability of being assigned to the kth cluster; W k is the parameter matrix in the modified softmax layer The k-th row of represents the clustering assignment parameter matrix for the k-th cluster.

[0045] In step S250, based on the inherent cluster distribution and / or the cluster assignment distribution, the transfer feature learning algorithm is adjusted to ensure that the feature representation obtained by the transfer feature learning retains the basic data structure of the target sample. The benchmark for adjusting the transfer feature learning algorithm based on the inherent cluster distribution and / or the cluster assignment distribution can be implemented in a variety of ways as needed, which will be referred to in Figure 4 When the clustering-based transfer feature learning method according to the exemplary embodiment of the present disclosure is applied to image processing, the feature representation obtained by learning is an image feature representation for the image sample.

[0046] In summary, the clustering-based transfer feature learning method according to the exemplary embodiment of the present disclosure introduces a clustering branch into the existing transfer feature learning framework, so that the feature representation obtained by the clustering-based transfer feature learning method can be domain-invariant with respect to known category samples and more discriminative with respect to known category samples and unknown category samples in the target domain, such as Figure 3 shown.

[0047] Figure 4 A schematic diagram of the total loss L of the clustering-based transfer feature learning method for image processing according to an exemplary embodiment of the present disclosure is shown.

[0048] KL divergence minimizes L KL 411

[0049] KL divergence (Kullback-Leibler divergence) is a method used to describe the difference between two probability distributions. The smaller the KL divergence, the more similar the two probability distributions are.

[0050] Specifically, in an exemplary embodiment of the present disclosure, the inherent cluster distribution and the cluster assignment distribution are aligned by minimizing the KL divergence between the inherent cluster distribution and the cluster assignment distribution, wherein the KL divergence is defined as:

[0051]

[0052] Minimizing KL divergence with constraints on relationships between clusters

[0053] In another example, adjusting the transfer feature learning algorithm based on the inherent cluster distribution and the cluster assignment distribution may also include: additionally introducing an inter-cluster relationship as a constraint to the above-mentioned KL divergence to retain the inherent relationship between the cluster assignment parameter matrices, so that the resulting feature representation is more robust to diverse samples. In the case of the inter-cluster relationship as a constraint, the KL divergence can be further expressed as:

[0054]

[0055] Mutual information maximization L MIM 412

[0056] Mutual information is a measure used to represent the information contained in one random variable about another random variable.

[0057] In one embodiment, adjusting the transfer feature learning algorithm based on the inherent cluster distribution and / or the cluster assignment distribution may also include maximizing the mutual information between the intermediate layer features, the predicted classification step output by the transfer feature learning algorithm, and the cluster assignment distribution. Mutual information can be further divided into global mutual information and local mutual information. If estimated by the Jensen-Shannon MI estimator, the global mutual information can be expressed as:

[0058]

[0059] in, The target sample is transformed through the convolution layer and the average pooling layer The output feature map of the last convolutional layer The global feature vector obtained by encoding is Indicates the target sample The prediction classification steps are: Indicates the target sample The clustering distribution of and Cascade and input the cascade features into the global mutual information discriminator for identification. is the output of the discriminator, which represents the probability of identifying a true input feature with a matching predicted classification step and cluster assignment distribution. In addition, Represents different target images The global eigenvector of ; Represents the softplus function.

[0060] The local mutual information can be expressed as:

[0061]

[0062] in, For the target sample The output feature map of the last convolutional layer, Indicates the target sample The prediction classification steps are: Indicates the target sample The clustering distribution of and Cascade and input the cascade features into the local mutual information discriminator for identification. is the output of the discriminator, where is the ith element in its output and represents the probability of identifying the true input local feature with a matching predicted classification step and cluster assignment distribution at the ith spatial location. In addition, Indicates that it is different from The target image The feature map of Represents the softplus function.

[0063] The mutual information estimation as a combination of the global mutual information estimation and the local mutual information estimation can be expressed as follows:

[0064]

[0065] Wherein, d is a trade-off parameter. In addition, the mutual information can be configured to include only local mutual information or global mutual information as required.

[0066] The total training loss L of the transfer feature learning algorithm T 413

[0067] In machine learning, the loss function is used to measure the inconsistency between the model and the test value and the true value, wherein the smaller the loss function, the more robust the learned model is. As described above, the inventive concept can actually be applied to various training algorithms such as self-training algorithms, multi-view training algorithms, and self-ensembling algorithms, wherein different models can have different total training losses L KL .

[0068] like Figure 4 As shown, the overall training goal of the clustering-based transfer feature learning method according to the embodiment of the present disclosure is to convert the total training loss L representing the transfer feature learning algorithm into T , the KL divergence loss L expressed by equation (3) or equation (4) KL , and the mutual information estimate L expressed in equation (7) MIM Input adder 420 to integrate it into the total loss L. That is, by comprehensively considering the losses of these three aspects, the total loss L of the clustering-based transfer feature learning method for image processing according to the exemplary embodiment of the present disclosure is determined, that is, it can be expressed as the following equation:

[0069] L = α × L T +γ×L KL +β×L MIM (8)

[0070] Among them, α, γ and β are weights assigned to each loss, and except that α cannot be set to 0, α, γ and β can be configured to any weights according to the actual requirements of the application, so that the clustering-based transfer feature learning method according to the exemplary embodiment of the present disclosure can focus on the losses caused by different reasons, and thus can be applicable to any specific environment.

[0071] Here, the KL divergence can refer to the KL divergence between the inherent cluster distribution and the cluster allocation distribution under the constraint of the relationship between clusters (i.e., Equation 4), or it can refer to the KL divergence between the inherent cluster distribution and the cluster allocation distribution without setting constraints (i.e., Equation 3), as needed.

[0072] The transfer feature learning algorithm is adjusted by maximizing the mutual information between the intermediate layer features, the predicted classification steps output by the transfer feature learning algorithm, and the cluster assignment distribution, so that the target features learned in an unsupervised manner can be strengthened.

[0073] In addition, it should be noted that the above disclosed embodiments of adjusting the transfer feature learning algorithm based on the inherent cluster distribution and / or the cluster assignment distribution can be implemented separately or in combination with each other or with other parameters as needed.

[0074] In summary, the clustering-based transfer feature learning method according to the exemplary embodiment of the present disclosure introduces a clustering branch into the existing transfer feature learning framework, and takes the KL divergence between the inherent cluster distribution and the cluster assignment distribution of the target sample as an additional constraint condition, so that the feature representation obtained by the clustering-based transfer feature learning method can be domain-invariant with respect to known category samples and more discriminative with respect to known category samples and unknown category samples in the target domain.

[0075] Figure 5 The schematic framework of the self-ensembling with category-agnostic cluster (SE-CC) feature learning method is shown.

[0076] In such Figure 5 In the self-ensembling transfer feature learning framework shown in the figure, the labeled source image samples (x s ,y s ) is input to the student model so that the classifier in the student model can be trained by cross entropy, where y s is the image sample x s The target image sample is transformed into two perturbation image samples and Input to the teacher model and student model respectively. Apply conditional entropy to the perturbed image samples in the student model And use the self-ensemble loss to align the classification predictions in the student model and the teacher model. In this case, the total training loss Can include supervised cross entropy loss for source image samples Self-integration loss of target image samples and conditional entropy loss

[0077] In this method, in addition to aligning the classification predictions in the student model and the teacher model through the self-ensemble loss, the cluster assignment distribution is also aligned with the inherent cluster distribution through methods such as KL divergence.

[0078] also, Figure 5The SE-CC transfer feature learning method shown also additionally includes a mutual information maximization branch, in which the robustness of the feature representation learned by the student model is enhanced and the most essential structural features of the target domain image samples are retained by maximizing the mutual information between the intermediate layer features in the student model, the predicted classification steps estimated by the student model, and the cluster assignment distribution estimated by the student model.

[0079] exist Figure 5 Under the framework shown, equation (8) is further specified as:

[0080]

[0081] Among them, β is a trade-off parameter.

[0082] In summary, a method for transfer feature learning is provided for domain adaptation in open set and closed set scenarios using category-agnostic clustering in the target domain. The clustering-based transfer feature learning technology according to the exemplary embodiment of the present disclosure can be integrated with various existing technologies as a plug-in, so that they can be applied to both closed scenarios and real scenarios. Specifically, the proposed clustering-based transfer feature learning method can be used for image processing, for example, first by performing unsupervised clustering in the target domain to learn some clusters that are independent of the category but represent the most essential structural features of the image samples; then, an additional clustering branch is introduced into the transfer feature learning technology, so that the cluster distribution predicted by the image feature representation obtained by the transfer feature learning is consistent with the original cluster distribution, so that the learned image features can have the most essential structural features of the target domain. Additionally, the robustness of the learned image feature representation and its essential structural features can be further enhanced by introducing a measure of maximizing mutual information. Since the clustering-based transfer feature learning technology according to the exemplary embodiments of the present disclosure performs category-agnostic clustering on target samples and generates multiple clusters that can convey identification information of known categories and unknown categories specific to the target domain, the clustering-based transfer feature learning technology according to the exemplary embodiments of the present disclosure is applicable not only to closed sets but also to open sets or a mixed scenario of the two.

[0083] Figure 6 A block diagram of a clustering-based transfer feature learning apparatus according to an exemplary embodiment of the present disclosure is schematically shown.

[0084] like Figure 6As shown, the clustering-based migration feature learning device according to the exemplary embodiment of the present disclosure may include: a clustering engine 610, a first cluster distribution calculation module 620-1, a migration feature learning module 630, a second cluster distribution calculation module 620-2 and an algorithm adjustment module 640. The clustering engine 610 can be configured to cluster the target sample to generate K clusters, where K is an integer and K≥1. The first cluster distribution calculation module 620-1 can be configured to calculate the inherent cluster distribution of each target sample based on the generated K clusters. The migration feature learning module 630 can be configured to perform migration feature learning on each target sample according to the source sample by using the migration feature learning algorithm, and obtain the intermediate layer features of the target sample. The second cluster distribution calculation module 620-3 can be configured to calculate the cluster allocation distribution of each target sample using the obtained intermediate layer features. The algorithm adjustment module 640 can be configured to adjust the migration feature learning algorithm based on the inherent cluster distribution and / or the cluster allocation distribution to ensure that the feature representation obtained by the migration feature learning retains the basic data structure of the target sample. Specifically, the algorithm adjustment module 640 can be configured to ensure that the feature representation obtained by transferring feature learning retains the basic data structure of the target sample by minimizing the KL divergence between the inherent cluster distribution and the cluster allocation distribution. Alternatively, the algorithm adjustment module 640 can also be configured to minimize the KL divergence between the inherent cluster distribution and the cluster allocation distribution under the premise of constraining the relationship between clusters to ensure that the feature representation obtained by transferring feature learning retains the basic data structure of the target sample. As another example, the algorithm adjustment module 640 can also be configured to maximize the mutual information between the intermediate layer features, the predicted classification steps output by the transfer feature learning algorithm, and the cluster allocation distribution to further enhance the robustness of the feature representation and retain the basic data structure.

[0085] According to the modules and units of the embodiments of the present disclosure, any multiple or at least part of the functions of any multiple thereof can be implemented in one module. According to the modules and units of the embodiments of the present disclosure, any one or more can be split into multiple modules for implementation. According to the modules and units of the embodiments of the present disclosure, any one or more can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware of any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, according to the modules and units of the embodiments of the present disclosure, one or more can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0086] For example, any of the clustering engine 610, the first clustering distribution calculation module 620-1, the migration feature learning module 630, the second clustering distribution calculation module 620-2, and the algorithm adjustment module 640 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the clustering engine 610, the first clustering distribution calculation module 620-1, the migration feature learning module 630, the second clustering distribution calculation module 620-2, and the algorithm adjustment module 640 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware, and firmware, or by a suitable combination of any of them. Alternatively, at least one of the clustering engine 610, the first cluster distribution calculation module 620-1, the migration feature learning module 630, the second cluster distribution calculation module 620-2 and the algorithm adjustment module 640 can be at least partially implemented as a computer program module, which can perform corresponding functions when executed.

[0087] Figure 7 A schematic diagram of a clustering-based transfer feature learning device according to an exemplary embodiment of the present disclosure is schematically shown. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0088] like Figure 7 As shown, the electronic device 700 includes a processor 710 and a computer-readable storage medium 720. The electronic device 700 can execute the method according to the embodiment of the present disclosure.

[0089] Specifically, the processor 710 may include, for example, a general-purpose microprocessor, an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 70 may also include an onboard memory for cache purposes. The processor 710 may be a single processing unit or multiple processing units for executing different actions of the method flow according to an embodiment of the present disclosure.

[0090] The computer-readable storage medium 720 may be, for example, a non-volatile computer-readable storage medium, and specific examples include but are not limited to: a magnetic storage device, such as a magnetic tape or a hard disk (HDD); an optical storage device, such as a compact disk (CD-ROM); a memory, such as a random access memory (RAM) or flash memory; and the like.

[0091] The computer-readable storage medium 720 may include a computer program 721 , which may include code / computer-executable instructions that, when executed by the processor 710 , cause the processor 710 to perform a method according to an embodiment of the present disclosure or any variation thereof.

[0092] The computer program 721 may be configured to have, for example, a computer program code including a computer program module. For example, in an example embodiment, the code in the computer program 721 may include one or more program modules, for example, including a module 721A, a module 721B, ... It should be noted that the division method and number of modules are not fixed, and those skilled in the art may use appropriate program modules or program module combinations according to actual conditions, and when these program module combinations are executed by the processor 710, the processor 710 may execute the method according to the embodiment of the present disclosure or any variation thereof.

[0093] According to an embodiment of the present disclosure, at least one of the clustering engine 610, the first clustering distribution calculation module 620-1, the second clustering distribution calculation module 620-2, the migration feature learning module 630, and the algorithm adjustment module 640 may be implemented as a reference. Figure 7 The computer program modules described herein, when executed by the processor 710, can implement the corresponding operations described above.

[0094] Figure 8 The flowchart of the image processing method 800 according to an exemplary embodiment of the present disclosure is schematically shown. Figure 8 As shown, the image processing method 800 may include receiving an image sample to be processed at step S810; processing the image sample by using the transfer feature learning method according to any one of the aforementioned exemplary embodiments to obtain a feature representation for the image sample at step 820; and outputting the image feature representation at step 830. Specifically, step 820 may be further refined to include: first clustering the image samples to generate K clusters, where K is an integer and K≥1; then calculating the inherent cluster distribution of each image sample based on the generated K clusters, performing transfer feature learning on each image sample by using the clustering-based transfer feature learning algorithm configured or adjusted by the aforementioned method, and obtaining the intermediate layer image features for each image sample, thereby obtaining the image feature representation for each image sample.

[0095] Fig. 9 Schematically shows a block diagram of an image processing apparatus 900 according to an exemplary embodiment of the present disclosure. Fig. 9 As shown, the image processing device 900 may include a communicator 910 and a migration feature learning device 920 (eg, Figure 6 or Figure 7 The transfer feature learning device 920 is shown in FIG. 1 . The communicator 910 may be configured to receive the image sample to be processed via wired or wireless communication. In addition, the communicator 910 may also be configured to output a feature representation for the image sample. For example, the communicator may be a USB interface, a WiFi interface, a Bluetooth interface, or various interfaces of a power line. The transfer feature learning device 920 may be configured to process the image sample from the communicator 910 to obtain a feature representation for the image sample.

[0096] In another implementation, the image processing device according to the image processing device of the exemplary embodiment of the present disclosure may also be implemented to include a processor and a memory. Specifically, the memory may be configured to store one or more executable instructions, and the processor may be connected to the memory and used to execute the executable instructions stored in the memory to execute the clustering-based migration feature learning method as described in the above exemplary embodiment or execute the image processing method as described in the above exemplary embodiment.

[0097] It can be seen that the image processing method and device according to the exemplary embodiment of the present disclosure can process image samples using a clustering-based transfer feature learning method, so that first, by performing unsupervised clustering in the target domain, some clusters that are independent of the category but represent the most essential structural features of the image samples are learned; then, additional clustering branches are introduced into the transfer feature learning technology, so that the cluster distribution predicted by the image feature representation obtained by the transfer feature learning is consistent with the original cluster distribution, so that the learned image features can have the most essential structural features of the target domain. In addition, the robustness of the learned image feature representation and its essential structural features can be further enhanced by introducing a measure of maximizing mutual information.

[0098] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0099] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0100] It will be appreciated by those skilled in the art that, although the present disclosure has been shown and described with reference to specific exemplary embodiments of the present disclosure, it will be appreciated by those skilled in the art that, without departing from the spirit and scope of the present disclosure as defined by the appended claims and their equivalents, various changes in form and detail may be made to the present disclosure. Therefore, the scope of the present disclosure should not be limited to the above-described embodiments, but should be determined not only by the appended claims, but also by the equivalents of the appended claims.

Claims

1. A clustering-based transfer feature learning method for image processing, comprising: Cluster the target samples to generate K clusters, where K is an integer and K ≥ 1; Based on the generated K clusters, calculate the inherent cluster distribution of each target sample; By using a transfer feature learning algorithm to perform transfer feature learning on each target sample based on the source sample, and obtaining the intermediate layer features of the target sample; Using the obtained intermediate layer features, calculate the clustering distribution of each target sample; as well as Based on the inherent cluster distribution and / or the cluster assignment distribution, adjusting the transfer feature learning algorithm to ensure that the feature representation obtained by the transfer feature learning retains the underlying data structure of the target sample; Wherein, adjusting the transfer feature learning algorithm based on the inherent cluster distribution and / or the cluster assignment distribution includes: minimizing the KL divergence between the inherent cluster distribution and the cluster assignment distribution; or maximizing the mutual information between the intermediate layer features, the predicted classification distribution output by the transfer feature learning algorithm, and the cluster assignment distribution; The total loss of the clustering-based transfer feature learning method is comprehensively determined based on the total training loss, KL divergence loss and mutual information estimation representing the transfer feature learning algorithm, and the total training loss, the KL divergence loss and the mutual information estimation each correspond to a weight for loss allocation.

2. The method for transfer feature learning according to claim 1, wherein adjusting the transfer feature learning algorithm based on the inherent cluster distribution and / or the cluster assignment distribution comprises: Under the premise of taking the relationship between clusters as a constraint, the KL divergence between the inherent cluster distribution and the cluster assignment distribution is minimized. 3 . The migration feature learning method according to claim 1 , wherein the target samples are clustered by using a k-means algorithm.

4. The transfer feature learning method according to claim 1, wherein the transfer feature learning algorithm is any one of a self-training algorithm, a multi-view training algorithm and a self-integration algorithm.

5. The migration feature learning method according to claim 1, wherein the mutual information includes at least one of global mutual information and local mutual information between the intermediate layer features, the predicted classification distribution and the cluster assignment distribution.

6. A clustering-based migration feature learning device for image processing, comprising: A clustering engine configured to cluster the target sample to generate K clusters, where K is an integer and K ≥ 1; A first cluster distribution calculation module is configured to calculate the inherent cluster distribution of each target sample based on the generated K clusters; A transfer feature learning module, configured to perform transfer feature learning on each target sample according to the source sample by using a transfer feature learning algorithm, and obtain an intermediate layer feature of the target sample; A second cluster distribution calculation module is configured to calculate the cluster allocation distribution of each target sample using the obtained intermediate layer features; as well as an algorithm adjustment module configured to adjust the transfer feature learning algorithm based on the inherent cluster distribution and / or the cluster assignment distribution to ensure that the feature representation obtained by the transfer feature learning retains the underlying data structure of the target sample; The algorithm adjustment module is configured to minimize the KL divergence between the inherent cluster distribution and the cluster assignment distribution; the algorithm adjustment module is also configured to maximize the mutual information between the predicted classification distribution output by the intermediate layer features and the transfer feature learning algorithm and the cluster assignment distribution; The total loss of the clustering-based transfer feature learning method is comprehensively determined based on the total training loss, KL divergence loss and mutual information estimation representing the transfer feature learning algorithm, and the total training loss, the KL divergence loss and the mutual information estimation each correspond to a weight for loss allocation.

7. An image processing method, comprising: Receive image samples to be processed, Processing the image sample by using the transfer feature learning method according to any one of claims 1 to 5 to obtain a feature representation for the image sample, and The feature representation is output.

8. An image processing device, comprising: a communicator configured to receive image samples to be processed; as well as The transfer feature learning device according to claim 6, configured to process the image sample to obtain a feature representation for the image sample, The communicator is further configured to output the feature representation.

9. An image processing device, comprising: a memory configured to store one or more executable instructions; as well as A processor is connected to the memory and is used to execute executable instructions stored in the memory to execute the migration feature learning method as described in any one of claims 1 to 5 or the image processing method as described in claim 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, which, when executed by a processor, implement the migration feature learning method according to any one of claims 1 to 5 or the image processing method according to claim 7.

Images