Image annotation method and device based on curriculum learning and deep learning

Abstract

The present invention discloses a method and device for image annotation based on curriculum learning and deep learning. The method comprises: acquiring and preprocessing unlabeled images to form a dataset; constructing an autoencoder, converting the dataset into embedded features, and pretraining the autoencoder; fine-tuning the autoencoder; and using the trained autoencoder to find the density kernel corresponding to each cluster based on the clustering results of the embedded features, and confirming the label category based on the density kernel to obtain the image annotation results. The present invention designs an autoencoder and proposes a new image annotation method through fine-tuning training including calculating difficulty scores, curriculum generation, and density kernel-based clustering. This method minimizes computing resource consumption while accelerating model convergence and improving algorithm robustness.

Description

Technical Field

[0001] The present invention relates to the technical field of image annotation, and in particular to an image annotation method and device based on curriculum learning and deep learning. Background Art

[0002] With the rapid development of information technology, information generation channels are becoming increasingly diverse, and the explosive growth of data has provided abundant data support for high-quality development across various industries. Analyzing high-value patterns from complex data is a prerequisite for high-quality development in all industries. Image data is one of the most common and most abundant data types generated by various industries. However, raw images generated by various industries often lack label information. Furthermore, due to the cost constraints and subjective operation factors of manual labeling, unsupervised data clustering has become a key method for image data labeling.

[0003] As a key technology in artificial intelligence, machine learning, data mining, and pattern recognition, clustering algorithms aim to divide target data (such as images and text) into distinct groups, ensuring that data within the same group is highly homogeneous while data from different groups is highly diverse. This can quickly reduce redundant and useless information within the raw data, revealing underlying patterns or structures, and enabling unsupervised data classification.

[0004] Traditional clustering methods often struggle to achieve high clustering accuracy when faced with high-dimensional, large-scale, complex data. This is primarily due to inherent limitations of algorithms when processing complex data such as images, such as the curse of dimensionality, artificially engineered features, and low-quality similarity metrics. Furthermore, the CPU-based computing platforms that traditional algorithms rely on are not ideal for processing image data. Recently, a clustering paradigm combined with deep learning (referred to as deep clustering (DC)) has garnered significant attention as it bridges the gap between traditional clustering methods and high-dimensional data. Deep clustering algorithms are unsupervised learning methods that combine deep learning with clustering techniques. They aim to learn the underlying feature representation of the data and map it to a low-dimensional space in which clustering operations can be performed. Compared to traditional clustering methods such as K-means and DBSCAN, deep clustering algorithms can more effectively handle complex nonlinear data structures and exhibit stronger generalization capabilities.

[0005] Curriculum learning is a promising machine learning method that focuses on the inherent properties of the current data, avoids dependence on external data or supervisory information, and minimizes the consumption of computing resources. Curriculum learning has the potential to accelerate model convergence and improve algorithm robustness. Summary of the Invention

[0006] The main purpose of this invention is to overcome the shortcomings and deficiencies of existing technologies and provide an image annotation method and apparatus based on curriculum learning and deep learning. This invention designs an autoencoder that, through fine-tuning training including difficulty score calculation, curriculum generation, and density kernel-based clustering, proposes a new image annotation method that minimizes computational resource consumption while accelerating model convergence and improving algorithm robustness.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] In a first aspect, the present invention provides an image annotation method based on curriculum learning and deep learning, comprising the following steps:

[0009] Obtain and preprocess unlabeled images to form a data set;

[0010] Construct an autoencoder, convert the dataset into embedded features in the embedding space, and pre-train the autoencoder. The autoencoder includes an encoder's image segmentation and encoding module, a self-attention encoding module, and a linear encoding module, and a decoder's linear decoding module, a self-attention module, and a transposition layer. The transposition layer restores the features to the shape of the original data.

[0011] Using the clustering layer, the autoencoder is fine-tuned to obtain a trained autoencoder and cluster the embedded features. Specifically, the density and difficulty score of the embedded features are calculated. The embedded features are initially pre-partitioned using kmeans, the proportion of selected samples is calculated, and the descending order of the embedded features within the cluster and the indicator variable of each sample within the cluster are iteratively calculated based on the difficulty score to obtain the training set. The cluster density kernel of the embedded features is iteratively calculated based on the set density kernel selection ratio, and the cluster assignment probability is predicted to form a soft label distribution. The embedded features are clustered and the iteration is terminated when the change in the clustering result is less than the threshold. The loss of the training set is iteratively calculated and the autoencoder is trained through backpropagation to obtain the trained autoencoder.

[0012] According to the clustering results of the embedded features, the trained autoencoder is used to find the density kernel corresponding to each cluster, and the label category is confirmed based on the density kernel to obtain the image annotation results.

[0013] As a preferred technical solution, the method of converting the dataset into embedded features of the embedded space and pre-training the autoencoder is as follows:

[0014] Use the patch encoding layer to convert the dimension I c *I h *I w The dataset is converted into dimension I p *I e Patch code, where I c Indicates the number of channels of the image, Ih Indicates the height of the image, I w Indicates the width of the image, I p Indicates the number of patch codes, I e Indicates the size of the patch encoding;

[0015] The patch code is compiled by the self-attention module and the absolute position code, and then the patch code is converted into an embedding feature Z={z1,z2,…,z n},in And i=1,…,n, d represents the dimension of the embedding space, i represents the i-th sample, n represents the number of embedded features, x represents the image block in the dataset, f W (·) indicates encoder;

[0016] The output result is obtained by the transpose layer of the decoder;

[0017] Calculate the loss of the output and train the autoencoder.

[0018] As a preferred technical solution, the density and difficulty score of the embedded features are calculated as follows:

[0019] Use the dataset X to continue to embed feature density ρ i ,

[0020]

[0021] where d c is the sampling radius for density calculation, j represents the jth sample;

[0022] According to the embedding feature density ρ i Calculating difficulty score δ i

[0023]

[0024]

[0025] Where j′ represents the j′th sample and λ1 represents the density sampling radius d c The corresponding ratio.

[0026] As a preferred technical solution, the ratio of selected samples is calculated, and the descending order of embedded features in the cluster and the indicator variable of each sample in the cluster are iteratively calculated according to the difficulty score to obtain the training set, specifically,

[0027] Calculate the proportion of selected samples ζ iter , as follows:

[0028]

[0029] t=iter

[0030] Where t represents the iteration round, iter represents the actual iteration round, 0 Indicates the initial proportion of selected samples, T grow represents the number of iterations required for the encoder to reach 100% of the value for the first time, ζ max Indicates the maximum proportion of samples selected by control;

[0031] Iteratively calculate the descending order of the clusters of embedded features according to the difficulty score, and calculate the indicator variable of each sample in the cluster of embedded features As follows:

[0032]

[0033] t=iter

[0034] Among them C k represents the kth cluster generated from the result of the (t-1)th iteration, Represents x i The difficulty score at iteration t, is an ordered sequence, It is cluster C k Corresponding to ζ t The value at the position;

[0035] Get the training set As follows:

[0036]

[0037] As a preferred technical solution, the cluster density kernel of the embedded feature is iteratively calculated according to the set density kernel selection ratio, and the cluster assignment probability is predicted to form a soft label distribution, cluster the embedded features, and the iteration is terminated when the change in the clustering result is less than the threshold. Specifically,

[0038] According to the set density kernel selection ratio λ2, the cluster density kernel of the embedded feature is calculated As follows:

[0039]

[0040] Where i represents the i-th sample, ρ i represents the local density of the i-th sample in the embedded feature space, Represents cluster C k The value of the λ2 position in the descending sequence of the density values ​​of all samples in , k represents the number of clusters of embedded features;

[0041] Use Student's T distribution to predict cluster assignment probability q ik , as follows:

[0042]

[0043] Where j represents the jth sample;

[0044] Form a soft label distribution and cluster the embedded features to obtain the clustering result Y, as shown in the following formula:

[0045]

[0046] Where n represents the number of embedded features;

[0047] Iterate the above steps. When the change of two consecutive clustering results Y is less than the threshold μ, the iteration terminates.

[0048] As a preferred technical solution, an auxiliary target variable p is also included. ik To repeatedly improve the model's predictions, as follows:

[0049]

[0050] As a preferred technical solution, the iterative calculation of the loss of the training set and the back-propagation training of the autoencoder are specifically as follows:

[0051] Using the training set Calculate clustering loss Reconstruction loss and the overall loss L t , as follows:

[0052]

[0053]

[0054]

[0055] where t represents the number of iterations, KL(·) represents the divergence between the cluster assignment matrix and the auxiliary distribution matrix, P represents the auxiliary distribution matrix, Q represents the cluster assignment matrix, and f W (·) represents the encoder mapping function, g U (·) represents the decoder mapping function, α represents the weight of the reconstruction loss;

[0056] Update training set And back-propagation trains the autoencoder. In a second aspect, the present invention provides an image annotation system based on curriculum learning and deep learning, which is applied to the image annotation method based on curriculum learning and deep learning, including an image acquisition module, a model building module, a fine-tuning training module and an image annotation module;

[0057] The image acquisition module acquires and preprocesses unlabeled images to form a data set;

[0058] The model building module constructs an autoencoder, uses the embedded features converted from the dataset into an embedding space and pre-trains the autoencoder; the autoencoder includes an image segmentation and encoding module, a self-attention encoding module, and a linear encoding module of the encoder, and a linear decoding module, a self-attention module, and a transposition layer of the decoder; the transposition layer restores the features to the shape of the original data;

[0059] The fine-tuning training module uses a clustering layer to fine-tune the autoencoder to obtain a trained autoencoder and cluster the embedded features. Specifically, the module calculates the density and difficulty score of the embedded features; initially pre-partitions the embedded features using kmeans, calculates the proportion of selected samples, and iteratively calculates the descending order of the embedded features within the cluster and the indicator variable of each sample within the cluster based on the difficulty score to obtain a training set; iteratively calculates the cluster density kernel of the embedded features based on the set density kernel selection ratio, predicts the cluster assignment probability, forms a soft label distribution, clusters the embedded features, and terminates the iteration when the change in the clustering result is less than a threshold; iteratively calculates the loss of the training set and back-propagates the training autoencoder to obtain a trained autoencoder;

[0060] The image annotation module uses the trained autoencoder to find the density kernel corresponding to each cluster based on the clustering results of the embedded features, and confirms the label category based on the density kernel to obtain the image annotation results.

[0061] In a third aspect, the present invention provides an electronic device, comprising:

[0062] at least one processor; and,

[0063] a memory communicatively connected to the at least one processor; wherein,

[0064] The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor to enable the at least one processor to perform the image annotation method based on curriculum learning and deep learning.

[0065] In a fourth aspect, the present invention provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the image annotation method based on curriculum learning and deep learning.

[0066] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0067] (1) This paper proposes a density kernel-based clustering method, which iteratively calculates the cluster density kernel of the embedded features according to the set density kernel selection ratio, predicts the cluster assignment probability, forms a soft label distribution, and clusters the embedded features, thereby achieving the best clustering performance;

[0068] (2) The present invention proposes an autoencoder based on a self-attention module, which minimizes the consumption of computing resources while accelerating model convergence and improving algorithm robustness through fine-tuning training including calculating difficulty scores, curriculum generation, and density kernel-based clustering. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0070] Figure 1 This is a workflow diagram of an image annotation method based on curriculum learning and deep learning according to an embodiment of the present invention;

[0071] Figure 2 Schematic diagram of convergence analysis in an embodiment of the present invention;

[0072] Figure 3 Schematic diagram of the structure of an image annotation system based on curriculum learning and deep learning in an embodiment of the present invention;

[0073] Figure 4 A structural diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0074] In order to enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0075] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0076] See also Figure 1 The image annotation method based on curriculum learning and deep learning in this embodiment includes the following steps:

[0077] S1. Obtain and preprocess unlabeled images to form a dataset;

[0078] S2. Construct an autoencoder, convert the dataset into embedded features in an embedding space, and pre-train the autoencoder. The autoencoder includes an encoder's image segmentation and encoding module, a self-attention encoding module, and a linear encoding module; a decoder's linear decoding module, a self-attention module, and a transposition layer. The transposition layer restores the features to the shape of the original data.

[0079] Optionally, in this step, n images are organized into batches of size 256 for training, and pre-training is performed on the loss corresponding to a single batch for 500 epochs.

[0080] Preferably, the method of converting the dataset into an embedding feature of an embedding space and pre-training the autoencoder is as follows:

[0081] S21. Use the patch coding layer to transform the dimension I c *I h *I w The dataset is converted into dimension I p *I e Patch code, where I c Indicates the number of channels of the image, I h Indicates the height of the image, I w Indicates the width of the image, I p Indicates the number of patch codes, I e Indicates the size of the patch encoding;

[0082] S22. The patch code is compiled by the self-attention module and the absolute position code, and then the patch code is converted into an embedding feature Z = {z1,z2,…,z n},in And i=1,…,n, d represents the dimension of the embedding space, i represents the i-th sample, n represents the number of embedded features, x represents the image block in the dataset, f W (·) indicates encoder;

[0083] S23. The output result is obtained by the transpose layer of the decoder;

[0084] S24. Calculate the loss of the output and train the autoencoder.

[0085] S3. Use the clustering layer to fine-tune the autoencoder to obtain a trained autoencoder and cluster the embedded features; specifically: calculate the density and difficulty score of the embedded features; initially use kmeans to pre-partition the embedded features, calculate the proportion of selected samples, and iteratively calculate the descending order of the embedded features in the cluster and the indicator variable of each sample in the cluster based on the difficulty score to obtain the training set; iteratively calculate the cluster density kernel of the embedded features according to the set density kernel selection ratio, and predict the cluster assignment probability to form a soft label distribution, cluster the embedded features, and terminate the iteration when the change in the clustering result is less than the threshold; iteratively calculate the loss of the training set and backpropagate the training autoencoder to obtain a trained autoencoder.

[0086] Optionally, step S3 specifically includes:

[0087] S31. Calculate the density and difficulty score of the embedded features;

[0088] Preferably, step S31 is specifically as follows:

[0089] S311. Use the dataset X to continue to change the embedding feature density ρ i ,

[0090]

[0091] where d c is the sampling radius for density calculation, j represents the jth sample;

[0092] S312. According to the embedded feature density ρ i Calculating difficulty score δ i , as follows:

[0093]

[0094]

[0095] Where j′ represents the j′th sample and λ1 represents the density sampling radius d c The corresponding ratio;

[0096] S32. Calculate the proportion of selected samples, and iteratively calculate the descending order of the clusters of embedded features and the indicator variable of each sample in the cluster of embedded features according to the difficulty score to obtain a training set;

[0097] Preferably, step S32 is specifically as follows:

[0098] S321. Calculate the proportion of selected samples ζ iter , as follows:

[0099]

[0100] t=iter

[0101] Where t represents the iteration round, iter represents the actual iteration round, 0 Indicates the initial proportion of selected samples, T grow represents the number of iterations required for the encoder to reach 100% of the value for the first time, ζ max Indicates the maximum proportion of samples selected by control;

[0102] S322. Iteratively calculate the descending order of the clusters of embedded features according to the difficulty score, and calculate the indicator variable of each sample in the cluster of embedded features As follows:

[0103]

[0104] t=iter

[0105] Among them C k represents the kth cluster generated from the result of the (t-1)th iteration, Represents x i The difficulty score at iteration t, is an ordered sequence, It is cluster C k Corresponding to ζ t The value at the position;

[0106] Furthermore, v i =1 means x i is an easy sample, otherwise, it is classified as a hard sample;

[0107] S323. Obtaining training set As follows:

[0108]

[0109] S33. Iteratively calculate the cluster density kernel of the embedded features according to the set density kernel selection ratio, and predict the cluster assignment probability to form a soft label distribution, cluster the embedded features, and terminate the iteration when the change in the clustering result is less than the threshold;

[0110] Preferably, step S33 is specifically as follows:

[0111] S331. Calculate the cluster density kernel of the embedded feature according to the set density kernel selection ratio λ2

[0112]

[0113] Where i represents the i-th sample, ρ i represents the local density of the i-th sample in the embedded feature space, Represents cluster Ck The value of the λ2 position in the descending sequence of the density values ​​of all samples in , k represents the number of clusters of embedded features;

[0114] Furthermore, the density kernel selection ratio λ2 is selected as 5%;

[0115] S332. Using Student's T distribution to predict cluster assignment probability q ik , as follows:

[0116]

[0117] Where j represents the jth sample;

[0118] S333. Form a soft label distribution and cluster the embedded features to obtain the clustering result Y, as shown in the following formula:

[0119]

[0120] Where n represents the number of embedded features;

[0121] S334. Iterate the above steps until the change in the clustering result Y is less than the threshold μ for two consecutive times, and then terminate the iteration.

[0122] Furthermore, step S3 also includes an auxiliary target variable p ik To repeatedly improve the model's predictions, as follows:

[0123]

[0124] S34. Iteratively calculate the loss of the training set and back-propagate the training autoencoder to obtain a trained autoencoder.

[0125] Preferably, step S34 is specifically as follows:

[0126] S341. Using the training set Calculate clustering loss Reconstruction loss and the overall loss L t , as follows:

[0127]

[0128]

[0129]

[0130] where t represents the number of iterations, KL(·) represents the divergence between the cluster assignment matrix and the auxiliary distribution matrix, P represents the auxiliary distribution matrix, Q represents the cluster assignment matrix, and f W (·) represents the encoder mapping function, g U(·) represents the decoder mapping function, α represents the weight of the reconstruction loss;

[0131] Furthermore, the weight α of the reconstruction loss is set to 0.1;

[0132] S342. Update training set And back-propagation trains the autoencoder;

[0133] S4. Based on the clustering results of the embedded features, use the trained autoencoder to find the density kernel corresponding to each cluster, and confirm the label category based on the density kernel to obtain the image annotation results.

[0134] In addition, the fine-tuning training of the present invention includes a course generator G, whose operation mode is shown in Table 1.

[0135] Table 1 Course Generator

[0136]

[0137] Furthermore, the fine-tuning training of the present invention also includes the following algorithm process, as shown in Table 2:

[0138] Table 2 Algorithm flow

[0139]

[0140]

[0141] The present invention selected MNIST, Emnist-Letters, Digits, GTSRB, YTF, and Reuters-10k data sets for verification, and compared the latest clustering algorithms at home and abroad. The detailed information of the data set is shown in Table 3 below, and the experimental results are shown in Table 4 below. The evaluation indicators used in the table are clustering accuracy ACC and normalized mutual information NMI (inside the brackets), and the values ​​are expressed as percentages. '-' indicates that the code cannot be run on the relevant data set. Overall, this algorithm has achieved the best clustering performance compared to the latest clustering algorithms at home and abroad on multiple complex data sets.

[0142] Table 3

[0143]

[0144] Table 4

[0145]

[0146] In order to verify the convergence of the algorithm, the present invention runs 200 iterations on the MNIST-test dataset. Figure 2 Visualize the convergence process. Figure 2Part (a) of the diagram shows the initial state of the data. The results show that the original data is chaotic and unguided, with several clusters crowded together. As the training progresses, the clusters formed become more reasonable. After 200 iterations, most data points in the same cluster are grouped together, as shown in Figure 2. Figure 2 The diagram of the dataset after 200 iterations is shown in part (b). Figure 2 In part (b) of , the darker dots represent the density core of each cluster. Figure 2 Parts (c) and (d) of the figure show the clustering metrics and training loss versus training epochs. It's clear that clustering performance significantly improved during the initial iterations, ultimately reaching a plateau. As these two figures illustrate, the present invention achieved convergence at iteration 100, demonstrating the effectiveness of the learning process.

[0147] In order to evaluate the efficiency of the present algorithm in processing large-scale and high-dimensional data, Table 4 shows the convergence time of the present method and other algorithms. The criterion used is the time taken to reach 90% of the maximum accuracy of the corresponding algorithm on the MNIST, LETTER AJ and STL datasets, respectively, in seconds. In general, the running times of DEC, IDEC, EDESC and the method proposed in the present invention on the three datasets are 941.6, 1059.4, 1103.4 and 919.5 seconds, respectively. From the perspective of computational convenience, the model of the present invention outperforms these state-of-the-art methods. In addition, the obvious time difference between the present invention and the degenerate model (which does not use the curriculum learning strategy designed in the present algorithm) verifies that the proposed curriculum learning scheme plays a vital role in the rapid convergence of the model.

[0148] Based on the same idea as the image annotation method based on curriculum learning and deep learning in the above-mentioned embodiment, the present invention also provides an image annotation system based on curriculum learning and deep learning, which can be used to execute the above-mentioned image annotation method based on curriculum learning and deep learning. For ease of explanation, the structural diagram of the embodiment of the image annotation system based on curriculum learning and deep learning only shows the parts related to the embodiment of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation of the device, and may include more or fewer components than shown in the diagram, or combine certain components, or arrange the components differently.

[0149] See also Figure 3 In another embodiment of the present application, an image annotation system 10 based on curriculum learning and deep learning is provided, the system comprising an image acquisition module 11, a model building module 12, a fine-tuning training module 13 and an image annotation module 14;

[0150] The image acquisition module 11 acquires and preprocesses unlabeled images to form a data set;

[0151] The model building module 12 constructs an autoencoder, uses the embedded features converted from the dataset into the embedding space and pre-trains the autoencoder; the autoencoder includes an image segmentation and encoding module, a self-attention encoding module, and a linear encoding module of the encoder, and a linear decoding module, a self-attention module, and a transposition layer of the decoder; the transposition layer restores the features to the shape of the original data;

[0152] The fine-tuning training module 13 uses a clustering layer to fine-tune the autoencoder to obtain a trained autoencoder and cluster the embedded features. Specifically, the module 13 calculates the density and difficulty score of the embedded features; initially pre-partitions the embedded features using kmeans, calculates the ratio of selected samples, and iteratively calculates the descending order of the embedded features in the cluster and the indicator variable of each sample in the cluster based on the difficulty score to obtain a training set; iteratively calculates the cluster density kernel of the embedded features based on the set density kernel selection ratio, predicts the cluster assignment probability, forms a soft label distribution, clusters the embedded features, and terminates the iteration when the change in the clustering result is less than a threshold; iteratively calculates the loss of the training set and back-propagates the training autoencoder to obtain a trained autoencoder;

[0153] The image annotation module 14 uses the trained autoencoder to find the density kernel corresponding to each cluster based on the clustering result of the embedded features, and confirms the label category based on the density kernel to obtain the image annotation result.

[0154] It should be noted that the image annotation system based on curriculum learning and deep learning of the present invention corresponds one-to-one to the image annotation method based on curriculum learning and deep learning of the present invention. The technical features and beneficial effects described in the above-mentioned embodiment of the image annotation method based on curriculum learning and deep learning are applicable to the embodiment of image annotation based on curriculum learning and deep learning. For specific contents, please refer to the description in the embodiment of the method of the present invention. No further details will be given here. This is hereby declared.

[0155] In addition, in the implementation of the image annotation system based on curriculum learning and deep learning in the above-mentioned embodiment, the logical division of each program module is only an example. In actual application, the above-mentioned functions can be assigned to different program modules as needed, for example, for the configuration requirements of the corresponding hardware or the convenience of software implementation. That is, the internal structure of the image annotation system based on curriculum learning and deep learning is divided into different program modules to complete all or part of the functions described above.

[0156] See also Figure 4In one embodiment, an electronic device for implementing an image annotation method based on curriculum learning and deep learning is provided. The electronic device 20 may include a first processor 21, a first memory 22 and a bus, and may also include a computer program stored in the first memory 22 and executable on the first processor 21, such as an image annotation program 23 based on curriculum learning and deep learning.

[0157] Among them, the first memory 22 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, SD or DX memory, etc.), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the first memory 22 can be an internal storage unit of the electronic device 20, such as a mobile hard disk of the electronic device 20. In other embodiments, the first memory 22 can also be an external storage device of the electronic device 20, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 20. Furthermore, the first memory 22 can also include both an internal storage unit of the electronic device 20 and an external storage device. The first memory 22 can not only be used to store application software and various types of data installed on the electronic device 20, such as the code of the image annotation program 23 based on course learning and deep learning, but can also be used to temporarily store data that has been output or is to be output.

[0158] In some embodiments, the first processor 21 may be composed of an integrated circuit, such as a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The first processor 21 is the control core (Control Unit) of the electronic device, connecting the various components of the entire electronic device using various interfaces and lines, and executing or executing programs or modules stored in the first memory 22, as well as calling data stored in the first memory 22, to perform various functions of the electronic device 20 and process data.

[0159] Figure 4 Only the electronic device with components is shown, and it can be understood by those skilled in the art that Figure 4 The structure shown does not constitute a limitation on the electronic device 20 , and the electronic device 20 may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0160] The image annotation program 23 based on curriculum learning and deep learning stored in the first memory 22 of the electronic device 20 is a combination of multiple instructions. When running in the first processor 21, it can achieve the following:

[0161] Obtain and preprocess unlabeled images to form a data set;

[0162] Constructing an autoencoder, using the embedded features converted from the dataset into an embedding space and pre-training the autoencoder; the autoencoder includes an encoder's image segmentation and encoding module, a self-attention encoding module, and a linear encoding module; a decoder's linear decoding module, a self-attention module, and a transposition layer; the transposition layer restores the features to the shape of the original data;

[0163] Using the clustering layer, the autoencoder is fine-tuned to obtain a trained autoencoder and cluster the embedded features. Specifically, the density and difficulty score of the embedded features are calculated. The embedded features are initially pre-partitioned using kmeans, the proportion of selected samples is calculated, and the descending order of the embedded features within the cluster and the indicator variable of each sample within the cluster are iteratively calculated based on the difficulty score to obtain the training set. The cluster density kernel of the embedded features is iteratively calculated based on the set density kernel selection ratio, and the cluster assignment probability is predicted to form a soft label distribution. The embedded features are clustered and the iteration is terminated when the change in the clustering result is less than the threshold. The loss of the training set is iteratively calculated and the autoencoder is trained through backpropagation to obtain the trained autoencoder.

[0164] According to the clustering results of the embedded features, the trained autoencoder is used to find the density kernel corresponding to each cluster, and the label category is confirmed based on the density kernel to obtain the image annotation results.

[0165] Furthermore, if the modules / units integrated in the electronic device 20 are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0166] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0167] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0168] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. An image annotation method based on curriculum learning and deep learning, characterized by: The following steps are involved: Obtain and preprocess unlabeled images to form a data set; Constructing an autoencoder, using the embedded features converted from the dataset into an embedding space and pre-training the autoencoder; the autoencoder includes an encoder's image segmentation and encoding module, a self-attention encoding module, and a linear encoding module; a decoder's linear decoding module, a self-attention module, and a transposition layer; the transposition layer restores the features to the shape of the original data; Using the clustering layer, fine-tune the autoencoder to obtain the trained autoencoder and cluster the embedded features; Specifically, the density and difficulty score of the embedded features are calculated; the embedded features are initially pre-partitioned using kmeans, the proportion of selected samples is calculated, and the descending order of the embedded features within the cluster and the indicator variable of each sample within the cluster are iteratively calculated based on the difficulty score to obtain the training set; the cluster density kernel of the embedded features is iteratively calculated based on the set density kernel selection ratio, and the cluster assignment probability is predicted to form a soft label distribution, and the embedded features are clustered. The iteration is terminated when the change in the clustering result is less than the threshold; the loss of the training set is iteratively calculated and the autoencoder is trained through backpropagation to obtain the trained autoencoder; According to the clustering results of the embedded features, the trained autoencoder is used to find the density kernel corresponding to each cluster, and the label category is confirmed based on the density kernel to obtain the image annotation results.

2. The image annotation method based on curriculum learning and deep learning according to claim 1, characterized in that: The method converts the dataset into the embedding features of the embedding space and pre-trains the autoencoder, specifically, Use the patch encoding layer to convert the dimension I c *I h *I w The dataset is converted into dimension I p *I e Patch code, where I c Indicates the number of channels of the image, I h Indicates the height of the image, I w Indicates the width of the image, I p Indicates the number of patch codes, I e Indicates the size of the patch encoding; The patch code is compiled by the self-attention module and the absolute position code, and then the patch code is converted into an embedding feature Z={z1,z2,…,z n },in And i=1,…,n, d represents the dimension of the embedding space, i represents the i-th sample, n represents the number of embedded features, x represents the image block in the dataset, f W (·) indicates encoder; The output result is obtained by the transpose layer of the decoder; Calculate the loss of the output and train the autoencoder.

3. The image annotation method based on curriculum learning and deep learning according to claim 2, characterized in that: The density and difficulty score of the calculated embedding features are specifically, Use the dataset X to continue to embed feature density ρ i , where d c is the sampling radius for density calculation, j represents the jth sample; According to the embedding feature density ρ i Calculating difficulty score δ i , as follows: Where j′ represents the j′th sample and λ1 represents the density sampling radius d c The corresponding ratio.

4. The image annotation method based on curriculum learning and deep learning according to claim 1, characterized in that: The calculation selects the proportion of samples, and iteratively calculates the descending order of the embedded features in the cluster and the indicator variable of each sample in the cluster according to the difficulty score to obtain the training set, specifically, Calculate the proportion of selected samples ζ iter , as follows: t=iter Where t represents the iteration round, iter represents the actual iteration round, 0 Indicates the initial proportion of selected samples, T grow represents the number of iterations required for the encoder to reach 100% of the value for the first time, ζ max Indicates the maximum proportion of samples selected by control; Iteratively calculate the descending order of the clusters of embedded features according to the difficulty score, and calculate the indicator variable of each sample in the cluster of embedded features As follows: t=iter Among them C k represents the kth cluster generated from the result of the (t-1)th iteration, Represents x i The difficulty score at iteration t, is an ordered sequence, It is cluster C k Corresponding to ζ t The value at the position; Get the training set As follows:

5. The image annotation method based on curriculum learning and deep learning according to claim 1, characterized in that: The cluster density kernel of the embedded feature is iteratively calculated according to the set density kernel selection ratio, and the cluster assignment probability is predicted to form a soft label distribution, cluster the embedded feature, and the iteration is terminated when the change of the clustering result is less than the threshold. Specifically, According to the set density kernel selection ratio λ2, the cluster density kernel of the embedded feature is calculated As follows: Where i represents the i-th sample, ρ i represents the local density of the i-th sample in the embedded feature space, Represents cluster C k The value of the λ2 position in the descending sequence of the density values ​​of all samples in , k represents the number of clusters of embedded features; Use Student's T distribution to predict cluster assignment probability q ik , as follows: Where j represents the jth sample; Form a soft label distribution and cluster the embedded features to obtain the clustering result Y, as shown in the following formula: Where n represents the number of embedded features; Iterate the above steps. When the change of two consecutive clustering results Y is less than the threshold μ, the iteration terminates.

6. The image annotation method based on curriculum learning and deep learning according to claim 5, characterized in that: Also includes an auxiliary target variable p ik To repeatedly improve the model's predictions, as follows:

7. The image annotation method based on curriculum learning and deep learning according to claim 6, characterized in that: The iterative calculation of the loss of the training set and back-propagation training of the autoencoder is as follows: Using the training set Calculate clustering loss Reconstruction loss and the overall loss L t , as follows: where t represents the number of iterations, KL(·) represents the divergence between the cluster assignment matrix and the auxiliary distribution matrix, P represents the auxiliary distribution matrix, Q represents the cluster assignment matrix, and f W (·) represents the encoder mapping function, g U (·) represents the decoder mapping function, α represents the weight of the reconstruction loss; Update training set And train the autoencoder by backpropagation.

8. Image annotation system based on curriculum learning and deep learning, characterized by: An image annotation method based on curriculum learning and deep learning, applied to any one of claims 1-7, comprising an image acquisition module, a model building module, a fine-tuning training module, and an image annotation module; The image acquisition module acquires and preprocesses unlabeled images to form a data set; The model building module constructs an autoencoder, uses the embedded features converted from the dataset into an embedding space and pre-trains the autoencoder; the autoencoder includes an image segmentation and encoding module, a self-attention encoding module, and a linear encoding module of the encoder, and a linear decoding module, a self-attention module, and a transposition layer of the decoder; the transposition layer restores the features to the shape of the original data; The fine-tuning training module uses the clustering layer to fine-tune the autoencoder to obtain a trained autoencoder and cluster embedding features; Specifically, the density and difficulty score of the embedded features are calculated; the embedded features are initially pre-partitioned using kmeans, the proportion of selected samples is calculated, and the descending order of the embedded features within the cluster and the indicator variable of each sample within the cluster are iteratively calculated based on the difficulty score to obtain the training set; the cluster density kernel of the embedded features is iteratively calculated based on the set density kernel selection ratio, and the cluster assignment probability is predicted to form a soft label distribution, and the embedded features are clustered. The iteration is terminated when the change in the clustering result is less than the threshold; the loss of the training set is iteratively calculated and the autoencoder is trained through backpropagation to obtain the trained autoencoder; The image annotation module uses the trained autoencoder to find the density kernel corresponding to each cluster based on the clustering results of the embedded features, and confirms the label category based on the density kernel to obtain the image annotation results.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the image annotation method based on curriculum learning and deep learning as described in any one of claims 1-7.

10. A computer-readable storage medium storing a program, characterized in that: When the program is executed by a processor, the image annotation method based on curriculum learning and deep learning according to any one of claims 1 to 7 is implemented.

Images