A model training method, device and storage medium for image classification

Abstract

The present application discloses a model training method, device and storage medium for image classification, which is used to solve the technical problem that the model obtained by the existing model training method cannot achieve a good image classification effect. The method includes: obtaining a visual feature vector of a sample image; extracting shallow semantic features and deep semantic features in the visual feature vector based on a preset algorithm, and integrating the shallow semantic features and the deep semantic features to obtain a joint semantic feature; performing semantic space alignment on the joint semantic feature to obtain a semantic alignment loss function; reconstructing the visual features, and determining the autoencoder loss function based on the reconstructed visual features; based on the semantic alignment loss function, the autoencoder loss function and the preset parameter regularization term, determining the objective function to train the neural network model. The present application achieves the improvement of the discriminability of the semantic embedding space and alleviates the domain bias problem of the zero-sample learning model through the above method.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a model training method, device and storage medium for image classification. Background Art

[0002] With the widespread use of deep learning model frameworks, supervised learning has achieved many outstanding results in the field of image recognition. Deep learning models can continuously improve the recognition accuracy of the model through sufficient training. However, existing supervised image recognition methods can only recognize categories that have appeared in the data set. In most practical application scenarios, it takes a lot of time to label a large amount of data. To solve this problem, researchers have proposed zero-shot learning to recognize categories that do not appear in the training set. Zero-shot learning aims to classify unseen categories through knowledge learned in visible categories.

[0003] At present, the most commonly used zero-shot learning classification method is based on the generative model. However, the existing generative models are difficult to converge in the optimization process, and the generated pseudo samples of unseen classes usually cannot achieve the ideal training effect. Summary of the invention

[0004] The embodiments of the present application provide a model training method, device and storage medium for image classification, which are used to solve the technical problem that the model trained by the existing model training method for image classification does not perform well when performing image classification.

[0005] On the one hand, an embodiment of the present application provides a model training method for image classification, the method comprising: obtaining a visual feature vector of a sample image; extracting shallow semantic features and deep semantic features in the visual feature vector based on a preset algorithm, and integrating the shallow semantic features and the deep semantic features to obtain a joint semantic feature; performing semantic space alignment on the joint semantic feature to obtain a semantic alignment loss function; reconstructing the visual feature vector based on the joint semantic feature, and determining an autoencoder loss function based on the reconstructed visual feature vector; determining an objective function of a neural network model based on the semantic alignment loss function, the autoencoder loss function, and a preset parameter regularization term; and training the neural network model based on the objective function.

[0006] In one implementation of the present application, the step of obtaining a visual feature vector of a sample image specifically includes: labeling the sample image; wherein the label is used to describe the category of the sample image; inputting the labeled sample image into a residual neural network to extract visual features in the sample image; and vectorizing the visual features to obtain a visual feature vector.

[0007] In one implementation of the present application, the extraction of shallow semantic features and deep semantic features in the visual feature vector based on a preset algorithm specifically includes: mapping the visual feature vector to a first semantic embedding space through a preset mapping matrix to obtain shallow semantic features; determining a projection matrix of the visual feature vector; and mapping the visual feature vector to a second semantic embedding space through the projection matrix to obtain deep semantic features.

[0008] In one implementation of the present application, the shallow semantic features and the deep semantic features are integrated to obtain a joint semantic feature, specifically including: multiplying the mapping matrix with the visual feature vector to obtain a shallow semantic product; multiplying the projection matrix with the visual feature vector to obtain a deep semantic product; and summing the shallow semantic product and the deep semantic product to obtain a joint semantic feature.

[0009] In one implementation of the present application, determining the projection matrix of the visual feature vector specifically includes: performing zero-mean processing on the visual feature vector to obtain a zero-mean visual feature vector; performing linear combination on the visual feature vector to obtain a combined visual feature matrix; calculating the covariance matrix of the visual feature vector; determining the eigenvalue and the first eigenvector based on the combined visual feature matrix, the covariance matrix and the zero-mean visual feature vector; arranging the first eigenvector in order from large to small according to the size of the eigenvalue to obtain a second eigenvector; and determining the projection matrix based on the second eigenvector.

[0010] In one implementation of the present application, the visual feature vector is reconstructed based on the joint semantic feature, and the autoencoder loss function is determined according to the reconstructed visual feature vector, specifically including: multiplying the joint semantic feature and a preset reconstruction matrix to obtain the reconstructed visual feature; based on the reconstructed visual feature vector, calculating the autoencoder loss function.

[0011] In one implementation of the present application, the training of the neural network model based on the objective function specifically includes: adjusting the parameter value of the preset parameter regularization term in the objective function; wherein the formula of the objective function is: in, is the autoencoder loss function, is the semantic alignment loss function, Ω(P, V, W) is the parameter regularization term, X is the visual feature vector, B is the reconstructed visual feature, A is the attribute description, S is the joint semantic feature, P is the projection matrix, V is the reconstruction matrix, and W is the mapping matrix; the optimal solution of the mapping matrix and the reconstruction matrix in the objective function is solved based on the preset matrix equation algorithm.

[0012] In one implementation of the present application, after calculating the optimal solution of the mapping matrix and the reconstruction matrix in the objective function, the method further includes: inputting a test sample into a trained neural network model and outputting a prediction result of the test sample; calculating the accuracy of the neural network model based on the prediction result and the actual result; and repeating the test until the accuracy of the neural network model reaches a preset value.

[0013] On the other hand, the present application also provides a model training device for image classification, the device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can: obtain a visual feature vector of a sample image; extract shallow semantic features and deep semantic features in the visual feature vector based on a preset algorithm, and integrate the shallow semantic features and the deep semantic features to obtain a joint semantic feature; perform semantic spatial alignment on the joint semantic feature to obtain a semantic alignment loss function; reconstruct the visual feature vector based on the joint semantic feature, and determine the autoencoder loss function based on the reconstructed visual feature vector; determine the objective function of the neural network model based on the semantic alignment loss function, the autoencoder loss function and a preset parameter regularization term; and train the neural network model based on the objective function.

[0014] The present application also provides a non-volatile computer storage medium for model training of image classification, storing computer executable instructions, wherein the computer executable instructions are configured to: obtain a visual feature vector of a sample image; extract shallow semantic features and deep semantic features in the visual feature vector based on a preset algorithm, and integrate the shallow semantic features and the deep semantic features to obtain a joint semantic feature; perform semantic spatial alignment on the joint semantic feature to obtain a semantic alignment loss function; reconstruct the visual feature vector based on the joint semantic feature, and determine an autoencoder loss function based on the reconstructed visual feature vector; determine an objective function of a neural network model based on the semantic alignment loss function, the autoencoder loss function, and a preset parameter regularization term; and train the neural network model based on the objective function.

[0015] The embodiment of the present application provides a model training method, device and storage medium for image classification, which obtains a visual feature vector by preprocessing an existing sample image, then extracts shallow semantic features from the visual feature vector through a mapping matrix, extracts deep semantic features through a principal component analysis algorithm to solve a projection matrix, recombines shallow semantic features and deep semantic features to obtain joint semantic features, and aligns the semantic space to obtain a semantic alignment loss function. The visual features are reconstructed using joint semantic features, and finally the loss function of the autoencoder is obtained. The model training is completed by solving the optimal solution of the mapping matrix and the reconstruction matrix through preset parameter regularization terms. The present application improves the discriminability of the semantic embedding space by mining the potential joint semantic representation in the visual features, alleviating the domain bias problem of the zero-shot learning model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 A flow chart of a model training method for image classification provided in an embodiment of the present application;

[0018] Figure 2 A brief flowchart of a model training method for image classification provided in an embodiment of the present application;

[0019] Figure 3 A diagram of the model structure for image classification provided in an embodiment of the present application;

[0020] Figure 4 A schematic diagram of a model training device for image classification provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0022] With the widespread use of deep learning model frameworks, supervised learning has achieved many outstanding results in the field of image recognition. Thanks to the increasing number of labeled data sets available for training, deep learning models can continuously improve the recognition accuracy of the model through sufficient training. However, existing supervised image recognition methods can only recognize categories that have appeared in the data set. In most practical application scenarios, it takes a lot of time and manpower to label a large amount of data. To solve this problem, researchers proposed zero-shot learning to recognize categories that did not appear in the training set. Zero-shot learning aims to classify unseen classes based on the knowledge learned in the visible classes.

[0023] There are three types of zero-shot learning classification methods, namely, attribute transfer-based methods, mapping-based methods, and generative model-based methods. The early zero-shot learning methods, direct attribute prediction and indirect model prediction, all establish a strict correspondence between attributes and class labels, and then use the Bayesian formula to calculate the posterior probability product to predict the label of the unseen class. However, the correspondence between learning attributes and labels usually leads to unsatisfactory recognition of unseen classes due to overfitting problems. Therefore, in order to improve the recognition accuracy of unseen classes, mapping-based methods aim to further learn the relationship between visual space and semantic space, and use semantic information as an intermediate bridge to classify unseen class samples. In order to solve the problem of overfitting of the model in visible class training, researchers proposed a zero-shot learning method based on generative models, which uses the semantic description of unseen classes to generate pseudo-sample visual features to alleviate the sample imbalance problem between visible and unseen classes.

[0024] The above-mentioned generative model usually generates unseen pseudo samples that cannot achieve the ideal training effect because the objective function is difficult to converge during the optimization process. An additional reconstruction error is added on the basis of the mapping model to alleviate the domain bias problem. However, in the process of reconstructing visual features, there is often a huge gap in the manifold structure between the visual space and the semantic space. Secondly, artificially designed semantic information (such as attributes) can only focus on the shallow semantic information (such as color, shape, etc.) corresponding to the visual features of the intuitive category, and ignore the potential deep semantic information in the actual model learning process. According to the artificially designed attribute description rules, we can only focus on the relatively intuitive semantics of the samples visually. However, in data of different categories, the discriminative information that can be provided by these shallow semantic information is often not enough to distinguish samples of different categories, and the model still has the problem of domain bias.

[0025] The embodiment of the present application provides a model training method for image classification, which is used to solve the technical problem that the model trained by the existing model training method cannot achieve good results in image classification.

[0026] The technical solution proposed in the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0027] Figure 1 Provided in the embodiments of this application. Figure 1 As shown, Figure 1 A flow chart of a model training method for image classification provided in an embodiment of the present application, the method mainly includes the following steps.

[0028] Step 101: Obtain visual features of sample images.

[0029] In the embodiment of the present application, a brief flowchart of the model training method for image classification is as follows: Figure 2 As shown in the figure, there are four steps: image preprocessing, learning joint semantic representation, reconstructing visual features, and model training. In the image preprocessing process, a certain number of image samples are provided in the training data set. First, these images are labeled, and these labels contain the categories of the sample images. At the same time, some attribute descriptions are pre-added to these sample images to describe the sample images.

[0030] Furthermore, the images in the dataset are input into a 101-layer residual neural network (ResNet101) to extract the visual features of the images, and then the visual features are vectorized to obtain a 2048-dimensional visual feature vector.

[0031] Step 102: extract shallow semantic features and deep semantic features from the visual feature vector, and integrate the shallow semantic features and the deep semantic features to obtain joint semantic features.

[0032] In the embodiment of the present application, the proposed model is divided into two parts: learning joint semantic representation and reconstructing visual features. Since shallow semantic information usually leads to unsatisfactory learning effect when classifying unseen classes due to insufficient inter-class distinction, deep semantic information of visual features is extracted based on the original semantic representation, and the shallow semantic representation and the deep semantic representation are recombined to obtain a joint semantic representation corresponding to the sample to enhance the discrimination of the semantic representation learned in the model.

[0033] In an embodiment of the present application, during the encoding process, the visual feature X of the visible class sample is mapped to the first semantic embedding space through a preset mapping matrix W to obtain a shallow semantic representation corresponding to the sample. In order to extract deep semantic information, the principal component analysis method is used to extract potential deep attributes in the visual features. The extracted deep attributes are mapped to the second semantic embedding space to obtain the deep semantic features of the sample image. The shallow semantic features and deep semantic features obtained in the first semantic space and the second semantic space are recombined to finally obtain a joint semantic representation.

[0034] Specifically, according to the PCA dimensionality reduction algorithm, the visual feature vector needs to be averaged, and each row of data in the visual feature vector X is zero-meaned (a row of data represents a sample visual feature), that is, the visual feature vector X of each sample is subtracted from the mean of its own vector. The calculation formula is as follows:

[0035]

[0036] Furthermore, the original visual features X are linearly combined to obtain the combined visual feature matrix X g , X g =RFX', F is the unit matrix, and X is calculated g The covariance matrix C of g The first eigenvector of g Then, the first eigenvector R is arranged from large to small according to the size of the eigenvalue to obtain the second eigenvector R', and the projection matrix P is obtained by taking the first k rows of the second eigenvector R'.

[0037] Furthermore, after the projection matrix P is obtained, the visual feature vector X is projected into the second semantic embedding space using the projection matrix P through the principal component analysis algorithm to extract the deep semantic features. The deep semantic features are linearly combined with the shallow semantic features to obtain the joint semantic representation S.

[0038] Specifically, the mapping matrix W is multiplied by the visual feature vector X to obtain the shallow semantic product WX. Then the projection matrix P is multiplied by the visual feature vector X to obtain the deep semantic product PX. Then the shallow semantic product WX and the deep semantic product PX are summed to obtain the joint semantic feature S. The calculation process formula is as follows:

[0039] S=PX+WX,

[0040] Where P is the projection matrix (P∈R k*d ), W is the mapping matrix, and X is the visual feature vector.

[0041] Step 103: perform semantic space alignment on the joint semantic features to obtain a semantic alignment loss function.

[0042] In the embodiment of the present application, in order to keep the manifold after the shallow semantic representation and the deep semantic representation reorganized consistent with the original semantic space manifold, we perform semantic space alignment on the joint semantic representation, and the semantic alignment loss function is:

[0043]

[0044] Where A is the attribute description, S is the joint semantic feature, is the Frobenius norm, which is used to calculate the Euclidean distance between the attribute description and the joint semantic representation, ensuring the consistency between the reorganized joint semantic representation manifold and the original semantic manifold.

[0045] Furthermore, there will be a dimensionality gap between the visual feature vector and the attribute description vector due to structural differences. In order to alleviate the variance attenuation caused by the dimensionality gap, the joint semantic representation is constrained to maintain the invariance of the visual space manifold. The semantic alignment loss function after the constraint is:

[0046]

[0047] Step 104: reconstruct the visual feature vector based on the joint semantic feature, and determine the autoencoder loss function according to the reconstructed visual feature vector.

[0048] In an embodiment of the present application, the autoencoder learns the input representation of the sample by minimizing the original visual features and reconstructing the visual features. The visible class sample is used as the input of the encoder. During the encoding process, the visual feature X is projected into the semantic embedding space to obtain the shallow semantic attributes corresponding to the visual feature in the semantic embedding space. In the decoder part, the visual feature X is reconstructed using the joint semantic representation S obtained by combining the shallow semantic attributes and deep visual attributes obtained by the encoder, and the association between the visual feature and the joint semantics is learned through the reconstruction error. The joint semantic representation S is multiplied by the preset reconstruction matrix V to obtain the reconstructed visual feature B. The calculation formula of the reconstructed visual feature B is:

[0049] B=VS,

[0050] Where V is the reconstruction matrix, V∈R k*d , S is the joint semantic feature.

[0051] Therefore, the initial loss function of the autoencoder is:

[0052]

[0053] Furthermore, due to the structural difference between the manifolds of the visual feature space and the semantic space, it is necessary to constrain the reconstructed visual features in the decoder to ensure that the visual feature B reconstructed by the joint semantic feature S has a similar Euclidean norm to the original visual feature X, thereby preventing the domain bias problem caused by the difference in manifold structure between the visual feature space and the semantic space when recognizing unseen class samples. The final loss function of the autoencoder is:

[0054]

[0055] in is the Frobenius norm, which is used to calculate the Euclidean distance between the attribute description and the joint semantic representation, ensuring the consistency between the reorganized joint semantic representation manifold and the original semantic manifold.

[0056] Step 105: Determine an objective function based on the semantic alignment loss function, the autoencoder loss function, and a preset parameter regularization term.

[0057] In the embodiment of the present application, the objective function of the general zero-shot learning framework is divided into three parts: the autoencoder loss Shallow semantic and deep semantic alignment loss And the preset parameter regularization term Ω. The purpose of adding the parameter regularization term Ω is to prevent the model from overfitting during the training process. By adding the L2 normal form constraint to penalize the standard weight decay of the projection matrix and reconstruction matrix to be learned, the generalization of the model is improved. The expression of the parameter regularization term Ω is:

[0058] Ω(P, V, W) = ‖P‖ 2 +‖V‖ 2 +‖W‖ 2 ,

[0059] The objective function is calculated as follows:

[0060]

[0061] Step 106: Based on the objective function, train the neural network model.

[0062] In the embodiment of the present application, the mapping matrix W and the reconstruction matrix V can be solved according to the objective function, and the specific calculation process is as follows:

[0063]

[0064] According to the norm definition

[0065] A T A-2A T WX+2(α+1)X T WPX+2(α+1)X T W T WX+λW T W

[0066] The partial derivative of the objective function with respect to W is:

[0067]

[0068] Setting the derivative to 0 gives the equation:

[0069] 2λW+(α+1)WXX T =2AX T -2(α+1)XX T P

[0070] It is known that the Sylvester equation is

[0071] MX+XK=Q

[0072] Where M = 2λ, K = (α + 1) XX T , Q = 2AX T -2(α+1)XX T P. The Bartels–Stewart algorithm is used to solve the equation MX+XK=Q and obtain the optimal parameter W.

[0073] Furthermore, the process of solving the reconstruction matrix V is as follows:

[0074]

[0075] First, we take the partial derivative of the objective function and set the derivative to 0 to obtain the Sylvester equation:

[0076] λV+(β+1)VSS T =XS T

[0077] We can solve the Sylvester equation λV+(β+1)VSS T =XS T , and obtain the optimal solution of the reconstruction matrix V. The solution algorithm process is as follows.

[0078]

[0079]

[0080] In the embodiment of the present application, the process of solving the optimal solution of the mapping matrix W and the reconstruction matrix V based on the objective function is actually a process of model training. After solving the optimal solution, it is necessary to test the degree of model training, that is, to detect how well the model recognizes samples of unseen classes. First, the sample image used for testing is input into the model, and the test sample, that is, the predicted result of the unseen class sample, is output. Then, the model accuracy is calculated based on the actual results and the predicted results. As the test process continues to accumulate, the accuracy continues to rise until it reaches the preset accuracy value, proving that the model has completed training. In this process, the model learns and analyzes the input unseen class samples, and assists in learning the data with reconstructed visual features as samples, and finally obtains a label for predicting image classification. The generation of this label indicates that the model can be used to identify unseen class sample data.

[0081] In the embodiment of the present application, the model structure used for image classification is as follows: Figure 3 As shown. First, the training sample X tr The feature extraction is then input into the encoder and PCA algorithm to extract shallow and deep semantic features, and finally the joint semantic representation is output, which is then reconstructed in the decoder to obtain the reconstructed training sample X re , the test sample X te Input the model, test the accuracy of the model, and reconstruct the training sample X re Compare and get the predicted label y for classifying the unseen class pro .

[0082] In the embodiments of the present application, the performance of the model was tested on three benchmark data sets (AWA1[6], AWA2[5], SUN[7]). In the three commonly used zero-shot learning benchmark data sets, we referred to the PS visible class / unseen class division method, so the PS V2.0 division scheme was used in all three benchmark data sets. At the same time, the 2048-dimensional visual features used in the model testing process were extracted from the ResNet-101 pre-trained model and tested under the setting, and the experimental results achieved the best performance. The detailed information of the data set is shown in Table 1 below.

[0083]

[0084] Table 1 Detailed information on samples and seen / unseen class divisions of 5 benchmark datasets

[0085] In order to evaluate the model as fairly as possible and compare it with the existing zero-shot learning model, we uniformly use the dataset used in PS V2.0. The benchmark dataset test results under the ZSL setting are shown in Table 2 below.

[0086]

[0087] Table 2 Top-1 recognition accuracy of unseen classes on benchmark datasets AWA1, AWA2, and SUN under zero-shot learning setting

[0088] In traditional zero-shot learning, the model proposed in the embodiment of the present application achieved the best results on the AWA1, AWA2 and SUN datasets.

[0089] Due to the dimensionality gap between visual features and semantic information, direct mapping will cause the feature space to collapse, and when visual features are mapped to the semantic space, they will fall into the pivot point problem. Our model uses the principal component analysis algorithm to average the visual features, alleviating the pivot point problem after mapping.

[0090] Most existing methods use attribute descriptions as semantic auxiliary information of categories, but ignore the limitations of artificially designed attribute descriptions and cannot capture potential deep semantic information well. Therefore, we extract deep semantic features from visual features in the visual-semantic mapping process, and map shallow semantic features and deep semantic features to the joint semantic embedding space to obtain the joint semantic representation of the sample. Not only does it mine the potential deep semantics corresponding to the visual features of the sample, but it also removes redundant features, effectively retaining the visual features.

[0091] In the model proposed in the embodiment of the present application, the joint semantic representation is constrained in the process of synthesizing the joint semantic representation to preserve the invariance of the visual space manifold. At the same time, L2-norm constraints are added to the projection matrix P and reconstruction matrix V that need to be learned in the encoding and decoding process, thereby enhancing the generalization of the model.

[0092] The above is an embodiment of a model training method for image classification provided in an embodiment of the present application. Based on the same inventive concept, the present application also provides a model training device for image classification, such as Figure 4 As shown, the device includes: at least one processor 401; and, a memory 402 communicatively connected to the at least one processor 401; wherein the memory 402 stores instructions executable by the at least one processor 401, and the instructions are executed by the at least one processor 401 so that the at least one processor 401 can: obtain a visual feature vector of a sample image; extract shallow semantic features and deep semantic features in the visual feature vector based on a preset algorithm, and integrate the shallow semantic features and the deep semantic features to obtain a joint semantic feature; perform semantic spatial alignment on the joint semantic feature to obtain a semantic alignment loss function; reconstruct the visual feature based on the joint semantic feature, and determine the autoencoder loss function based on the reconstructed visual feature; determine the objective function based on the semantic alignment loss function, the autoencoder loss function and a preset parameter regularization term; and train a neural network model based on the objective function.

[0093] In addition, an embodiment of the present application provides a non-volatile computer storage medium for model training for image classification, storing computer executable instructions, wherein the computer executable instructions are configured to: obtain a visual feature vector of a sample image; extract shallow semantic features and deep semantic features in the visual feature vector based on a preset algorithm, and integrate the shallow semantic features and the deep semantic features to obtain a joint semantic feature; perform semantic spatial alignment on the joint semantic feature to obtain a semantic alignment loss function; reconstruct the visual features based on the joint semantic feature, and determine an autoencoder loss function based on the reconstructed visual features; determine an objective function based on the semantic alignment loss function, the autoencoder loss function, and a preset parameter regularization term; and train a neural network model based on the objective function.

[0094] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0095] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0096] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0097] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0098] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0099] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0100] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0101] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0102] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A model training method for image classification, characterized in that: The method comprises: Get the visual feature vector of the sample image; Extracting shallow semantic features and deep semantic features in the visual feature vector based on a preset algorithm, and integrating the shallow semantic features and the deep semantic features to obtain a joint semantic feature; Performing semantic space alignment on the joint semantic features to obtain a semantic alignment loss function; Reconstructing the visual feature vector based on the joint semantic feature, and determining an autoencoder loss function according to the reconstructed visual feature vector; Determining the objective function of the neural network model based on the semantic alignment loss function, the autoencoder loss function and the preset parameter regularization term; Based on the objective function, the neural network model is trained.

2. The model training method for image classification according to claim 1, characterized in that: The obtaining of the visual feature vector of the sample image specifically includes: Performing labeling processing on the sample images; wherein the labels are used to describe the categories of the sample images; Inputting the labeled sample images into a residual neural network to extract visual features from the sample images; Vectorization is performed on the visual feature to obtain a visual feature vector.

3. The model training method for image classification according to claim 1, characterized in that: The extracting of shallow semantic features and deep semantic features in the visual feature vector based on a preset algorithm specifically includes: Mapping the visual feature vector into a first semantic embedding space through a preset mapping matrix to obtain the shallow semantic feature; Determining a projection matrix of the visual feature vector; The visual feature vector is mapped into a second semantic embedding space through the projection matrix to obtain the deep semantic feature.

4. A model training method for image classification according to claim 3, characterized in that: The shallow semantic features and the deep semantic features are integrated to obtain the joint semantic features, specifically including: Multiplying the mapping matrix with the visual feature vector to obtain a shallow semantic product; wherein the shallow semantic product is a value of the shallow semantic feature; Multiplying the projection matrix with the visual feature vector to obtain a deep semantic product; wherein the deep semantic product is a value of the deep semantic feature; The shallow semantic product and the deep semantic product are summed to obtain the joint semantic feature.

5. The model training method for image classification according to claim 3, characterized in that: The determining of the projection matrix of the visual feature vector specifically includes: Performing zero-mean processing on the visual feature vector to obtain a zero-mean visual feature vector; Performing linear combination on the visual feature vectors to obtain a combined visual feature matrix; Calculating the covariance matrix of the visual feature vector; Determining an eigenvalue and a first eigenvector of the combined visual feature matrix based on the combined visual feature matrix, the covariance matrix, and the zero-mean visual feature vector; Arrange the first eigenvectors in order from large to small according to the eigenvalues ​​to obtain a second eigenvector; Based on the second eigenvector, the projection matrix is ​​determined.

6. A model training method for image classification according to claim 1, characterized in that: The reconstructing the visual feature vector based on the joint semantic feature and determining the autoencoder loss function according to the reconstructed visual feature vector specifically includes: Multiplying the joint semantic feature and a preset reconstruction matrix to obtain a reconstructed visual feature vector; Based on the reconstructed visual feature vector, the autoencoder loss function is calculated.

7. The model training method for image classification according to claim 1, characterized in that: The training of the neural network model based on the objective function specifically includes: Adjust the parameter value of the preset parameter regularization term in the objective function; wherein the objective function is: in, is the autoencoder loss function, is the semantic alignment loss function, Ω(P, V, W) is the parameter regularization term, X is the visual feature vector, B is the reconstructed visual feature, A is the attribute description, S is the joint semantic feature, P is the projection matrix, V is the reconstruction matrix, and W is the mapping matrix; The optimal solution of the mapping matrix and the reconstruction matrix in the objective function is solved based on a preset matrix equation algorithm.

8. A model training method for image classification according to claim 7, characterized in that: The optimal solution of the mapping matrix and the reconstruction matrix in the objective function is solved based on the preset matrix equation algorithm: Inputting the test sample into the trained neural network model, and outputting the prediction result of the test sample; Based on the predicted results and the actual results, calculating the accuracy of the neural network model; The test is repeated until the accuracy of the neural network model reaches a preset value.

9. A model training device for image classification, characterized in that: The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to: Get the visual feature vector of the sample image; Extracting shallow semantic features and deep semantic features in the visual feature vector based on a preset algorithm, and integrating the shallow semantic features and the deep semantic features to obtain a joint semantic feature; Performing semantic space alignment on the joint semantic features to obtain a semantic alignment loss function; Reconstructing the visual feature vector based on the joint semantic feature, and determining an autoencoder loss function according to the reconstructed visual feature vector; Determining the objective function of the neural network model based on the semantic alignment loss function, the autoencoder loss function and the preset parameter regularization term; Based on the objective function, the neural network model is trained.

10. A non-volatile computer storage medium for model training for image classification, storing computer executable instructions, characterized in that: The computer executable instructions are configured to: Get the visual feature vector of the sample image; Extracting shallow semantic features and deep semantic features in the visual feature vector based on a preset algorithm, and integrating the shallow semantic features and the deep semantic features to obtain a joint semantic feature; Performing semantic space alignment on the joint semantic features to obtain a semantic alignment loss function; Reconstructing the visual feature vector based on the joint semantic feature, and determining an autoencoder loss function according to the reconstructed visual feature vector; Determining the objective function of the neural network model based on the semantic alignment loss function, the autoencoder loss function and the preset parameter regularization term; Based on the objective function, the neural network model is trained.

Images