Multi-mark image classification method and device based on saliency features and readable medium

Abstract

The invention discloses a multi-mark image classification method and device based on saliency features and a readable medium, and the method comprises the steps: constructing a regression frame based on a feature selection matrix, a feature space matrix and a pseudo-mark matrix, constructing a fitting model in combination with a traction matrix, a non-negative mark sparse matrix, a mark space matrix and the pseudo-mark matrix, constructing a mark correlation model through the pseudo mark matrix; constructing a nuclear norm model of the feature selection matrix to construct an objective function, and solving the objective function to obtain a final optimized feature selection matrix; determining an image feature subset based on the finally optimized feature selection matrix, and training an image classification model based on the image feature subset to obtain a trained image classification model; and the features corresponding to the image feature subsets in the image are input into the trained image classification model to obtain an image classification result, so that the correlation between category marks is learned, and the influence of low-density marks on feature selection is reduced.

Description

Technical Field

[0001] The present invention relates to the field of image classification, and in particular to a multi-label image classification method, device and readable medium based on significant features. Background Art

[0002] With the development of information acquisition technology, a large proportion of application data faces a common problem, namely the rapid growth of labeled and high-dimensional data. High-dimensional multi-label data may contain a lot of noise and redundancy, resulting in low computational efficiency and performance degradation. Existing multi-label feature selection methods can be roughly divided into three categories: filtering methods, packaging methods, and embedded methods. The filtering method does not rely on any learning algorithm in the process of generating feature subsets, so it is impossible to select more ideal features. The packaging method uses an evolutionary algorithm to search for feature subsets, but the number of iterations of this method is usually tens of thousands, and the computational time overhead is expensive. The embedded method also combines the learning model to generate a feature selection matrix, which directly determines the feature importance ranking.

[0003] There are two problems with traditional embedded methods: First, traditional embedded methods fit the original label space matrix by constructing a pseudo label matrix, and then use the pseudo label matrix to mine the correlation between class labels, but the obtained label correlation coefficient has a certain deviation. In order to approximate the logical value in the original label space matrix, the pseudo label matrix will ignore the correlation between the feature and the class label. Second, the data distribution of each label in multi-label data is unbalanced. For low-density labels, the weight of the corresponding feature will be affected in the feature selection process. Existing graph-based embedding methods are widely studied for their simplicity and efficiency to deal with the problem of unbalanced label density, but in high-dimensional data, noise information will lead to inaccurate construction of the graph matrix. Summary of the invention

[0004] The purpose of this application is to propose a multi-label image classification method, device and readable medium based on salient features to address the above-mentioned technical problems.

[0005] In a first aspect, the present invention provides a multi-label image classification method based on salient features, comprising the following steps:

[0006] S1, obtain several images with known labels and construct a feature space matrix and a label space matrix, randomly generate an initial feature selection matrix from the feature space matrix, randomly initialize and generate a pseudo label matrix with the same spatial size as the label space matrix, and construct a regression framework based on the feature selection matrix, the feature space matrix and the pseudo label matrix;

[0007] S2, construct a traction matrix for guiding the sparse direction, design a non-negative labeled sparse matrix to learn the sparsity, combine the traction matrix, non-negative labeled sparse matrix, label space matrix and pseudo-label matrix to build a fitting model; construct a label correlation model through the pseudo-label matrix; construct a nuclear norm model of the feature selection matrix; construct an objective function according to the regression framework, fitting model, label correlation model and nuclear norm model of the feature selection matrix, solve the objective function through an alternating iterative optimization algorithm, and judge whether the objective function converges. If so, the feature selection matrix after the last round of optimization is used as the final optimized feature selection matrix, otherwise repeat step S2, determine the image feature subset based on the final optimized feature selection matrix, and construct training data based on the image feature subset and the label space matrix;

[0008] S3, constructing an image classification model, and using the training data to train the image classification model to obtain a trained image classification model;

[0009] S4, acquiring an image, and inputting features in the image corresponding to the image feature subset into a trained image classification model to obtain an image classification result.

[0010] Preferably, the feature space matrix is ​​expressed as in is the vector of elements in the i-th row of the feature space matrix, x i Each element in is the feature of the i-th image, including the color mean feature, color standard deviation feature, shape feature, 4 Gaussian filters and texture features extracted by 12 directional filters corresponding to each of the 8 regions of the image. represents the real number space, d represents the d-dimensional feature, n represents the total number of images, and the label space matrix contains the labels corresponding to n images, which is expressed as The i-th image x i ∈X(1≤i≤n) is associated with a row vector in the label space matrix Y Represents the element in the i-th row and j-th column of the tag space matrix, which is marked as the category of each object in the image. If the i-th image is associated with the j-th tag, the corresponding element in the tag space matrix If the i-th image is not associated with the j-th tag, then the element in the i-th row and j-th column of the tag space matrix is

[0011] As a preferred method, a traction matrix for guiding the sparse direction is constructed, a non-negative labeled sparse matrix is ​​designed to learn the sparsity, a fitting model is constructed by combining the traction matrix, the non-negative labeled sparse matrix, the label space matrix and the pseudo-label matrix; a label correlation model is constructed through the pseudo-label matrix; a nuclear norm model of the feature selection matrix is ​​constructed; and an objective function is constructed according to the regression framework, the fitting model, the label correlation model and the nuclear norm model of the feature selection matrix, specifically including:

[0012] The regression framework is expressed as:

[0013]

[0014] Where α is ||W|| 2,1 The trade-off parameter, W represents the feature selection matrix, V represents the pseudo-label matrix, X represents the feature space matrix, ||·|| F represents the Frobenius norm, ||·|| 2,1 Indicates l 2,1 norm;

[0015] Combine the traction matrix B, the non-negative marker sparse matrix S and the marker space matrix Y to fit the pseudo marker matrix V and construct a fitting model, which is expressed as:

[0016]

[0017] in, represents the matrix Hadamard product operator, Defined as: The element B in the i-th row and j-th column of each traction matrix B ij Represents a relaxation direction, i.e., “+1” represents a positive sparse direction and “-1” represents a negative sparse direction;

[0018] Based on the second-order strategy, the exploration of tag correlation is introduced and a tag correlation model is constructed, which is expressed as:

[0019]

[0020] Among them, ||·||2 represents the l2 norm, V :p is the pseudo-labeling result of the pth label, V :q is the pseudo-labeling result of the qth label, L = AG, A is a diagonal matrix, represented by diag(a1, a2, ..., a c ), a j represents the element in row j and column j in A, g pq is the element in the p-th row and q-th column of G, p,q=1,2,…,c; G represents the affinity matrix, and the affinity matrix is ​​initially generated by random initialization;

[0021] The nuclear norm model of the feature selection matrix is ​​expressed as:

[0022]

[0023] Among them, ||·|| * represents kernel function, Tr represents trace;

[0024] The objective function is expressed as:

[0025]

[0026] where β, γ, and λ represent the trade-off parameters of the fitting model, the marker correlation model, and the nuclear norm model representation of the feature selection matrix, respectively.

[0027] Preferably, the objective function is solved by an alternating iterative optimization algorithm, specifically including:

[0028] Set W = [w 1 ,w 2 ,...,w d ] T , h=1,2,…,d, the objective function is equivalent to:

[0029]

[0030] in, D is a diagonal matrix, D hh represents the element in the hth row and hth column of D, w h Represents the row vector consisting of the elements of the hth row in the feature selection matrix W after the previous round of optimization;

[0031] Fix V and S, and solve W after optimization in the current round. Formula (6) is simplified to:

[0032]

[0033] Where Δ represents the simplified objective function after fixing V and S;

[0034] make According to the singular value decomposition W = U∑P T , W=U∑P T Using the feature selection matrix W after the previous round of optimization, we can get:

[0035]

[0036] Among them, U and P represent the first variable and the second variable obtained by solving the feature selection matrix W after the current round of optimization, W = (X T X+αD) -1 (XT V-λUP T ) is the optimized W of the current round;

[0037] Fix W and S, and solve V after optimization in the current round. Formula (6) is simplified to:

[0038]

[0039] Where Θ represents the simplified objective function after fixing W and S;

[0040] make We can get:

[0041]

[0042] Where I is the identity matrix;

[0043] Fix W and V, set H = VY, Y = VH, Solving for S after optimization in the current round, formula (6) is simplified to:

[0044]

[0045] The optimal solution of S can be calculated as:

[0046]

[0047] As a preference, the l2 norm of the row vector consisting of the elements of each row in the final optimized feature selection matrix is ​​calculated || w h ||2, and follow ||w h ||2(h=1→d) in descending order, and select the row vector composed of the elements of each row in the final optimized feature selection matrix ranked in the top Q% as the image feature subset.

[0048] Preferably, the image classification model comprises an MLKNN model.

[0049] In a second aspect, the present invention provides a multi-label image classification method based on salient features, comprising the following steps:

[0050] A regression framework construction module is configured to obtain a number of images with known labels and construct a feature space matrix and a label space matrix, randomly generate an initial feature selection matrix from the feature space matrix, randomly initialize and generate a pseudo label matrix with the same spatial size as the label space matrix, and construct a regression framework based on the feature selection matrix, the feature space matrix and the pseudo label matrix;

[0051] The training data construction module is configured to construct a traction matrix for guiding the sparsity direction, design a non-negative labeled sparse matrix to learn sparsity, and construct a fitting model by combining the traction matrix, the non-negative labeled sparse matrix, the label space matrix and the pseudo-label matrix; construct a label correlation model through the pseudo-label matrix; construct a nuclear norm model of the feature selection matrix; construct an objective function according to the regression framework, the fitting model, the label correlation model and the nuclear norm model of the feature selection matrix, solve the objective function through an alternating iterative optimization algorithm, and judge whether the objective function converges. If so, the feature selection matrix after the last round of optimization is used as the final optimized feature selection matrix, otherwise the training data construction module is repeatedly executed, and the image feature subset is determined based on the final optimized feature selection matrix, and the training data is constructed based on the image feature subset and the label space matrix;

[0052] A model building module is configured to build an image classification model, and train the image classification model using training data to obtain a trained image classification model;

[0053] The classification module is configured to acquire an image, input features in the image corresponding to the image feature subset into a trained image classification model, and obtain an image classification result.

[0054] In a third aspect, the present invention provides an electronic device comprising one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0055] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0056] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] (1) The multi-label image classification method based on salient features proposed in the present invention utilizes a label sparsity mechanism to obtain a sparse pseudo-label matrix for learning reliable correlations between category labels.

[0059] (2) The multi-label image classification method based on significant features proposed in the present invention utilizes the nuclear norm model of the feature selection matrix to mine the shared information of multi-label learning, which can deal with the problem of label density imbalance.

[0060] (3) The multi-label image classification method based on significant features proposed in the present invention combines the label space matrix with the non-negative label sparse matrix, learns the correlation between category labels, and uses the feature selection matrix to capture the shared information of different related labels, thereby reducing the impact of low-density labels on feature selection, thereby improving the classification performance of multi-label image data. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0062] Figure 1 A flowchart of a multi-label image classification method based on significant features according to an embodiment of the present application;

[0063] Figure 2 A schematic diagram of a feature selection process of a multi-label image classification method based on significant features according to an embodiment of the present application;

[0064] Figure 3 A schematic diagram of a multi-label image classification device based on salient features according to an embodiment of the present application;

[0065] Figure 4 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] Figure 1 A multi-label image classification method based on salient features provided in an embodiment of the present application is shown, comprising the following steps:

[0068] S1, construct several images with known labels and construct feature space matrix and label space matrix, randomly generate an initial feature selection matrix from the feature space matrix, randomly initialize and generate a pseudo label matrix with the same spatial size as the label space matrix, and construct a regression framework based on the feature selection matrix, feature space matrix and pseudo label matrix.

[0069] In a specific embodiment, the feature space matrix is ​​expressed as in is the vector of elements in the i-th row of the feature space matrix, x i Each element in is the feature of the i-th image, including the color mean feature, color standard deviation feature, shape feature, 4 Gaussian filters and texture features extracted by 12 directional filters corresponding to each of the 8 regions of the image. represents the real number space, d represents the d-dimensional feature, n represents the total number of images, and the label space matrix contains the labels corresponding to n images, which is expressed as The i-th image x i ∈X(1≤i≤n) is associated with a row vector in the label space matrix Y Represents the element in the i-th row and j-th column of the tag space matrix, which is marked as the category of each object in the image. If the i-th image is associated with the j-th tag, the corresponding element in the tag space matrix If the i-th image is not associated with the j-th tag, then the element in the i-th row and j-th column of the tag space matrix is

[0070] Specifically, assuming is a feature space matrix with d-dimensional features,

[0071] is a label space matrix with c category labels. Each image is divided into 8 regions, and each region calculates the color mean feature, shape feature, color standard deviation feature, 4 Gaussian filters and 12 directional filters to extract texture features. Each element in the label space matrix is ​​a label for the object category in the image, including any object that may exist in natural images, such as the sun, sky, sea, waves, buildings, people, cats, dogs, etc. i ∈X(1≤i≤n) is associated with a set of tags in Indicates that the jth tag is related to the i-th image. If not, it is represented as

[0072] For any matrix The trace of Z is denoted as Tr(Z), and the transpose of Z is denoted as Z T , the Frobenius norm of Z is defined as and Z's l 2,1 The norm is defined as where z fl Represents the element in the fth row and lth column of the matrix Z.

[0073] S2, construct a traction matrix for guiding the sparse direction, design a non-negative labeled sparse matrix to learn sparsity, combine the traction matrix, non-negative labeled sparse matrix, label space matrix and pseudo-label matrix to build a fitting model; construct a label correlation model through the pseudo-label matrix; construct a nuclear norm model of the feature selection matrix; construct an objective function based on the regression framework, fitting model, label correlation model and nuclear norm model of the feature selection matrix, solve the objective function through an alternating iterative optimization algorithm to obtain an optimized feature selection matrix, determine the image feature subset based on the optimized feature selection matrix, and construct training data based on the image feature subset and the label space matrix.

[0074] In a specific embodiment, a traction matrix for guiding the sparse direction is constructed, a non-negative labeled sparse matrix is ​​designed to learn the sparsity, a fitting model is constructed by combining the traction matrix, the non-negative labeled sparse matrix, the label space matrix and the pseudo label matrix; a label correlation model is constructed through the pseudo label matrix; a nuclear norm model of the feature selection matrix is ​​constructed; and an objective function is constructed according to the regression framework, the fitting model, the label correlation model and the nuclear norm model of the feature selection matrix, specifically including:

[0075] The regression framework is expressed as:

[0076]

[0077] Where α is ||W|| 2,1 The trade-off parameter, W represents the feature selection matrix, V represents the pseudo-label matrix, X represents the feature space matrix, ||·|| F represents the Frobenius norm, ||·|| 2,1 Indicates l 2,1 norm;

[0078] Combine the traction matrix B, the non-negative marker sparse matrix S and the marker space matrix Y to fit the pseudo marker matrix V and construct a fitting model, which is expressed as:

[0079]

[0080] in, represents the matrix Hadamard product operator, Defined as: The element B in the i-th row and j-th column of each traction matrix B ij Represents a relaxation direction, i.e., “+1” represents a positive sparse direction and “-1” represents a negative sparse direction;

[0081] Based on the second-order strategy, the exploration of tag correlation is introduced and a tag correlation model is constructed, which is expressed as:

[0082]

[0083] Among them, ||·||2 represents the l2 norm, V :p is the pseudo-labeling result of the pth label, V :q is the pseudo-labeling result of the qth label, L = AG, A is a diagonal matrix, represented by diag(a1, a2, ..., a c ), a j represents the element in row j and column j of A, g pq is the element in the p-th row and q-th column of G, p,q=1,2,…,c; G represents the affinity matrix, and the affinity matrix is ​​initially generated by random initialization;

[0084] The nuclear norm model of the feature selection matrix is ​​expressed as:

[0085]

[0086] Among them, ||·|| * represents kernel function, Tr represents trace;

[0087] The objective function is expressed as:

[0088]

[0089] where β, γ, and λ represent the trade-off parameters of the fitting model, the marker correlation model, and the nuclear norm model representation of the feature selection matrix, respectively.

[0090] Specifically, refer to Figure 2 First, we build a regression framework to explore the mapping relationship between the feature space matrix and the label space matrix. We construct a feature selection matrix. The initial feature selection matrix is ​​randomly generated from the feature space matrix and iteratively optimized in each subsequent round. We calculate each row vector [w h (h=1→d)], and then determine the row vector [w h (h=1→d)], the l1 norm of the feature is reflected in the row vector [w h(h=1→d)]. The l2,1 norm is applied to the feature selection matrix, and the pseudo-label matrix V and the feature selection matrix W are constructed through the regression framework to process the label information and the unbalanced label density respectively. The traction matrix B is created to guide the sparse direction, and the non-negative label sparse matrix S is designed to learn the sparsity. The traction matrix B and the non-negative label sparse matrix S are combined with the label space matrix Y to fit the pseudo-label matrix V and construct a fitting model. The label correlation is explored through the fitted pseudo-label matrix. If the correlation between two labels is stronger, the distance between the two labels should be closer. Conversely, the smaller the correlation, the farther the distance between the labels. Based on the second-order strategy, the label correlation model is introduced to explore the label correlation. The feature selection matrix W is used to explore the shared information between multiple category labels, and the important shared information of multiple category labels is obtained by minimizing the nuclear norm model of the feature selection matrix W. The final objective function is constructed by combining the above regression framework, fitting model, label correlation model and nuclear norm model of the feature selection matrix W to achieve feature selection.

[0091] In a specific embodiment, the objective function is solved by an alternating iterative optimization algorithm to obtain an optimized feature selection matrix, which specifically includes:

[0092] Set W = [w 1 ,w 2 ,...,w d ] T , h=1,2,…,d, the objective function is equivalent to:

[0093]

[0094] in, D is a diagonal matrix, D hh represents the element in the hth row and hth column of D, w h Represents the row vector consisting of the elements of the hth row in the feature selection matrix W;

[0095] Fix V and S, and solve W after optimization in the current round. Formula (6) is simplified to:

[0096]

[0097] Where Δ represents the simplified objective function after fixing V and S;

[0098] make According to the singular value decomposition W = U∑P T , W=U∑P T Using the feature selection matrix W after the previous round of optimization, we can get:

[0099]

[0100] Among them, U and P represent the first variable and the second variable obtained by solving the feature selection matrix W after the current round of optimization, W = (X T X+αD) -1 (X T V-λUP T ) is the optimized W of the current round;

[0101] Fix W and S, and solve V after optimization in the current round. Formula (6) is simplified to:

[0102]

[0103] Where Θ represents the simplified objective function after fixing W and S;

[0104] make We can get:

[0105]

[0106] Where I is the identity matrix;

[0107] Fix W and V, set H = VY, Y = VH, Solving for S after optimization in the current round, formula (6) is simplified to:

[0108]

[0109] The optimal solution of S can be calculated as:

[0110]

[0111] Specifically, the objective function is solved by an alternating iterative optimization algorithm until the objective function converges to obtain the final optimized feature selection matrix. If the objective function does not converge, the objective function is solved again based on the S, V, and W after the previous round of optimization to obtain the S, V, and W after the current round of optimization.

[0112] In a specific embodiment, the l2 norm of the row vector composed of the elements of each row in the final optimized feature selection matrix is ​​calculated. h ||2, and follow ||w h ||2(h=1→d) in descending order, and select the row vector composed of the elements of each row in the final optimized feature selection matrix ranked in the top Q% as the image feature subset.

[0113] Specifically, in the embodiments of the present application, according to || w h The top 20% features are selected as the image feature subset in descending order of ||2(h=1→d). In other embodiments, other Q values ​​may also be selected.

[0114] S3, constructing an image classification model, and using the training data to train the image classification model to obtain a trained image classification model.

[0115] In a specific embodiment, the image classification model includes an MLKNN model.

[0116] Specifically, in the embodiment of the present application, the MLKNN model is selected as the image classification model. In other embodiments, other classification models may also be selected, which are not limited here. The optimized feature selection matrix obtained by the final solution is used to select the top-ranked image feature subset to characterize the image, and the quality of the selected features is tested by the MLKNN model.

[0117] The MLKNN model is used to test the selected image feature subset, and the average performance of an image dataset Corel16k on two indicators (Hamming loss and Micro-F1) is recorded through five-fold cross validation. Among them, LSMFS is the method proposed in the embodiment of the present application. The lower the Hamming loss value, the better the performance, and the higher the Micro-F1 value, the better the performance.

[0118] Table 1: Detailed characteristics of a multi-label image dataset

[0119] Dataset Number of images Number of features Number of markers Mark Cardinality Marker density Corel16k 13761 500 164 2.882 0.018

[0120] Table 2: Performance of LSMFS with 6 comparison methods on two indicators under the MLKNN model

[0121]

[0122] The above results show that the image feature subset selected by the feature selection method adopted in the embodiment of the present application has a better classification effect on the MLKNN model.

[0123] S4, acquiring an image, and inputting features in the image corresponding to the image feature subset into a trained image classification model to obtain an image classification result.

[0124] Specifically, the trained image classification model is deployed and applied, features corresponding to the image feature subset in any image are obtained, and the features are input into the trained image classification model to obtain image classification results.

[0125] Further references Figure 3 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a multi-label image classification device based on salient features. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0126] The present invention provides a method for multi-label image classification based on salient features, comprising the following steps:

[0127] A regression framework construction module 1 is configured to obtain a number of images with known labels and construct a feature space matrix and a label space matrix, randomly generate an initial feature selection matrix from the feature space matrix, randomly initialize and generate a pseudo label matrix with the same spatial size as the label space matrix, and construct a regression framework based on the feature selection matrix, the feature space matrix and the pseudo label matrix;

[0128] The training data construction module 2 is configured to construct a traction matrix for guiding the sparsity direction, design a non-negative labeled sparse matrix to learn the sparsity, and construct a fitting model by combining the traction matrix, the non-negative labeled sparse matrix, the label space matrix and the pseudo-label matrix; construct a label correlation model through the pseudo-label matrix; construct a nuclear norm model of the feature selection matrix; construct an objective function according to the regression framework, the fitting model, the label correlation model and the nuclear norm model of the feature selection matrix, solve the objective function through an alternating iterative optimization algorithm, and judge whether the objective function converges. If so, the feature selection matrix after the last round of optimization is used as the final optimized feature selection matrix, otherwise, the training data construction module 2 is repeatedly executed to determine the image feature subset based on the final optimized feature selection matrix, and construct training data based on the image feature subset and the label space matrix;

[0129] The model building module 3 is configured to build an image classification model, and train the image classification model using training data to obtain a trained image classification model;

[0130] The classification module 4 is configured to acquire an image, input features in the image corresponding to the image feature subset into a trained image classification model, and obtain an image classification result.

[0131] Figure 4 Schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present invention. Figure 4 As shown, the electronic device of this embodiment includes: a processor 401 and a memory 402; wherein the memory 402 is used to store computer-executable instructions; the processor 401 is used to execute the computer-executable instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0132] Optionally, the memory 402 may be independent or integrated with the processor 401 .

[0133] When the memory 402 is independently provided, the electronic device further includes a bus 403 for connecting the memory 402 and the processor 401 .

[0134] The embodiment of the present invention further provides a computer storage medium, in which computer execution instructions are stored. When the processor 401 executes the computer execution instructions, the above method is implemented.

[0135] The embodiment of the present invention further provides a computer program product, including a computer program. When the computer program is executed by the processor 401, the above method is implemented.

[0136] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0137] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of this embodiment.

[0138] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each module may exist physically separately, or two or more modules may be integrated into one unit. The unit formed by the above modules may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0139] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor 401 to perform some steps of the methods of various embodiments of the present application.

[0140] It should be understood that the processor 401 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or the processor 401 may be any conventional processor 401, etc. The steps of the method disclosed in the invention may be directly embodied in the hardware processor 401 for execution, or may be executed by a combination of hardware and software modules in the processor 401.

[0141] The memory 402 may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk, or an optical disk.

[0142] The bus 403 may be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 403 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus 403 in the drawings of the present application is not limited to only one bus 403 or one type of bus 403.

[0143] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0144] An exemplary storage medium is coupled to the processor 401, so that the processor 401 can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor 401. The processor 401 and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor 401 and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0145] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0146] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-label image classification method based on salient features, characterized in that: The following steps are involved: S1, obtaining a number of images with known labels and constructing a feature space matrix and a label space matrix, randomly generating an initial feature selection matrix from the feature space matrix, randomly initializing and generating a pseudo label matrix with the same spatial size as the label space matrix, and constructing a regression framework based on the feature selection matrix, the feature space matrix and the pseudo label matrix; S2, constructing a traction matrix for guiding the sparsity direction, designing a non-negative label sparse matrix to learn the sparsity, and combining the traction matrix, the non-negative label sparse matrix, the label space matrix and the pseudo label matrix to construct a fitting model; Constructing a marker correlation model through the pseudo-marker matrix; constructing a nuclear norm model of the feature selection matrix; constructing an objective function according to the regression framework, the fitting model, the marker correlation model and the nuclear norm model of the feature selection matrix, solving the objective function through an alternating iterative optimization algorithm, and judging whether the objective function converges, if so, taking the feature selection matrix after the last round of optimization as the final optimized feature selection matrix, otherwise repeating step S2, determining an image feature subset based on the final optimized feature selection matrix, and constructing training data based on the image feature subset and the marker space matrix; S3, constructing an image classification model, and using the training data to train the image classification model to obtain a trained image classification model; S4, acquiring an image, and inputting features in the image corresponding to the image feature subset into the trained image classification model to obtain an image classification result.

2. The multi-label image classification method based on salient features according to claim 1, characterized in that: The feature space matrix is ​​expressed as in is the vector consisting of the elements of the i-th row in the feature space matrix, x i Each element in is a feature of the i-th image, and the features include color mean features, color standard deviation features, shape features, 4 Gaussian filters, and texture features extracted by 12 directional filters corresponding to each of the 8 regions of the image. represents the real number space, d represents the d-dimensional feature, n represents the total number of images, and the label space matrix contains the labels corresponding to n images, which is expressed as The i-th image x i ∈X(1≤i≤n) are associated with a row vector in the label space matrix Y represents the element in the i-th row and j-th column of the tag space matrix. The tag is the category of each object in the image. If the i-th image is associated with the j-th tag, the corresponding element in the tag space matrix is If the i-th image is not associated with the j-th tag, then the element in the i-th row and j-th column of the tag space matrix is 3. The multi-label image classification method based on salient features according to claim 1, characterized in that: Constructing a traction matrix for guiding the sparsity direction, designing a non-negative labeled sparse matrix to learn the sparsity, and combining the traction matrix, the non-negative labeled sparse matrix, the labeled space matrix and the pseudo labeled matrix to construct a fitting model; Constructing a marker correlation model through the pseudo marker matrix; constructing a nuclear norm model of the feature selection matrix; constructing an objective function according to the regression framework, the fitting model, the marker correlation model and the nuclear norm model of the feature selection matrix, specifically including: The regression framework is expressed as: Where α is ||W|| 2,1 , W represents the feature selection matrix, V represents the pseudo label matrix, X represents the feature space matrix, ||·||F represents the Frobenius norm, ‖·‖ 2,1 Indicates e 2,1 norm; Combine the traction matrix B, the non-negative marker sparse matrix S and the marker space matrix Y to fit the pseudo marker matrix V and construct a fitting model, which is expressed as: in, represents the matrix Hadamard product operator, Defined as: The element Bij in the i-th row and j-th column of each traction matrix B represents a relaxation direction, that is, "+1" represents a positive sparse direction, and "-1" represents a negative sparse direction; Based on the second-order strategy, the exploration of tag correlation is introduced and a tag correlation model is constructed, which is expressed as: Among them, ||·||2 represents the l2 norm, V :p is the pseudo-labeling result of the pth label, V :q is the pseudo-labeling result of the qth label, L = AG, A is a diagonal matrix, represented by diag(a1, a2, ..., a c ), a j represents the element in row j and column j in A, g pq is the element in the p-th row and q-th column of G, p,q=1,2,…,c; G represents the affinity matrix, and the affinity matrix is ​​initially generated by random initialization; The nuclear norm model of the feature selection matrix is ​​expressed as: Among them, ||·||* * represents kernel function, Tr represents trace; The objective function is expressed as: where β, γ, and λ represent the trade-off parameters of the fitting model, the marker correlation model, and the nuclear norm model representation of the feature selection matrix, respectively.

4. The multi-label image classification method based on salient features according to claim 3 is characterized in that: The objective function is solved by an alternating iterative optimization algorithm, specifically including: Set W = [w 1 ,w 2 ,...,w d ] T , h=1,2,…,d, the objective function is equivalent to: in, D is a diagonal matrix, D hh represents the element in the hth row and hth column of D, w h Represents the row vector consisting of the elements of the hth row in the feature selection matrix W after the previous round of optimization; Fix V and S, and solve W after optimization in the current round. Formula (6) is simplified to: Where Δ represents the simplified objective function after fixing V and S; make According to the singular value decomposition W = U∑P T , W=U∑P T Using the feature selection matrix W after the previous round of optimization, we can get: Among them, U and P represent the first variable and the second variable obtained by solving the feature selection matrix W after the current round of optimization, W = (X T X+αD) -1 (X T V-λUP T ) is the optimized W of the current round; Fix W and S, and solve V after optimization in the current round. Formula (6) is simplified to: Where Θ represents the simplified objective function after fixing W and S; make We can get: Where I is the identity matrix; Fix W and V, set H = VY, Y = VH, Solving for S after optimization in the current round, formula (6) is simplified to: The optimal solution of S can be calculated as:

5. The multi-label image classification method based on salient features according to claim 1, characterized in that: Calculate the row vector composed of the elements of each row in the final optimized feature selection matrix Norm||w h ||2, and follow ||w h ||2(h=1→d) in descending order, and select the row vector composed of the elements of each row in the final optimized feature selection matrix that ranks in the top Q% as the image feature subset.

6. The multi-label image classification method based on salient features according to claim 1, characterized in that: The image classification model includes an MLKNN model.

7. A multi-label image classification method based on salient features, characterized in that: The following steps are involved: A regression framework construction module is configured to obtain a number of images with known labels and construct a feature space matrix and a label space matrix, randomly generate an initial feature selection matrix from the feature space matrix, randomly initialize and generate a pseudo label matrix with the same spatial size as the label space matrix, and construct a regression framework based on the feature selection matrix, the feature space matrix and the pseudo label matrix; A training data construction module is configured to construct a traction matrix for guiding the sparsity direction, design a non-negative labeled sparse matrix to learn the sparsity, and construct a fitting model by combining the traction matrix, the non-negative labeled sparse matrix, the label space matrix and the pseudo-label matrix; Constructing a marker correlation model through the pseudo-marker matrix; constructing a nuclear norm model of the feature selection matrix; constructing an objective function according to the regression framework, the fitting model, the marker correlation model and the nuclear norm model of the feature selection matrix, solving the objective function through an alternating iterative optimization algorithm, and judging whether the objective function converges, if so, taking the feature selection matrix after the last round of optimization as the final optimized feature selection matrix, otherwise repeating the training data construction module, determining an image feature subset based on the final optimized feature selection matrix, and constructing training data based on the image feature subset and the marker space matrix; A model building module is configured to build an image classification model, and train the image classification model using the training data to obtain a trained image classification model; The classification module is configured to acquire an image, input features in the image corresponding to the image feature subset into the trained image classification model, and obtain an image classification result.

8. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images