Implicit semantic association modeling method, zero-sample image classification method and device

Abstract

An implicit semantic association modeling method, a zero-shot image classification method and apparatus of the present invention belong to the technical field of image processing. The zero-shot image classification method includes the following steps: Step 1, obtain a test image and a source image, where the test image is an unlabeled target image and the source image is a labeled image; Step 2, use the source image for model training to obtain a semantic correlation prediction model g(*); Step 3, calculate the implicit semantic representation of the test image; Step 4, calculate the semantic association matrix H between the test image and each source image i,t ; Step 5, use the semantic correlation prediction model g(*) to predict the semantic consistency between the test image and each source image, and classify the test image into the category with the largest average semantic consistency. The present invention performs excellently on multiple public data sets and continuously improves the classification accuracy, and has good discriminative power and generalization ability.

Description

Technical Field

[0001] The invention relates to an implicit semantic association modeling method, a zero-sample image classification method and a device, and belongs to the technical field of image processing. Background Art

[0002] Zero-shot image classification is a challenging problem in computer vision. In traditional image classification tasks, a large number of labeled samples are required to train the classification model. However, in many practical applications, it is very difficult to obtain a large number of labeled samples. Zero-shot image classification aims to classify unlabeled target category images using only labeled source category images, thus avoiding the need to obtain a large number of labeled samples.

[0003] At present, zero-shot image classification methods are mainly divided into two categories: methods based on explicit semantics and methods based on implicit semantics. Methods based on explicit semantics require the acquisition of semantic description information of the image, such as attributes, labels, etc. Such methods usually require a large amount of manual annotation and have high requirements on the accuracy of semantic description. However, in practical applications, it is very difficult to obtain accurate semantic descriptions, which limits the practicality of such methods. Methods based on implicit semantics attempt to automatically learn semantic information from the image itself, such as using deep learning models to learn semantic representations from image features. Such methods avoid the need for manual annotation, but place higher requirements on model training and design, and in many cases, the implicitly learned semantic information is still not accurate and reliable enough.

[0004] In addition, existing zero-shot image classification methods usually treat each image as an independent individual when processing images, and do not fully utilize the semantic association information between images. This independent modeling approach leads to insufficient generalization ability of the classification model and makes it difficult to handle complex and changing data distributions. Summary of the invention

[0005] In order to solve the above problems, the present invention proposes an implicit semantic association modeling method, a zero-sample image classification method and a device, which can improve the classification accuracy and have good discrimination and generalization capabilities.

[0006] The technical solution adopted by the present invention to solve the technical problem is:

[0007] In a first aspect, an embodiment of the present invention provides an implicit semantic association modeling method, comprising the following steps:

[0008] Extract the images to be classified and mark them, and build a source image set;

[0009] A classification model f(*) is established based on a convolutional neural network, and the classification model f(*) is trained using a source image set. Each dimension of the classification model f(*) represents the semantic consistency between the image and the corresponding class, which is called the implicit semantic representation of the image.

[0010] For image x i and image x j Two different source images, along image x i and image x j The lines between them are evenly divided into Q new sampling points, and the implicit semantic representation h of these sampling points is calculated. i,q,j =f(x i,q,j ), and concatenate them into an implicit semantic relevance matrix H i,j , q≤Q-1;

[0011] Calculate image x i and image x j Semantic consistency between tags , construct a training set (H i,j , );

[0012] Using the training set (H i,j , ) is used to train the residual neural network and obtain the semantic relevance prediction model g(*).

[0013] As a possible implementation of this embodiment, the convolutional neural network of the classification model f(*) includes a ResNet neural network, a VGG neural network, or a GoogLeNet neural network.

[0014] As a possible implementation of this embodiment, the residual neural network includes a ResNet18 neural network.

[0015] As a possible implementation of this embodiment, the value range of Q is 5-50.

[0016] As a possible implementation of this embodiment, the implicit semantic association modeling method further includes:

[0017] Perform performance test on the semantic relevance prediction model g(*);

[0018] As a possible implementation of this embodiment, the performance test of the semantic relevance prediction model g(*) includes:

[0019] When performing zero-shot image classification, the separation accuracy is used acc Performance test of semantic relevance prediction model g(*):

[0020] ,

[0021] In the formula, N correct is the number of correctly classified images, N total is the total number of images;

[0022] When performing generalized zero-shot image classification, the harmonic mean is used HM Performance test of semantic relevance prediction model g(*):

[0023] ,

[0024] In the formula, for generalized zero-shot image classification, acc s is the classification accuracy calculated on the source image, acc u is the classification accuracy calculated on the target image.

[0025] In a second aspect, an embodiment of the present invention provides an implicit semantic association modeling device, comprising:

[0026] The source image construction module is used to extract and label the images to be classified and construct a source image set;

[0027] A classification model building module is used to build a classification model f(*) based on a convolutional neural network, and use a source image set to train the classification model f(*). Each dimension of the classification model f(*) represents the semantic consistency between the image and the corresponding class, which is called the implicit semantic representation of the image.

[0028] Correlation matrix calculation module, used for image x i and image x j Two different source images, along image x i and image x j The line between them is evenly divided into Q segments by setting q sampling points, and the implicit semantic representation h of these sampling points is calculated. i,q,j =f(x i,q,j ), and concatenate them into an implicit semantic relevance matrix H i,j , q≤Q-1;

[0029] The training set builds the module for computing the image x i and image x j Semantic consistency between tags , construct a training set (H i,j , );

[0030] Model training module, used to use the training set (H i,j , ) is used to train the residual neural network and obtain the semantic relevance prediction model g(*).

[0031] In a third aspect, an embodiment of the present invention provides a zero-shot image classification method based on implicit semantic association, comprising the following steps:

[0032] Step 1, obtaining a test image and a source image, wherein the test image is an unlabeled target image, and the source image is a labeled image;

[0033] Step 2: Use the source image to perform model training to obtain a semantic relevance prediction model g(*);

[0034] Step 3, calculate the implicit semantic representation of the test image;

[0035] Step 4: Calculate the semantic association matrix H between the test image and each source image i,t ;

[0036] Step 5: Use the semantic relevance prediction model g(*) to predict the semantic consistency between the test image and each source image, and classify the test image into the category with the largest average semantic consistency.

[0037] As a possible implementation of this embodiment, step 2 includes:

[0038] Step 21, establishing a classification model f(*) based on a convolutional neural network, and using the source image set to train the classification model f(*), each dimension of the classification model f(*) represents the semantic consistency between the image and the corresponding class, which is called the implicit semantic representation of the image;

[0039] Step 22, evenly divide a number of points along the line between the two source images, calculate the implicit semantic representations of these points, and splice them into an implicit semantic correlation matrix;

[0040] Step 23, calculating the semantic consistency labels between the two source images and constructing a training set;

[0041] Step 24, use the training set to perform residual neural network training to obtain a semantic relevance prediction model g(*).

[0042] As a possible implementation of this embodiment, step 22 includes:

[0043] For an image x, let is the implicit semantic description of image x. If two images belong to the same category, their implicit semantic descriptions are similar and consistent.

[0044] Using h i and h jRepresents the image x i and image x j Implicit semantic description of two source images;

[0045] Along the image x i and image x j The lines between them set q sampling points to transform the image x i and image x j The line between them is evenly divided into Q segments, and each sampling point x i,q,j Obtained by interpolation calculation:

[0046] = +q( ) / Q,

[0047] Where q = 1, ..., Q-1;

[0048] The implicit semantic description of the q sampling points is:

[0049] ;

[0050] h i and h i,q,j Concatenate into implicit semantic correlation matrix H i,j :

[0051] ,

[0052] In the formula, H i,j The size of is C×(Q+1), where C is the number of target domain categories.

[0053] As a possible implementation of this embodiment, step 23 includes:

[0054] Calculate image x i and image x j Semantic consistency between tags :

[0055] ,

[0056] In the formula, For image x i Semantically consistent labels;

[0057] Semantically consistent labels With the implicit semantic correlation matrix H i,j Construct a training set (H i,j , ).

[0058] As a possible implementation of this embodiment, step 4 includes:

[0059] Along the test image x t and the source image x i The connection between the two sets q sampling points to test the image x t and the source image x i The line between them is evenly divided into Q segments, and each sampling point x t,q,i Obtained by interpolation calculation:

[0060] = +q( ) / Q,

[0061] Where q = 1, ..., Q-1;

[0062] The implicit semantic description of the q sampling points is:

[0063] ;

[0064] h i and h i,q,j Concatenate into implicit semantic correlation matrix H t,i :

[0065] ,

[0066] In the formula, H i,j The size of is C×(Q+1), where C is the number of target domain categories.

[0067] As a possible implementation of this embodiment, step 5 includes:

[0068] The implicit semantic correlation matrix H t,i The input semantic relevance prediction model g(*) predicts the semantic consistency between the test image and each source image;

[0069] Classify the test image into category c with the largest average semantic consistency:

[0070] ,

[0071] In the formula, is the number of images of category c.

[0072] In a fourth aspect, an embodiment of the present invention provides a zero-sample image classification device based on implicit semantic association, comprising:

[0073] A data acquisition module, used to acquire a test image and a source image, wherein the test image is an unlabeled target image and the source image is a labeled image;

[0074] A prediction model training module is used to perform model training using source images to obtain a semantic relevance prediction model g(*);

[0075] Latent semantic representation module, used to calculate the latent semantic representation of the test image;

[0076] The association matrix calculation module is used to calculate the semantic association matrix H between the test image and each source image. i,t ;

[0077] The image classification module is used to predict the semantic consistency between the test image and each source image using the semantic relevance prediction model g(*), and classify the test image into the category with the largest average semantic consistency.

[0078] In a fifth aspect, an embodiment of the present invention provides a computer device, comprising a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory through the bus, and the processor executes the machine-readable instructions to perform steps of any of the above-mentioned implicit semantic association modeling methods and / or zero-sample image classification methods based on implicit semantic associations.

[0079] In a sixth aspect, an embodiment of the present invention provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned implicit semantic association modeling methods and / or zero-sample image classification methods based on implicit semantic associations are executed.

[0080] The beneficial effects of the technical solution of the embodiment of the present invention are as follows:

[0081] The present invention makes full use of the semantic association information between images to improve the classification accuracy; it does not need to manually annotate semantic information, reducing the dependence on data annotation; the present invention is not only computationally efficient, but also achieves fast classification through matrix operations and deep learning acceleration; it also has strong generalization ability and is applicable to different types of image data sets. The present invention is easy to expand and combines advanced technologies such as attention mechanism and graph convolutional network to further improve the classification performance.

[0082] The present invention is applicable to general zero-shot image classification tasks, and can also be applied to generalized zero-shot image classification tasks, effectively improving the accuracy of image classification. The present invention performs well on multiple public data sets and continuously improves classification accuracy, with good discrimination and generalization capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 is a flow chart of an implicit semantic association modeling method according to an exemplary embodiment;

[0084] Figure 2 is a structural schematic diagram of an implicit semantic association modeling device according to an exemplary embodiment;

[0085] Figure 3 is a flow chart of a zero-shot image classification method based on implicit semantic association according to an exemplary embodiment;

[0086] Figure 4 The diagram is a structural diagram of a zero-sample image classification device based on implicit semantic association according to an exemplary embodiment. DETAILED DESCRIPTION

[0087] In order to more clearly illustrate the technical features of the solution of the present invention, the present invention is described in detail below through specific implementation methods and in conjunction with the accompanying drawings.

[0088] like Figure 1 As shown, an implicit semantic association modeling method provided by an embodiment of the present invention includes the following steps:

[0089] Extract the images to be classified and mark them, and build a source image set;

[0090] A classification model f(*) is established based on a convolutional neural network, and the classification model f(*) is trained using a source image set. Each dimension of the classification model f(*) represents the semantic consistency between the image and the corresponding class, which is called the implicit semantic representation of the image.

[0091] For image x i and image x j Two different source images, along image x i and image x j The lines between them are evenly divided into Q new sampling points, and the implicit semantic representation h of these sampling points is calculated. i,q,j =f(x i,q,j ), and concatenate them into an implicit semantic relevance matrix H i,j , q≤Q-1;

[0092] Calculate image x i and image x j Semantic consistency between tags , construct a training set (H i,j , );

[0093] Using the training set (H i,j , ) is used to train the residual neural network and obtain the semantic relevance prediction model g(*).

[0094] The present invention improves the classification accuracy of the semantic relevance prediction model by modeling the semantic relevance between images.

[0095] As a possible implementation of this embodiment, the convolutional neural network of the classification model f(*) includes a ResNet neural network, a VGG neural network, or a GoogLeNet neural network.

[0096] As a possible implementation of this embodiment, the residual neural network includes a ResNet18 neural network.

[0097] As a possible implementation of this embodiment, the value range of Q is 5-50.

[0098] As a possible implementation of this embodiment, the implicit semantic association modeling method further includes:

[0099] Perform performance test on the semantic relevance prediction model g(*);

[0100] As a possible implementation of this embodiment, the performance test of the semantic relevance prediction model g(*) includes:

[0101] When performing zero-shot image classification, the separation accuracy is used acc Performance test of semantic relevance prediction model g(*):

[0102] ,

[0103] In the formula, N correct is the number of correctly classified images, N total is the total number of images;

[0104] When performing generalized zero-shot image classification, the harmonic mean is used HM Performance test of semantic relevance prediction model g(*):

[0105] ,

[0106] In the formula, for generalized zero-shot image classification, acc s is the classification accuracy calculated on the source image, acc u is the classification accuracy calculated on the target image.

[0107] like Figure 2 As shown, an implicit semantic association modeling device provided by an embodiment of the present invention includes:

[0108] The source image construction module is used to extract and label the images to be classified and construct a source image set;

[0109] A classification model building module is used to build a classification model f(*) based on a convolutional neural network, and use a source image set to train the classification model f(*). Each dimension of the classification model f(*) represents the semantic consistency between the image and the corresponding class, which is called the implicit semantic representation of the image.

[0110] Correlation matrix calculation module, used for image x i and image x j Two different source images, along image x i and image x j The line between them is evenly divided into Q segments by setting q sampling points, and the implicit semantic representation h of these sampling points is calculated. i,q,j =f(x i,q,j ), and concatenate them into an implicit semantic relevance matrix H i,j , q≤Q-1;

[0111] The training set builds the module for computing the image x i and image x j Semantic consistency between tags , construct a training set (H i,j , );

[0112] Model training module, used to use the training set (H i,j , ) is used to train the residual neural network and obtain the semantic relevance prediction model g(*).

[0113] like Figure 3 As shown, a zero-sample image classification method based on implicit semantic association provided by an embodiment of the present invention includes the following steps:

[0114] Step 1, obtaining a test image and a source image, wherein the test image is an unlabeled target image, and the source image is a labeled image;

[0115] Step 2: Use the source image to perform model training to obtain a semantic relevance prediction model g(*);

[0116] Step 3, calculate the implicit semantic representation of the test image;

[0117] Step 4: Calculate the semantic association matrix H between the test image and each source image i,t ;

[0118] Step 5: Use the semantic relevance prediction model g(*) to predict the semantic consistency between the test image and each source image, and classify the test image into the category with the largest average semantic consistency.

[0119] As a possible implementation of this embodiment, step 2 includes:

[0120] Step 21, establishing a classification model f(*) based on a convolutional neural network, and using the source image set to train the classification model f(*), each dimension of the classification model f(*) represents the semantic consistency between the image and the corresponding class, which is called the implicit semantic representation of the image;

[0121] Step 22, evenly divide a number of points along the line between the two source images, calculate the implicit semantic representations of these points, and splice them into an implicit semantic correlation matrix;

[0122] Step 23, calculating the semantic consistency labels between the two source images and constructing a training set;

[0123] Step 24, use the training set to perform residual neural network training to obtain a semantic relevance prediction model g(*).

[0124] As a possible implementation of this embodiment, step 22 includes:

[0125] For an image x, let is the implicit semantic description of image x. If two images belong to the same category, their implicit semantic descriptions are similar and consistent.

[0126] Using h i and h j Represents the image x i and image x j Implicit semantic description of two source images;

[0127] Along the image x i and image x j The lines between them set q sampling points to transform the image x i and image x j The line between them is evenly divided into Q segments, and each sampling point x i,q,j Obtained by interpolation calculation:

[0128] = +q( ) / Q,

[0129] Where q = 1, ..., Q-1;

[0130] The implicit semantic description of the q sampling points is:

[0131] ;

[0132] h i and h i,q,j Concatenate into implicit semantic correlation matrix H i,j :

[0133] ,

[0134] In the formula, H i,j The size of is C×(Q+1), where C is the number of target domain categories.

[0135] As a possible implementation of this embodiment, step 23 includes:

[0136] Calculate image x i and image x j Semantic consistency between tags :

[0137] ,

[0138] In the formula, For image x i Semantically consistent labels;

[0139] Semantically consistent labels With the implicit semantic correlation matrix H i,j Construct a training set (H i,j , ).

[0140] As a possible implementation of this embodiment, step 4 includes:

[0141] Along the test image x t and the source image x i The connection between the two sets q sampling points to test the image x t and the source image x i The line between them is evenly divided into Q segments, and each sampling point x t,q,i Obtained by interpolation calculation:

[0142] = +q( ) / Q,

[0143] Where q = 1, ..., Q-1;

[0144] The implicit semantic description of the q sampling points is:

[0145] ;

[0146] hi and h i,q,j Concatenate into implicit semantic correlation matrix H t,i :

[0147] ,

[0148] In the formula, H i,j The size of is C×(Q+1), where C is the number of target domain categories.

[0149] As a possible implementation of this embodiment, step 5 includes:

[0150] The implicit semantic correlation matrix H t,i The input semantic relevance prediction model g(*) predicts the semantic consistency between the test image and each source image;

[0151] Classify the test image into category c with the largest average semantic consistency:

[0152] ,

[0153] In the formula, is the number of images of category c.

[0154] like Figure 4 As shown, an embodiment of the present invention provides a zero-sample image classification device based on implicit semantic association, comprising:

[0155] A data acquisition module, used to acquire a test image and a source image, wherein the test image is an unlabeled target image and the source image is a labeled image;

[0156] A prediction model training module is used to perform model training using source images to obtain a semantic relevance prediction model g(*);

[0157] Latent semantic representation module, used to calculate the latent semantic representation of the test image;

[0158] The association matrix calculation module is used to calculate the semantic association matrix H between the test image and each source image. i,t ;

[0159] The image classification module is used to predict the semantic consistency between the test image and each source image using the semantic relevance prediction model g(*), and classify the test image into the category with the largest average semantic consistency.

[0160] The process of zero-sample image classification in the present invention mainly includes the following steps:

[0161] Step 1: Latent semantic representation learning.

[0162] First, use the labeled source images to train a classification model f(*). f(*) can be ResNet, VGG or GoogLeNet, or other suitable deep learning models.

[0163] The output dimension of f(*) is consistent with the number of categories in the target domain to reflect the implicit semantic representation of the image. The class corresponding to the maximum response will be selected. However, this strategy does not fully utilize f(*). The present invention attempts to use this information more appropriately. For an image, each dimension of f(*) represents the semantic consistency between the image and the corresponding class, which is called the implicit semantic representation of the image. In this way, the discriminant information of the image can be obtained.

[0164] Step 2: Implicit semantic correlation matrix H i,j Calculation.

[0165] For an image x, let is the implicit semantic description of image x. If two images belong to the same class, their implicit semantic descriptions should be similar and consistent. However, due to the imperfection of the pre-learned classification model, implicit semantic consistency does not always lead to class consistency. The implicit semantic consistency between two images should be explored jointly.

[0166] For two images x i and x j ,h i and h j In order to explore the implicit semantic relationship between them, the present invention first converts x i and x j The line between them is evenly divided into Q segments, and the number of sampling points Q can be selected according to the actual situation. i,q,j Obtained through interpolation calculation, the formula is:

[0167] = +q( ) / Q,

[0168] Where q = 1, ..., Q-1. Then the implicit semantics of these Q-1 points can be described as:

[0169] ,

[0170] h i and h i,q,j Concatenate into matrix H t,i :

[0171] ,

[0172] The matrix Ht,i The size of is C×(Q+1), where C is the number of target domain categories.

[0173] Step 3: Semantic association prediction model training.

[0174] Using the latent semantic association matrix H i,j Train a semantic association prediction model g(*) that can determine the semantic consistency between two images. g(*) can use a deep neural network, such as ResNet18. The input of g(*) is the matrix H i,j , the output is a semantic consistency label, which can be 1 or -1. Using the training data set {(H i,j , )} train g(*), where For x i The semantic consistency label can be calculated as follows:

[0175] .

[0176] Step 4: Test image classification.

[0177] For zero-shot image classification, the test image x t belongs only to the target class. To predict its category, first use f(x t ) to predict its implicit semantic representation. Then calculate x t and each labeled source image x i The semantic consistency matrix H between i,t . g( H i,t ) is used to predict x t and x i Then, x t can be classified into class c with the largest average semantic consistency:

[0178] .

[0179] For generalized zero-shot image classification, the test image x t It may belong to either the source class or the target class. In order to predict its category, its implicit semantic representation can be predicted and the semantic consistency matrix g(H) can be calculated in a similar way to zero-shot image classification. i , t ) is used to predict x t and x i The semantic consistency between the source class and the target class is calculated. Then, x t Can be classified into the class with the maximum response.

[0180] The following is a direct comparison of the performance of other zero-shot classification methods. The test indicators include the following two.

[0181] 1. Accuracy is a common indicator for evaluating classification models. For zero-shot image classification, the classification accuracy acc is used to evaluate the zero-shot classification model. It is defined as:

[0182] ,

[0183] Where: N correct is the number of correctly classified images. N total is the total number of images. For generalized zero-shot image classification, acc s is the classification accuracy calculated on the source image, acc u is the classification accuracy calculated on the target image, acc s and acc u Can be calculated in a similar way to acc.

[0184] 2. Harmonic mean. Harmonic mean is also used to evaluate the performance of generalized zero-shot image classification methods. It is achieved by jointly considering acc s and acc u It tries to balance the performance on the source and target images:

[0185] ,

[0186] The performance data comparison of the zero-shot image classification method based on implicit semantic association and other baseline image classification methods for zero-shot image classification on aPY, Aw A2, CUB and SUN datasets are shown in Tables 1 and 2. The values ​​in the table indicate the classification accuracy of these methods on the four datasets. The classification accuracy is used for quantitative comparison.

[0187] Table 1: Classification accuracy of the present invention on aPY, AwA2, CUB and SUN datasets

[0188]

[0189] Table 2: Classification accuracy of other baseline methods on aPY, AwA2, CUB and SUN datasets

[0190]

[0191] Due to the modeling of semantically related images, zero-shot image classification based on implicit semantic associations has improved accuracy over other baseline methods.

[0192] When the present invention forms a system, it can be implemented based on a deep learning framework, such as TensorFlow, PyTorch, etc., and can be extended to various types of image data sets, such as face recognition, target detection, scene classification, etc.

[0193] A computer device provided by an embodiment of the present invention includes a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the device is running, the processor communicates with the memory through the bus, and the processor executes the machine-readable instructions to perform steps of any of the above-mentioned implicit semantic association modeling methods and / or zero-sample image classification methods based on implicit semantic associations.

[0194] Specifically, the above-mentioned memory and processor are general-purpose memories and processors. When the processor runs the computer program stored in the memory, it can execute any of the above-mentioned implicit semantic association modeling methods and / or zero-sample image classification methods based on implicit semantic associations.

[0195] Corresponding to the method for starting the above-mentioned application, an embodiment of the present invention also provides a storage medium, on which a computer program is stored. When the computer program is run by a processor, the steps of any of the above-mentioned implicit semantic association modeling methods and / or zero-sample image classification methods based on implicit semantic association are executed.

[0196] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for implicit semantic association modeling, characterized in that: The steps include: Extract the images to be classified and mark them, and build a source image set; A classification model f(*) is established based on a convolutional neural network, and the classification model f(*) is trained using a source image set. Each dimension of the classification model f(*) represents the semantic consistency between the image and the corresponding class, which is called the implicit semantic representation of the image. For image x i and image x j Two different source images, along image x i and image x j The line between them is evenly divided into Q segments by setting q sampling points, and the implicit semantic representation h of these sampling points is calculated. i,q,j =f(x i,q,j ), and concatenate them into an implicit semantic relevance matrix H i,j , q≤Q-1; Calculate image x i and image x j Semantic consistency between tags , construct a training set (H i,j , ); Using the training set (H i,j , ) is used to train the residual neural network and obtain the semantic relevance prediction model g(*).

2. The implicit semantic association modeling method according to claim 1, characterized in that: The convolutional neural network of the classification model f(*) includes a ResNet neural network, a VGG neural network or a GoogLeNet neural network.

3. The implicit semantic association modeling method according to claim 1, characterized in that: The residual neural network includes a ResNet18 neural network.

4. The implicit semantic association modeling method according to claim 1, characterized in that: The value range of Q is 5-50.

5. The implicit semantic association modeling method according to any one of claims 1 to 4, characterized in that: Also includes: The performance of the semantic relevance prediction model g(*) is tested.

6. The implicit semantic association modeling method according to claim 5, characterized in that: The performance test of the semantic relevance prediction model g(*) includes: When performing zero-shot image classification, the separation accuracy is used acc Performance test of semantic relevance prediction model g(*): ， In the formula, N correct is the number of correctly classified images, N total is the total number of images; When performing generalized zero-shot image classification, the harmonic mean is used HM Performance test of semantic relevance prediction model g(*): ， In the formula, for generalized zero-shot image classification, acc s is the classification accuracy calculated on the source image, acc u is the classification accuracy calculated on the target image.

7. An implicit semantic association modeling device, characterized in that: include: The source image construction module is used to extract and label the images to be classified and construct a source image set; A classification model building module is used to build a classification model f(*) based on a convolutional neural network, and use a source image set to train the classification model f(*). Each dimension of the classification model f(*) represents the semantic consistency between the image and the corresponding class, which is called the implicit semantic representation of the image. Correlation matrix calculation module, used for image x i and image x j Two different source images, along image x i and image x j The line between them is evenly divided into Q segments by setting q sampling points, and the implicit semantic representation h of these sampling points is calculated. i,q,j =f(x i,q,j ), and concatenate them into an implicit semantic relevance matrix H i,j , q≤Q-1; The training set builds the module for computing the image x i and image x j Semantic consistency between tags , construct a training set (H i,j , ); Model training module, used to use the training set (H i,j , ) is used to train the residual neural network and obtain the semantic relevance prediction model g(*).

8. A zero-shot image classification method based on implicit semantic association, characterized in that: The steps include: Step 1, obtaining a test image and a source image, wherein the test image is an unlabeled target image, and the source image is a labeled image; Step 2: Use the source image to perform model training to obtain a semantic relevance prediction model g(*); Step 3, calculate the implicit semantic representation of the test image; Step 4: Calculate the semantic association matrix H between the test image and each source image i,t ; Step 5, using the semantic relevance prediction model g(*) to predict the semantic consistency between the test image and each source image, and classifying the test image into the category with the largest average semantic consistency; The step 2 comprises: Step 21, establishing a classification model f(*) based on a convolutional neural network, and using the source image set to train the classification model f(*), each dimension of the classification model f(*) represents the semantic consistency between the image and the corresponding class, which is called the implicit semantic representation of the image; Step 22, evenly divide a number of points along the line between the two source images, calculate the implicit semantic representations of these points, and splice them into an implicit semantic correlation matrix; Step 23, calculating the semantic consistency labels between the two source images and constructing a training set; Step 24, use the training set to perform residual neural network training to obtain a semantic relevance prediction model g(*).

9. The zero-shot image classification method based on implicit semantic association according to claim 8, characterized in that: The step 22 comprises: For an image x, let is the implicit semantic description of image x. If two images belong to the same category, their implicit semantic descriptions are similar and consistent. Using h i and h j Represents the image x i and image x j Implicit semantic description of two source images; Along the image x i and image x j The lines between them set q sampling points to transform the image x i and image x j The line between them is evenly divided into Q segments, and each sampling point x i,q,j Obtained by interpolation calculation: = +q( ) / Q, Where q = 1, ..., Q-1; The implicit semantic description of the q sampling points is: ； h i and h i,q,j Concatenate into implicit semantic correlation matrix H i,j : ， In the formula, H i,j The size of is C×(Q+1), where C is the number of target domain categories.

10. The zero-shot image classification method based on implicit semantic association according to claim 9, characterized in that: The step 23 comprises: Calculate image x i and image x j Semantic consistency between tags : ， In the formula, For image x i Semantically consistent labels; Semantically consistent labels With the implicit semantic correlation matrix H i,j Construct a training set (H i,j , ).

11. The zero-shot image classification method based on implicit semantic association according to any one of claims 8 to 10, characterized in that: The step 4 comprises: Along the test image x t and the source image x i The connection between the two sets q sampling points to test the image x t and the source image x i The line between them is evenly divided into Q segments, and each sampling point x t,q,i Obtained by interpolation calculation: = +q( ) / Q, Where q = 1, ..., Q-1; The implicit semantic description of the q sampling points is: ； h i and h i,q,j Concatenate into implicit semantic correlation matrix H t,i : ， In the formula, H i,j The size of is C×(Q+1), where C is the number of target domain categories.

12. The zero-shot image classification method based on implicit semantic association according to claim 11, characterized in that: The step 5 comprises: The implicit semantic correlation matrix H t,i The input semantic relevance prediction model g(*) predicts the semantic consistency between the test image and each source image; Classify the test image into category c with the largest average semantic consistency: ， In the formula, is the number of images of category c.

13. A zero-shot image classification device based on implicit semantic association, characterized in that: include: A data acquisition module, used to acquire a test image and a source image, wherein the test image is an unlabeled target image and the source image is a labeled image; A prediction model training module is used to perform training using source images to obtain a semantic relevance prediction model g(*); Latent semantic representation module, used to calculate the latent semantic representation of the test image; The association matrix calculation module is used to calculate the semantic association matrix H between the test image and each source image. i,t ; An image classification module is used to predict the semantic consistency between the test image and each source image using the semantic relevance prediction model g(*), and classify the test image into the category with the largest average semantic consistency; The prediction model training module uses the source image for training, and the specific process of obtaining the semantic relevance prediction model g(*) is as follows: Step 21, establishing a classification model f(*) based on a convolutional neural network, and using the source image set to train the classification model f(*), each dimension of the classification model f(*) represents the semantic consistency between the image and the corresponding class, which is called the implicit semantic representation of the image; Step 22, evenly divide a number of points along the line between the two source images, calculate the implicit semantic representations of these points, and splice them into an implicit semantic correlation matrix; Step 23, calculating the semantic consistency labels between the two source images and constructing a training set; Step 24, use the training set to perform residual neural network training to obtain a semantic relevance prediction model g(*).

14. A computer device, characterized in that: It includes a processor, a memory and a bus, the memory stores machine-readable instructions executable by the processor, when the computer device is running, the processor and the memory communicate through the bus, and the processor executes the machine-readable instructions to perform the steps of the implicit semantic association modeling method as described in any one of claims 1-6 and / or the zero-sample image classification method based on implicit semantic association as described in any one of claims 8-12.

15. A storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, executes the steps of the implicit semantic association modeling method as described in any one of claims 1 to 6 and / or the zero-sample image classification method based on implicit semantic association as described in any one of claims 8 to 12.

Images