Classification, model training, information search method, system and equipment

Abstract

The embodiments of the present application provide a classification, model training, information search method, system and device. The method includes: using the image to be classified as the input of the neural network model to obtain the classification result; the model is used to determine the classification result according to the image to be classified and the description information of multiple categories; the model is also used to extract features of sample images corresponding to multiple categories respectively to obtain sample features corresponding to multiple categories; combining the sample features corresponding to multiple categories to determine multiple semantic cluster centers; determining the description information of the first category according to the distribution information of the sample features corresponding to the first category around the multiple semantic cluster centers. The technical solution provided by the embodiments of the present application describes each category based on multiple semantic cluster centers common to multiple categories, and can describe each category in a global and generalized manner, which helps to improve the prediction accuracy.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a classification, model training, information search method, system and device. Background Art

[0002] At present, many neural network models have been widely used in various fields to solve various problems, such as image classification problems.

[0003] Some neural network models need to describe multiple categories to be identified, and then classify the images to be classified according to the description information of the multiple categories to be identified.

[0004] However, in the prior art, the prediction accuracy of these neural network models is poor, especially in small sample scenarios where training samples are extremely scarce. Summary of the invention

[0005] In view of the above problems, the present application is proposed to provide a classification, model training, information search method, system and device that solve the above problems or at least partially solve the above problems.

[0006] Therefore, in one embodiment of the present application, an image classification method is provided. The method comprises:

[0007] Get the image to be classified;

[0008] Using the image to be classified as the input of the trained neural network model, executing the neural network model, and obtaining a classification result;

[0009] The neural network model is used to determine the classification result according to the image to be classified and the description information of multiple categories;

[0010] The neural network model is also used to extract features of sample images corresponding to the multiple categories to obtain sample features corresponding to the multiple categories; determine multiple semantic cluster centers by combining the sample features corresponding to the multiple categories; determine description information of the first category based on distribution information of sample features corresponding to the first category around the multiple semantic cluster centers; wherein the first category is included in the multiple categories.

[0011] In another embodiment of the present application, a model training method is provided. The method comprises:

[0012] Using the sample image to be classified corresponding to the first sample category as input of the neural network model, executing the neural network model, and obtaining a classification prediction result;

[0013] Optimizing the neural network model according to the classification prediction result and the first sample category;

[0014] Wherein, the neural network model is used to determine the classification prediction result according to the sample image to be classified and the description information of multiple sample categories;

[0015] The neural network model is also used to extract features of sample images corresponding to the multiple sample categories to obtain sample features corresponding to the multiple sample categories; determine multiple semantic cluster centers by combining the sample features corresponding to the multiple sample categories; determine description information of the second sample category based on distribution information of sample features corresponding to the second sample category around the multiple semantic cluster centers; and include the second sample category in the multiple sample categories.

[0016] In another embodiment of the present application, an image classification method is provided. The method comprises:

[0017] Get the image to be classified;

[0018] Using the image to be classified as the input of the trained neural network model, executing the neural network model, and obtaining a classification result;

[0019] Among them, the neural network model is used to extract features of the image to be classified to obtain image features; the image features are respectively fused with description information of multiple categories to obtain multiple fused features; and the classification result is obtained by combining the multiple fused features and the trained network parameters in the neural network model.

[0020] In another embodiment of the present application, a model training method is provided. The method comprises:

[0021] Using the sample image to be classified corresponding to the first sample category as input of the neural network model, executing the neural network model, and obtaining a classification prediction result;

[0022] Optimizing the neural network model according to the classification prediction result and the first sample category;

[0023] Among them, the neural network model is used to extract features of the sample image to be classified to obtain image features; the image features are respectively fused with the description information of multiple sample categories to obtain multiple fused features; and the classification prediction result is obtained by combining the multiple fused features and the network parameters to be trained in the neural network model.

[0024] In another embodiment of the present application, a neural network system is provided. The system comprises: at least one first neural network layer, at least one second neural network layer and a computing module; wherein:

[0025] The at least one first neural network layer is used to extract features from the image to be classified to obtain image features; and is also used to extract features from sample images corresponding to multiple categories to obtain sample features corresponding to the multiple categories;

[0026] The at least one second neural network layer is used to determine a plurality of semantic cluster centers by synthesizing the sample features corresponding to the plurality of categories; and is also used to determine the description information of the first category according to the distribution information of the sample features corresponding to the first category around the plurality of semantic cluster centers; wherein the first category is one of the plurality of categories;

[0027] The calculation module is used to determine the classification result according to the image features and the description information of the multiple categories.

[0028] In another embodiment of the present application, a neural network system is provided. The system comprises: at least one first neural network layer and at least one third neural network layer; wherein:

[0029] The at least one first neural network layer is used to extract features of the image to be classified to obtain image features;

[0030] The at least one third neural network layer is used to fuse the image features with the description information of multiple categories respectively to obtain multiple fused features; and to obtain a classification result by combining the multiple fused features and the trained network parameters.

[0031] In another embodiment of the present application, a neural network system is provided. The system comprises: at least one first neural network layer, at least one third neural network layer and a computing module; wherein:

[0032] The at least one first neural network layer is used to extract features of the image to be classified to obtain image features;

[0033] The at least one third neural network layer is used to fuse the image features with the description information of multiple categories respectively to obtain multiple fused features; and obtain a first classification result according to the multiple fused features and the trained network parameters;

[0034] The calculation module is used to respectively calculate the distance between the image feature and the description information of the multiple categories; determine the second classification result according to the distance; and determine the final classification result by combining the first classification result and the second classification result.

[0035] In another embodiment of the present application, an electronic device is provided. The device includes: a memory and a processor, wherein:

[0036] The memory is used to store programs;

[0037] The processor is coupled to the memory and is configured to execute the program stored in the memory to:

[0038] Get the image to be classified;

[0039] Using the image to be classified as the input of the trained neural network model, executing the neural network model, and obtaining a classification result;

[0040] The neural network model is used to determine the classification result according to the image to be classified and the description information of multiple categories;

[0041] The neural network model is also used to extract features of sample images corresponding to the multiple categories to obtain sample features corresponding to the multiple categories; determine multiple semantic cluster centers by combining the sample features corresponding to the multiple categories; determine description information of the first category based on distribution information of sample features corresponding to the first category around the multiple semantic cluster centers; wherein the first category is included in the multiple categories.

[0042] In another embodiment of the present application, an electronic device is provided. The device includes: a memory and a processor, wherein:

[0043] The memory is used to store programs;

[0044] The processor is coupled to the memory and is configured to execute the program stored in the memory to:

[0045] Using the sample image to be classified corresponding to the first sample category as input of the neural network model, executing the neural network model, and obtaining a classification prediction result;

[0046] Optimizing the neural network model according to the classification prediction result and the first sample category;

[0047] Wherein, the neural network model is used to determine the classification prediction result according to the sample image to be classified and the description information of multiple sample categories;

[0048] The neural network model is also used to extract features of sample images corresponding to the multiple sample categories to obtain sample features corresponding to the multiple sample categories; determine multiple semantic cluster centers by combining the sample features corresponding to the multiple sample categories; describe the second sample category based on distribution information of sample features corresponding to the second sample category around the multiple semantic cluster centers; and include the second sample category in the multiple sample categories.

[0049] In another embodiment of the present application, an electronic device is provided. The device includes: a memory and a processor, wherein:

[0050] The memory is used to store programs;

[0051] The processor is coupled to the memory and is configured to execute the program stored in the memory to:

[0052] Get the image to be classified;

[0053] Using the image to be classified as the input of the trained neural network model, executing the neural network model, and obtaining a classification result;

[0054] Among them, the neural network model is used to extract features of the image to be classified to obtain image features; the image features are respectively fused with description information of multiple categories to obtain multiple fused features; and the classification result is obtained by combining the multiple fused features and the trained network parameters in the neural network model.

[0055] In another embodiment of the present application, an electronic device is provided. The device includes: a memory and a processor, wherein:

[0056] The memory is used to store programs;

[0057] The processor is coupled to the memory and is configured to execute the program stored in the memory to:

[0058] Using the sample image to be classified corresponding to the first sample category as input of the neural network model, executing the neural network model, and obtaining a classification prediction result;

[0059] Optimizing the neural network model according to the classification prediction result and the first sample category;

[0060] Among them, the neural network model is used to extract features of the sample image to be classified to obtain image features; the image features are respectively fused with the description information of multiple sample categories to obtain multiple fused features; and the classification prediction result is obtained by combining the multiple fused features and the network parameters to be trained in the neural network model.

[0061] In another embodiment of the present application, an information search method is provided. The method comprises:

[0062] Obtaining an image to be classified as a search basis;

[0063] Using the image to be classified as the input of the trained neural network model, executing the neural network model, and obtaining a classification result;

[0064] Searching to obtain search results according to the classification results;

[0065] The neural network model is used to determine the classification result according to the image to be classified and the description information of multiple categories;

[0066] The neural network model is also used to extract features of sample images corresponding to the multiple categories to obtain sample features corresponding to the multiple categories; determine multiple semantic cluster centers by combining the sample features corresponding to the multiple categories; determine description information of the first category based on distribution information of sample features corresponding to the first category around the multiple semantic cluster centers; wherein the first category is included in the multiple categories.

[0067] In another embodiment of the present application, an information search method is provided. The method comprises:

[0068] Obtaining an image to be classified as a search basis;

[0069] Using the image to be classified as the input of the trained neural network model, executing the neural network model, and obtaining a classification result;

[0070] Searching to obtain search results according to the classification results;

[0071] Among them, the neural network model is used to extract features of the image to be classified to obtain image features; the image features are respectively fused with description information of multiple categories to obtain multiple fused features; and the classification result is obtained by combining the multiple fused features and the trained network parameters in the neural network model.

[0072] In another embodiment of the present application, a data classification method is provided. The method includes: obtaining data to be classified;

[0073] Using the data to be classified as input of a trained neural network model, executing the neural network model, and obtaining a classification result;

[0074] Wherein, the neural network model is used to determine the classification result according to the data to be classified and the description information of multiple categories;

[0075] The neural network model is also used to extract features from sample data corresponding to the multiple categories to obtain sample features corresponding to the multiple categories; determine multiple semantic cluster centers by integrating the sample features corresponding to the multiple categories; determine description information of the first category based on distribution information of sample features corresponding to the first category around the multiple semantic cluster centers; wherein the first category is included in the multiple categories.

[0076] In another embodiment of the present application, a data classification method is provided. The method includes:

[0077] Obtain data to be classified;

[0078] Using the data to be classified as input of a trained neural network model, executing the neural network model, and obtaining a classification result;

[0079] Among them, the neural network model is used to extract features of the data to be classified to obtain data features; the data features are respectively fused with the description information of multiple categories to obtain multiple fused features; and the classification result is obtained by combining the multiple fused features and the trained network parameters in the neural network model.

[0080] In another embodiment of the present application, an electronic device is provided. The device includes: a memory and a processor, wherein:

[0081] The memory is used to store programs;

[0082] The processor is coupled to the memory and is configured to execute the program stored in the memory to:

[0083] Obtaining an image to be classified as a search basis;

[0084] Using the image to be classified as the input of the trained neural network model, executing the neural network model, and obtaining a classification result;

[0085] Searching to obtain search results according to the classification results;

[0086] The neural network model is used to determine the classification result according to the image to be classified and the description information of multiple categories;

[0087] The neural network model is also used to extract features of sample images corresponding to the multiple categories to obtain sample features corresponding to the multiple categories; determine multiple semantic cluster centers by combining the sample features corresponding to the multiple categories; determine description information of the first category based on distribution information of sample features corresponding to the first category around the multiple semantic cluster centers; wherein the first category is included in the multiple categories.

[0088] In another embodiment of the present application, an electronic device is provided. The device includes: a memory and a processor, wherein:

[0089] The memory is used to store programs;

[0090] The processor is coupled to the memory and is configured to execute the program stored in the memory to:

[0091] Obtaining an image to be classified as a search basis;

[0092] Using the image to be classified as the input of the trained neural network model, executing the neural network model, and obtaining a classification result;

[0093] Searching to obtain search results according to the classification results;

[0094] Among them, the neural network model is used to extract features of the image to be classified to obtain image features; the image features are respectively fused with description information of multiple categories to obtain multiple fused features; and the classification result is obtained by combining the multiple fused features and the trained network parameters in the neural network model.

[0095] In another embodiment of the present application, an electronic device is provided. The device includes: a memory and a processor, wherein:

[0096] The memory is used to store programs;

[0097] The processor is coupled to the memory and is configured to execute the program stored in the memory to:

[0098] Obtain data to be classified;

[0099] Using the data to be classified as input of a trained neural network model, executing the neural network model, and obtaining a classification result;

[0100] Wherein, the neural network model is used to determine the classification result according to the data to be classified and the description information of multiple categories;

[0101] The neural network model is also used to extract features from sample data corresponding to the multiple categories to obtain sample features corresponding to the multiple categories; determine multiple semantic cluster centers by integrating the sample features corresponding to the multiple categories; determine description information of the first category based on distribution information of sample features corresponding to the first category around the multiple semantic cluster centers; wherein the first category is included in the multiple categories.

[0102] In another embodiment of the present application, an electronic device is provided. The device includes: a memory and a processor, wherein:

[0103] The memory is used to store programs;

[0104] The processor is coupled to the memory and is configured to execute the program stored in the memory to:

[0105] Obtain data to be classified;

[0106] Using the data to be classified as input of a trained neural network model, executing the neural network model, and obtaining a classification result;

[0107] Among them, the neural network model is used to extract features of the data to be classified to obtain data features; the data features are respectively fused with the description information of multiple categories to obtain multiple fused features; and the classification result is obtained by combining the multiple fused features and the trained network parameters in the neural network model.

[0108] The technical solution provided in the embodiment of the present application integrates the sample features corresponding to each category in multiple categories to determine multiple semantic cluster centers common to multiple categories. The distribution information of the sample features corresponding to the first category around each semantic cluster center can indicate the expression of the first category on the semantic cluster center, which is the description of the first category based on the semantic cluster center. It can be seen that the technical solution provided in the embodiment of the present application describes each category separately based on the multiple semantic cluster centers shared by multiple categories, and can provide a global and generalized description of each category, which helps to improve the prediction accuracy.

[0109] The technical solution provided in the embodiment of the present application uses trainable network parameters to calculate the similarity between image features and description information of multiple categories, which can make the neural network model adapt to different sample scenarios, with greater robustness and adaptability, thereby improving the prediction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0110] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0111] Figure 1a A schematic diagram of a scene of an image classification method provided in an embodiment of the present application;

[0112] Figure 1b A schematic diagram of the internal structure and data processing flow of a neural network model provided in one embodiment of the present application;

[0113] Figure 1c A flowchart of an image classification method provided in one embodiment of the present application;

[0114] Figure 2 A flowchart of a model training method provided in another embodiment of the present application;

[0115] Figure 3A structural block diagram of an image classification device provided in one embodiment of the present application;

[0116] Figure 4 A structural block diagram of a model training device provided in another embodiment of the present application;

[0117] Figure 5 A structural block diagram of an image classification device provided in one embodiment of the present application;

[0118] Figure 6 A structural block diagram of a model training device provided in another embodiment of the present application;

[0119] Figure 7 A structural block diagram of an electronic device provided in one embodiment of the present application;

[0120] Figure 8 It is a schematic diagram of the first method of describing a class in the prior art;

[0121] Fig. 9 It is a schematic diagram of the second method of describing a class in the prior art;

[0122] Fig.10 A schematic diagram of a method for describing a class provided in an embodiment of the present application;

[0123] Fig.11 A visualization of different types of descriptive information obtained using the trained neural network model provided by this application;

[0124] Fig.12 A flowchart of a data classification method provided in accordance with an embodiment of the present application. DETAILED DESCRIPTION

[0125] In the existing technology, most neural network models need a large number of labeled samples for training to work. However, in the real world, there are many problems that do not have so much labeled data, and the cost of obtaining labeled data is also very high, such as in the medical field and security field. In the security field, there are long-tail risk images with rich categories but few in number. The number of samples of long-tail risk images is insufficient to support the training requirements of existing neural network models.

[0126] In the prior art, meta-learning technology is usually used to solve the above-mentioned sample shortage problem (i.e., small sample problem). However, there is no generalized description of the class in the prior art, which leads to weak generalization of the classifier and affects the classification effect. In the prior art, the Prototypical Networks method based on meta-learning is usually used to solve the above-mentioned small sample problem.

[0127] Among them, the prototype network based on meta-learning only needs a few sample images for each category. The prototype network maps multiple sample images corresponding to each category into a space and extracts their "mean" (such as Figure 8 As shown in the figure, the triangle, solid circle, five-pointed star, and square in the figure represent multiple features corresponding to a category; the cross in the figure represents the mean of multiple features corresponding to a category), or the maximum value (such as Fig. 9 The cross shown is the maximum value of multiple features corresponding to a category) to represent the prototype of the category, that is, as a description of the category. Use Euclidean distance or cosine distance as the distance metric, and train so that the distance between the image of this category and the prototype of this category is the closest, and the distance to the prototypes of other categories is farther. During testing, the distance between the sample image to be classified and the prototypes of each category is processed by normalized exponential softmax to determine the category label of the sample image to be classified.

[0128] It can be seen that the prototype network averages or maximizes the features of multiple sample images of each category to obtain a description of each category. However, obtaining a description of a category by averaging or maximizing the features will weaken or lose some important information, and there is no generalized description of the class, which results in a weak generalization of the classifier and affects the classification effect.

[0129] In addition, the original network uses a fixed distance standard, such as Euclidean distance or cosine distance, to judge the classification results. This distance standard is fixed and cannot adapt to training samples in different scenarios, so it has poor robustness and adaptability.

[0130] In the process of implementing the technical solution provided by the embodiment of the present application, the inventor discovered that when humans recognize an object, they usually analyze the object from multiple different aspects. For example, when recognizing a bird, they will analyze which category the bird belongs to from aspects such as the bird's head, feathers, and wings. Another example is that when humans recognize a person's emotions through speech, they usually analyze from multiple different frequencies of the speech. Based on this, the inventor proposed the present application, that is, from different aspects (i.e., the following multiple semantic clustering centers, such as Fig.10 Each cross shown in the figure is a semantic clustering center) to describe a category, then a more generalized description can be obtained.

[0131] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below according to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0132] In addition, some of the processes described in the specification, claims and the above-mentioned figures of the present application include multiple operations that appear in a specific order, and these operations may not be executed in the order in which they appear in this article or may be executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0133] Fig.12 The flowchart of the data classification method provided by an embodiment of the present application is shown. The execution subject of the method can be a client or a server. The client can be a hardware with an embedded program integrated in the terminal, or an application software installed in the terminal, or a tool software embedded in the terminal operating system, etc., which is not limited in the embodiment of the present application. The terminal can be any terminal device including a mobile phone, a tablet computer, a smart wearable device, etc. The server can be a common server, a cloud or a virtual server, etc. Fig.12 As shown, the method includes:

[0134] 1201. Obtain data to be classified.

[0135] 1202. Use the data to be classified as input to a trained neural network model, execute the neural network model, and obtain a classification result.

[0136] The neural network model is used to determine the classification result based on the data to be classified and the description information of multiple categories; the neural network model is also used to extract features of sample data corresponding to the multiple categories to obtain sample features corresponding to the multiple categories; multiple semantic cluster centers are determined by integrating the sample features corresponding to the multiple categories; the description information of the first category is determined based on the distribution information of the sample features corresponding to the first category around the multiple semantic cluster centers; the multiple categories include the first category.

[0137] In the above 1201, the data to be classified may be an image to be classified, a voice to be classified, or a video to be classified. When the execution subject of the method provided in this embodiment is a client, the above data to be classified input by the user in the client interface may be received; when the execution subject of the method provided in this embodiment is a server, the above data to be classified input by the user sent by the client may be received.

[0138] In the above 1202, the neural network model is used to determine the classification result based on the data to be classified and the description information of multiple categories. The neural network model can extract data features by performing feature extraction (for example, convolution operation) on the data to be classified; and determine the classification result based on the data features and the description information of the multiple categories. In an achievable scheme, the distance between the data features and the description information of multiple categories can be calculated, for example, Euclidean distance or cosine distance; and the classification result is determined based on the calculated distance. Specifically, the category with the smallest distance can be used as the category to which the data to be classified belongs. Among them, the calculation implementation of the Euclidean distance or the cosine distance can refer to the prior art and will not be described in detail here.

[0139] In actual application, before the neural network model classifies the data to be classified, it is necessary to obtain the description information of multiple categories that need to be identified in the application. Specifically, the neural network model is also used to extract features of the sample data corresponding to the multiple categories to obtain sample features corresponding to the multiple categories; determine multiple semantic cluster centers based on the sample features corresponding to the multiple categories; determine the description information of the first category based on the distribution information of the sample features corresponding to the first category around the multiple semantic cluster centers; wherein the multiple categories include the first category. The first category refers to any one of the multiple categories.

[0140] The number of sample data corresponding to each of the above categories can be one or more, which can be set according to actual needs. The neural network model can perform convolution operations on the sample data corresponding to each category in multiple categories through the feature extraction subnetwork to extract sample features corresponding to each category. Among them, one sample data corresponds to one sample feature, that is, a convolution operation is performed on one sample data to extract one sample feature. In addition, the feature extraction subnetwork in the neural network model can be selected from: ordinary convolutional network, VGG-16, ResNet-12, ResNet-34, etc. In the small sample problem, the number of samples is small, and the feature extraction subnetwork in the neural network model can specifically use a wide residual network, which can achieve better results.

[0141] The multiple semantic cluster centers can be understood as multiple bases constituting the feature space of a class, and the multiple bases are used in the feature space of the class to describe each of the multiple categories. The number of semantic cluster centers can be set according to actual needs. The multiple semantic cluster centers are all common semantic cluster centers of the multiple categories.

[0142] In practical applications, sample features are in the form of feature graphs, which can be expressed as a D*H*W structure. H*W represent the length and width of the feature graph, respectively, and D represents the depth of the feature graph (that is, the number of channels of the feature graph). Among them, D, H, and W are greater than or equal to 1. In addition, the semantic cluster center is also expressed in the form of a feature graph, that is, expressed as a D*H*W structure. It should be noted that when the data to be classified is speech to be classified, H and W are both 1.

[0143] In an achievable solution, the neural network model can directly perform convolution operations on the sample features corresponding to each of the multiple categories to obtain multiple semantic cluster centers common to the multiple categories. Specifically, the sample features corresponding to the multiple categories can be combined together by channel and input as input data into at least one convolution layer in the neural network model to obtain multiple semantic cluster centers common to the multiple categories. The network parameters in the at least one convolution layer (for example, the parameters in the convolution kernel) are all trained when training the neural network model. Obtaining multiple semantic cluster centers through convolution operation learning can effectively reduce the amount of calculation and save computing time.

[0144] In another feasible solution, the above-mentioned “combining the sample features corresponding to the multiple categories to determine multiple semantic cluster centers” can be implemented by the following steps:

[0145] S210, respectively calculating the mean features of the sample features corresponding to each category in the multiple categories.

[0146] S211, combining the mean features corresponding to each category in the multiple categories by channel to obtain a combined mean feature;

[0147] S212: performing convolution processing on the combined mean feature to obtain a plurality of semantic cluster centers common to the plurality of categories.

[0148] In the above S210, the above multiple categories include the first category. The number of sample features corresponding to the first category is multiple; the multiple sample features corresponding to the first category are element-wise added (Element-wise Addition) to obtain samples and features; each element in the samples and features is divided by the number of the multiple sample features corresponding to the first category to obtain the mean feature corresponding to the first category. Element-wise addition means adding the elements at the same position in the multiple sample features corresponding to the first category, that is, adding the elements with the same subscript.

[0149] In the above S211 and S212, the mean features corresponding to each category in the multiple categories are combined together by channel and input into at least one convolution layer in the neural network model as input data to obtain multiple semantic cluster centers common to the multiple categories. The network parameters in the at least one convolution layer (for example, the parameters in the convolution kernel) are all trained when training the neural network model.

[0150] The technical solution provided in the embodiment of the present application integrates the sample features corresponding to each category in multiple categories to determine multiple semantic cluster centers common to multiple categories. The distribution information of the sample features corresponding to the first category around each semantic cluster center can indicate the expression of the first category on the semantic cluster center, which is also the description of the first category based on the semantic cluster center. It can be seen that the technical solution provided in the embodiment of the present application describes each category separately based on multiple shared semantic cluster centers, and can provide a global and generalized description of each category, which helps to improve the prediction accuracy.

[0151] The description information of the first category is determined based on the distribution information of the sample features corresponding to the first category around the multiple semantic cluster centers, that is, the description information of the first category is determined based on the distribution information of the sample features corresponding to the first category around each semantic cluster center in the multiple semantic cluster centers.

[0152] In one example, the above “determining the description information of the first category according to the distribution information of the sample features corresponding to the first category around the centers of the multiple semantic clusters” can be implemented by the following steps:

[0153] S221 . Determine sub-description information of the first category based on the first semantic cluster center according to distribution information of sample features corresponding to the first category around the first semantic cluster center.

[0154] S222: Determine description information of the first category based on sub-description information of the first category by each semantic cluster center in the multiple semantic cluster centers.

[0155] The plurality of semantic cluster centers include the first semantic cluster center. It should be noted that the first semantic cluster center refers to any one of the plurality of semantic cluster centers.

[0156] In the above S221, the distribution information of the sample features corresponding to the first category around the center of the first semantic cluster indicates the expression of the sample features corresponding to the first category on the center of the first semantic cluster, which is the sub-description information of the first category based on the center of the first semantic cluster. For ease of understanding, taking the bird recognition scene as an example, assuming that the center of the first semantic cluster refers to the bird's head, the distribution information of the sample features corresponding to the first category around the semantic cluster center of the bird's head can indicate the specific shape, size, color and other detailed information of the bird's head.

[0157] In an achievable solution, the residual between each sample feature in the sample features corresponding to the first category and the center of the first semantic cluster can be used as the distribution information of the sample features corresponding to the first category around the center of the first semantic cluster. That is, the distribution information of the sample features corresponding to the first category around the center of the first semantic cluster includes the residual between each sample feature in the sample features corresponding to the first category and the center of the first semantic cluster.

[0158] During specific implementation, the above neural network model is also used to calculate the residual between each sample feature in the sample features corresponding to the first category and the first semantic cluster center. That is, the sample feature is subtracted from the first semantic cluster center element-wise to obtain the residual. Element-wise subtraction means subtracting the second element at the same position in the first semantic cluster center from the first element at a certain position in the sample feature, wherein the subscripts of the first element and the second element are the same. It should be noted that the residual is also expressed in the form of a feature graph of a D*H*W structure.

[0159] In the above S221, the step of "determining sub-description information of the first category based on the first semantic cluster center according to the distribution information of the sample features corresponding to the first category around the first semantic cluster center" can be implemented in one or more of the following ways:

[0160] Method 1: summing the residuals between each sample feature in the sample features corresponding to the first category and the first semantic cluster center to obtain sub-description information of the first category based on the first semantic cluster center.

[0161] Specifically, multiple residuals corresponding to the first category are added element by element to obtain a residual sum, and the residual sum is used as sub-description information of the first category based on the center of the first semantic cluster.

[0162] In the above method 1, it is assumed that each sample feature corresponding to the first category contributes the same degree to the sub-description information. However, in actual applications, the inventors found that the closer the sample features are to the center of the semantic cluster, the more they can reflect the expression of the corresponding category on the center of the semantic cluster. Therefore, the following method 2 can be used to achieve this.

[0163] Method 2: Calculate the allocation weight corresponding to each sample feature based on the distance between each sample feature in the sample features corresponding to the first category and the first semantic cluster center; and aggregate to obtain sub-description information of the first category based on the first semantic cluster center based on the allocation weight and the residual between each sample feature in the sample features corresponding to the first category and the first semantic cluster center.

[0164] Among them, the distance between each sample feature and the center of the first semantic cluster can be a Euclidean distance. The distance between each sample feature in the sample features corresponding to the first category and the center of the first semantic cluster can be summed to obtain the sum of the distances. The sample features corresponding to the first category include the first sample feature, and the ratio of the distance between the first sample feature and the center of the first semantic cluster to the sum of the distances is the allocation weight corresponding to the first sample feature. For example: the distance between the first sample feature and the center of the first semantic cluster is 2, and the sum of the distances is 10, then the allocation weight is 0.2.

[0165] Among them, the aggregation operation is specifically: multiplying each element of the residual between the first sample feature and the first semantic cluster center by the assigned weight corresponding to the first sample feature to obtain the weighted residual corresponding to the first sample feature; adding the weighted residuals corresponding to each sample feature in the multiple sample features corresponding to the first category element by element to obtain sub-description information of the first category based on the first semantic cluster center.

[0166] For example: the first category corresponds to two sample features: feature map A and feature map B. The residual between feature map A and the center of the first semantic cluster is feature map R, and the residual between feature map B and the center of the first semantic cluster is feature map T. The sub-description information of the first category based on the center of the first semantic cluster is feature map G. The allocation weight corresponding to feature map A is α, and the allocation weight corresponding to feature map B is β. Among them, feature maps A, B, R, T and G are all D*H*W structures.

[0167] The elements in the feature map R are denoted as r dhw , where the subscript d represents the dth channel (i.e., depth) of the element in the feature map R, and the subscripts h and w represent the hth row and wth column of the dth page in the feature map R. The element in the feature map T is represented by t dhw , where the subscript d represents the dth channel (i.e., depth) of the element in the feature map T, and the subscripts h and w represent the hth row and wth column of the dth page in the feature map T. The element in the feature map G is represented by g dhw , where the subscript d represents the element is located in the dth channel (i.e., depth) of the feature map G, and the subscripts h and w represent the element is located in the hth row and wth column of the dth channel of the feature map G. dhw It can be calculated using the following formula (1):

[0168] g dhw =α*r dhw +β*t dhw (1)

[0169] In the above S222, the sub-description information of the first category based on each of the multiple semantic cluster centers can be combined by channel to obtain combined sub-description information; and the combined sub-description information is used as the description information of the first category. Assuming that the number of semantic cluster centers is n, and the sub-description information is a D*H*W structure (i.e., the number of channels of the sub-description information is D), the combined sub-description information is an nD*H*W structure (i.e., the number of channels of the combined sub-description information is nD).

[0170] In practical applications, since the data features corresponding to the above-mentioned data to be classified are also of D*H*W structure, that is, the depth of the data features is also D. In order to facilitate the comparison of data features with description information of multiple categories at the same scale. Therefore, the above-mentioned S222 "comprehensively based on the sub-description information of the first category by each semantic cluster center in the multiple semantic cluster centers, determine the description information of the first category", can be implemented in the following steps:

[0171] A21. Combining the sub-description information of the first category based on each semantic cluster center in the multiple semantic cluster centers by channel to obtain combined sub-description information.

[0172] A22. Perform dimensionality reduction processing on the combined sub-description information to obtain description information of the first category.

[0173] The depth of the description information of the first category obtained after dimensionality reduction is the same as the depth of the data feature, and the width and height are also the same. Specifically, the dimensionality reduction can be achieved by convolution. Of course, other methods can also be used, and this embodiment does not specifically limit this.

[0174] In another example, the above “determining the classification result according to the data to be classified and the description information of the multiple categories” can be implemented by the following steps:

[0175] S230a, extracting features from the data to be classified to obtain data features.

[0176] S231a. Fusing the data features with description information of multiple categories respectively to obtain multiple fused features.

[0177] S232a. Combining the multiple fusion features and the trained network parameters in the neural network model, obtain the classification result.

[0178] For the above S230a, reference may be made to the corresponding contents in the above embodiments, which will not be described in detail here.

[0179] In the above S231a, the multiple categories include the first category. The data feature and the description information of the first category may be element-wise added to obtain the fused feature; or the data feature and the description information of the first category may be element-wise multiplied (element-wise product) to obtain the fused feature.

[0180] In specific implementation, the fusion method between the data features and the description information of the first category can also be realized by other methods other than the two methods mentioned above, which are not specifically limited here. In the above S232a, the neural network model includes at least one neural network layer based on the trained network parameters; the fused features are used as the input of the at least one neural network layer, and the at least one neural network layer predicts the similarity between the data features and the description information of multiple categories respectively, so as to obtain the classification result.

[0181] Specifically, the trained network parameters may include: at least one first network parameter and at least one second network parameter. The neural network model includes at least one convolutional layer based on the at least one first network parameter and at least one fully connected layer based on the at least one second network parameter (that is, the at least one neural network layer includes at least one convolutional layer and at least one fully connected layer). The multiple fused features may be input into the at least one convolutional layer to obtain post-convolution features; the post-convolution features may be input into the at least one fully connected layer to obtain the classification result.

[0182] In this embodiment, the classification result is obtained by using the trainable network parameters in the neural network model, that is, the trainable network parameters in the neural network model are used to determine which category's description information the data feature is most similar to, so as to replace the method of using a fixed distance standard to judge the similarity in the prior art. This trainable network parameter can adapt to different sample scenarios and has stronger robustness and adaptability.

[0183] In order to further improve the prediction accuracy of the neural network model, the above network parameters and a fixed distance standard can be combined to determine the classification result. Specifically, the above S232a can be implemented by the following steps:

[0184] b21. Obtain a first classification result based on the multiple fusion features and the trained network parameters in the neural network model.

[0185] b22. Calculate the distances between the data feature and the description information of the multiple categories respectively.

[0186] b23. Determine a second classification result based on the distance.

[0187] b24. Determine the classification result by combining the first classification result and the second classification result.

[0188] In the above b21, the trained network parameters include: at least one first network parameter and at least one second network parameter; the neural network model includes at least one convolutional layer based on the at least one first network parameter and at least one fully connected layer based on the at least one second network parameter; the multiple fused features are input into the at least one convolutional layer to obtain convolutional features; the convolutional features are input into the at least one fully connected layer to obtain the first classification result.

[0189] For b22 and b23, reference may be made to the corresponding contents in the above embodiment and will not be repeated here.

[0190] In the above b24, the multiple categories include the second category; the second category refers to any one of the above multiple categories. The first classification result includes the first probability q1 that the data to be classified belongs to the second category; the second classification result includes the second probability q2 that the data to be classified belongs to the second category; the classification result includes the third probability q that the data to be classified belongs to the second category. The above b4 is specifically: combining the first probability q1 and the second probability q2 to determine the third probability q3, specifically, q3 = λ1*q1+λ2*q2, wherein the specific values ​​of the weight λ1 and the weight λ2 can be set according to actual needs and are not specifically limited here.

[0191] In order to solve the small sample problem, the above neural network model can be based on meta-learning technology. The neural network model based on meta-learning technology is trained according to multiple different meta-tasks. A support set and a query set can be randomly sampled from a training set to train the neural network model as a unary task. For example: the query set includes N sample categories, each sample category has M sample data, that is, the N-way-M-shot mode; accordingly, the query set should also include these N sample categories, and each sample category includes M sample data that do not intersect with the query set. Multiple meta-tasks can be obtained through multiple random sampling to train the neural network model, for example: sampling 600 times. Among them, N is greater than 1, and M is greater than or equal to 1.

[0192] The above multiple categories refer to the multiple categories that need to be identified when the trained neural network model based on meta-learning technology is applied. Usually, the above multiple categories are not involved in the training process.

[0193] In this way, there is a more generalized global description of the class, which can effectively improve the prediction accuracy in small sample scenarios.

[0194] In the embodiment of the present application, multiple common semantic cluster centers are used to generalize the features of the description class, and these semantic cluster centers are used to construct the feature space of the class, and the feature space of the class is used for classification, which can effectively improve the classification efficiency. In addition, the use of trainable network parameters to calculate the similarity between data features and description information of multiple categories can make the neural network model adapt to different sample scenarios, and it is more robust and adaptable.

[0195] In one example, the training process of the neural network model is as follows: the sample data to be classified corresponding to the first sample category is used as the input of the neural network model, and the neural network model is executed to obtain a classification prediction result. The neural network model is optimized according to the classification prediction result and the first sample category. The neural network model is used to determine the classification prediction result according to the sample data to be classified and the description information of multiple sample categories. The neural network model is also used to extract features from the sample data corresponding to the multiple sample categories to obtain sample features corresponding to the multiple sample categories; to determine multiple semantic cluster centers by combining the sample features corresponding to the multiple sample categories; to determine the description information of the second sample category according to the distribution information of the sample features corresponding to the second sample category around the multiple semantic cluster centers; and the multiple sample categories include the second sample category. The second sample category refers to any one of the multiple sample categories.

[0196] In the above “optimizing the neural network model according to the classification prediction result and the first sample category”, a loss function may be used to calculate a loss value according to the classification prediction result and the first sample category; and parameters of the neural network model may be optimized according to the loss value. For example, cross-entropy loss may be used as a loss function to calculate the loss value. In addition, regularization may be added to regulate the training of network parameters, so as to fine-tune the network parameters in the neural network model. Regularization can prevent overfitting.

[0197] Among them, the specific implementation of the above-mentioned "determining the classification prediction result based on the description information of the sample data to be classified and multiple sample categories" can refer to the corresponding content of "determining the classification result based on the description information of the data to be classified and multiple categories" in the above-mentioned embodiments, which will not be repeated here.

[0198] The specific implementation of the above-mentioned "performing feature extraction on the sample data corresponding to the multiple sample categories to obtain sample features corresponding to the multiple sample categories" can refer to the corresponding content of "performing feature extraction on the sample data corresponding to the multiple categories to obtain sample features corresponding to the multiple categories" in the above-mentioned embodiments.

[0199] The specific implementation of the above “combining the sample features corresponding to the multiple sample categories to determine the multiple semantic cluster centers” may refer to the corresponding content of “combining the sample features corresponding to the multiple categories to determine the multiple semantic cluster centers” in the above embodiments.

[0200] The specific implementation of the above-mentioned "determining the description information of the second sample category based on the distribution information of the sample features corresponding to the second sample category around the centers of the multiple semantic clusters" can refer to the corresponding content of "determining the description information of the first category based on the distribution information of the sample features corresponding to the first category around the centers of the multiple semantic clusters" in the above-mentioned embodiments.

[0201] It should be noted that the forward calculation process of the above-mentioned neural network model in the training stage is the same as that in the application stage. Therefore, the data processing flow in the training stage can be specifically referred to the corresponding contents in the above-mentioned embodiments, which will not be repeated here.

[0202] Figure 1c The flowchart of the image classification method provided by an embodiment of the present application is shown. The execution subject of the method can be a client or a server. The client can be a hardware with an embedded program integrated on the terminal, or an application software installed in the terminal, or a tool software embedded in the terminal operating system, etc., which is not limited in the embodiment of the present application. The terminal can be any terminal device including a mobile phone, a tablet computer, a smart wearable device, etc. The server can be a common server, a cloud or a virtual server, etc. Figure 1c As shown, the method includes:

[0203] 101. Obtain an image to be classified.

[0204] 102. Use the image to be classified as input of a trained neural network model, execute the neural network model, and obtain a classification result.

[0205] The neural network model is used to determine the classification result according to the image to be classified and the description information of multiple categories;

[0206] The neural network model is also used to extract features of sample images corresponding to the multiple categories to obtain sample features corresponding to the multiple categories; determine multiple semantic cluster centers by combining the sample features corresponding to the multiple categories; determine description information of the first category based on distribution information of sample features corresponding to the first category around the multiple semantic cluster centers; wherein the first category is included in the multiple categories.

[0207] In this embodiment, the data to be classified in the above embodiments are specifically images to be classified; accordingly, the data features in the above embodiments are specifically image features; and the sample data in the above embodiments are specifically sample images.

[0208] The specific implementation of the above 101 and 102 can refer to the corresponding content in the above embodiments, which will not be repeated here.

[0209] In one feasible solution, the above “combining the sample features corresponding to the multiple categories to determine multiple semantic cluster centers” can be implemented by the following steps:

[0210] S10. Calculate the mean features of the sample features corresponding to each category in the multiple categories respectively.

[0211] S11, combining the mean features corresponding to each category in the multiple categories by channel to obtain a combined mean feature;

[0212] S12. Perform convolution processing on the combined mean feature to obtain multiple semantic cluster centers common to the multiple categories.

[0213] The specific implementation of the above S10, S11 and S12 can refer to the corresponding content in the above embodiments, which will not be repeated here.

[0214] The technical solution provided in the embodiment of the present application integrates the sample features corresponding to each category in multiple categories to determine multiple semantic cluster centers common to multiple categories. The distribution information of the sample features corresponding to the first category around each semantic cluster center can indicate the expression of the first category on the semantic cluster center, which is the description of the first category based on the semantic cluster center. It can be seen that the technical solution provided in the embodiment of the present application describes each category separately based on multiple shared semantic cluster centers, and can provide a global and generalized description of each category, which helps to improve the prediction accuracy.

[0215] In one example, the above “determining the description information of the first category according to the distribution information of the sample features corresponding to the first category around the centers of the multiple semantic clusters” can be implemented by the following steps:

[0216] S21. Determine sub-description information of the first category based on the first semantic cluster center according to distribution information of sample features corresponding to the first category around the first semantic cluster center.

[0217] S22: Determine description information of the first category based on the sub-description information of the first category by each semantic cluster center in the multiple semantic cluster centers.

[0218] The plurality of semantic cluster centers include the first semantic cluster center. It should be noted that the first semantic cluster center refers to any one of the plurality of semantic cluster centers.

[0219] The specific implementation of the above S21 and S22 can refer to the corresponding content in the above embodiments, which will not be repeated here.

[0220] In an achievable solution, the residual between each sample feature in the sample features corresponding to the first category and the center of the first semantic cluster can be used as the distribution information of the sample features corresponding to the first category around the center of the first semantic cluster. That is, the distribution information of the sample features corresponding to the first category around the center of the first semantic cluster includes the residual between each sample feature in the sample features corresponding to the first category and the center of the first semantic cluster.

[0221] During specific implementation, the above neural network model is also used to calculate the residual between each sample feature in the sample features corresponding to the first category and the first semantic cluster center. That is, the sample feature is subtracted from the first semantic cluster center element by element to obtain the residual. Element-wise subtraction means subtracting the second element at the same position in the first semantic cluster center from the first element at a certain position in the sample feature, wherein the subscripts of the first element and the second element are the same. It should be noted that the residual is also expressed in the form of a feature graph of a D*H*W structure.

[0222] In the above S21, the step of "determining sub-description information of the first category based on the first semantic cluster center according to the distribution information of the sample features corresponding to the first category around the first semantic cluster center" can be implemented in one or more of the following ways:

[0223] Method 1: summing the residuals between each sample feature in the sample features corresponding to the first category and the first semantic cluster center to obtain sub-description information of the first category based on the first semantic cluster center.

[0224] Specifically, multiple residuals corresponding to the first category are added element by element to obtain a residual sum, and the residual sum is used as sub-description information of the first category based on the center of the first semantic cluster.

[0225] In the above method 1, it is assumed that each sample feature corresponding to the first category contributes the same degree to the sub-description information. However, in actual applications, the inventors found that the closer the sample features are to the center of the semantic cluster, the more they can reflect the expression of the corresponding category on the center of the semantic cluster. Therefore, the following method 2 can be used to achieve this.

[0226] Method 2: Calculate the allocation weight corresponding to each sample feature based on the distance between each sample feature in the sample features corresponding to the first category and the first semantic cluster center; and aggregate to obtain sub-description information of the first category based on the first semantic cluster center based on the allocation weight and the residual between each sample feature in the sample features corresponding to the first category and the first semantic cluster center.

[0227] In practical applications, since the image features corresponding to the above-mentioned image to be classified are also of D*H*W structure, that is, the depth of the image features is also D. In order to facilitate the comparison of image features and description information of multiple categories at the same scale. Therefore, the above-mentioned S22 "comprehensively based on the sub-description information of the first category by each semantic cluster center in the multiple semantic cluster centers, determine the description information of the first category", can be implemented in the following steps:

[0228] a1. Combining the sub-description information of the first category based on each semantic cluster center in the multiple semantic cluster centers by channel to obtain combined sub-description information.

[0229] a2. Perform dimensionality reduction processing on the combined sub-description information to obtain the description information of the first category.

[0230] The depth of the description information of the first category obtained after dimensionality reduction is the same as the depth of the image feature, and the width and height are also the same.

[0231] The inventors have verified through experimental comparisons in small sample scenarios that the prediction accuracy of the neural network model provided by this embodiment is higher than that of the neural network model in the prior art based on the same data set and the same N-way-M-shot mode. Taking the dataset MiniImageNet as an example, under the same 5-way-1-shot mode, the accuracy of the neural network model with the highest accuracy in the prior art is (61.76±0.08)%, and the accuracy of the neural network model provided by this application is (62.97±0.16)%; under the same 5-way-5-shot mode, the accuracy of the neural network model with the highest accuracy in the prior art is (77.59±0.12)%, and the accuracy of the neural network model provided by this application is (78.87±0.15)%. Taking the dataset CUB as an example, under the same 5-way-1-shot mode, the accuracy of the neural network model with the highest accuracy in the prior art is (63.52±0.25)%, and the accuracy of the neural network model provided in the present application is (66.80±0.22)%; under the same 5-way-5-shot mode, the accuracy of the neural network model with the highest accuracy in the prior art is (79.06±0.20)%, and the accuracy of the neural network model provided in the present application is (82.0±0.15)%.

[0232] In another example, the above “determining the classification result according to the image to be classified and the description information of the multiple categories” can be implemented by the following steps:

[0233] S30a, extracting features of the image to be classified to obtain image features.

[0234] S31a, fusing the image features with description information of multiple categories respectively to obtain multiple fused features.

[0235] S32a, combining the multiple fusion features and the trained network parameters in the neural network model to obtain the classification result.

[0236] The specific implementation of the above S30a, S31a and S32a can refer to the corresponding content in the above embodiments, which will not be repeated here.

[0237] In this embodiment, the classification result is obtained by using the trainable network parameters in the neural network model, that is, the trainable network parameters in the neural network model are used to determine which category of description information the image feature is most similar to, so as to replace the method of using a fixed distance standard to judge the similarity in the prior art. This trainable network parameter can adapt to different sample scenarios and has stronger robustness and adaptability.

[0238] In order to further improve the prediction accuracy of the neural network model, the above network parameters and a fixed distance standard can be combined to determine the classification result. Specifically, the above S32a can be implemented by the following steps:

[0239] b1. Obtain a first classification result based on the multiple fusion features and the trained network parameters in the neural network model.

[0240] b2. Calculate the distances between the image features and the description information of the multiple categories respectively.

[0241] b3. Determine a second classification result based on the distance.

[0242] b4. Determine the classification result by combining the first classification result and the second classification result.

[0243] For the above b1, b2, b3 and b4, reference may be made to the corresponding contents in the above embodiments and will not be repeated here.

[0244] The inventors have verified through experimental comparisons in small sample scenarios that the prediction accuracy of the neural network model provided by this embodiment is higher than that of the neural network model in the prior art based on the same data set and the same N-way-M-shot mode. Taking the dataset MiniImageNet as an example, under the same 5-way-1-shot mode, the accuracy of the neural network model with the highest accuracy in the prior art is (61.76±0.08)%, and the accuracy of the neural network model provided by this application is (63.37±0.14)%; under the same 5-way-5-shot mode, the accuracy of the neural network model with the highest accuracy in the prior art is (77.59±0.12)%, and the accuracy of the neural network model provided by this application is (79.03±0.16)%. Taking the dataset CUB as an example, under the same 5-way-1-shot mode, the accuracy of the neural network model with the highest accuracy in the prior art is (63.52±0.25)%, and the accuracy of the neural network model provided in the present application is (67.85±0.24)%; under the same 5-way-5-shot mode, the accuracy of the neural network model with the highest accuracy in the prior art is (79.06±0.20)%, and the accuracy of the neural network model provided in the present application is (83.53±0.16)%.

[0245] In order to solve the small sample problem, the above neural network model can be based on meta-learning technology. The neural network model based on meta-learning technology is trained according to multiple different meta-tasks. A support set and a query set can be randomly sampled from a training set to train the neural network model as a unary task. For example: the query set includes N sample categories, each sample category has M sample images, that is, the N-way-M-shot mode; accordingly, the query set should also include these N sample categories, and each sample category includes M sample images that do not intersect with the query set. Multiple meta-tasks can be obtained through multiple random sampling to train the neural network model, for example: sampling 600 times. Among them, N is greater than 1, and M is greater than or equal to 1.

[0246] The above multiple categories refer to the multiple categories that need to be identified when the trained neural network model based on meta-learning technology is applied. Usually, the above multiple categories are not involved in the training process.

[0247] In this way, there is a more generalized global description of the class, which can effectively improve the prediction accuracy in small sample scenarios.

[0248] In the embodiment of the present application, multiple common semantic cluster centers are used to generalize the features of the description class, and these semantic cluster centers are used to construct the feature space of the class, and the feature space of the class is used for classification, which can effectively improve the classification efficiency. In addition, the use of trainable network parameters to calculate the similarity between image features and description information of multiple categories can make the neural network model adapt to different sample scenarios, with stronger robustness and adaptability, thereby improving the prediction accuracy.

[0249] It should be noted here that: for the contents of each step not fully described in detail in the method provided in the embodiment of the present application, please refer to the corresponding contents in the above embodiment, which will not be repeated here. In addition, in addition to the above steps, the method provided in the embodiment of the present application may also include other parts or all of the steps in the above embodiments, which can be specifically referred to the corresponding contents in the above embodiments, which will not be repeated here.

[0250] Figure 2 FIG. 2 is a flow chart showing a training method for the neural network model provided by another embodiment of the present application. Figure 2 As shown, the method comprises:

[0251] 201. Using the sample image to be classified corresponding to the first sample category as input of a neural network model, executing the neural network model, and obtaining a classification prediction result.

[0252] 202. Optimize the neural network model according to the classification prediction result and the first sample category.

[0253] The neural network model is used to determine the classification prediction result according to the sample image to be classified and the description information of multiple sample categories. The neural network model can extract image features by performing feature extraction (e.g., convolution operation) on the sample image to be classified; and determine the classification prediction result according to the image features and the description information of multiple sample categories. For determining the classification prediction result according to the image features and the description information of multiple sample categories, reference can be made to the corresponding content of "determining the classification result according to the image features and the description information of multiple categories" in the above embodiment, which will not be repeated here.

[0254] The neural network model is also used to extract features from sample images corresponding to the multiple sample categories to obtain sample features corresponding to the multiple sample categories; determine multiple semantic cluster centers by combining the sample features corresponding to the multiple sample categories; determine description information of the second sample category based on distribution information of sample features corresponding to the second sample category around the multiple semantic cluster centers; the multiple sample categories include the second sample category. The second sample category refers to any one of the multiple sample categories.

[0255] The number of sample images corresponding to each of the above sample categories can be one or more, which can be set according to actual needs. The neural network model can perform convolution operations on sample images corresponding to each of the multiple sample categories through a feature extraction subnetwork to extract sample features corresponding to each sample category. Among them, one sample image corresponds to one sample feature, that is, a convolution operation is performed on one sample image to extract one sample feature. In the small sample problem, the number of samples is small, and the feature extraction subnetwork in the neural network model can specifically use a wide residual network to achieve better results.

[0256] The multiple semantic cluster centers can be understood as multiple bases constituting the feature space of a class, and the multiple bases are used in the feature space of the class to describe each sample category in the multiple sample categories. The number of semantic cluster centers can be set according to actual needs.

[0257] Among them, the specific implementation of "combining the sample features corresponding to the multiple sample categories to determine multiple semantic cluster centers" can refer to the corresponding content of "combining the sample features corresponding to the multiple categories to determine multiple semantic cluster centers" in the above embodiments, which will not be repeated here.

[0258] According to the distribution information of the sample features corresponding to the second sample category around the multiple semantic cluster centers, the description information of the second sample category is determined, that is, according to the distribution information of the sample features corresponding to the second sample category around each semantic cluster center among the multiple semantic cluster centers, the description information of the second sample category is determined. The specific implementation of this step can refer to the corresponding content of "determining the description information of the first category according to the distribution information of the sample features corresponding to the first category around each semantic cluster center among the multiple semantic cluster centers" in the above-mentioned embodiments, which will not be repeated here.

[0259] In the above 202, a loss function may be used to calculate a loss value based on the classification prediction result and the first sample category. For example, a cross-entropy loss may be used as a loss function to calculate the loss value. The neural network model parameters are optimized based on the loss value. In addition, regularization may be added to regulate the training of network parameters, so as to fine-tune the network parameters in the neural network model. Regularization can prevent overfitting.

[0260] The technical solution provided in the embodiment of the present application integrates the sample features corresponding to each category in multiple categories to determine multiple semantic cluster centers common to multiple categories. The distribution information of the sample features corresponding to the first category around each semantic cluster center can indicate the expression of the first category on the semantic cluster center, which is the description of the first category based on the semantic cluster center. It can be seen that the technical solution provided in the embodiment of the present application describes each category separately based on multiple shared semantic cluster centers, and can provide a global and generalized description of each category, which helps to improve the prediction accuracy.

[0261] In practical applications, before using the meta-task to train the neural network model, the data in the training set of the above embodiment can be selected to pre-train the feature extraction sub-network. In this way, a relatively robust pre-training model can be obtained, which helps to improve the training speed and classification accuracy.

[0262] Furthermore, the above-mentioned “determining the classification prediction result according to the description information of the sample image to be classified and the multiple sample categories” can be implemented by the following steps:

[0263] S40: extracting features of the sample image to be classified to obtain image features.

[0264] The specific implementation of the above S40 may refer to the corresponding content of “extracting features from the image to be classified” in the above embodiments, which will not be described in detail here.

[0265] S41, fusing the image features with description information of multiple sample categories respectively to obtain multiple fused features.

[0266] The specific implementation of the above S41 can refer to the corresponding content of "merging the image features with the description information of multiple categories respectively" in the above embodiments, which will not be repeated here.

[0267] S42. Combining the multiple fusion features and the network parameters to be trained in the neural network model, obtain the classification prediction result.

[0268] The specific implementation of the above S42 can refer to the corresponding content of "combining the multiple fusion features and the trained network parameters in the neural network model to obtain the classification prediction result" in the above embodiments, which will not be repeated here.

[0269] Furthermore, in the above S42, “combining the multiple fusion features and the network parameters to be trained in the neural network model to obtain the classification prediction result” can be specifically implemented by the following steps:

[0270] c1. Obtain a first classification prediction result based on the multiple fusion features and the network parameters to be trained in the neural network model.

[0271] c2. Calculate the distances between the image features and the description information of the multiple sample categories respectively.

[0272] c3. Determine a second classification prediction result based on the distance.

[0273] c4. Determine the classification prediction result by combining the first classification prediction result and the second classification prediction result.

[0274] The above steps c1, c2, c3 and c4 may be specifically referred to the corresponding contents in the above embodiments, which will not be described in detail here.

[0275] In practical applications, in order to solve the small sample problem, the above neural network model can be based on meta-learning technology. The neural network model based on meta-learning technology is trained according to multiple different meta-tasks. A support set and a query set can be randomly sampled from a training set to train the neural network model as a unary task. For example: the query set includes N sample categories, each sample category has M sample images, that is, N-way-M-shot; accordingly, the query set should also include these N sample categories, and each sample category includes M sample images that do not intersect with the query set. Multiple meta-tasks can be obtained through multiple random sampling to train the neural network model, for example: sampling 600 times. Among them, N is greater than 1, and M is greater than or equal to 1.

[0276] The sample image to be classified corresponding to the first sample category in 201 is a sample image in the query set. The sample images corresponding to the multiple sample categories are all sample images in the support set.

[0277] In the above 202, in practical applications, the network parameters in the neural network model are usually adjusted once based on a meta-task. Therefore, in the above 201, all the sample images to be classified in the query set can be input into the neural network model to obtain the classification prediction result corresponding to each sample image to be classified.

[0278] The neural network model is optimized by combining the classification prediction result and the real category corresponding to each sample image to be classified. For example, the difference (i.e., loss value) between the classification prediction result and the real category corresponding to each sample image to be classified can be calculated respectively, and the neural network model is optimized according to the sum of the differences.

[0279] Of course, the network parameters in the neural network model may also be adjusted once according to each image to be classified in the query set, and this embodiment does not specifically limit this.

[0280] In this way, there is a more generalized global description of the class, which can effectively improve the prediction accuracy in small sample scenarios.

[0281] The following is an introduction to the image classification method provided by the embodiment of the present application in combination with actual business scenarios:

[0282] like Figure 1a As shown, assuming that the trained neural network model needs to be put into the image classification scene of category X (for example, flamingo) and category Y (for example, sparrow), it is necessary to first obtain a small number of sample images of category X and a small number of sample images of category Y. On the server side, based on the sample images of category X and the sample images of category Y, multiple semantic cluster centers are determined using the methods described in the above embodiments, for example: Figure 1a The semantic cluster centers of the bird head, the bird wings and the bird feet in the image are obtained; and based on these multiple semantic cluster centers, the description information of category X and category Y is obtained for backup. When the server receives the image to be classified sent by the client and input by the user, the image features are extracted through the feature extraction subnetwork in the neural network model; the image features are respectively fused with the description information of category X and category Y through the fusion subnetwork in the neural network model to obtain fusion features; the fusion features are input into the classifier in the neural network model to classify the image to be classified, and the classification results are returned to the client for output display.

[0283] Of course, when the computing power of the terminal where the client is located is sufficient, the above processing flows can be implemented by the client. Figure 1b The internal structure and internal processing flow of the neural network model provided by the embodiment of the present application are shown. Among them, the feature extraction subnetwork is used to extract features from sample images corresponding to multiple sample categories in the support set and sample images to be classified in the query set; the feature extraction subnetwork inputs sample images corresponding to multiple sample categories in the support set into the encoder, and the encoder outputs the combined sub-description information D corresponding to the i-th sample category i , the combined sub-description information D corresponding to the i-th sample category i Input to the decoder, the decoder outputs the description information C of the i-th sample category i . The image features of the sample images to be classified in the query set output by the feature extraction subnetwork are fused with the description information of the K sample categories to obtain multiple fused features. The multiple fused features are input into the similarity subnetwork to obtain the first classification prediction result. The Euclidean distance between the image features of the sample images to be classified in the query set output by the feature extraction subnetwork and the K sample categories is calculated; the second classification prediction result is obtained according to the Euclidean distance. The loss value is determined according to the first classification prediction result and the second classification prediction result; the network parameters in the neural network model are optimized according to the loss value.

[0284] Fig.11 This is a visualization of the description information of different classes obtained by using the trained neural network model provided by this application. Multiple sample images corresponding to each class are input into the neural network model in batches to obtain multiple category descriptions corresponding to each class; multiple category descriptions corresponding to each class are visualized to obtain Fig.11 . Fig.11 In the example, multiple category descriptions corresponding to the same class are clustered together to form a cluster, and the clusters corresponding to different classes are clearly separated, unlike the prior art where category descriptions corresponding to different classes are mixed together and cannot be distinguished. In other words, the neural network model provided in the embodiment of the present application can effectively express various different classes, and the class description has good generalization, so that the neural network model can have a better classification effect.

[0285] It should be noted that the forward calculation process of the above-mentioned neural network model in the training stage is the same as that in the application stage. Therefore, the data processing flow in the training stage can be specifically referred to the corresponding contents in the above-mentioned embodiments, which will not be repeated here.

[0286] Fig.12A flow chart of a data classification method provided by another embodiment of the present application is shown. The execution subject of the method can be a client or a server. The client can be a hardware with an embedded program integrated in the terminal, or an application software installed in the terminal, or a tool software embedded in the terminal operating system, etc., which is not limited in the embodiments of the present application. The terminal can be any terminal device including a mobile phone, a tablet computer, a smart wearable device, etc. The server can be a common server, a cloud or a virtual server, etc. Fig.12 As shown, the method comprises:

[0287] 1201. Obtain data to be classified.

[0288] 1202. Use the data to be classified as input to a trained neural network model, execute the neural network model, and obtain a classification result.

[0289] Among them, the neural network model is used to extract features of the data to be classified to obtain data features; the data features are respectively fused with the description information of multiple categories to obtain multiple fused features; and the classification result is obtained by combining the multiple fused features and the trained network parameters in the neural network model.

[0290] The description information of multiple categories may be obtained by averaging in the prior art, or by the methods provided in the above embodiments, which is not specifically limited in this embodiment.

[0291] The specific implementation of the above steps 1201 and 1202 can refer to the corresponding content in the above embodiments, which will not be repeated here.

[0292] In this embodiment, trainable network parameters are used to calculate the similarity between data features and description information of multiple categories, so that the neural network model can adapt to different sample scenarios and be more robust and adaptable.

[0293] Furthermore, the above “combining the multiple fusion features and the trained network parameters in the neural network model to obtain the classification result” can be implemented by the following steps:

[0294] S251. Obtain a first classification result based on the multiple fusion features and the trained network parameters in the neural network model.

[0295] S252: Calculate the distances between the data feature and the description information of the multiple categories respectively.

[0296] S253: Determine a second classification result according to the distance.

[0297] S254: Determine the classification result by combining the first classification result and the second classification result.

[0298] The specific implementation of the above steps S251 to S254 can refer to the corresponding contents in the above embodiments, which will not be repeated here.

[0299] In one example, the training process of the neural network model is as follows: taking the sample data to be classified corresponding to the first sample category as the input of the neural network model, executing the neural network model, and obtaining a classification prediction result; optimizing the neural network model according to the classification prediction result and the first sample category; wherein the neural network model is used to extract features of the sample data to be classified to obtain data features; fusing the data features with the description information of multiple sample categories respectively to obtain multiple fused features; combining the multiple fused features and the network parameters to be trained in the neural network model to obtain the classification prediction result.

[0300] The specific implementation of the above-mentioned “extracting features from the sample data to be classified to obtain data features” may refer to the corresponding contents of “extracting features from the data to be classified to obtain data features” in the above-mentioned embodiments.

[0301] The specific implementation of the above-mentioned “fusing the data features with the description information of multiple sample categories respectively to obtain multiple fused features” can refer to the corresponding content of “fusing the data features with the description information of multiple categories respectively to obtain multiple fused features” in the above-mentioned embodiments.

[0302] The specific implementation of the above-mentioned "combining the multiple fusion features and the network parameters to be trained in the neural network model to obtain the classification prediction result" can refer to the corresponding content of "combining the multiple fusion features and the trained network parameters in the neural network model to obtain the classification result" in the above-mentioned embodiments.

[0303] It should be noted that the forward calculation process of the above-mentioned neural network model in the training stage is the same as that in the application stage. Therefore, the data processing flow in the training stage can be specifically referred to the corresponding contents in the above-mentioned embodiments, which will not be repeated here.

[0304] It should be noted here that: for the contents of each step not fully described in detail in the method provided in the embodiment of the present application, please refer to the corresponding contents in the above embodiment, which will not be repeated here. In addition, in addition to the above steps, the method provided in the embodiment of the present application may also include other parts or all of the steps in the above embodiments, which can be specifically referred to the corresponding contents in the above embodiments, which will not be repeated here.

[0305] Figure 1cA flowchart of an image classification method provided by another embodiment of the present application is shown. The execution subject of the method can be a client or a server. The client can be a hardware with an embedded program integrated on the terminal, or an application software installed in the terminal, or a tool software embedded in the terminal operating system, etc., which is not limited in the present application embodiment. The terminal can be any terminal device including a mobile phone, a tablet computer, a smart wearable device, etc. The server can be a common server, a cloud or a virtual server, etc. Figure 1c As shown, the method comprises:

[0306] 101. Obtain an image to be classified.

[0307] 102. Use the image to be classified as input of a trained neural network model, execute the neural network model, and obtain a classification result.

[0308] Among them, the neural network model is used to extract features of the image to be classified to obtain image features; the image features are respectively fused with description information of multiple categories to obtain multiple fused features; and the classification result is obtained by combining the multiple fused features and the trained network parameters in the neural network model.

[0309] The description information of multiple categories may be obtained by averaging in the prior art, or by the methods provided in the above embodiments, which is not specifically limited in this embodiment.

[0310] The specific implementation of the above steps 101 and 102 can refer to the corresponding contents in the above embodiments, which will not be repeated here.

[0311] In this embodiment, trainable network parameters are used to calculate the similarity between image features and description information of multiple categories, so that the neural network model can adapt to different sample scenarios and be more robust and adaptable.

[0312] Furthermore, the above “combining the multiple fusion features and the trained network parameters in the neural network model to obtain the classification result” can be implemented by the following steps:

[0313] S51. Obtain a first classification result according to the multiple fusion features and the trained network parameters in the neural network model.

[0314] S52: Calculate the distances between the image feature and the description information of the multiple categories respectively.

[0315] S53: Determine a second classification result according to the distance.

[0316] S54: Determine the classification result by combining the first classification result and the second classification result.

[0317] The specific implementation of the above steps S51 to S54 can refer to the corresponding contents in the above embodiments, which will not be repeated here.

[0318] It should be noted here that: for the contents of each step not fully described in detail in the method provided in the embodiment of the present application, please refer to the corresponding contents in the above embodiment, which will not be repeated here. In addition, in addition to the above steps, the method provided in the embodiment of the present application may also include other parts or all of the steps in the above embodiments, which can be specifically referred to the corresponding contents in the above embodiments, which will not be repeated here.

[0319] Figure 2 FIG. 2 is a flow chart showing a training method for the neural network model provided by another embodiment of the present application. Figure 2 As shown, the method includes:

[0320] 201. Using the sample image to be classified corresponding to the first sample category as input of a neural network model, executing the neural network model, and obtaining a classification prediction result.

[0321] 202. Optimize the neural network model according to the classification prediction result and the first sample category.

[0322] Among them, the neural network model is used to extract features of the sample image to be classified to obtain image features; the image features are respectively fused with the description information of multiple sample categories to obtain multiple fused features; and the classification prediction result is obtained by combining the multiple fused features and the network parameters to be trained in the neural network model.

[0323] The specific implementation of the above steps 201 and 202 can refer to the corresponding contents in the above embodiments, which will not be repeated here.

[0324] In this embodiment, trainable network parameters are used to calculate the similarity between image features and description information of multiple categories, so that the neural network model can adapt to different sample scenarios and be more robust and adaptable.

[0325] Furthermore, the above “combining the multiple fusion features and the network parameters to be trained in the neural network model to obtain the classification prediction result” can be implemented by the following steps:

[0326] S61. Obtain a first classification prediction result according to the multiple fusion features and the network parameters to be trained in the neural network model.

[0327] S62: Calculate the distances between the image feature and the description information of the multiple sample categories respectively.

[0328] S63. Determine a second classification prediction result according to the distance.

[0329] S64: Determine the classification prediction result by combining the first classification prediction result and the second classification prediction result.

[0330] The specific implementation of the above steps S61 to S64 can refer to the corresponding contents in the above embodiments, which will not be repeated here.

[0331] It should be noted here that: for the contents of each step not fully described in detail in the method provided in the embodiment of the present application, please refer to the corresponding contents in the above embodiment, which will not be repeated here. In addition, in addition to the above steps, the method provided in the embodiment of the present application may also include other parts or all of the steps in the above embodiments, which can be specifically referred to the corresponding contents in the above embodiments, which will not be repeated here.

[0332] The embodiment of the present application also provides an information search method. The execution subject of the method can be a client or a server. The client can be a hardware with an embedded program integrated in the terminal, or an application software installed in the terminal, or a tool software embedded in the terminal operating system, etc., which is not limited in the embodiment of the present application. The terminal can be any terminal device including a mobile phone, a tablet computer, a smart wearable device, etc. The server can be a common server, a cloud or a virtual server, etc. The method includes:

[0333] 901. Obtain an image to be classified as a search basis.

[0334] 902. Use the image to be classified as input of a trained neural network model, execute the neural network model, and obtain a classification result.

[0335] 903. Search and obtain search results according to the classification results.

[0336] The neural network model is used to determine the classification result based on the image to be classified and the description information of multiple categories; the neural network model is also used to extract features of sample images corresponding to the multiple categories to obtain sample features corresponding to the multiple categories; multiple semantic cluster centers are determined by integrating the sample features corresponding to the multiple categories; the description information of the first category is determined based on the distribution information of the sample features corresponding to the first category around the multiple semantic cluster centers; the multiple categories include the first category.

[0337] In the above 901, the above image to be classified may be input by the user through the client search interface. For example, the e-commerce client provides a search interface for the user to input the image to be classified as the search basis.

[0338] When the execution subject of the method provided in this embodiment is a server, the client can upload the image to be classified input by the user to the server.

[0339] The specific implementation of the above 902 can refer to the corresponding content in the above embodiments, which will not be repeated here.

[0340] In the above 903, the classification result can be used as a search keyword to search for search results related to the classification result.

[0341] Take the e-commerce scenario as an example: if the classification result is dress, the search results are product search results related to dress.

[0342] Taking the image search scenario as an example, if the classification result is dog, the search results will be images related to dogs.

[0343] The technical solution provided in the embodiment of the present application integrates the sample features corresponding to each category in multiple categories to determine multiple semantic cluster centers common to multiple categories. The distribution information of the sample features corresponding to the first category around each semantic cluster center can indicate the expression of the first category on the semantic cluster center, and the expression is also the description of the first category based on the semantic cluster center. It can be seen that the technical solution provided in the embodiment of the present application describes each category separately based on the multiple semantic cluster centers shared by multiple categories, and can globally and generally describe each category, which helps to improve the prediction accuracy, and then improve the search accuracy.

[0344] It should be noted here that: for the contents of each step not fully described in detail in the method provided in the embodiment of the present application, please refer to the corresponding contents in the above embodiment, which will not be repeated here. In addition, in addition to the above steps, the method provided in the embodiment of the present application may also include other parts or all of the steps in the above embodiments, which can be specifically referred to the corresponding contents in the above embodiments, which will not be repeated here.

[0345] The embodiment of the present application also provides an information search method. The execution subject of the method can be a client or a server. The client can be a hardware with an embedded program integrated in the terminal, or an application software installed in the terminal, or a tool software embedded in the terminal operating system, etc., which is not limited in the embodiment of the present application. The terminal can be any terminal device including a mobile phone, a tablet computer, a smart wearable device, etc. The server can be a common server, a cloud or a virtual server, etc. The method includes:

[0346] 1001. Obtain an image to be classified as a search basis.

[0347] 1002. Use the image to be classified as input of a trained neural network model, execute the neural network model, and obtain a classification result.

[0348] 1003. Search and obtain search results according to the classification results.

[0349] Among them, the neural network model is used to extract features of the image to be classified to obtain image features; the image features are respectively fused with description information of multiple categories to obtain multiple fused features; and the classification result is obtained by combining the multiple fused features and the trained network parameters in the neural network model.

[0350] In the above 1001, the above image to be classified may be input by the user through the client search interface. For example, the e-commerce client provides a search interface for the user to input the image to be classified as the search basis.

[0351] When the execution subject of the method provided in this embodiment is a server, the client can upload the image to be classified input by the user to the server.

[0352] The specific implementation of the above 1002 can refer to the corresponding content in the above embodiments, which will not be repeated here.

[0353] In the above 1003, the classification result can be used as a search keyword to search for search results related to the classification result.

[0354] Take the e-commerce scenario as an example: if the classification result is dress, the search results are product search results related to dress.

[0355] Taking the image search scenario as an example, if the classification result is dog, the search results will be images related to dogs.

[0356] In this embodiment, trainable network parameters are used to calculate the similarity between image features and description information of multiple categories, so that the neural network model can adapt to different sample scenarios, be more robust and adaptable, and improve prediction accuracy, thereby improving search accuracy.

[0357] An embodiment of the present application provides a neural network system. The system includes: at least one first neural network layer, at least one second neural network layer and a computing module; wherein:

[0358] The at least one first neural network layer is used to extract features from the image to be classified to obtain image features; and is also used to extract features from sample images corresponding to multiple categories to obtain sample features corresponding to the multiple categories;

[0359] The at least one second neural network layer is used to determine a plurality of semantic cluster centers by synthesizing the sample features corresponding to the plurality of categories; and is also used to determine the description information of the first category according to the distribution information of the sample features corresponding to the first category around the plurality of semantic cluster centers; wherein the first category is one of the plurality of categories;

[0360] The calculation module is used to determine the classification result according to the image features and the description information of the multiple categories.

[0361] Among them, at least one first neural network layer constitutes a feature extraction subnetwork, which can be specifically a WRN (Wide Residual Network) network.

[0362] Among them, the at least one second neural network layer may include at least one convolutional layer, at least one softmax layer and / or at least one normalization layer.

[0363] The technical solution provided in the embodiment of the present application integrates the sample features corresponding to each category in multiple categories to determine multiple semantic cluster centers common to multiple categories. The distribution information of the sample features corresponding to the first category around each semantic cluster center can indicate the expression of the first category on the semantic cluster center, which is the description of the first category based on the semantic cluster center. It can be seen that the technical solution provided in the embodiment of the present application describes each category separately based on multiple shared semantic cluster centers, and can provide a global and generalized description of each category, which helps to improve the prediction accuracy.

[0364] It should be noted here that for the contents not fully described in the specific processing steps of the neural network layer provided in the embodiments of the present application, please refer to the corresponding contents in the above embodiments and will not be repeated here.

[0365] When the neural network systems in the above embodiments need to be applied to small sample classification scenarios, the neural network systems in the above embodiments are all based on meta-learning technology, which can obtain a more generalized global class description and effectively improve the prediction accuracy in small sample classification scenarios.

[0366] An embodiment of the present application provides a neural network system. The system includes: at least one first neural network layer and at least one third neural network layer; wherein:

[0367] The at least one first neural network layer is used to extract features of the image to be classified to obtain image features;

[0368] The at least one third neural network layer is used to fuse the image features with the description information of multiple categories respectively to obtain multiple fused features; and to obtain a classification result by combining the multiple fused features and the trained network parameters.

[0369] In this embodiment, trainable network parameters are used to calculate the similarity between image features and description information of multiple categories, so that the neural network model can adapt to different sample scenarios and be more robust and adaptable.

[0370] Furthermore, the at least one first neural network layer is further used to extract features from sample images corresponding to the multiple categories respectively, to obtain sample features corresponding to the multiple categories;

[0371] The system further comprises:

[0372] At least one second neural network layer is used to integrate the sample features corresponding to the multiple categories to determine multiple semantic cluster centers; it is also used to determine the description information of the first category based on the distribution information of the sample features corresponding to the first category around the multiple semantic cluster centers; wherein the first category is one of the multiple categories.

[0373] It should be noted here that for the contents not fully described in the specific processing steps of the neural network layer provided in the embodiments of the present application, please refer to the corresponding contents in the above embodiments and will not be repeated here.

[0374] When the neural network systems in the above embodiments need to be applied to small sample classification scenarios, the neural network systems in the above embodiments are all based on meta-learning technology, which can obtain a more generalized global class description and effectively improve the prediction accuracy in small sample classification scenarios.

[0375] An embodiment of the present application provides a neural network system. The system includes: at least one first neural network layer, at least one third neural network layer and a computing module; wherein:

[0376] The at least one first neural network layer is used to extract features of the image to be classified to obtain image features;

[0377] The at least one third neural network layer is used to fuse the image features with the description information of multiple categories respectively to obtain multiple fused features; and obtain a first classification result according to the multiple fused features and the trained network parameters;

[0378] The calculation module is used to respectively calculate the distance between the image feature and the description information of the multiple categories; determine the second classification result according to the distance; and determine the final classification result by combining the first classification result and the second classification result.

[0379] In this embodiment, trainable network parameters are used to calculate the similarity between image features and description information of multiple categories, so that the neural network model can adapt to different sample scenarios and be more robust and adaptable. In addition, a fixed distance standard is combined to further improve classification accuracy.

[0380] Furthermore, the at least one first neural network layer is further used to extract features from sample images corresponding to the multiple categories respectively, to obtain sample features corresponding to the multiple categories;

[0381] The system further comprises:

[0382] At least one second neural network layer is used to integrate the sample features corresponding to the multiple categories to determine multiple semantic cluster centers; it is also used to determine the description information of the first category based on the distribution information of the sample features corresponding to the first category around the multiple semantic cluster centers; wherein the first category is one of the multiple categories.

[0383] It should be noted here that for the contents not fully described in the specific processing steps of the neural network layer provided in the embodiments of the present application, please refer to the corresponding contents in the above embodiments and will not be repeated here.

[0384] When the neural network systems in the above embodiments need to be applied to small sample classification scenarios, the neural network systems in the above embodiments are all based on meta-learning technology, which can obtain a more generalized global class description and effectively improve the prediction accuracy in small sample classification scenarios.

[0385] Figure 3 The structure block diagram of the image classification device provided by an embodiment of the present application is shown. The device includes: a first acquisition module 501 and a first execution module 502. Among them,

[0386] A first acquisition module 501 is used to acquire an image to be classified;

[0387] A first execution module 502, configured to use the image to be classified as an input of a trained neural network model, execute the neural network model, and obtain a classification result;

[0388] The neural network model is used to determine the classification result according to the image to be classified and the description information of multiple categories;

[0389] The neural network model is also used to extract features of sample images corresponding to the multiple categories to obtain sample features corresponding to the multiple categories; determine multiple semantic cluster centers by combining the sample features corresponding to the multiple categories; determine description information of the first category based on distribution information of sample features corresponding to the first category around the multiple semantic cluster centers; wherein the first category is included in the multiple categories.

[0390] Figure 3 The image classification device can perform Figure 1c The implementation principle and technical effect of the image classification method described in the illustrated embodiment will not be described in detail. The specific manner in which each module and the neural network model perform operations in the image classification device in the above embodiment has been described in detail in the embodiment of the method, and will not be described in detail here.

[0391] Figure 4 The structural block diagram of the model training device provided by an embodiment of the present application is shown. The device includes: a second execution module 601 and a first optimization module 602. Among them,

[0392] The second execution module 601 is used to use the sample image to be classified corresponding to the first sample category as the input of the neural network model, execute the neural network model, and obtain the classification prediction result;

[0393] A first optimization module 602, configured to optimize the neural network model according to the classification prediction result and the first sample category;

[0394] Wherein, the neural network model is used to determine the classification prediction result according to the sample image to be classified and the description information of multiple sample categories;

[0395] The neural network model is also used to extract features of sample images corresponding to the multiple sample categories to obtain sample features corresponding to the multiple sample categories; determine multiple semantic cluster centers by combining the sample features corresponding to the multiple sample categories; describe the second sample category based on distribution information of sample features corresponding to the second sample category around the multiple semantic cluster centers; and include the second sample category in the multiple sample categories.

[0396] Figure 4 The model training device can perform Figure 2 The implementation principle and technical effect of the model training method described in the illustrated embodiment will not be described in detail. The specific manner in which each module and the neural network model perform operations in the model training device in the above embodiment has been described in detail in the embodiment of the method, and will not be elaborated here.

[0397] Figure 5 The structure block diagram of the image classification device provided by an embodiment of the present application is shown. The device includes: a second acquisition module 701 and a third execution module 702. Among them,

[0398] The second acquisition module 701 is used to acquire the image to be classified;

[0399] A third execution module 702 is used to use the image to be classified as an input of the trained neural network model, execute the neural network model, and obtain a classification result;

[0400] Among them, the neural network model is used to extract features of the image to be classified to obtain image features; the image features are respectively fused with description information of multiple categories to obtain multiple fused features; and the classification result is obtained by combining the multiple fused features and the trained network parameters in the neural network model.

[0401] Figure 5 The image classification device can perform Figure 1c The implementation principle and technical effect of the image classification method described in the illustrated embodiment will not be described in detail. The specific manner in which each module and the neural network model perform operations in the image classification device in the above embodiment has been described in detail in the embodiment of the method, and will not be described in detail here.

[0402] Figure 6 The structure block diagram of an image classification device provided by an embodiment of the present application is shown. The device includes:

[0403] The fourth execution module 801 is used to use the sample image to be classified corresponding to the first sample category as the input of the neural network model, execute the neural network model, and obtain the classification prediction result;

[0404] A second optimization module 802, configured to optimize the neural network model according to the classification prediction result and the first sample category;

[0405] Among them, the neural network model is used to extract features of the sample image to be classified to obtain image features; the image features are respectively fused with the description information of multiple sample categories to obtain multiple fused features; and the classification prediction result is obtained by combining the multiple fused features and the network parameters to be trained in the neural network model.

[0406] Figure 6 The model training device can perform Figure 2The implementation principle and technical effect of the model training method described in the illustrated embodiment will not be described in detail. The specific manner in which each module and the neural network model perform operations in the model training device in the above embodiment has been described in detail in the embodiment of the method, and will not be elaborated here.

[0407] An information search device provided by an embodiment of the present application includes: a third acquisition module, a fifth execution module and a first search module.

[0408] A third acquisition module is used to acquire the image to be classified as a search basis;

[0409] A fifth execution module, used to use the image to be classified as an input of the trained neural network model, execute the neural network model, and obtain a classification result;

[0410] The first search module is used to search and obtain search results according to the classification results.

[0411] The neural network model is used to determine the classification result according to the image to be classified and the description information of multiple categories;

[0412] The neural network model is also used to extract features of sample images corresponding to the multiple categories to obtain sample features corresponding to the multiple categories; determine multiple semantic cluster centers by combining the sample features corresponding to the multiple categories; determine description information of the first category based on distribution information of sample features corresponding to the first category around the multiple semantic cluster centers; wherein the first category is included in the multiple categories.

[0413] The information search device in this embodiment can execute the information search method described in the above corresponding embodiment, and its implementation principle and technical effect will not be repeated. For the information search device in the above embodiment, the specific methods of each module and neural network model have been described in detail in the embodiment of the method, and will not be elaborated here.

[0414] An information search device provided by an embodiment of the present application includes: a third acquisition module, a fifth execution module and a first search module.

[0415] A third acquisition module is used to acquire the image to be classified as a search basis;

[0416] A fifth execution module, used to use the image to be classified as an input of the trained neural network model, execute the neural network model, and obtain a classification result;

[0417] The first search module is used to search and obtain search results according to the classification results.

[0418] Among them, the neural network model is used to extract features of the image to be classified to obtain image features; the image features are respectively fused with description information of multiple categories to obtain multiple fused features; and the classification result is obtained by combining the multiple fused features and the trained network parameters in the neural network model.

[0419] The information search device in this embodiment can execute the information search method described in the above corresponding embodiment, and its implementation principle and technical effect will not be repeated. For the information search device in the above embodiment, the specific methods of each module and neural network model have been described in detail in the embodiment of the method, and will not be elaborated here.

[0420] Figure 3 The structure block diagram of the data classification device provided by an embodiment of the present application is shown. The device includes: a first acquisition module 501 and a first execution module 502. Among them,

[0421] A first acquisition module 501 is used to acquire data to be classified;

[0422] A first execution module 502 is used to use the data to be classified as the input of the trained neural network model, execute the neural network model, and obtain a classification result;

[0423] Wherein, the neural network model is used to determine the classification result according to the data to be classified and the description information of multiple categories;

[0424] The neural network model is also used to extract features from sample data corresponding to the multiple categories to obtain sample features corresponding to the multiple categories; determine multiple semantic cluster centers by integrating the sample features corresponding to the multiple categories; determine description information of the first category based on distribution information of sample features corresponding to the first category around the multiple semantic cluster centers; wherein the first category is included in the multiple categories.

[0425] Figure 3 The data classification device can execute the image classification method described in the above corresponding embodiment, and its implementation principle and technical effect will not be repeated. The specific way in which each module and neural network model perform operations in the data classification device in the above embodiment has been described in detail in the embodiment of the method, and will not be elaborated here.

[0426] Figure 5 The structure block diagram of the data classification device provided by an embodiment of the present application is shown. The device includes: a second acquisition module 701 and a third execution module 702. Among them,

[0427] The second acquisition module 701 is used to acquire data to be classified;

[0428] The third execution module 702 is used to use the data to be classified as the input of the trained neural network model, execute the neural network model, and obtain a classification result;

[0429] Among them, the neural network model is used to extract features of the data to be classified to obtain data features; the data features are respectively fused with the description information of multiple categories to obtain multiple fused features; and the classification result is obtained by combining the multiple fused features and the trained network parameters in the neural network model.

[0430] Figure 5 The data classification device can execute the data classification method described in the above corresponding embodiment, and its implementation principle and technical effect will not be repeated. The specific way in which each module and neural network model perform operations in the data classification device in the above embodiment has been described in detail in the embodiment of the method, and will not be elaborated here.

[0431] Figure 7 FIG. 1 is a schematic diagram showing the structure of an electronic device provided by an embodiment of the present application. Figure 7 As shown, the electronic device includes a memory 1101 and a processor 1102. The memory 1101 can be configured to store various other data to support operations on the electronic device. Examples of these data include instructions for any application or method for operating on the electronic device. The memory 1101 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk or an optical disk.

[0432] The memory 1101 is used to store programs;

[0433] The processor 1102 is coupled to the memory 1101 and is used to execute the program stored in the memory 1101 to implement the image classification method, model training method, information search method or data classification method provided by the above-mentioned method embodiments.

[0434] In particular, when the processor 1102 executes the program in the memory 1101, in addition to the above functions, other functions may also be implemented, and details may be referred to the description of the previous embodiments.

[0435] Further, if Figure 7As shown, the electronic device also includes: a communication component 1103, a display 1104, a power component 1105, an audio component 1106 and other components. Figure 7 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 7 Components shown.

[0436] Accordingly, an embodiment of the present application also provides a computer-readable storage medium storing a computer program, which, when executed by a computer, can implement the steps or functions of the image classification method, model training method, information search method or data classification method provided in the above embodiments.

[0437] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0438] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0439] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An image classification method, characterized in that: include: Get the image to be classified; Using the image to be classified as the input of the trained neural network model, executing the neural network model, and obtaining a classification result; The neural network model is used to determine the classification result according to the image to be classified and the description information of multiple categories, including: extracting features of the image to be classified to obtain image features; fusing the image features with the description information of multiple categories respectively to obtain multiple fused features; combining the multiple fused features and the trained network parameters in the neural network model to obtain the classification result; The neural network model is also used to extract features of sample images corresponding to the multiple categories to obtain sample features corresponding to the multiple categories; determine multiple semantic cluster centers by combining the sample features corresponding to the multiple categories; determine description information of the first category based on distribution information of sample features corresponding to the first category around the multiple semantic cluster centers; wherein the first category is included in the multiple categories.

2. The method according to claim 1, characterized in that Determining description information of the first category according to distribution information of sample features corresponding to the first category around the centers of the plurality of semantic clusters includes: Determine sub-description information of the first category based on the first semantic cluster center according to distribution information of sample features corresponding to the first category around the first semantic cluster center; wherein the first semantic cluster center is included in the plurality of semantic cluster centers; Determine the description information of the first category based on the sub-description information of the first category by each semantic cluster center in the multiple semantic cluster centers.

3. The method according to claim 2, characterized in that The neural network model is also used for: Calculating the residual between each sample feature in the sample features corresponding to the first category and the first semantic clustering center; The distribution information of the sample features corresponding to the first category around the center of the first semantic cluster includes the residual between each sample feature in the sample features corresponding to the first category and the center of the first semantic cluster.

4. The method according to claim 3, characterized in that Determining sub-description information of the first category based on the first semantic cluster center according to distribution information of sample features corresponding to the first category around the first semantic cluster center includes: Calculating the allocation weight corresponding to each sample feature according to the distance between each sample feature in the sample features corresponding to the first category and the center of the first semantic cluster; Sub-description information of the first category based on the first semantic cluster center is obtained by aggregation according to the allocation weight and the residual between each sample feature in the sample features corresponding to the first category and the first semantic cluster center.

5. The method according to any one of claims 2 to 4, characterized in that Determining description information of the first category based on sub-description information of the first category by each semantic cluster center in the plurality of semantic cluster centers includes: Combining the sub-description information of the first category based on each of the multiple semantic cluster centers by channel to obtain combined sub-description information; Perform dimensionality reduction processing on the combined sub-description information to obtain the description information of the first category.

6. The method according to any one of claims 1 to 4, characterized in that Combining the sample features corresponding to the multiple categories, multiple semantic cluster centers are determined, including: Calculate the mean features of the sample features corresponding to each category in the multiple categories respectively; Combining the mean features corresponding to each category in the multiple categories by channel to obtain a combined mean feature; The combined mean feature is convolved to obtain a plurality of semantic cluster centers common to the plurality of categories.

7. The method according to any one of claims 1 to 4, characterized in that Combining the multiple fusion features and the trained network parameters in the neural network model to obtain the classification result includes: Obtaining a first classification result according to the multiple fusion features and the trained network parameters in the neural network model; respectively calculating the distances between the image features and the description information of the multiple categories; Determining a second classification result according to the distance; The first classification result and the second classification result are combined to determine the classification result.

8. The method according to claim 7, characterized in that the plurality of categories comprising a second category; The first classification result includes a first probability that the image to be classified belongs to the second category; The second classification result includes a second probability that the image to be classified belongs to the second category; The classification result includes a third probability that the image to be classified belongs to the second category; Combining the first classification result and the second classification result to determine the classification result includes: The third probability is determined by combining the first probability and the second probability.

9. The method according to claim 7, characterized in that: The trained network parameters include: at least one first network parameter and at least one second network parameter; The neural network model includes at least one convolutional layer based on the at least one first network parameter and at least one fully connected layer based on the at least one second network parameter; According to the multiple fusion features and the trained network parameters in the neural network model, a first classification result is obtained, including: Inputting the multiple fused features into the at least one convolutional layer to obtain convolutional features; The convolutional features are input into the at least one fully connected layer to obtain the first classification result.

10. The method according to any one of claims 1 to 4, characterized in that The neural network model is based on meta-learning technology.

11. A model training method, characterized in that: include: Using the sample image to be classified corresponding to the first sample category as input of the neural network model, executing the neural network model, and obtaining a classification prediction result; Optimizing the neural network model according to the classification prediction result and the first sample category; The neural network model is used to determine the classification prediction result according to the sample image to be classified and the description information of multiple sample categories, including: extracting features of the sample image to be classified to obtain image features; fusing the image features with the description information of multiple sample categories respectively to obtain multiple fused features; combining the multiple fused features and the network parameters in the neural network model to obtain the classification prediction result; The neural network model is also used to extract features of sample images corresponding to the multiple sample categories to obtain sample features corresponding to the multiple sample categories; determine multiple semantic cluster centers by combining the sample features corresponding to the multiple sample categories; determine description information of the second sample category based on distribution information of sample features corresponding to the second sample category around the multiple semantic cluster centers; and include the second sample category in the multiple sample categories.

12. The method according to claim 11, characterized in that Determining the classification prediction result according to the sample image to be classified and the description information of the multiple sample categories includes: Extracting features of the sample image to be classified to obtain image features; Fusing the image features with description information of multiple sample categories respectively to obtain multiple fused features; The classification prediction result is obtained by combining the multiple fusion features and the network parameters to be trained in the neural network model.

13. The method according to claim 11 or 12, characterized in that: The neural network model is based on meta-learning technology.

14. A neural network system, characterized in that: include: At least one first neural network layer, at least one second neural network layer and a computing module; wherein, The at least one first neural network layer is used to perform feature extraction on the image to be classified to obtain image features; and is also used to perform feature extraction on sample images corresponding to multiple categories to obtain sample features corresponding to the multiple categories; The at least one second neural network layer is used to determine a plurality of semantic cluster centers by synthesizing the sample features corresponding to the plurality of categories; and is also used to determine the description information of the first category according to the distribution information of the sample features corresponding to the first category around the plurality of semantic cluster centers; wherein the first category is one of the plurality of categories; The calculation module is used to determine the classification result based on the image features and the description information of the multiple categories, including: fusing the image features with the description information of the multiple categories respectively to obtain multiple fused features; combining the multiple fused features and the trained network parameters in the neural network model to obtain the classification result.

15. A neural network system, characterized in that: include: at least one first neural network layer, at least one second neural network layer, at least one third neural network layer and a computing module; wherein, The at least one first neural network layer is used to perform feature extraction on the image to be classified to obtain image features; and is also used to perform feature extraction on sample images corresponding to multiple categories to obtain sample features corresponding to the multiple categories; The at least one second neural network layer is used to determine a plurality of semantic cluster centers by synthesizing the sample features corresponding to the plurality of categories; and is also used to determine the description information of the first category according to the distribution information of the sample features corresponding to the first category around the plurality of semantic cluster centers; wherein the first category is one of the plurality of categories; The at least one third neural network layer is used to fuse the image features with the description information of multiple categories respectively to obtain multiple fused features; and obtain a first classification result according to the multiple fused features and the trained network parameters; The calculation module is used to respectively calculate the distance between the image feature and the description information of the multiple categories; determine the second classification result according to the distance; and determine the final classification result by combining the first classification result and the second classification result.

16. An electronic device, characterized in that: include: A memory and a processor, wherein The memory is used to store programs; The processor is coupled to the memory and is configured to execute the program stored in the memory to: Get the image to be classified; The image to be classified is used as an input of a trained neural network model, and the neural network model is executed to obtain a classification result, including: extracting features of the image to be classified to obtain image features; fusing the image features with description information of multiple categories to obtain multiple fused features; and combining the multiple fused features with the trained network parameters in the neural network model to obtain the classification result; The neural network model is used to determine the classification result according to the image to be classified and the description information of multiple categories; The neural network model is also used to extract features of sample images corresponding to the multiple categories to obtain sample features corresponding to the multiple categories; determine multiple semantic cluster centers by combining the sample features corresponding to the multiple categories; determine description information of the first category based on distribution information of sample features corresponding to the first category around the multiple semantic cluster centers; wherein the first category is included in the multiple categories.

17. An electronic device, characterized in that: include: A memory and a processor, wherein The memory is used to store programs; The processor is coupled to the memory and is configured to execute the program stored in the memory to: Using the sample image to be classified corresponding to the first sample category as input of the neural network model, executing the neural network model, and obtaining a classification prediction result; Optimizing the neural network model according to the classification prediction result and the first sample category; The neural network model is used to determine the classification prediction result according to the sample image to be classified and the description information of multiple sample categories, including: extracting features of the sample image to be classified to obtain image features; fusing the image features with the description information of multiple sample categories respectively to obtain multiple fused features; combining the multiple fused features and the network parameters in the neural network model to obtain the classification prediction result; The neural network model is also used to extract features of sample images corresponding to the multiple sample categories to obtain sample features corresponding to the multiple sample categories; determine multiple semantic cluster centers by combining the sample features corresponding to the multiple sample categories; determine description information of the second sample category based on distribution information of sample features corresponding to the second sample category around the multiple semantic cluster centers; and include the second sample category in the multiple sample categories.

18. An information search method, characterized in that: include: Obtaining an image to be classified as a search basis; Using the image to be classified as the input of the trained neural network model, executing the neural network model, and obtaining a classification result; Searching to obtain search results according to the classification results; The neural network model is used to determine the classification result according to the image to be classified and the description information of multiple categories, including: extracting features of the image to be classified to obtain image features; fusing the image features with the description information of multiple categories respectively to obtain multiple fused features; combining the multiple fused features and the trained network parameters in the neural network model to obtain the classification result; The neural network model is also used to extract features of sample images corresponding to the multiple categories to obtain sample features corresponding to the multiple categories; determine multiple semantic cluster centers by combining the sample features corresponding to the multiple categories; determine description information of the first category based on distribution information of sample features corresponding to the first category around the multiple semantic cluster centers; wherein the first category is included in the multiple categories.

19. An electronic device, characterized in that: include: A memory and a processor, wherein The memory is used to store programs; The processor is coupled to the memory and is configured to execute the program stored in the memory to: Obtaining an image to be classified as a search basis; Using the image to be classified as the input of the trained neural network model, executing the neural network model, and obtaining a classification result; Searching to obtain search results according to the classification results; The neural network model is used to determine the classification result according to the image to be classified and the description information of multiple categories, including: extracting features of the image to be classified to obtain image features; fusing the image features with the description information of multiple categories respectively to obtain multiple fused features; combining the multiple fused features and the trained network parameters in the neural network model to obtain the classification result; The neural network model is also used to extract features of sample images corresponding to the multiple categories to obtain sample features corresponding to the multiple categories; determine multiple semantic cluster centers by combining the sample features corresponding to the multiple categories; determine description information of the first category based on distribution information of sample features corresponding to the first category around the multiple semantic cluster centers; wherein the first category is included in the multiple categories.

20. A data classification method, characterized in that: include: Obtain data to be classified; Using the data to be classified as input of a trained neural network model, executing the neural network model, and obtaining a classification result; The neural network model is used to determine the classification result according to the data to be classified and the description information of multiple categories, including: extracting features of the data to be classified to obtain data features; fusing the data features with the description information of multiple categories to obtain multiple fusion features; combining the multiple fusion features and the trained network parameters in the neural network model to obtain the classification result; The neural network model is also used to extract features from sample data corresponding to the multiple categories to obtain sample features corresponding to the multiple categories; determine multiple semantic cluster centers by integrating the sample features corresponding to the multiple categories; determine description information of the first category based on distribution information of sample features corresponding to the first category around the multiple semantic cluster centers; wherein the first category is included in the multiple categories.

21. An electronic device, characterized in that: include: A memory and a processor, wherein The memory is used to store programs; The processor is coupled to the memory and is configured to execute the program stored in the memory to: Obtain data to be classified; Using the data to be classified as input of a trained neural network model, executing the neural network model, and obtaining a classification result; The neural network model is used to determine the classification result according to the data to be classified and the description information of multiple categories, including: extracting features of the data to be classified to obtain data features; fusing the data features with the description information of multiple categories to obtain multiple fusion features; combining the multiple fusion features and the trained network parameters in the neural network model to obtain the classification result; The neural network model is also used to extract features from sample data corresponding to the multiple categories to obtain sample features corresponding to the multiple categories; determine multiple semantic cluster centers by integrating the sample features corresponding to the multiple categories; determine description information of the first category based on distribution information of sample features corresponding to the first category around the multiple semantic cluster centers; wherein the first category is included in the multiple categories.

Images