Image classification method, device, electronic device and storage medium

Abstract

The embodiment of the present application discloses an image classification method, device, electronic device and storage medium, including: performing a first feature extraction process on the image to be processed to obtain image features; for each of the multiple categories at the terminal level, combining the learnable prompt information of the category, the learnable prompt information of the category at the upper level, and the category into category information; performing a second feature extraction process on the category information to obtain category features; determining the similarity between the image features and multiple category features; determining the category of the image to be processed based on the category corresponding to the category feature with the highest similarity among the multiple category features. The embodiment of the present application integrates the correlation between the upper and lower levels of the category into the feature learning process, thereby making the image classification more accurate. And in the training stage, only the learnable prompt information corresponding to each category needs to be trained, the training parameters are relatively small, and the training efficiency is relatively high. The embodiment of the present application can achieve a balance between training efficiency and classification accuracy.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to an image classification method, device, electronic device and storage medium. Background Art

[0002] When implementing image classification, the existing technology often uses raw data to train a deep learning model, and then uses the trained deep learning model to perform image classification.

[0003] However, the above method requires more parameters to be trained. If the training amount is small, the accuracy of image classification is low; if the accuracy of image classification is improved, the training time consumed is long. Existing technologies often cannot achieve a balance between training efficiency and classification accuracy. Summary of the invention

[0004] The embodiments of the present application provide an image classification method, device, electronic device and storage medium, which can improve the problem that the prior art cannot balance training efficiency and classification accuracy.

[0005] The present application provides an image classification method, the method comprising:

[0006] Performing a first feature extraction process on the image to be processed to obtain image features;

[0007] For each of the plurality of categories at the terminal level, combining the learnable hint information of the category, the learnable hint information of the category at the upper level of the category, and the category into category information;

[0008] Performing a second feature extraction process on the category information to obtain a category feature;

[0009] Determining the similarity between the image feature and a plurality of the category features;

[0010] The category of the image to be processed is determined based on the category corresponding to the category feature with the highest similarity among the multiple category features.

[0011] The present application provides an image classification device, the device comprising:

[0012] An image feature unit is used to perform a first feature extraction process on the image to be processed to obtain an image feature;

[0013] an information combining unit, for combining, for each of a plurality of categories at a terminal level, learnable prompt information of the category, learnable prompt information of categories at a level above the category, and the category into category information;

[0014] A category feature unit, used for performing a second feature extraction process on the category information to obtain a category feature;

[0015] a similarity determination unit, used to determine the similarity between the image feature and a plurality of the category features;

[0016] The category determination unit is used to determine the category of the image to be processed based on the category corresponding to the category feature with the highest similarity among the multiple category features.

[0017] In one embodiment, the category determination unit includes:

[0018] A category acquisition subunit, used to acquire the category corresponding to the category feature with the highest similarity, and the category of the upper level of the category;

[0019] The category determination subunit is used to take the category corresponding to the category feature with the highest similarity and the category of the upper level of the category as the category of the image to be processed.

[0020] In one implementation, the category determination unit is specifically configured to: determine the category corresponding to the category feature with the highest similarity as the category of the image to be processed.

[0021] In one implementation, the categories have N levels, and for any two adjacent levels in the N levels: the i-1th level and the i-th level, the i-1th level has a first number of categories, the i-th level has a second number of categories, and the parent categories of one or more categories in the second number of categories exist in the first number of categories, and the first number is not greater than the second number; i is not greater than N, and i and N are both positive integers greater than 1; accordingly, the device further includes:

[0022] A first feature extraction unit, used to perform a first feature extraction process on the training image to obtain a training image feature;

[0023] A probability distribution calculation unit, configured to calculate, for each of the N levels, probability distributions corresponding to the multiple categories of the level based on the training category features corresponding to the multiple categories of the level and the training image features;

[0024] A first loss calculation unit, configured to calculate a first loss function corresponding to the level based on probability distributions corresponding to the multiple categories of the level;

[0025] A second loss calculation unit is used to calculate a second loss function of an i-1th level for an i-th level among the N levels based on the probability distributions corresponding to the multiple categories of the i-th level and the probability distribution corresponding to the parent category of each category of the i-th level;

[0026] A total loss calculation unit, used to determine a total loss function based on the first loss function and the second loss function;

[0027] The parameter updating unit is used to update the parameters of the learnable prompt information corresponding to the multiple categories of each level in the N levels based on the total loss function until convergence, so as to obtain a trained image classification model.

[0028] In one embodiment, the probability distribution calculation unit includes:

[0029] a training feature determination subunit, configured to determine, based on a position of the level among the N levels, training category features corresponding to a plurality of categories of the level;

[0030] The probability distribution calculation subunit is used to calculate the probability distribution corresponding to the multiple categories of the level based on the training category features corresponding to the multiple categories of the level, the training image features and the set temperature coefficient.

[0031] In one embodiment, training the feature determination subunit includes:

[0032] a first combining sub-subunit, configured to, when the level is the highest level among the N levels, combine, for each category in a plurality of categories of the level, the learnable prompt information of the category with the category into training category information;

[0033] The feature extraction sub-subunit performs a second feature extraction process on the training category information to obtain a training category feature.

[0034] In one embodiment, the training feature determination subunit further includes:

[0035] a second combining sub-subunit, for combining, for each of a plurality of categories of the level, learnable hint information of the category, learnable hint information of categories of a level above the category, and the category into training category information when the level is a non-highest level among the N levels;

[0036] The extraction processing sub-subunit performs a second feature extraction process on the training category information to obtain a training category feature.

[0037] In one embodiment, the second loss calculation unit includes:

[0038] An alignment level establishing subunit, used to establish an alignment level of level i-1, the number of categories of the alignment level being the same as the number of categories of the level i, and the categories of the alignment level corresponding one-to-one to the categories of the level i;

[0039] an alignment level probability determination subunit, configured to determine, for each category of the i-th level, a probability distribution of a category corresponding to the category at the alignment level based on the probability distribution of a parent category of the category;

[0040] The second loss subunit is used to calculate the second loss function of the i-1th level based on the probability distribution of each category of the i-th level and the probability distribution of the corresponding category of the alignment level.

[0041] In the image classification method provided in the embodiment of the present application, a first feature extraction process can be performed on the image to be processed to obtain image features corresponding to the image to be processed. For each category at the terminal level, the category, the learnable prompt information of the category, and the learnable prompt information of the category at the upper level of the category (that is, the category to which the category belongs) can be combined to obtain category information; and a second feature extraction process is performed on the category information to obtain category features. There are multiple category features, and the number of category features is the same as the number of categories at the terminal level. Then, the similarity between the image feature and the multiple category features is calculated to obtain the category feature with the highest similarity, and based on the category corresponding to the category feature with the highest similarity, the category of the image to be processed is determined.

[0042] In an embodiment of the present application, when generating category information based on multiple categories at the terminal level, in addition to combining the learnable prompt information of the category, the learnable prompt information of the category to which the category belongs is also combined. The embodiment of the present application integrates the correlation between the upper and lower levels of the category into the feature learning process, thereby making the image classification more accurate. And in the training stage, the embodiment of the present application only needs to train the learnable prompt information corresponding to each category, with fewer training parameters and higher training efficiency. Therefore, the embodiment of the present application achieves a balance between training efficiency and classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0044] Figure 1a It is a schematic diagram of an application scenario of the image classification method provided by this application;

[0045] Figure 1bIt is a flowchart of the image classification method provided in the embodiment of the present application;

[0046] Figure 1c A schematic diagram showing the affiliation of multiple categories at three levels;

[0047] Figure 1d (1) shows the correspondence between the categories at the i-1th level and the categories at the i-th level;

[0048] Figure 1d (2) in the figure shows the correspondence between the alignment level and the i-1th level and the ith level;

[0049] Figure 1e A schematic diagram of a business application scenario corresponding to the image classification method provided by the present application is shown;

[0050] Figure 1f A schematic structural block diagram of the generation process of training category features at each level is shown;

[0051] Figure 2 It is a flowchart of an image classification method provided in a specific embodiment of the present application;

[0052] Figure 3 is a structural schematic diagram of an image classification device provided by an embodiment of the present application;

[0053] Figure 4 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0055] Embodiments of the present application provide an image classification method, device, electronic device, and storage medium.

[0056] The image classification device can be integrated into an electronic device, which can be a terminal, a server, or other device. The terminal can be a mobile phone, a tablet computer, a smart Bluetooth device, a laptop computer, or a desktop computer. The server can be a single server, or a server cluster or a cloud server composed of multiple servers.

[0057] In some embodiments, the image classification device may also be integrated into multiple electronic devices. For example, the image classification device may be integrated into multiple servers, and the image classification method of the present application may be implemented by multiple servers.

[0058] In some embodiments, the terminal may also be used as a server to implement part or all of the functions of the server.

[0059] For more information, please see Figure 1a , the method provided by the embodiment of the present application may include: performing a first feature extraction process on the image to be processed to obtain image features. Figure 1a For each of the m categories shown: category 1, category 2, ..., category m), the learnable hint information of the category, the learnable hint information of the category at the upper level of the category, and the category are combined into category information. Figure 1a , taking category i among m categories as an example, category i, the learnable prompt information corresponding to category i, and the learnable prompt information corresponding to the category of the upper level of category i can be combined to obtain category information i. The same operation can be performed on other categories to obtain m category information: category information 1, category information 2, ..., category information m. Perform the second feature extraction process on the category information to obtain the category feature. For details, please refer to Figure 1a , performing second feature extraction processing on the m category information respectively, m category features can be obtained: category feature 1, category feature 2, ..., category feature m. Determine the similarity between the image feature and the multiple category features; determine the category of the image to be processed based on the category corresponding to the category feature with the highest similarity among the multiple category features.

[0060] In the above method, when generating category information based on multiple categories at the terminal level, in addition to combining the learnable prompt information of the category, the learnable prompt information of the category to which the category belongs can also be combined, and the correlation between the upper and lower levels of the category can be integrated into the feature learning process to improve the accuracy of image classification. In addition, in the training stage, the embodiment of the present application only needs to train the learnable prompt information corresponding to each category, and the training efficiency is high, achieving a balance between training efficiency and classification accuracy.

[0061] The embodiments of the present application can be applied to business scenarios such as video application programs (App) or news apps. Specifically, the image classification method provided in the embodiments of the present application can be used to set type labels for user-generated content (UGC). UGC can be videos or pictures uploaded by users through video apps or news apps. Setting type labels using the image classification method provided in the embodiments of the present application (i.e., machine labeling) can be combined with manually set labels (i.e., manual labeling) to achieve a balance between labeling efficiency and labeling accuracy. For details, please see Figure 1e .

[0062] It is understandable that in the embodiments of the present application, related data such as user information is involved. When the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0063] It should be noted that the serial numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

[0064] In this embodiment, an image classification method is provided. Figure 1b As shown, the image classification method is applied to an electronic device. The specific process of the method may include the following steps 110 to 150:

[0065] 110. Perform a first feature extraction process on the image to be processed to obtain image features.

[0066] The image to be processed is an image to be subjected to image classification processing.

[0067] The first feature extraction process is a process of extracting image features. The first feature extraction process can be implemented by an image encoder. Optionally, the image encoder can be an image encoder of a trained image classification model. The training process of the image classification model will be described in detail below.

[0068] 120. For each of the multiple categories at the terminal level, combine the learnable hint information of the category, the learnable hint information of the category at the upper level of the category, and the category into category information.

[0069] The terminal level is the level at the end of multiple levels. Levels are used to characterize the granularity of categories. Multiple levels can be arranged in order of granularity from coarse to fine, that is, the terminal level is the level with the finest granularity among multiple levels. There are categories at each level. For ease of description, let's assume that there are N levels of categories. For any two adjacent levels among the N levels: the i-1th level and the i-th level, there are a first number of categories in the i-1th level, there are a second number of categories in the i-th level, and the parent category of one or more categories in the second number of categories exists in the first number of categories, and the first number is not greater than the second number; i is not greater than N, and i and N are both positive integers greater than 1.

[0070] Alternatively, let N be 3. For details, see Figure 1c , there are three levels: Level 1, Level 2 and Level 3. Level 1 includes four categories: games, sports, musical instruments, and fine arts; Level 2 includes eight categories: action games, strategy games, sandbox games, ball games, string instruments, wind instruments, abstract art, and impressionism; Level 3 includes thirteen categories: Game 1, Game 2, Game 3, Game 4, Game 5, football, table tennis, guitar, guzheng, oboe, painting 1, painting 2, and painting 3. Among them, the categories included in Level 3 are all at the terminal level.

[0071] Level 1, Level 2, and Level 3 are arranged in order from coarse to fine granularity of the categories. Figure 1c , taking the first-level category "sports" as an example, we can see that the category granularity of "sports" is coarser than the second-level category "ball games"; similarly, the category granularity of the second-level category "ball games" is coarser than the third-level categories "football" and "table tennis". That is, the parent category of "football" and "table tennis" is "ball games"; the parent category of "ball games" is "sports".

[0072] Among them, among the eight categories included in the second level, the parent categories of the three categories of action games, strategy games, and sandbox games exist in the four categories included in the first level, specifically the categories in the first level: games.

[0073] Among the eight categories included in the second level, the parent category of the ball game category exists in the four categories included in the first level, specifically the category in the first level: sports.

[0074] Among the eight categories included in the second level, the parent categories of string instruments and wind instruments exist in the four categories included in the first level, specifically the categories in the first level: musical instruments.

[0075] Among the eight categories included in the second level, the parent categories of the two categories of abstractionism and impressionism exist in the four categories included in the first level, specifically the categories in the first level: fine arts.

[0076] Among the thirteen categories included in the third level, the parent category of the Game 1 category exists in the eight categories included in the second level, specifically the category in the second level: Action Game.

[0077] Among the thirteen categories included in the third level, the parent categories of Game 2, Game 3, and Game 4 exist in the eight categories included in the second level, specifically the categories in the second level: strategy games.

[0078] Among the thirteen categories included in the third level, the parent category of the game 5 category exists in the eight categories included in the second level, specifically the category in the second level: sandbox game.

[0079] Among the thirteen categories included in the third level, the parent categories of football and table tennis exist in the eight categories included in the second level, specifically the categories in the second level: ball games.

[0080] Among the thirteen categories included in the third level, the parent categories of guitar and guzheng exist in the eight categories included in the second level, specifically the categories in the second level: string instruments.

[0081] Among the thirteen categories included in the third level, the parent category of the oboe category exists in the eight categories included in the second level, specifically the category in the second level: wind instruments.

[0082] Among the thirteen categories included in the third level, the parent category of the category Painting 1 exists in eight categories included in the second level, specifically the category in the second level: Abstract.

[0083] Among the thirteen categories included in the third level, the parent categories of Painting 2 and Painting 3 exist in the eight categories included in the second level, specifically the categories in the second level: Impressionism.

[0084] Learnable prompts refer to prompts that can be optimized during training. Learnable prompts are usually composed of a set of special tokens, whose embedding vectors are trainable parameters. Tokens refer to a series of text units that make up prompts. These text units are usually processed as independent minimum meaningful units in natural language processing (NLP) tasks.

[0085] Each category has its own corresponding learnable hint information. The learnable hint information of each category can be randomly initialized at the beginning of training, and the parameters are continuously updated as the training process progresses until the training is completed.

[0086] The categories at the parent level of a category refer to the categories that are in the same category branch as the category and are higher in level than the category. For example, see Figure 1c For example, taking the second-level category "Sandbox Games", the category that is in the same category branch as "Sandbox Games" and is higher in level than "Sandbox Games" is "Games", so the category of the parent level of "Sandbox Games" is "Games". For another example, taking the third-level category "Guitar", the categories that are in the same category branch as "Guitar" and are higher in level than "Guitar" include "Strings" and "Musical Instruments", so the categories of the parent level of "Guitar" are "Strings" and "Musical Instruments". It should be understood that if there are multiple categories in the parent level, one or more can be selected from the multiple categories.

[0087] For example, taking the category "guitar" of the terminal level as an example, in the process of obtaining the category of the parent level of the "guitar" category, you can select any one of the corresponding multiple parent levels as the "parent level category" in step 120, such as the parent level category of the "guitar" category can be "string music" or "musical instrument"; you can also use all of its corresponding parent levels as the "parent level category" in step 120, that is, the parent level category of "guitar" is "string music" and "musical instrument"; you can also specify the parent category of the "guitar" category (that is, "string music") as the "parent level category" in step 120. It should be understood that the method of determining the category of the parent level of a certain category should not be understood as a limitation on the present application. For the sake of ease of description, let's take the example that the parent category of a certain category is the parent category of the category for explanation.

[0088] For more information, please see Figure 1c , taking the category "guitar" at the terminal level as an example, the description is continued. The learnable prompt information of "guitar", the learnable prompt information of the category "strings" at the upper level of "guitar", and "guitar" can be combined to obtain category information.

[0089] The above combination method can be splicing. For example, feature extraction can be performed on "guitar" to obtain a feature vector; then the learnable prompt information of "guitar", the learnable prompt information of "string instrument" and the feature vector corresponding to "guitar" can be spliced ​​to obtain category information.

[0090] In addition to splicing, other combinations are also possible. For example, the learnable prompt information of "guitar" and the learnable prompt information of "strings" can be weighted summed to obtain the sum result; then, the above sum result is spliced ​​with the feature vector corresponding to "guitar" to obtain category information. It should be understood that the specific method of determining category information based on the learnable prompt information of the category, the learnable prompt information of the category of the parent level of the category, and the category should not be understood as a limitation of the present application.

[0091] Optionally, in one embodiment, the image classification method provided in the embodiment of the present application is implemented using a trained image classification model. Before step 120, the untrained image classification model may be trained to obtain a trained image classification model. The training process of training the untrained image classification model may include the following steps A1 to A6:

[0092] A1. Perform a first feature extraction process on the training image to obtain training image features.

[0093] In step A1, the image encoder of the pre-trained model may be used to perform a first feature extraction process on the training image to obtain training image features.

[0094] The pre-trained model is obtained by pre-training based on a large-scale data set. Optionally, in one embodiment, the pre-trained model may be a pre-trained Contrastive Language–Image Pre-training (CLIP) model.

[0095] A2. For each of the N levels, based on the training category features and the training image features respectively corresponding to the multiple categories of the level, calculate the probability distributions respectively corresponding to the multiple categories of the level.

[0096] For each of the N levels, there are multiple categories. Each of the multiple categories has its own corresponding training category features. Continue with the above example. For details, please refer to Figure 1c , N takes the value of 3, and there are four categories corresponding to the first level. Each of the four categories has its own corresponding training category features; similarly, the eight categories corresponding to the second level and the thirteen categories corresponding to the third level also have their own corresponding training category features.

[0097] Why not Represents the training category features. Among them, i Indicates the level to which the category corresponding to the training category feature belongs; j Indicates that the category corresponding to the training category feature is in iThe sequence number of the level. Continue with the above example. For details, see Figure 1c ,For example, Indicates that the category corresponding to the training category feature is the second category of the first level, combined with Figure 1c It can be seen that the second category of the first level is "sports", that is, is the training category feature corresponding to the category "sports". For example, Indicates that the category corresponding to the training category feature is the sixth category of the third level, combined with Figure 1c It can be seen that the sixth category of the third level is "football", that is It is the training category feature corresponding to the category "soccer".

[0098] The method for obtaining the training category features corresponding to each category will be described in detail below.

[0099] Optionally, in one implementation, step A2 may specifically include the following steps A21 to A22:

[0100] A21. Based on the position of the level in the N levels, determine the training category features corresponding to the multiple categories of the level.

[0101] The position of the level in the N levels is different, and the corresponding method of determining the training category feature is also different. Optionally, in one embodiment, step A21 may specifically include the following steps A211 to A214:

[0102] A211. If the level is the highest level among the N levels, for each category in the multiple categories of the level, combine the learnable prompt information of the category with the category into training category information.

[0103] The highest level is the topmost level among multiple levels. Multiple levels can be arranged in order from coarse to fine granularity, that is, the highest level is the coarsest level among multiple levels. Figure 1c , Figure 1c The first level is the highest level.

[0104] For each of the multiple categories at the highest level, the category may be combined with the learnable hint information of the category to obtain training category information.

[0105] The above-mentioned combination method can be splicing. Take the first level "game" as an example. You can extract features from "game" to obtain a feature vector; then splice the learnable prompt information of "game" and the feature vector corresponding to "game" to obtain category information. In addition to splicing, other combinations are also possible. For example, you can perform a linear transformation on the learnable prompt information of "game" to obtain a transformation result, and then splice the above-mentioned transformation result with the feature vector corresponding to "game" to obtain category information. It should be understood that the specific combination method of combining categories with learnable prompt information of categories should not be understood as a limitation on this application.

[0106] A212. Perform a second feature extraction process on the training category information to obtain a training category feature.

[0107] In step A212, a text encoder of a pre-trained model may be used to perform a second feature extraction process on the training category information to obtain a training category feature. The pre-trained model may be a pre-trained CLIP model.

[0108] A213. If the level is not the highest level among the N levels, for each category in the multiple categories of the level, the learnable prompt information of the category, the learnable prompt information of the category in the upper level of the category, and the category are combined into training category information.

[0109] A non-top level is a level other than the top level among multiple levels. Figure 1c , Figure 1c The 2nd and 3rd levels are not the highest levels.

[0110] The category at the parent level of a category refers to a category that is in the same category branch as the category and is higher in level than the category. This is consistent with the above description and will not be elaborated here.

[0111] For each of the multiple categories of the non-highest level, the category, the learnable hint information of the category, and the learnable hint information of the category of the upper level of the category may be combined to obtain the training category information.

[0112] The above combination can be spliced, for example, see Figure 1c, feature extraction can be performed on "Game 5" to obtain a feature vector; then the learnable prompt information of "Game 5", the learnable prompt information of "Sandbox Game", and the feature vector corresponding to "Game 5" can be spliced ​​to obtain training category information. In addition to splicing, other combinations are also possible. For example, a weighted summation can be performed on the learnable prompt information of "Game 5" and the learnable prompt information of "Sandbox Game" to obtain a sum result; then, the above sum result is spliced ​​with the feature vector corresponding to "Game 5" to obtain training category information. It should be understood that the specific method of determining training category information based on the learnable prompt information of the category, the learnable prompt information of the category at the upper level of the category, and the category should not be understood as a limitation on the present application.

[0113] A214. Perform a second feature extraction process on the training category information to obtain a training category feature.

[0114] The specific processing process of step A214 is the same as that of step A212, and will not be repeated here.

[0115] Figure 1f The schematic structural diagram of the generation process of training category features at each level is shown. Let N be 3, then there are 3 levels in total. For details, please refer to Figure 1c and Figure 1f , let’s assume that the category branches to be processed are: Game-Strategy Game-Game 2.

[0116] For the category "game" at the first level, feature extraction is performed to obtain a first-level feature vector, and then the first-level feature vector is concatenated with the learnable prompt information of "game" to obtain the corresponding training category information (not shown in the figure). The training category information is then input into the text encoder for the second feature extraction process to obtain the first-level training category feature, see Figure 1f .

[0117] For the category "strategy game" at the second level, feature extraction is performed to obtain a secondary feature vector, and then the secondary feature vector, the learnable prompt information of "strategy game", and the learnable prompt information of "game" are spliced ​​to obtain the corresponding training category information (not shown in the figure), and then the training category information is input into the text encoder for the second feature extraction process to obtain the secondary training category feature, see Figure 1f .

[0118] For the category "Game 2" at the third level, feature extraction is performed to obtain a three-level feature vector, and then the three-level feature vector, the learnable prompt information of "Game 2", and the learnable prompt information of "Strategy Game" are spliced ​​to obtain the corresponding training category information (not shown in the figure), and then the training category information is input into the text encoder for the second feature extraction process to obtain the three-level training category features, see Figure 1f .

[0119] The training category information in the above steps A211 to A214 can also be expressed as follows:

[0120] [v i,j ][CLASS],i=1

[0121] [v i-1,idx(i,j) ][v i,j ][CLASS],i>1

[0122] Among them, [v i,j ] represents the learnable hint information corresponding to the jth category of the i-th level. [CLASS] represents the feature vector obtained by feature extraction of the jth category of the i-th level. idx(i,j) represents the sequence number of the parent category of the jth category of the i-th level at the i-1 level. Figure 1d (1) in the figure: idx(i,4)=2, which means: the 4th category of the i-th level (i.e. Figure 1d 4 shown in (1) i ) has a parent category at level i-1 with a serial number of 2 (i.e. Figure 1d 2 shown in (1) i-1 ). i=1 indicates that the i-th level is the highest level. i>1 indicates that the i-th level is not the highest level. In the embodiment of the present application, the number of tokens of the learnable prompt information can be fixed, which is L, and L is a positive integer. Its specific value should not be understood as a limitation of the present application. For example, L can be 16 or 18.

[0123] It should be understood that during the training process, Figure 1f The parameters in the picture encoder and text encoder shown may be fixed and no longer updated.

[0124] In the above-mentioned implementation, the calculation method of the training category information corresponding to the multiple categories of the level can be determined based on whether the level is the highest level. If the level is the highest level, then for each category of the level, the learnable prompt information of the category is combined with the category to obtain the training category information. If the level is not the highest level, then for each category of the level, in addition to combining the learnable prompt information of the category with the category, it is also necessary to combine with the learnable prompt information of the category of the upper level of the category to obtain the training category information. That is, for a level with an upper level, when calculating the training category information of the category of the level, it is necessary to combine with the learnable prompt information of the category of the upper level. This approach can integrate the correlation between the upper and lower levels of the category into the feature learning process, so that the trained model can be more accurate in image classification.

[0125] A22. Calculate the probability distributions corresponding to the multiple categories of the level based on the training category features corresponding to the multiple categories of the level, the training image features, and the set temperature coefficient.

[0126] Optionally, in one implementation, the probability distributions corresponding to a plurality of categories at a certain level may be calculated specifically by the following formula.

[0127]

[0128] in, For the i Level j The probability distribution corresponding to the categories is To train image features, For the i Level j The training category features corresponding to the categories are Representation calculation and The inner product of To set the temperature coefficient, the specific value of the temperature coefficient can be set by the developer based on work experience. Optionally, Specifically, it may be the temperature coefficient of the CLIP model.

[0129] For more information, please see Figure 1f , Figure 1f The diagram shows the first-level probability distribution obtained by the training image features and the first-level training category features, the second-level probability distribution obtained by the training image features and the second-level training category features, and the third-level probability distribution obtained by the training image features and the third-level training category features. The set temperature coefficients used to calculate the first-level probability distribution, the second-level probability distribution, and the third-level probability distribution are respectively same.

[0130] A3. Calculate a first loss function corresponding to the level based on the probability distribution corresponding to the multiple categories of the level.

[0131] The following formula can be used to calculate the i The first loss function of the layer:

[0132] (1)

[0133] in, Indicates i The first loss function of the layer. is the one-hot tag of the word, indicating: i Level j When a category is a word with a set position in the preset vocabulary, The value is 1; i Level j If a category is not a word in the preset vocabulary, The value is 0. The preset vocabulary can be generated based on the training data set. For example, suppose that all the texts in the training data set can have 100,000 different words. Then, these 100,000 different words can be sorted in descending order according to the frequency of occurrence of the words, and the first r words are taken to form the preset vocabulary. r is a positive integer, and the value of r can be set by the developer based on work experience. For example, r can be 15,000 or 24,000. It should be understood that the specific value of r should not be understood as a limitation to the present application.

[0134] It should be understood that the first loss function of each level in the N levels can be calculated by formula (1) and averaged within a batch, and the first loss functions of each level can be independent of each other. Optionally, the first loss function can specifically be a cross entropy loss function, or a mean square error loss function, a logarithmic loss function, or a multi-class hinge loss function. It should be understood that the specific loss function type of the first loss function should not be understood as a limitation on the present application.

[0135] A4. For the i-th level among the N levels, based on the probability distributions corresponding to the multiple categories of the i-th level and the probability distribution corresponding to the parent category of each category of the i-th level, calculate the second loss function of the i-1-th level.

[0136] The second loss function of the i-1th level can be performed based on the probability distributions corresponding to the multiple categories of the i-th level and the probability distributions corresponding to the parent categories of each category of the i-th level (i.e., the multiple categories of the i-1th level). Optionally, in one embodiment, step A4 can specifically include the following steps A41 to A43:

[0137] A41. Establish an alignment level of the i-1th level, the number of categories of the alignment level is the same as the number of categories of the i-th level, and the categories of the alignment level correspond one-to-one with the categories of the i-th level.

[0138] For more information, please see Figure 1d , Figure 1d (1) in shows the correspondence between the categories at the i-1th level and the categories at the ith level. Figure 1d From (1) in the above, we can see that category 1 (i.e. Figure 1d 1 shown in (1) i ), Category 2 (i.e. Figure 1d 2 shown in (1) i ) is the parent category of the i-1th level category 1 (i.e. Figure 1d 1 shown in (1) i-1 ); Category 3 of level i (i.e. Figure 1d 3 shown in (1) i ), Category 4 (i.e. Figure 1d 4 shown in (1) i ), Category 5 (i.e. Figure 1d 5 shown in (1) i ) is the parent category of the i-1th level category 2 (i.e. Figure 1d 2 shown in (1) i-1 ); Category 6 of level i (i.e. Figure 1d The 6 shown in (1) i ) is the parent category of the i-1th level category 3 (i.e. Figure 1d 3 shown in (1) i-1 ); Category 7 of level i (i.e. Figure 1d 7 shown in (1) i ), Category 8 (i.e. Figure 1d 8 shown in (1) i ) is the parent category of the i-1th level category 4 (i.e. Figure 1d 4 shown in (1) i-1 ).

[0139] Figure 1d (2) in shows the correspondence between the alignment level of the i-1th level and the category of the i-th level. Figure 1d From (2) in the figure, we can see that the number of categories in the i-1th level of alignment is the same as that in the i-th level, which is 8; and the categories in the i-1th level of alignment correspond one-to-one to the categories in the i-th level. For details, see Figure 1d In (2), the 8 categories of the alignment level at level i-1 are: , , , , , , , .and with 1 i correspond, with 2 i correspond, with 3 i correspond, With 4 i correspond, with 5 i correspond, with 6 i correspond, with 7 i correspond, with 8 i For details, please see Figure 1d (2) in the above. The same applies below. For the sake of narrative fluency, the description is omitted. A42. For each category of the i-th level, based on the probability distribution of the parent category of the category, determine the probability distribution of the category corresponding to the category at the alignment level.

[0140] For any category at the i-th level, the probability distribution of the category corresponding to the category at the alignment level can be determined based on the probability distribution of the parent category of the category (that is, the category to which the category belongs at the i-1 level).

[0141] Continue with the above example. For details, see Figure 1d In (2), for category 3 at level i (i.e. Figure 1d 3 shown i ), the category 3 of the alignment level (i.e. Figure 1d Shown ) probability distribution.

[0142] In one embodiment, the category 2 (i.e. Figure 1d 2 shown i-1 ) is assigned to category 3 of the alignment level (i.e. Figure 1d Shown ). Similarly, for each category at the i-th level, the value of the probability distribution of its parent category at the i-1th level can be assigned to the category corresponding to itself in the alignment level, and then normalized to obtain the probability distribution corresponding to each category at the alignment level.

[0143] For example, see Figure 1d In (2):

[0144] For category 1 at the i-th level, the value of the probability distribution of its parent category (i.e., category 1 at the i-1th level) can be assigned to category 1 at the alignment level;

[0145] For category 2 at the i-th level, the value of the probability distribution of its parent category (i.e., category 1 at the i-1th level) can be assigned to category 2 at the alignment level;

[0146] For category 4 at the i-th level, the value of the probability distribution of its parent category (i.e., category 2 at the i-1th level) can be assigned to category 4 at the alignment level;

[0147] For category 5 at the i-th level, the value of the probability distribution of its parent category (i.e., category 2 at the i-1th level) can be assigned to category 5 at the alignment level;

[0148] For category 6 at the i-th level, the value of the probability distribution of its parent category (i.e., category 3 at the i-1th level) can be assigned to category 6 at the alignment level;

[0149] For category 7 at the i-th level, the value of the probability distribution of its parent category (i.e., category 4 at the i-1-th level) can be assigned to category 7 at the alignment level;

[0150] For category 8 at the i-th level, the numerical value of the probability distribution of its parent category (i.e., category 4 at the i-1-th level) can be assigned to category 8 at the alignment level.

[0151] After completing the above-mentioned assignment operation, the assigned values ​​of all categories of the alignment level are normalized to obtain the probability distribution corresponding to all categories of the alignment level.

[0152] In another implementation, the probability distribution of the parent category may be evenly divided according to the number of its corresponding subcategories, and the evenly divided result is assigned to the alignment level, thereby obtaining the probability distribution of each category of the alignment level.

[0153] For example, see Figure 1d In (2):

[0154] For category 1 and category 2 at the i-th level, their parent category (i.e. category 1 at the i-1th level) has two subcategories: category 1 at the i-th level and category 2 at the i-th level. Therefore, the value of the probability distribution of category 1 at the i-1th level can be divided into two parts, and the two values ​​are assigned to category 1 and category 2 at the alignment level respectively;

[0155] For category 3, category 4 and category 5 at the i-th level, their parent category (i.e. category 2 at the i-1th level) has three subcategories: category 3 at the i-th level, category 4 at the i-th level, and category 5 at the i-th level. Therefore, the value of the probability distribution of category 2 at the i-1th level can be divided into three equal parts, and the three values ​​are assigned to category 3, category 4 and category 5 at the alignment level respectively;

[0156] For category 6 at the i-th level, its parent category (i.e. category 3 at the i-1th level) has a total of one subcategory: category 6 at the i-th level. Therefore, the value of the probability distribution of category 3 at the i-1th level can be directly assigned to category 6 at the alignment level;

[0157] For categories 7 and 8 at the i-th level, their parent category (i.e., category 4 at the i-1th level) has two subcategories: category 7 at the i-th level and category 8 at the i-th level. Therefore, the value of the probability distribution of category 4 at the i-1th level can be divided into two parts, and the two values ​​are assigned to categories 7 and 8 at the alignment level respectively.

[0158] After completing the above operations, the probability distribution of each category at the alignment level can be obtained.

[0159] For ease of description, we may use Indicates the alignment level of level i-1 j The probability distribution of categories. j Indicates the first of multiple categories of the alignment level of the i-1th level j categories.

[0160] A43. Calculate the second loss function of the i-1th level based on the probability distribution of each category of the i-th level and the probability distribution of the corresponding category of the alignment level.

[0161] In most cases, the number of categories at the i-th level is different from the number of categories at the i-1th level, resulting in the inconsistency between the lengths of the probability distribution at the i-th level and the probability distribution at the i-1th level, and the inability to directly compare the similarity of the two probability distributions. To solve this problem, the embodiment of the present application introduces an alignment level at the i-1th level, thereby indirectly comparing the similarity between the probability distribution at the i-th level and the probability distribution at the i-1th level.

[0162] Optionally, the second loss function of the i-1th level can be calculated by the following formula:

[0163] (2)

[0164] in, represents the second loss function of the i-1th level; Indicates the alignment level of level i-1j The probability distribution of the categories; Indicates the first level of j The probability distribution of the categories.

[0165] In the above implementation, a second loss function, namely the loss function of the consistency of probabilities of adjacent levels, is introduced to make the training process more accurate and efficient.

[0166] A5. Determine a total loss function based on the first loss function and the second loss function.

[0167] Alternatively, by combining equations (1) and (2), the total loss function can be calculated by the following formula:

[0168] (3)

[0169] in, Indicates i The first loss function of the layer is The first accumulation function representing the first loss functions of the 1st level to the Nth level, that is, the accumulation of N first loss functions; represents the second loss function of the i-1th level, then The second cumulative function representing the second loss functions of the first level to the N-1th level, that is, the accumulation of N-1 second loss functions. The total loss function is the sum of the first cumulative function and the second cumulative function.

[0170] A6. Based on the total loss function, update the parameters of the learnable prompt information corresponding to the multiple categories in each level of the N levels until convergence to obtain a trained image classification model.

[0171] During the training process, the parameters of the image encoder and text encoder of the pre-trained model can be kept fixed, and only the parameters of the learnable prompt information corresponding to multiple categories in each of the N levels are updated until the total loss function converges to obtain a trained image classification model.

[0172] In the above implementation, the encoder parameters in the pre-trained model can be fixed during the training process, and only the learnable prompt information corresponding to each category at each level can be modified, thereby greatly reducing the parameters that need to be trained, achieving the goal of efficiently fine-tuning the model using a small amount of data, and quickly migrating to the target task using a small amount of training samples, thereby improving the efficiency of training. In addition, while the embodiment of the present application classifies each level separately based on the first loss function, it introduces the constraint of the consistency of the probability distribution of adjacent levels based on the second loss function, and integrates the correlation and hierarchical relationship between candidate categories into the feature learning process, which can improve the accuracy of classification.

[0173] Optionally, in addition to using the constraint of probability distribution consistency, other constraints can be used to ensure that the generated data has similar statistical properties to the real data. Other constraints include maximum likelihood estimation, least squares method, etc.

[0174] 130. Perform a second feature extraction process on the category information to obtain a category feature.

[0175] The second feature extraction process is a process of extracting text features. The second feature extraction process can be implemented by a text encoder. Optionally, the text encoder can be a text encoder of a trained image classification model. The number of category features is consistent with the number of categories at the terminal level.

[0176] 140. Determine the similarity between the image feature and the plurality of category features.

[0177] For each category feature among multiple category features, its similarity with the image feature can be calculated. Optionally, the cosine similarity between the category feature and the image feature can be calculated, and the cosine similarity can be used as the similarity between the two. In addition to cosine similarity, other methods can be used to calculate the similarity between the category feature and the image feature. For example, the Euclidean distance between the category feature and the image feature can be calculated, and the Euclidean distance can be used as the similarity between the two; the Manhattan distance between the category feature and the image feature can also be calculated, and the Manhattan distance can be used as the similarity between the two. It should be understood that the specific method of calculating the similarity should not be understood as a limitation to this application.

[0178] 150. Determine the category of the image to be processed based on the category corresponding to the category feature with the highest similarity among the multiple category features.

[0179] After calculating the similarity of each of the multiple category features, the category corresponding to the category feature with the highest similarity value can be obtained, and the category of the image to be processed is determined based on the category.

[0180] Optionally, in a specific implementation, step 150 may specifically include the following steps S1 to S2:

[0181] S1. Obtain the category corresponding to the category feature with the highest similarity, and the category of the upper level of the category.

[0182] The category corresponding to the category feature with the highest similarity is the category at the terminal level. In addition to obtaining the category, the category at the parent level of the category can also be obtained.

[0183] Optionally, the categories of the upper level in step S1 may be the categories of all the upper levels of the categories of the terminal level. For details, see Figure 1c, let’s assume that the category corresponding to the category feature with the highest similarity is “football” at the third level, then we can obtain all the categories at the parent level of the category “football”: the category “ball games” at the second level and the category “sports” at the first level.

[0184] Optionally, the category of the upper level in step S1 may also be any category of all the categories of the upper level of the category of the terminal level. Continuing with the above example, it is assumed that the category corresponding to the category feature with the highest similarity is "football" at the third level, then the category of the upper level of the category "football" can be any one of the categories "ball games" at the second level and "sports" at the first level.

[0185] For the convenience of description, the following description will be continued by taking the case where the categories of the upper level in step S1 are all the categories of the upper level of the category of the terminal level as an example.

[0186] S2. The category corresponding to the category feature with the highest similarity and the category at the upper level of the category are taken as the category of the image to be processed.

[0187] The category corresponding to the category feature with the highest similarity and the category of the upper level of the category can be taken as the category of the image to be processed.

[0188] Continuing with the example above, the category "football" corresponding to the category feature with the highest similarity, together with the category "balls" and the category "sports" can be used as the category of the image to be processed. The specific output format can be as follows: the category of the image to be processed is sports-balls-football.

[0189] In the above implementation, the category of the terminal level together with the category of the upper level can be used as the category of the image to be processed, so that the user can know the classification of the image to be processed at different scales in more detail and accuracy.

[0190] Optionally, in another specific implementation, step 150 may specifically include the following steps: determining the category corresponding to the category feature with the highest similarity as the category of the image to be processed.

[0191] Continue with the above example. For details, see Figure 1c , let’s assume that the category corresponding to the category feature with the highest similarity is “football” at the third level, then the category “football” corresponding to the category feature with the highest similarity can be directly used as the category of the image to be processed. The specific output format can be as follows: the category of the image to be processed is football.

[0192] In the above implementation, the category of the terminal level may be directly used as the category of the image to be processed, so that the user can more intuitively know the classification of the image to be processed.

[0193] In the image classification method provided in the embodiment of the present application, the first feature extraction process can be performed on the image to be processed to obtain the image features corresponding to the image to be processed. For each category at the terminal level, the category, the learnable prompt information of the category, and the learnable prompt information of the category of the upper level of the category (that is, the category to which the category belongs) can be combined to obtain category information; and the category information is subjected to a second feature extraction process to obtain category features. There are multiple category features, and the number of category features is the same as the number of categories at the terminal level. Then the similarity between the image feature and the multiple category features is calculated to obtain the category feature with the highest similarity, and the category of the image to be processed is determined based on the category corresponding to the category feature with the highest similarity. In the embodiment of the present application, when generating category information based on multiple categories at the terminal level, in addition to combining the learnable prompt information of the category, the learnable prompt information of the category to which the category belongs is also combined. The embodiment of the present application integrates the correlation between the upper and lower levels of the category into the process of feature learning, thereby making the image classification more accurate. In addition, in the training stage, the embodiment of the present application only needs to train the learnable prompt information corresponding to each category, with fewer training parameters and higher training efficiency.

[0194] The embodiments of the present application can achieve a balance between training efficiency and classification accuracy.

[0195] In this embodiment, the method of the embodiment of the present application is described in detail by taking the value of N as 3 as an example. The image classification method provided in the embodiment of the present application can be executed by an electronic device. In this embodiment, the electronic device is a server as an example. Figure 2 As shown, the specific process of an image classification method is as follows:

[0196] 201. The server performs a first feature extraction process on the training image to obtain a training image feature.

[0197] 202. The server determines, based on the position of the level in the three levels, training category features corresponding to the multiple categories of the level.

[0198] Optionally, in one implementation, step 202 may specifically include the following steps:

[0199] If the level is the highest level among the three levels, for each category in the multiple categories of the level, the learnable prompt information of the category is combined with the category into training category information; and a second feature extraction process is performed on the training category information to obtain training category features.

[0200] If the level is not the highest level among the three levels, for each of the multiple categories of the level, the learnable prompt information of the category, the learnable prompt information of the category of the upper level of the category, and the category are combined into training category information; the second feature extraction process is performed on the training category information to obtain training category features.

[0201] 203. The server calculates probability distributions corresponding to the multiple categories of the level based on the training category features corresponding to the multiple categories of the level, the training image features, and the set temperature coefficient.

[0202] 204. The server calculates a first loss function corresponding to the level based on probability distributions corresponding to the multiple categories of the level.

[0203] 205. The server establishes an alignment level of level (i-1), the number of categories of the alignment level being the same as the number of categories of the level (i), and the categories of the alignment level corresponding one-to-one to the categories of the level (i).

[0204] 206. The server determines, for each category at the i-th level, based on the probability distribution of a parent category of the category, a probability distribution of a category corresponding to the category at the alignment level.

[0205] 207. The server calculates a second loss function of the (i-1)th level based on the probability distribution of each category of the i-th level and the probability distribution of the corresponding category of the alignment level.

[0206] 208. The server determines a total loss function based on the first loss function and the second loss function.

[0207] 209. Based on the total loss function, the server updates the parameters of the learnable prompt information corresponding to the multiple categories in each of the three levels until convergence, thereby obtaining a trained image classification model.

[0208] 210. The server uses the trained image classification model to perform a first feature extraction process on the image to be processed to obtain image features.

[0209] 211. The server uses the trained image classification model to combine the learnable prompt information of each category in the multiple categories at the terminal level, the learnable prompt information of the category at the upper level of the category, and the category into category information.

[0210] 212. The server uses the trained image classification model to perform a second feature extraction process on the category information to obtain category features.

[0211] 213. The server uses the trained image classification model to determine the similarity between the image feature and the plurality of category features.

[0212] 214. The server determines the category of the image to be processed by using the trained image classification model based on the category corresponding to the category feature with the highest similarity among the multiple category features.

[0213] Optionally, in one implementation, step 214 includes:

[0214] The category corresponding to the category feature with the highest similarity and the category of the upper level of the category are obtained; and the category corresponding to the category feature with the highest similarity and the category of the upper level of the category are taken together as the category of the image to be processed.

[0215] Optionally, in another implementation, step 214 includes:

[0216] The category corresponding to the category feature with the highest similarity is determined as the category of the image to be processed.

[0217] The specific execution process of step 201 to step 214 has been described in detail above and will not be repeated here.

[0218] In the image classification method provided in the embodiment of the present application, the first feature extraction process can be performed on the image to be processed to obtain the image features corresponding to the image to be processed. For each category at the terminal level, the category, the learnable prompt information of the category, and the learnable prompt information of the category of the upper level of the category (that is, the category to which the category belongs) can be combined to obtain category information; and the category information is subjected to a second feature extraction process to obtain category features. There are multiple category features, and the number of category features is the same as the number of categories at the terminal level. Then the similarity between the image feature and the multiple category features is calculated to obtain the category feature with the highest similarity, and the category of the image to be processed is determined based on the category corresponding to the category feature with the highest similarity. In the embodiment of the present application, when generating category information based on multiple categories at the terminal level, in addition to combining the learnable prompt information of the category, the learnable prompt information of the category to which the category belongs is also combined. The embodiment of the present application integrates the correlation between the upper and lower levels of the category into the process of feature learning, thereby making the image classification more accurate. In addition, in the training stage, the embodiment of the present application only needs to train the learnable prompt information corresponding to each category, with fewer training parameters and higher training efficiency.

[0219] The embodiments of the present application can achieve a balance between training efficiency and classification accuracy.

[0220] In order to better implement the above method, the present application embodiment also provides an image classification device, such as Figure 3 As shown, the device comprises:

[0221] An image feature unit 301 is used to perform a first feature extraction process on the image to be processed to obtain image features;

[0222] An information combining unit 302, for combining, for each of the multiple categories at the terminal level, the learnable prompt information of the category, the learnable prompt information of the category at the upper level of the category, and the category into category information;

[0223] A category feature unit 303, configured to perform a second feature extraction process on the category information to obtain a category feature;

[0224] A similarity determination unit 304, configured to determine the similarity between the image feature and a plurality of the category features;

[0225] The category determination unit 305 is configured to determine the category of the image to be processed based on the category corresponding to the category feature with the highest similarity among the plurality of category features.

[0226] In one implementation, the category determination unit 305 includes:

[0227] A category acquisition subunit, used to acquire the category corresponding to the category feature with the highest similarity, and the category of the upper level of the category;

[0228] The category determination subunit is used to take the category corresponding to the category feature with the highest similarity and the category of the upper level of the category as the category of the image to be processed.

[0229] In one implementation, the category determination unit 305 is specifically configured to: determine the category corresponding to the category feature with the highest similarity as the category of the image to be processed.

[0230] In one implementation, the categories have N levels, and for any two adjacent levels in the N levels: the i-1th level and the i-th level, the i-1th level has a first number of categories, the i-th level has a second number of categories, and the parent categories of one or more categories in the second number of categories exist in the first number of categories, and the first number is not greater than the second number; i is not greater than N, and i and N are both positive integers greater than 1; accordingly, the device further includes:

[0231] A first feature extraction unit, used to perform a first feature extraction process on the training image to obtain a training image feature;

[0232] A probability distribution calculation unit, configured to calculate, for each of the N levels, probability distributions corresponding to the multiple categories of the level based on the training category features corresponding to the multiple categories of the level and the training image features;

[0233] A first loss calculation unit, configured to calculate a first loss function corresponding to the level based on probability distributions corresponding to the multiple categories of the level;

[0234] A second loss calculation unit is used to calculate a second loss function of an i-1th level for an i-th level among the N levels based on the probability distributions corresponding to the multiple categories of the i-th level and the probability distribution corresponding to the parent category of each category of the i-th level;

[0235] A total loss calculation unit, used to determine a total loss function based on the first loss function and the second loss function;

[0236] The parameter updating unit is used to update the parameters of the learnable prompt information corresponding to the multiple categories of each level in the N levels based on the total loss function until convergence, so as to obtain a trained image classification model.

[0237] In one embodiment, the probability distribution calculation unit includes:

[0238] a training feature determination subunit, configured to determine, based on a position of the level among the N levels, training category features corresponding to a plurality of categories of the level;

[0239] The probability distribution calculation subunit is used to calculate the probability distribution corresponding to the multiple categories of the level based on the training category features corresponding to the multiple categories of the level, the training image features and the set temperature coefficient.

[0240] In one embodiment, training the feature determination subunit includes:

[0241] a first combining sub-subunit, configured to, when the level is the highest level among the N levels, combine, for each category in a plurality of categories of the level, the learnable prompt information of the category with the category into training category information;

[0242] The feature extraction sub-subunit performs a second feature extraction process on the training category information to obtain a training category feature.

[0243] In one embodiment, the training feature determination subunit further includes:

[0244] a second combining sub-subunit, for combining, for each of a plurality of categories of the level, learnable hint information of the category, learnable hint information of categories of a level above the category, and the category into training category information when the level is a non-highest level among the N levels;

[0245] The extraction processing sub-subunit performs a second feature extraction process on the training category information to obtain a training category feature.

[0246] In one embodiment, the second loss calculation unit includes:

[0247] An alignment level establishing subunit, used to establish an alignment level of level i-1, the number of categories of the alignment level being the same as the number of categories of the level i, and the categories of the alignment level corresponding one-to-one to the categories of the level i;

[0248] an alignment level probability determination subunit, configured to determine, for each category of the i-th level, a probability distribution of a category corresponding to the category at the alignment level based on the probability distribution of a parent category of the category;

[0249] The second loss subunit is used to calculate the second loss function of the i-1th level based on the probability distribution of each category of the i-th level and the probability distribution of the corresponding category of the alignment level.

[0250] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can refer to the previous method embodiments, which will not be repeated here.

[0251] In the image classification method provided in the embodiment of the present application, the first feature extraction process can be performed on the image to be processed to obtain the image features corresponding to the image to be processed. For each category at the terminal level, the category, the learnable prompt information of the category, and the learnable prompt information of the category of the upper level of the category (that is, the category to which the category belongs) can be combined to obtain category information; and the category information is subjected to a second feature extraction process to obtain category features. There are multiple category features, and the number of category features is the same as the number of categories at the terminal level. Then the similarity between the image feature and the multiple category features is calculated to obtain the category feature with the highest similarity, and the category of the image to be processed is determined based on the category corresponding to the category feature with the highest similarity. In the embodiment of the present application, when generating category information based on multiple categories at the terminal level, in addition to combining the learnable prompt information of the category, the learnable prompt information of the category to which the category belongs is also combined. The embodiment of the present application integrates the correlation between the upper and lower levels of the category into the process of feature learning, thereby making the image classification more accurate. In addition, in the training stage, the embodiment of the present application only needs to train the learnable prompt information corresponding to each category, with fewer training parameters and higher training efficiency.

[0252] The embodiments of the present application can achieve a balance between training efficiency and classification accuracy.

[0253] The present application also provides an electronic device. In this embodiment, the electronic device of this embodiment is described in detail by taking the server as an example. For example, Figure 4 As shown, it shows a schematic diagram of the structure of the electronic device involved in the embodiment of the present application, specifically:

[0254] The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, an input module 404, and a communication module 405. Those skilled in the art will appreciate that Figure 4 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0255] The processor 401 is the control center of the electronic device, and uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 402, and calling data stored in the memory 402. In some embodiments, the processor 401 may include one or more processing cores; in some embodiments, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 401.

[0256] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and image classification model training by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0257] The electronic device also includes a power supply 403 for supplying power to various components. In some embodiments, the power supply 403 can be logically connected to the processor 401 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 403 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0258] The electronic device may further include an input module 404, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0259] The electronic device may further include a communication module 405. In some embodiments, the communication module 405 may include a wireless module. The electronic device may perform short-range wireless transmission through the wireless module of the communication module 405, thereby providing the user with wireless broadband Internet access. For example, the communication module 405 may be used to help the user send and receive emails, browse web pages, and access streaming media.

[0260] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 401 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 402 according to the following instructions, and the processor 401 will run the application programs stored in the memory 402, thereby realizing various functions in the various method embodiments of the present application.

[0261] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0262] To this end, an embodiment of the present application provides a computer-readable storage medium, in which instructions are stored, and the instructions can be loaded by a processor to execute the steps in any image classification method provided in the embodiment of the present application. For example, the instructions can execute each step in each method embodiment of the present application.

[0263] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0264] According to one aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program including instructions, the instructions being stored in a computer-readable storage medium. A processor of a computer device reads the instructions from the computer-readable storage medium, and the processor executes the instructions, so that the computer device executes the methods provided in the various optional implementations provided in the above embodiments.

[0265] Since the instructions stored in the storage medium can execute the steps in any image classification method provided in the embodiments of the present application, the beneficial effects that can be achieved by any image classification method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0266] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0267] The above is a detailed introduction to an image classification method, device, electronic device and computer-readable storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An image classification method, characterized in that: The image classification method is implemented by using a trained image classification model, and the method comprises: Performing a first feature extraction process on the image to be processed to obtain image features; For each of the multiple categories at the terminal level, the learnable prompt information of the category, the learnable prompt information of the category at the upper level of the category, and the category are combined into category information; the category has N levels, and for any two adjacent levels in the N levels: the i-1th level and the i-th level, the i-1th level has a first number of categories, the i-th level has a second number of categories, and the parent category of one or more categories in the second number of categories exists in the first number of categories, and the first number is not greater than the second number; i is not greater than N, and i and N are both positive integers greater than 1; before this step, the method further includes: Perform a first feature extraction process on the training image to obtain training image features; for each of the N levels, based on the training category features corresponding to the multiple categories of the level and the training image features, calculate the probability distributions corresponding to the multiple categories of the level; based on the probability distributions corresponding to the multiple categories of the level, calculate the first loss function corresponding to the level; for the i-th level of the N levels, based on the probability distributions corresponding to the multiple categories of the i-th level and the probability distribution corresponding to the parent category of each category of the i-th level, calculate the second loss function of the i-1-th level; based on the first loss function and the second loss function, determine the total loss function; based on the total loss function, update the parameters of the learnable prompt information corresponding to the multiple categories of each level of the N levels until convergence, and obtain a trained image classification model; Performing a second feature extraction process on the category information to obtain a category feature; Determining the similarity between the image feature and a plurality of the category features; The category of the image to be processed is determined based on the category corresponding to the category feature with the highest similarity among the multiple category features.

2. The method according to claim 1, characterized in that The step of determining the category of the image to be processed based on the category corresponding to the category feature with the highest similarity among the plurality of category features comprises: Obtaining the category corresponding to the category feature with the highest similarity, and the category of the parent level of the category; The category corresponding to the category feature with the highest similarity and the category at the upper level of the category are taken together as the category of the image to be processed.

3. The method according to claim 1, characterized in that The step of determining the category of the image to be processed based on the category corresponding to the category feature with the highest similarity among the plurality of category features comprises: The category corresponding to the category feature with the highest similarity is determined as the category of the image to be processed.

4. The method according to claim 1, characterized in that The step of calculating, for each of the N levels, probability distributions corresponding to the multiple categories of the level based on the training category features and the training image features corresponding to the multiple categories of the level, comprises: Determining, based on the position of the level among the N levels, training category features corresponding to a plurality of categories of the level; Based on the training category features respectively corresponding to the multiple categories of the hierarchy, the training image features and the set temperature coefficient, the probability distributions respectively corresponding to the multiple categories of the hierarchy are calculated.

5. The method according to claim 4, characterized in that The determining, based on the position of the level in the N levels, training category features corresponding to the multiple categories of the level, respectively, includes: If the level is the highest level among the N levels, for each of the multiple categories of the level, combining the learnable prompt information of the category with the category into training category information; A second feature extraction process is performed on the training category information to obtain a training category feature.

6. The method according to claim 4, characterized in that The determining, based on the position of the level in the N levels, training category features corresponding to the multiple categories of the level, respectively, further includes: If the level is a non-highest level among the N levels, for each of the multiple categories of the level, combining the learnable prompt information of the category, the learnable prompt information of the category of the upper level of the category, and the category into training category information; A second feature extraction process is performed on the training category information to obtain a training category feature.

7. The method according to claim 1, characterized in that The step of calculating, for the i-th level of the N levels, a second loss function of the i-1th level based on the probability distributions respectively corresponding to the multiple categories of the i-th level and the probability distribution corresponding to the parent category of each category of the i-th level, comprises: Establishing an alignment level of level i-1, the number of categories of the alignment level being the same as the number of categories of the level i, and the categories of the alignment level corresponding one-to-one to the categories of the level i; For each category of the i-th level, determining the probability distribution of the category corresponding to the category at the alignment level based on the probability distribution of the parent category of the category; Based on the probability distribution of each category of the i-th level and the probability distribution of the corresponding category of the alignment level, a second loss function of the i-1th level is calculated.

8. An image classification device, characterized in that: The image classification device is implemented using a trained image classification model, and the device comprises: An image feature unit is used to perform a first feature extraction process on the image to be processed to obtain an image feature; An information combination unit is used to combine the learnable prompt information of the category, the learnable prompt information of the category of the upper level of the category, and the category into category information for each category in the multiple categories at the terminal level; the category has N levels, and for any two adjacent levels in the N levels: the i-1th level and the i-th level, the i-1th level has a first number of categories, the i-th level has a second number of categories, and the parent category of one or more categories in the second number of categories exists in the first number of categories, and the first number is not greater than the second number; i is not greater than N, and i and N are both positive integers greater than 1; the device is also used to: Perform a first feature extraction process on the training image to obtain training image features; for each of the N levels, based on the training category features corresponding to the multiple categories of the level and the training image features, calculate the probability distributions corresponding to the multiple categories of the level; based on the probability distributions corresponding to the multiple categories of the level, calculate the first loss function corresponding to the level; for the i-th level of the N levels, based on the probability distributions corresponding to the multiple categories of the i-th level and the probability distribution corresponding to the parent category of each category of the i-th level, calculate the second loss function of the i-1-th level; based on the first loss function and the second loss function, determine the total loss function; based on the total loss function, update the parameters of the learnable prompt information corresponding to the multiple categories of each level of the N levels until convergence, and obtain a trained image classification model; A category feature unit, used for performing a second feature extraction process on the category information to obtain a category feature; a similarity determination unit, used to determine the similarity between the image feature and a plurality of the category features; The category determination unit is used to determine the category of the image to be processed based on the category corresponding to the category feature with the highest similarity among the multiple category features.

9. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores instructions; the processor loads instructions from the memory to execute the steps in the image classification method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, which are suitable for a processor to load to execute the steps in the image classification method according to any one of claims 1 to 7.

11. A computer program product, characterized in that The method comprises instructions which, when executed by a processor, implement the steps of the image classification method according to any one of claims 1 to 7.

Images