Hierarchical classification and model training method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a hierarchical classification and model training method and device, electronic equipment and storage medium. The model training method comprises: extracting a plurality of image features of a first image sample based on a plurality of classification levels respectively, wherein the higher the classification level, the larger the classification granularity; respectively fusing a plurality of semantic labels of the plurality of classification levels and the plurality of image features to obtain a plurality of sets of fusion data corresponding to the plurality of classification levels; training a first sequence neural network by taking the plurality of sets of fusion data sorted from high to low according to the classification level as input and taking the plurality of semantic labels as output to obtain a hierarchical classification model. The scheme of the embodiments of the present application still gives a reliable classification result through the dependency relationship between the classification levels even in the case of a small number of image samples, and realizes reliable small sample learning.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a hierarchical classification and model training method, apparatus, electronic device and storage medium. Background Technology

[0002] With the rapid development of the global internet, social networks, e-commerce, live streaming, and short videos generate massive amounts of data every day. Some potentially risky information is also mixed in with this data. The rapid spread of the internet allows prohibited information to proliferate quickly, leading to negative social values ​​and even economic losses. Furthermore, these risks possess unique characteristics that increase the difficulty of identification and prevention.

[0003] While a large amount of data samples exist in a vast amount of multimedia information, the label data used for model training and optimization is limited in complex content governance scenarios due to the complexity of governance risk categories, the scarcity of black data for some special risks, the rapid changes in risks, and the high difficulty of manual annotation.

[0004] Therefore, for models in scenarios with limited labeled data, there is an urgent need for an effective few-shot learning method to meet the requirements of scenario customization. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a hierarchical classification and model training method, apparatus, electronic device, and storage medium to at least partially solve the above-mentioned problems.

[0006] According to a first aspect of the present invention, a model training method is provided, comprising: extracting multiple image features of a first image sample based on multiple classification levels, wherein the higher the classification level, the larger the classification granularity; fusing multiple semantic labels of the multiple classification levels with the multiple image features to obtain multiple sets of fused data corresponding to the multiple classification levels; using the multiple sets of fused data sorted from high to low classification levels as input and the multiple semantic labels as output to train a first sequence neural network to obtain a hierarchical classification model.

[0007] In another implementation of the present invention, the step of fusing multiple semantic labels of the multiple classification levels with the multiple image features to obtain multiple sets of fused data corresponding to the multiple classification levels includes: obtaining the semantic feature vectors of each of the multiple semantic labels of the multiple classification levels and the image feature vectors of each of the multiple image features; concatenating the semantic feature vector of each semantic label with the corresponding image feature vector to obtain the concatenation vector of each classification level; and determining the concatenation vector of each classification level as a set of fused data for each classification level.

[0008] In another implementation of the present invention, obtaining the semantic feature vectors of each of the plurality of semantic tags includes: inputting the plurality of semantic tags into a pre-trained semantic vector encoding model to obtain the semantic feature vectors of each of the plurality of semantic tags.

[0009] In another implementation of the present invention, the first sequence neural network is a long short-term memory neural network composed of multiple propagation units. The step of using multiple sets of fused data sorted from high to low classification level as input and the multiple semantic labels as output to train the first sequence neural network to obtain a hierarchical classification model includes: using multiple sets of fused data sorted from high to low classification level as input to the multiple propagation units respectively, and using the multiple semantic labels as output to the multiple propagation units respectively, to train the long short-term memory neural network, wherein the unit state and hidden state of each propagation unit are determined based on the input and the unit state and hidden state of the upstream propagation units, and the output of each propagation unit is determined based on the corresponding hidden state.

[0010] In another implementation of the present invention, the step of extracting multiple image features of a first image sample based on multiple semantic tags includes: inputting the first image sample into a feature extraction model to obtain multiple image features of the first image sample, wherein the feature extraction model is used to extract features based on multiple classification levels from high to low.

[0011] In another implementation of the present invention, the feature extraction model includes a backbone network and multiple branch networks, wherein the output of the backbone network is connected to the input of each of the multiple branch networks. The method further includes: while locking the parameters of the pre-trained backbone network, using the first image sample as the input of the backbone network, and using the multiple semantic labels as the outputs of the multiple branch networks respectively, to train the multiple branch networks.

[0012] In another implementation of the present invention, the method further includes: unlocking the various parameters of the backbone network; using the first image sample as input to the backbone network and the plurality of semantic labels as outputs of the plurality of branch networks to further train the backbone network and the plurality of branch networks.

[0013] In another implementation of the present invention, the method further includes: extracting image features of the second image sample and the third image sample based on their respective target classification levels, wherein the target classification levels of the second image sample and the third image sample share a common classification level; fusing the image features of the second image sample and the third image sample to obtain fused image data; determining fused hierarchical semantic labels based on the second-level semantic labels between the target classification level of the second image sample and the common classification level, and the third-level semantic labels between the target classification level of the third image sample and the common classification level, wherein the second-level semantic labels and the third-level semantic labels share the label of the common classification level; and training the hierarchical classification model by using the fused image data as input to each propagation unit and using each label in the fused hierarchical semantic labels as output to the corresponding propagation unit.

[0014] In another implementation of the present invention, the hierarchical classification model further includes a semantic vector encoding model, wherein the input of the semantic vector encoding model is connected to the output of each propagation unit, and the input of the semantic vector encoding model is connected to the corresponding label in the fusion hierarchical semantic label.

[0015] According to a second aspect of the present invention, a hierarchical classification method is provided, comprising: extracting multiple image features of a target image and multiple classification levels corresponding to the multiple image features based on multiple classification levels, wherein the higher the classification level, the larger the classification granularity; inputting multiple sets of fused data sorted from high to low according to the classification level into a hierarchical classification model to obtain classification results belonging to the multiple classification levels, wherein the hierarchical classification model is trained according to the method described in the first aspect.

[0016] According to a third aspect of the present invention, a model training apparatus is provided, comprising: an extraction module, which extracts multiple image features of a first image sample based on multiple semantic labels, wherein the higher the classification level corresponding to the semantic label, the larger the classification granularity; a fusion module, which fuses the multiple semantic labels and the multiple image features to obtain multiple sets of fused data corresponding to the multiple semantic labels; and a training module, which takes the multiple sets of fused data sorted from high to low classification level as input, takes the multiple semantic labels as output, and trains a first sequence neural network to obtain a hierarchical classification model.

[0017] According to a fourth aspect of the present invention, a hierarchical classification apparatus is provided, comprising: an extraction module, which extracts multiple image features of a target image and multiple classification levels corresponding to the multiple image features based on multiple classification levels, wherein the higher the classification level, the larger the classification granularity; and a fusion module, which inputs multiple sets of fused data sorted from high to low according to the classification levels into a hierarchical classification model to obtain classification results belonging to the multiple classification levels, wherein the hierarchical classification model is trained according to the method described in the first aspect.

[0018] According to a fifth aspect of the present invention, an electronic device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, wherein the executable instruction causes the processor to perform an operation corresponding to the method described in the first or second aspect.

[0019] According to a sixth aspect of the present invention, a computer storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method as described in the first or second aspect.

[0020] In the embodiment of the present invention, the semantic labels of the classification hierarchy introduce prior knowledge of the classification hierarchy in addition to prior knowledge of the classification labels. The semantic labels of the classification hierarchy and image features are fused in multiple sets of fused data corresponding to multiple classification levels. This is beneficial for associating the input fused data and semantic labels in the training of the first sequence neural model. Furthermore, when training the first sequence model, the prior knowledge of the classification hierarchy and image features are temporally associated, so that the trained hierarchical classification model can reliably perform hierarchical classification prediction of images. Even when there are few image samples, reliable classification results are still given through the dependency relationship between classification levels, thus achieving reliable few-shot learning. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0022] Figure 1 This is a flowchart of the steps of a model training method according to an embodiment of the present invention.

[0023] Figure 2 This is a flowchart of the steps of a hierarchical classification method according to another embodiment of the present invention.

[0024] Figure 3A This is a schematic diagram of hierarchical classification according to another embodiment of the present invention.

[0025] Figure 3B This is a schematic diagram of the classification result of a hierarchical classification method according to another embodiment of the present invention.

[0026] Figure 3C This is a schematic diagram illustrating the training process of an exemplary hierarchical classification model according to another embodiment of the present invention.

[0027] Figure 4 This is a schematic diagram of a model training method according to another embodiment of the present invention.

[0028] Figure 5 This is a structural block diagram of a model training device according to another embodiment of the present invention.

[0029] Figure 6 This is a structural block diagram of a hierarchical classification device according to another embodiment of the present invention.

[0030] Figure 7 This is a schematic diagram of the structure of an electronic device according to another embodiment of the present invention. Detailed Implementation

[0031] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and thoroughly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art should fall within the protection scope of the present invention.

[0032] The specific implementation of the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0033] Figure 1This is a flowchart illustrating the steps of a model training method according to an embodiment of the present invention. The solution of this embodiment can be applied to any suitable electronic device with data processing capabilities, including but not limited to: servers, mobile terminals (such as mobile phones, tablets, etc.), and PCs. For example, in each model training stage, a computing device (e.g., a data center) configured with a CPU (processor unit) + GPU (acceleration unit) architecture can be used to train the encoder-decoder model based on training samples. The computing device, such as a data center, can be deployed in a cloud server such as a private cloud, a hybrid cloud, or a dedicated cloud. Correspondingly, in the inference stage of each trained model, a computing device configured with a CPU (processor unit) + GPU (acceleration unit) architecture can also be used for inference operations. The model training method of this embodiment includes:

[0034] S110: Extract multiple image features of the first image sample based on multiple classification levels, where the higher the classification level, the larger the classification granularity.

[0035] It should be understood that images, such as pictures, can have multiple levels of tags at different granularities, and there are hierarchical relationships between these tags, for example, clothing, menswear, shirts, plaid shirts, etc. The collection of semantic tags at multiple levels is constructed in the form of a tag tree, such as... Figure 3A As shown, toys belong to household goods, while barbecue belongs to food; they are different nodes within the same category level. Further, the next category level for toys is toy birds, and the next category level for toy birds is toy flamingos and toy seagulls; the next category level for barbecue is roasted poultry, and the next category level for roasted poultry is roasted chicken.

[0036] S120: Merge multiple semantic labels and multiple image features from multiple classification levels to obtain multiple sets of fused data corresponding to multiple classification levels.

[0037] It should be understood that multiple semantic tags constitute hierarchical semantic tags. For example, multiple semantic tags can be semantic text composed of characters, including but not limited to characters from various language families or languages, such as Sino-Tibetan language family characters such as Chinese characters, or Germanic language family characters such as words.

[0038] It should also be understood that, Figure 3A In the hierarchical tag tree, unlike the tag for toy flamingos, the hierarchical semantic tag for images including toy flamingos can be toy-toy birds-toy flamingos.

[0039] It should also be understood that feature fusion includes feature superposition or concatenation. For example, concatenating the vector of semantic labels at a classification level with the vector of image features to form a fusion vector, or processing the vector of semantic labels at a classification level and the vector of image features into two vectors with the same number of elements, and then adding the corresponding elements of the two vectors to obtain the superimposed vector.

[0040] S130: Take multiple sets of fused data sorted from high to low classification level as input, take multiple semantic labels as output, train the first sequence neural network, and obtain the hierarchical classification model.

[0041] It should be understood that the first sequence neural network can be a recurrent neural network (RNN) or a long short-term memory (LSTM) network. When multiple semantic labels constitute hierarchical semantic labels, the classification level of the hierarchical semantic labels is from high to low, consistent with the sequence direction of the first sequence neural network.

[0042] In the embodiment of the present invention, the semantic labels of the classification hierarchy introduce prior knowledge of the classification hierarchy in addition to prior knowledge of the classification labels. The semantic labels of the classification hierarchy and image features are fused in multiple sets of fused data corresponding to multiple classification levels. This is beneficial for associating the input fused data and semantic labels in the training of the first sequence neural model. Furthermore, when training the first sequence model, the prior knowledge of the classification hierarchy and image features are temporally associated, so that the trained hierarchical classification model can reliably perform hierarchical classification prediction of images. Even when there are few image samples, reliable classification results are still given through the dependency relationship between the levels, thus achieving reliable few-shot learning.

[0043] Accordingly, the trained hierarchical classification model can be used to perform hierarchical classification. Figure 2 This is a flowchart illustrating the steps of a hierarchical classification method according to another embodiment of the present invention. The hierarchical classification method of this embodiment includes:

[0044] S210: Extract multiple image features of the target image and multiple classification levels corresponding to the multiple image features based on multiple classification levels respectively. The higher the classification level, the larger the classification granularity.

[0045] S220: Input multiple sets of fused data, sorted from high to low according to classification level, into the hierarchical classification model to obtain classification results belonging to multiple classification levels.

[0046] Furthermore, such as Figure 3B As shown, Path 1 and Path 2 are for hierarchical classification results in, for example, Figure 3AExamples of paths in a hierarchical label tree. Hollow circles represent paths that correspond to a level in the hierarchical label tree, indicating correct hierarchical classification; solid circles represent paths that do not correspond to a level in the hierarchical label tree, indicating incorrect hierarchical classification. The solution of this embodiment of the invention, through the dependencies between levels, can provide the most reliable hierarchical classification results possible, ensuring that incorrect hierarchical classification results are concentrated to the lowest levels, and greatly improving the confidence of higher-level hierarchical classification results.

[0047] In other examples, multiple semantic labels and image features from multiple classification levels are fused to obtain multiple sets of fused data corresponding to multiple classification levels. This includes: obtaining the semantic feature vectors of each semantic label and the image feature vectors of each image feature from multiple classification levels; concatenating the semantic feature vector of each semantic label with its corresponding image feature vector to obtain the concatenated vector for each classification level; and determining the concatenated vector for each classification level as a set of fused data for each classification level. The semantic feature vector can be composed of word vectors of the semantic labels. By concatenating the semantic feature vector and the image feature vector, data processing is simplified. Incorporating semantic label information into the input side of the first sequence neural network is beneficial for reliably learning the association between semantic labels and image features.

[0048] In other examples, obtaining the semantic feature vectors of multiple semantic labels involves inputting multiple semantic labels into a pre-trained semantic vector encoding model to obtain the semantic feature vectors of each semantic label. The semantic vector encoding model can be used to perform positional encoding and contextual semantic encoding on the word vectors of semantic labels; for example, the semantic vector encoding model can have a BERT-based or transformer-based structure.

[0049] In other examples, the first sequence neural network is a long short-term memory neural network composed of multiple propagation units.

[0050] Multiple sets of fused data, sorted from high to low classification level, are used as input, and multiple semantic labels are used as output. A first-sequence neural network is trained to obtain a hierarchical classification model, including:

[0051] Multiple sets of fused data, sorted from high to low according to classification level, are used as inputs to multiple propagation units. Multiple semantic labels are used as outputs to train a long short-term memory neural network. The unit state and hidden state of each propagation unit are determined based on the input and the unit state and hidden state of the upstream propagation unit, and the output of each propagation unit is determined based on the corresponding hidden state.

[0052] For example, Figure 3CAs shown, the backbone network 310 and branch networks 311, 312, and 313 constitute the feature extraction model. A first image sample can be input into the feature extraction model to obtain multiple image features of the first image sample. The feature extraction model is used for feature extraction based on multiple classification levels from high to low. That is, the classification granularity of the image features extracted by branch networks 311, 312, and 313 (e.g., represented as feature maps) is arranged from large to small, thereby extracting multiple image features of the first image sample.

[0053] The input of the backbone network 310 is used as the input of the feature extraction model. The output of the backbone network 310 is connected to the input of the branch networks 311, 312 and 313. The outputs of the branch networks 311, 312 and 313 are used as the output of the feature extraction model.

[0054] The backbone network 310 and branch networks 311, 312, and 313 can all be different convolutional neural networks. Training the feature extraction model can include training the backbone network 310 and training the branch networks 311, 312, and 313. Before training the branch networks, the backbone network 310 needs to be pre-trained. Furthermore, branch networks 311, 312, and 313 can also be pre-trained; for example, while locking the parameters of the pre-trained backbone network, a first image sample can be used as the input to the backbone network, and multiple semantic labels can be used as the outputs of multiple branch networks to train multiple branch networks.

[0055] When training the branch networks, joint training of the backbone network 310 and branch networks 311, 312 and 313 can be performed (i.e., end-to-end training of the feature extraction model). Specifically, the locking of each parameter of the backbone network can be released, and then the first image sample is used as the input of the backbone network, and multiple semantic labels are used as the output of multiple branch networks to further train the backbone network and multiple branch networks.

[0056] The first sequence neural network is a long short-term memory neural network 320 composed of multiple propagation units, including various propagation units 321, 322 and 323.

[0057] Specifically, the unit state c1 and hidden state h1 of propagation unit 1 are determined based on the input and the unit states (e.g., initial unit state) and hidden states (e.g., initial hidden state) of its upstream propagation unit (e.g., the initial propagation unit). The unit state c2 and hidden state h2 of propagation unit 2 are determined based on the input and the unit state c1 and hidden state h1 of its upstream propagation unit 1. The unit state c3 and hidden state h3 of propagation unit 3 are determined based on the input and the unit state c2 and hidden state h2 of its upstream propagation unit 2. The output of propagation unit 1 is determined based on hidden state 1, the output of propagation unit 2 is determined based on hidden state 2, and the output of propagation unit 3 is determined based on hidden state 3. In other words, the unit states of each propagation unit change in the sequence direction (e.g., from propagation unit 321 to propagation unit 323), reflecting the dependence of the processing of the propagation unit on its upstream and downstream propagation units. The hidden states of the propagation units reflect their influence on the output of the propagation units.

[0058] During the training phase of the Long Short-Term Memory Neural Network 320, e1 is the encoding result of the initial semantic label (e.g., through encoding layer 1), e2 is the encoding result of semantic label 1 (e.g., through encoding layer 2), and e3 is the encoding result of semantic label 2 (e.g., through encoding layer 3). Image features f1, f2, and f3 are the feature extraction results of branch networks 1, 2, and 3 on the image (as image samples during the training phase and as target images to be classified during the inference phase), respectively.

[0059] Furthermore, the hierarchical classification model also includes a semantic vector encoding model, which connects the outputs of each propagation unit 321, 322, and 323 with each semantic label. That is, each encoding layer connects the hidden state and the semantic label. In other words, the encoding layer of each classification level is used to convert the semantic label into a semantic feature vector (word vector). This semantic feature vector serves both as the output of the hidden state of that classification level and is fused with the image features of the next classification level. More specifically, the concatenation of e1 and f1 serves as the input of propagation unit 1, and the semantic feature vector of semantic label 1 serves as the output of propagation unit 1. The concatenation of e2 and f2 serves as the input of propagation unit 2, and the semantic feature vector of semantic label 2 serves as the output of propagation unit 2. The concatenation of e3 and f3 serves as the input of propagation unit 3, and the semantic feature vector of semantic label 3 serves as the output of propagation unit 3. Encoding layers 1, 2, and 3 are examples of the semantic vector encoding model.

[0060] In the inference phase of the Long Short-Term Memory Neural Network 320, the concatenation of e1 and f1 serves as the input to propagation unit 1, yielding classification level 1, which is then the output of propagation unit 1. Next, the encoding e2 for classification level 1 is obtained, and the concatenation of e2 and f2 serves as the input to propagation unit 2, yielding classification level 2, which is then the output of propagation unit 2. Then, the encoding e3 for classification level 2 is obtained, and the concatenation of e3 and f3 serves as the input to propagation unit 3, yielding classification level 3, which is then the output of propagation unit 3. This process continues, with classification levels 1, 2, 3… constituting the hierarchical classification result.

[0061] It should be understood that the hierarchical classification model, the semantic vector encoding model, and the fusion operator of image features and semantic labels (e.g., the concat operator) constitute the backbone of the hierarchical classification, while the feature extraction model constitutes the preprocessing part. Joint training of the preprocessing part and the backbone can also be performed. For example, end-to-end training of the feature extraction model and the hierarchical classification model can be performed. That is, while locking the parameters of the pre-trained backbone network and the semantic vector encoding model, the parameters of each branch network and the hierarchical classification model (i.e., the first sequence neural network) can be adjusted. Then, after unlocking the parameters of the backbone network, the parameters of the feature extraction model and the hierarchical classification model can be further adjusted. In other words, the parameters of the hierarchical classification model can be adjusted independently, or they can be adjusted uniformly with the parameters of the backbone network, the parameters of each branch network, or the parameters of the backbone network and the parameters of each branch network to perform joint training. In this case, the output of joint training is the same as the output of the independently trained hierarchical classification model, that is, the hierarchical semantic labels formed by multiple semantic labels themselves; unlike the input of the independently trained hierarchical classification model, the input of joint training is the first image sample.

[0062] In other examples, the model training method further includes: extracting image features from the second and third image samples based on their respective target classification levels, where the target classification levels of the second and third image samples share a common classification level; fusing the image features of the second and third image samples to obtain fused image data; determining fused hierarchical semantic labels based on the second-level semantic labels between the target classification level and the common classification level of the second image sample, and the third-level semantic labels between the target classification level and the common classification level of the third image sample, where the second-level semantic labels and the third-level semantic labels share a common classification level; using the fused image data as input to each propagation unit, and using each label in the fused hierarchical semantic labels as output to the corresponding propagation unit, thus training the hierarchical classification model. Through the training process in this example, the features of each label in the fused hierarchical semantic labels are aligned with the features of the second and third image samples. Specifically, the following combines... Figure 4 The model training process according to another embodiment of the present invention will be described. Figure 4 The model training process can be performed in Figure 1 Before the training process, the feature extraction model 410 may include a backbone network and multiple branch networks. Image x (the second image sample) and image y (the third image sample) are input into the feature extraction model 410 to obtain the target classification level features fx and fy of image x and image y, respectively. The target classification level of image x and image y can be the same or different. For example, the target classification level of image x and image y can both be level 4, and their shared label is level 2. That is, the second-level semantic label is level 2 shared label - level 3 x label - level 4 x label, and the third-level semantic label is level 2 shared label - level 3 y label - level 4 y label. Correspondingly, the fusion level semantic label is level 4 x label - level 3 x label - level 2 shared label - level 3 y label - level 4 y label, corresponding to propagation units 1, 2, 3, 4, and 5, respectively. The input to each propagation unit is the fused data of image x and image y. Generally, the target classification level of the second image sample and the target classification level of the third image sample can be the same or different. Furthermore, the shared classification level is higher than the target classification level of the second image sample and the target classification level of the third image sample.

[0063] It should be understood that the semantic labels of the fusion hierarchy can also be reversed to obtain another labeled sample of the fused image data. Generally speaking, the labels of the second image sample and the labels of the third image sample belong to the same hierarchical label tree, so the labels of the second image sample and the labels of the third image sample have the same classification level. Based on the hierarchical label tree, a large number of labels for the second image sample and the third image sample can be constructed, increasing the actual number of training samples.

[0064] Furthermore, the hierarchical classification model also includes a semantic vector encoding model 430. The input of the semantic vector encoding model 430 is connected to the output of each propagation unit, and the input of the semantic vector encoding model 430 is connected to the corresponding label in the fusion hierarchical semantic label. This facilitates the alignment of the vectors of each semantic label with the vectors of the corresponding image features, thereby improving the accuracy of the classification. Figure 1 The reliability of the trained hierarchical classification model. It should be understood that the number of propagation units involved in the training process of this example is related to the number of labels in the fused hierarchical semantic labels, and can be compared with... Figure 1 The number of propagation units in the examples can be the same or different. Through training with a large number of training samples in this example, the matching between the semantic vector encoding model and the various propagation units of the long short-term memory network is improved.

[0065] Figure 5 This is a structural block diagram of a model training apparatus according to another embodiment of the present invention. The solution of this embodiment can be applied to any suitable electronic device with data processing capabilities, including but not limited to: servers, mobile terminals (such as mobile phones, PADs, etc.), and PCs. For example, in each model training stage, a computing device (e.g., a data center) configured with a CPU (processor unit) + GPU (acceleration unit) architecture can be used to train the encoder-decoder model based on training samples. The computing device, such as a data center, can be deployed in a cloud server such as a private cloud, a hybrid cloud, or a dedicated cloud. Correspondingly, in the inference stage of each trained model, inference operations can also be performed using a computing device configured with a CPU (processor unit) + GPU (acceleration unit) architecture. The model training apparatus of this embodiment includes:

[0066] The extraction module 510 extracts multiple image features of the first image sample based on multiple semantic labels. The higher the classification level corresponding to the semantic label, the larger the classification granularity.

[0067] The fusion module 520 fuses the multiple semantic tags and the multiple image features respectively to obtain multiple sets of fused data corresponding to the multiple semantic tags.

[0068] The training module 530 takes multiple sets of fused data sorted from high to low classification level as input and the multiple semantic labels as output to train the first sequence neural network to obtain a hierarchical classification model.

[0069] In the embodiment of the present invention, the semantic labels of the classification hierarchy introduce prior knowledge of the classification hierarchy in addition to prior knowledge of the classification labels. The semantic labels of the classification hierarchy and image features are fused in multiple sets of fused data corresponding to multiple classification levels. This is beneficial for associating the input fused data and semantic labels in the training of the first sequence neural model. Furthermore, when training the first sequence model, the prior knowledge of the classification hierarchy and image features are temporally associated, so that the trained hierarchical classification model can reliably perform hierarchical classification prediction of images. Even when there are few image samples, reliable classification results are still given through the dependency relationship between classification levels, thus achieving reliable few-shot learning.

[0070] In other examples, the fusion module is specifically used to: obtain the semantic feature vectors of each semantic label of the multiple classification levels and the image feature vectors of each image feature; concatenate the semantic feature vector of each semantic label and the corresponding image feature vector to obtain the concatenation vector of each classification level; and determine the concatenation vector of each classification level as a set of fused data for each classification level.

[0071] In other examples, the fusion module is specifically used to: input the multiple semantic labels into a pre-trained semantic vector encoding model to obtain the semantic feature vectors of each of the multiple semantic labels.

[0072] In other examples, the first sequence neural network is a long short-term memory neural network composed of multiple propagation units. The training module is specifically used to: use multiple sets of fused data sorted from high to low classification level as inputs to the multiple propagation units, and use the multiple semantic labels as outputs to the multiple propagation units, thereby training the long short-term memory neural network. The unit state and hidden state of each propagation unit are determined based on the input and the unit states and hidden states of the upstream propagation units, and the output of each propagation unit is determined based on its corresponding hidden state.

[0073] In other examples, the extraction module is specifically used to: input a first image sample into a feature extraction model to obtain multiple image features of the first image sample, wherein the feature extraction model is used to extract features based on multiple classification levels from high to low.

[0074] In other examples, the feature extraction model includes a backbone network and multiple branch networks, with the output of the backbone network connected to the inputs of the respective branch networks. The training module is further configured to: while locking the parameters of the pre-trained backbone network, use the first image sample as input to the backbone network and the multiple semantic labels as outputs of the multiple branch networks, respectively, to train the multiple branch networks.

[0075] In other examples, the training module is also used to: unlock the various parameters of the backbone network; use the first image sample as the input of the backbone network and the multiple semantic labels as the output of the multiple branch networks to further train the backbone network and the multiple branch networks.

[0076] In other examples, the training module is further configured to: extract image features of the second and third image samples based on their respective target classification levels, wherein the target classification levels of the second and third image samples share a common classification level; fuse the image features of the second and third image samples to obtain fused image data; determine fused hierarchical semantic labels based on the second-level semantic labels between the target classification level of the second image sample and the common classification level, and the third-level semantic labels between the target classification level of the third image sample and the common classification level, wherein the second-level semantic labels and the third-level semantic labels share the common classification level; and train the hierarchical classification model by using the fused image data as input to each propagation unit and each label in the fused hierarchical semantic labels as output to the corresponding propagation unit.

[0077] In other examples, the hierarchical classification model further includes a semantic vector encoding model, the input of which is connected to the output of each propagation unit, and the input of which is connected to the corresponding label in the fusion hierarchical semantic label.

[0078] Figure 6 This is a structural block diagram of a hierarchical classification device according to another embodiment of the present invention. The hierarchical classification device of this embodiment includes:

[0079] The extraction module 610 extracts multiple image features of the target image and multiple classification levels corresponding to the multiple image features based on multiple classification levels, wherein the higher the classification level, the larger the classification granularity.

[0080] The fusion module 620 inputs multiple sets of fused data sorted from high to low according to the classification level into the hierarchical classification model to obtain the classification results belonging to the multiple classification levels.

[0081] The apparatus of this embodiment is used to implement the corresponding methods in the foregoing method embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here. Furthermore, the functional implementation of each module in the apparatus of this embodiment can be referred to the description of the corresponding part in the foregoing method embodiments, which will also not be repeated here.

[0082] Reference Figure 5 The diagram shows a schematic of an electronic device according to another embodiment of the present invention. The specific embodiments of the present invention do not limit the specific implementation of the electronic device.

[0083] like Figure 5 As shown, the electronic device may include: a processor 502, a communications interface 504, a memory 506 storing a program 510, and a communications bus 508.

[0084] The processor, communication interface, and memory communicate with each other via a communication bus.

[0085] A communication interface is used to communicate with other electronic devices or servers.

[0086] The processor is used to execute programs, specifically the relevant steps in the above method embodiments.

[0087] Specifically, the program may include program code, which includes at least one executable instruction.

[0088] The processor may be a CPU, an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The one or more processors included in a smart device may be of the same type, such as one or more CPUs; or they may be of different types, such as one or more CPUs and one or more ASICs.

[0089] Memory is used to store programs. Memory may include high-speed RAM, and may also include non-volatile memory, such as at least one disk drive.

[0090] Specifically, the program can be used to cause the processor to execute the following: Figure 1 Model training methods or Figure 2 The hierarchical classification method.

[0091] Furthermore, the specific implementation of each step in the program can be found in the corresponding descriptions of the steps and units in the above method embodiments, and will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0092] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of the present invention can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present invention.

[0093] The methods described above according to embodiments of the present invention can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code originally stored on a remote recording medium or a non-transitory machine-readable medium and subsequently stored on a local recording medium, downloaded via a network. Thus, the methods described herein can be processed by software stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the methods shown herein.

[0094] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of the embodiments of the present invention.

[0095] The above embodiments are only used to illustrate the embodiments of the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of the present invention. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of the present invention, and the patent protection scope of the embodiments of the present invention should be defined by the claims.

Claims

1. A model training method, comprising: Based on multiple classification levels, multiple image features of the first image sample are extracted, where the higher the classification level, the larger the classification granularity. Multiple semantic labels are input into a pre-trained semantic vector encoding model to obtain the semantic feature vectors of each semantic label. The semantic feature vector of each semantic label and the image feature vector of each image feature are concatenated separately to obtain the concatenated vector for each classification level; Multiple sets of fused data, sorted from high to low according to classification level, are used as inputs to multiple propagation units. The multiple semantic labels are used as outputs to the multiple propagation units. A long short-term memory neural network is trained. The unit state and hidden state of each propagation unit are determined based on the input and the unit state and hidden state of the upstream propagation unit. The output of each propagation unit is determined based on the corresponding hidden state. Among them, a set of fused data for each classification level is the concatenation vector of that classification level.

2. The method according to claim 1, wherein, The extraction of multiple image features from the first image sample based on multiple classification levels includes: The first image sample is input into the feature extraction model to obtain multiple image features of the first image sample. The feature extraction model is used to extract features based on multiple classification levels from high to low.

3. The method according to claim 2, wherein, The feature extraction model includes a backbone network and multiple branch networks, and the output of the backbone network is connected to the input of each of the multiple branch networks. The method further includes: While locking the parameters of the pre-trained backbone network, the first image sample is used as the input of the backbone network, and the multiple semantic labels are used as the outputs of the multiple branch networks to train the multiple branch networks.

4. The method according to claim 3, wherein, The method further includes: Release the lock on each parameter of the backbone network; The first image sample is used as the input to the backbone network, and the multiple semantic labels are used as the outputs of the multiple branch networks to further train the backbone network and the multiple branch networks.

5. The method according to claim 1, wherein, The method further includes: Based on the target classification levels of the second and third image samples, image features of the second and third image samples are extracted. The target classification levels of the second and third image samples share a common classification level. By fusing the image features of the second image sample and the third image sample, fused image data is obtained; Based on the second-level semantic label between the target classification level and the shared classification level of the second image sample, and the third-level semantic label between the target classification level and the shared classification level of the third image sample, a fusion-level semantic label is determined, wherein the second-level semantic label and the third-level semantic label have the label of the shared classification level in the fusion-level semantic label; The fused image data is used as the input to each propagation unit, and each label in the fused hierarchical semantic label is used as the output of the corresponding propagation unit to train the hierarchical classification model.

6. The method according to claim 5, wherein, The hierarchical classification model also includes a semantic vector encoding model, the input of which is connected to the output of each propagation unit, and the input of which is connected to the corresponding label in the fusion hierarchical semantic label.

7. A hierarchical classification method, comprising: Based on multiple classification levels, multiple image features of the target image and multiple classification levels corresponding to the multiple image features are extracted, wherein the higher the classification level, the larger the classification granularity; Multiple sets of fused data, sorted from high to low according to classification level, are input into a hierarchical classification model to obtain classification results belonging to the multiple classification levels. The hierarchical classification model is trained according to any one of claims 1 to 6, wherein each set of fused data is obtained by concatenating each image feature with the encoding result of the corresponding classification level's encoding layer.

8. A model training device, comprising: The extraction module extracts multiple image features of the first image sample based on multiple semantic labels. The higher the classification level of the semantic label, the larger the classification granularity. The fusion module inputs the multiple semantic labels into a pre-trained semantic vector encoding model to obtain the semantic feature vectors of each semantic label. It then concatenates the semantic feature vector of each semantic label with the image feature vector of each image feature to obtain the concatenated vector for each classification level. The training module uses multiple sets of fused data sorted from high to low classification level as input to multiple propagation units, and uses the multiple semantic labels as output to the multiple propagation units to train a long short-term memory neural network. The unit state and hidden state of each propagation unit are determined based on the input and the unit state and hidden state of the upstream propagation unit, and the output of each propagation unit is determined based on the corresponding hidden state. Among them, a set of fused data for each classification level is the concatenation vector of that classification level.

9. A hierarchical classification device, comprising: The extraction module extracts multiple image features of the target image and the multiple classification levels corresponding to the multiple image features based on multiple classification levels, wherein the higher the classification level, the larger the classification granularity; The fusion module inputs multiple sets of fused data, sorted from high to low according to classification level, into a hierarchical classification model to obtain classification results belonging to the multiple classification levels. The hierarchical classification model is trained according to the method of any one of claims 1 to 6, wherein each set of fused data is obtained by concatenating each image feature with the encoding result of the corresponding classification level's encoding layer.

10. An electronic device, comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the operation corresponding to the method as described in any one of claims 1-7.

11. A computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of claims 1-7.

Images