Image classification model training method, image classification method, device and equipment

Abstract

The embodiment of the invention discloses an image classification model training method, an image classification method, an image classification device and equipment. A specific embodiment of the method comprises the steps of obtaining a target label set; performing semantic classification on the target label set to obtain at least one classification label group; for each classification label group in the at least one classification label group, allocating the classification label group to a corresponding data annotation end, so that the data annotation end generates a training image sample set corresponding to the classification label group; and according to the obtained at least one training image sample set, performing model training on the initial image classification model to obtain an image classification model. The embodiment is related to artificial intelligence, and the labeled image sample set for the target label set can be accurately and efficiently generated.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to an image classification model training method, an image classification method, a device, and an apparatus. Background Art

[0002] At present, image classification tasks are the foundation and important research direction of deep learning and computer vision. In practice, image classification neural network models are often used to implement image classification. For the model training of image classification neural network models, a large amount of labeled data is required. For the generation of labeled image samples, the usual method is: for unlabeled images, relevant labelers manually perform multi-label labeling to obtain labeled image samples.

[0003] However, the inventors have found that when the above method is used to generate label image samples, the following technical problems often occur:

[0004] For multi-label image annotation, since there are many label categories and the category annotations corresponding to different label categories correspond to different annotation standards, the manual annotation cost is high, inaccurate, and not efficient enough.

[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0006] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0007] Some embodiments of the present disclosure propose image classification model training methods, image classification methods, devices, electronic devices, computer-readable media and program products to solve the technical problems mentioned in the above background technology section.

[0008] In a first aspect, some embodiments of the present disclosure provide an image classification model training method, comprising: obtaining a target label set; performing semantic classification on the target label set to obtain at least one classification label group; for each classification label group in the at least one classification label group, assigning the classification label group to a corresponding data annotation end, so that the data annotation end generates a training image sample set corresponding to the classification label group; and performing model training on an initial image classification model based on the at least one training image sample set obtained to obtain an image classification model.

[0009] Optionally, the above-mentioned method of training the initial image classification model based on the at least one training image sample set obtained to obtain the image classification model includes: selecting training image samples from the at least one training image sample set as target training image samples, and performing the following training steps: inputting the image included in the target training image sample into the image feature extraction model to generate image feature information; using the above-mentioned initial image classification model, generating a label feature information group corresponding to each classification label group in the above-mentioned at least one classification label group, and obtaining at least one label feature information group; using the above-mentioned initial image classification model, generating classification loss information based on the above-mentioned at least one label feature information group and the above-mentioned image feature information; determining whether the model training of the above-mentioned initial image classification model is completed based on the above-mentioned classification loss information; in response to the completion of the training, determining the above-mentioned initial image classification model as the image classification model.

[0010] Optionally, the method further includes: in response to determining that the initial image classification model has not been trained, adjusting relevant model parameters in the initial image classification model, and reselecting target training image samples from the at least one training image sample set, and continuing to execute the training steps.

[0011] Optionally, the above-mentioned initial image classification model includes: an initial encoding model based on a first attention mechanism; and the above-mentioned generation of a label feature information group corresponding to each classification label group in the above-mentioned at least one classification label group includes: inputting each classification label in the above-mentioned classification label group into the above-mentioned initial encoding model based on the first attention mechanism to generate the above-mentioned label feature information group.

[0012] Optionally, the above-mentioned initial image classification model also includes: an initial decoding model based on a second attention mechanism; and the above-mentioned generation of classification loss information based on the above-mentioned at least one label feature information group and the above-mentioned image feature information, including: inputting the above-mentioned at least one label feature information group and the above-mentioned image feature information into the above-mentioned initial decoding model based on the second attention mechanism to output classification results for each label category; generating the above-mentioned classification loss information based on the label category information included in the above-mentioned target training image sample and the above-mentioned classification result.

[0013] In a second aspect, some embodiments of the present disclosure provide an image classification model training device, comprising: a first acquisition unit, configured to acquire a target label set; a semantic classification unit, configured to perform semantic classification on the target label set to obtain at least one classification label group; an allocation unit, configured to allocate the classification label group to a corresponding data annotation end for each classification label group in the at least one classification label group, so that the data annotation end generates a training image sample set corresponding to the classification label group; a training unit, configured to perform model training on an initial image classification model according to the at least one training image sample set obtained to obtain an image classification model.

[0014] Optionally, the training unit can be configured to: select training image samples from the above-mentioned at least one training image sample set as target training image samples, and perform the following training steps: input the image included in the above-mentioned target training image samples into an image feature extraction model to generate image feature information; use the above-mentioned initial image classification model to generate a label feature information group corresponding to each classification label group in the above-mentioned at least one classification label group, and obtain at least one label feature information group; use the above-mentioned initial image classification model to generate classification loss information based on the above-mentioned at least one label feature information group and the above-mentioned image feature information; determine whether the above-mentioned initial image classification model has completed model training based on the above-mentioned classification loss information; in response to the completion of training, determine the above-mentioned initial image classification model as the image classification model.

[0015] Optionally, the training unit can be configured to: in response to determining that the initial image classification model has not been trained, adjust relevant model parameters in the initial image classification model, and reselect target training image samples from the above at least one training image sample set to continue executing the above training steps.

[0016] Optionally, the above-mentioned initial image classification model includes: an initial encoding model based on a first attention mechanism; and the training unit can be configured to: input each classification label in the above-mentioned classification label group into the above-mentioned initial encoding model based on the first attention mechanism to generate the above-mentioned label feature information group.

[0017] Optionally, the above-mentioned initial image classification model also includes: an initial decoding model based on a second attention mechanism; and the training unit can be configured to: input the above-mentioned at least one label feature information group and the above-mentioned image feature information into the above-mentioned initial decoding model based on the second attention mechanism to output classification results for each label category; generate the above-mentioned classification loss information according to the label category information included in the above-mentioned target training image sample and the above-mentioned classification results.

[0018] In a third aspect, some embodiments of the present disclosure provide an image classification method, comprising: obtaining at least one target label feature information group corresponding to a target image and at least one classification label group, wherein the at least one target label feature information group is pre-generated based on an image classification model, and the image classification model is generated based on the image classification model training method corresponding to the first aspect; generating image feature information for the target image as target image feature information; and generating an image classification result for the target image based on the at least one target label feature information group and the target image feature information using the image classification model.

[0019] Optionally, the above-mentioned image classification model includes: an encoding model based on a first attention mechanism; and the above-mentioned at least one target label feature information group is generated by the following steps: for each classification label group in the above-mentioned at least one classification label group, each classification label in the above-mentioned classification label group is input into the above-mentioned encoding model based on the first attention mechanism to generate a label feature information group as the target label feature information group.

[0020] Optionally, the above-mentioned image classification model includes: a decoding model based on a second attention mechanism; and based on the above-mentioned at least one target label feature information group and the above-mentioned target image feature information, using the above-mentioned image classification model to generate an image classification result for the above-mentioned target image, including: inputting the above-mentioned at least one target label feature information group and the above-mentioned target image feature information into the above-mentioned decoding model based on the second attention mechanism to generate the above-mentioned image classification result.

[0021] In a fourth aspect, some embodiments of the present disclosure provide an image classification device, comprising: a second acquisition unit, configured to acquire a target image and at least one target label feature information group corresponding to at least one classification label group, wherein the at least one target label feature information group is pre-generated based on an image classification model, and the image classification model is generated based on the image classification model training method corresponding to the first aspect; a first generation unit, configured to: generate image feature information for the target image as target image feature information; and a second generation unit, configured to generate an image classification result for the target image based on the at least one target label feature information group and the target image feature information and using the image classification model.

[0022] Optionally, the above-mentioned image classification model includes: an encoding model based on a first attention mechanism; and the above-mentioned at least one target label feature information group is generated by the following steps: for each classification label group in the above-mentioned at least one classification label group, each classification label in the above-mentioned classification label group is input into the above-mentioned encoding model based on the first attention mechanism to generate a label feature information group as the target label feature information group.

[0023] Optionally, the above-mentioned image classification model includes: a decoding model based on a second attention mechanism; and the second generation unit can be configured to: input the above-mentioned at least one target label feature information group and the above-mentioned target image feature information into the above-mentioned decoding model based on the second attention mechanism to generate the above-mentioned image classification result.

[0024] In a fifth aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any one of the implementation methods in the first and third aspects.

[0025] In a sixth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any one of the implementation modes of the first aspect and the third aspect is implemented.

[0026] In a seventh aspect, some embodiments of the present disclosure provide a computer program product, including a computer program, which, when executed by a processor, implements the method described in any one of the implementation modes of the first and third aspects above.

[0027] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the image classification model training method of some embodiments of the present disclosure, an image sample set annotated for the target label set can be accurately and efficiently generated. Specifically, the reason why the generation of the relevant image sample set is not accurate and efficient is that for multi-label image annotation, due to the large number of label categories and the different labeling standards corresponding to the category annotations of different label categories, the manual annotation cost is large, inaccurate, and inefficient. Based on this, the image classification model training method of some embodiments of the present disclosure first obtains the target label set to subsequently determine the data style corresponding to the training data set, which is convenient for the subsequent training of the image classification model. Then, the above-mentioned target label set is semantically classified to obtain at least one classification label group. Here, the label semantics corresponding to each classification label group obtained are similar. Through the semantic classification method, it can be sent to the data annotation end in a targeted manner, so that the corresponding data annotation end only generates the training data set corresponding to the corresponding classification label group, and does not need to annotate all the classification labels. In this way, the annotation cost of data annotation can be greatly reduced and the annotation efficiency of data annotation can be improved. Furthermore, for each classification label group in the at least one classification label group, the classification label group is assigned to the corresponding data annotation end, so that the data annotation end can generate a training image sample set corresponding to the classification label group. Here, through the corresponding assignment of classification label groups, the data annotation end can only specifically annotate the label category of the corresponding image, which greatly improves the annotation efficiency. Finally, according to the obtained at least one training image sample set, the initial image classification model is trained to obtain an image classification model with more accurate image classification. In summary, through the semantic classification of the target label set and the annotation assignment of the corresponding classification label group, the annotation efficiency is greatly improved and the annotation cost is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0029] Figure 1 is a schematic diagram of an application scenario of an image classification model training method according to some embodiments of the present disclosure;

[0030] Figure 2 is a flowchart of some embodiments of the image classification model training method according to the present disclosure;

[0031] Figure 3 is a flowchart of other embodiments of the image classification model training method according to the present disclosure;

[0032] Figure 4 is a flow chart of some embodiments of the image classification method according to the present disclosure;

[0033] Figure 5 is a schematic diagram of the structure of some embodiments of the image classification model training device according to the present disclosure;

[0034] Figure 6 is a schematic diagram of the structure of some embodiments of the image classification device according to the present disclosure;

[0035] Figure 7 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0036] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0037] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0038] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0039] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0040] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0041] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0042] Figure 1 It is a schematic diagram of an application scenario of the image classification model training method according to some embodiments of the present disclosure.

[0043] exist Figure 1In the application scenario, first, the electronic device 101 can obtain the target tag set 102. In this application scenario, the target tag set 102 includes: "solid color", "stripes", "color matching", "high collar", "square collar", "Hong Kong style" and "Korean style". Then, the electronic device 101 can semantically classify the above target tag set 102 to obtain at least one classification tag group 103. In this application scenario, at least one classification tag group 103 may include: a first classification tag group consisting of "solid color", "stripes" and "color matching", a second classification tag group consisting of "high collar" and "square collar", and a third classification tag group consisting of "Hong Kong style" and "Korean style". Then, the electronic device 101 can assign the above classification tag group to the corresponding data annotation end for each classification tag group in the above at least one classification tag group 103, so that the above data annotation end generates a training image sample set corresponding to the above classification tag group. In this application scenario, for the first classification tag group, a corresponding first training image sample set is generated. For the second classification tag group, a corresponding second training image sample set is generated. For the third classification label group, a corresponding third training image sample set is generated. Finally, the electronic device 101 can perform model training on the initial image classification model 105 according to the obtained at least one training image sample set 104 to obtain an image classification model 106 .

[0044] It should be noted that the electronic device 101 can be hardware or software. When the electronic device is hardware, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or it can be implemented as a single server or a single terminal device. When the electronic device is embodied as software, it can be installed in the hardware devices listed above. It can be implemented as multiple software or software modules for providing distributed services, for example, or it can be implemented as a single software or software module. No specific limitation is made here.

[0045] It should be understood that Figure 1 The number of electronic devices in the embodiment is only for illustration. Any number of electronic devices may be provided according to implementation requirements.

[0046] Continue to refer Figure 2 , shows a process 200 of some embodiments of the image classification model training method according to the present disclosure. The image classification model training method comprises the following steps:

[0047] Step 201, obtaining a target tag set.

[0048] In some embodiments, the execution subject of the above-mentioned image classification model training method (for example Figure 1The electronic device 101 shown can obtain the target label set through a wired connection or a wireless connection. The target label in the target label set can be a label of the corresponding training image sample set to be generated. That is, the sample label corresponding to the training image sample set is the corresponding target label. For example, if the target label is "pure color", the sample label corresponding to the corresponding training image sample set is "pure color".

[0049] Step 202: semantically classify the target tag set to obtain at least one classification tag group.

[0050] In some embodiments, the execution subject may semantically classify the target tag set to obtain at least one classification tag group. The tags corresponding to the classification tags in each classification tag group are similar or identical in semantics. For example, the classification tag group may include: "solid color", "stripes" and "color matching".

[0051] As an example, first, the execution subject may input the target tag in the target tag set into the word embedding model to output the tag vector to obtain a tag vector set. Then, the tag vector set is subjected to vector clustering processing to obtain at least one tag vector group. Finally, at least one classification tag group is determined for the at least one tag vector group.

[0052] Step 203: for each classification label group in the at least one classification label group, the classification label group is allocated to a corresponding data annotation end, so that the data annotation end generates a training image sample set corresponding to the classification label group.

[0053] In some embodiments, the execution subject may assign the classification label group to a corresponding data annotation terminal for each classification label group in the at least one classification label group, so that the data annotation terminal can generate a training image sample set corresponding to the classification label group. The data annotation terminal may be an operation terminal for labeling the initial image. In practice, the data annotation terminal may be an operation terminal for the relevant operator to label. The training image samples in the training image sample set all have corresponding labels. Each classification label group has a unique corresponding data annotation terminal to perform the generation of the training image sample set corresponding to the subsequent classification label group.

[0054] As an example, first, the execution subject may determine the label annotation information corresponding to each data annotation end. The label annotation information may be information that the label corresponds to the image sample responsible for generating the information. Then, the classification label group is assigned to the corresponding data annotation end, so that the data annotation end generates a training image sample set corresponding to the classification label group.

[0055] Step 204: Perform model training on the initial image classification model based on the obtained at least one training image sample set to obtain an image classification model.

[0056] In some embodiments, the execution subject may perform model training on the initial image classification model according to the obtained at least one training image sample set to obtain the image classification model. The initial image classification model may be an image classification model for which the model training has not yet been completed. The image classification model may be a neural network model for generating a category corresponding to the image content of the image. For example, the image classification model may be a multi-layer serially connected convolutional neural network (CNN) model.

[0057] As an example, the execution subject may use at least one training image sample set as a training data set to perform model training on the initial image classification model to obtain an image classification model, wherein the specific training method is not described in detail.

[0058] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the image classification model training method of some embodiments of the present disclosure, an image sample set annotated for the target label set can be accurately and efficiently generated. Specifically, the reason why the generation of the relevant image sample set is not accurate and efficient is that for multi-label image annotation, due to the large number of label categories and the different labeling standards corresponding to the category annotations of different label categories, the manual annotation cost is large, inaccurate, and inefficient. Based on this, the image classification model training method of some embodiments of the present disclosure first obtains the target label set to subsequently determine the data style corresponding to the training data set, which is convenient for the subsequent training of the image classification model. Then, the above-mentioned target label set is semantically classified to obtain at least one classification label group. Here, the label semantics corresponding to each classification label group obtained are similar. Through the semantic classification method, it can be sent to the data annotation end in a targeted manner, so that the corresponding data annotation end only generates the training data set corresponding to the corresponding classification label group, and does not need to annotate all the classification labels. In this way, the annotation cost of data annotation can be greatly reduced and the annotation efficiency of data annotation can be improved. Furthermore, for each classification label group in the at least one classification label group, the classification label group is assigned to the corresponding data annotation end, so that the data annotation end can generate a training image sample set corresponding to the classification label group. Here, through the corresponding assignment of classification label groups, the data annotation end can only specifically annotate the label category of the corresponding image, which greatly improves the annotation efficiency. Finally, according to the obtained at least one training image sample set, the initial image classification model is trained to obtain an image classification model with more accurate image classification. In summary, through the semantic classification of the target label set and the annotation assignment of the corresponding classification label group, the annotation efficiency is greatly improved and the annotation cost is reduced.

[0059] Further references Figure 3 , shows a process 300 of another embodiment of the image classification model training method according to the present disclosure. The image classification model training method comprises the following steps:

[0060] Step 301, obtaining a target tag set.

[0061] Step 302: semantically classify the target tag set to obtain at least one classification tag group.

[0062] Step 303: for each classification label group in the at least one classification label group, the classification label group is allocated to a corresponding data annotation end, so that the data annotation end generates a training image sample set corresponding to the classification label group.

[0063] In some embodiments, the specific implementation of steps 301-303 and the technical effects thereof can be referred to in Figure 2 The steps 201-203 in the corresponding embodiment are not described in detail here.

[0064] Step 304: Select a training image sample from the at least one training image sample set as a target training image sample, and perform the following training steps:

[0065] Step 3041: input the image included in the target training image sample into an image feature extraction model to generate image feature information.

[0066] In some embodiments, an execution entity (e.g. Figure 1 The electronic device 101 shown) can input the image included in the above-mentioned target training image sample into an image feature extraction model to generate image feature information. Among them, the image feature extraction model can be an image feature extraction model whose model has not been trained yet. The above-mentioned image feature extraction model can be a pre-trained neural network model, or a neural network model that has not been trained yet. The above-mentioned image feature extraction model can be a model that extracts content feature information corresponding to the image. Specifically, for the image feature extraction model that is a neural network model that has not been trained yet, the subsequent image feature extraction model is trained and updated together with the initial image classification model. In practice, the image feature extraction model can be a residual neural network model with multiple layers connected in series. The image feature information can characterize the image content feature information of the image included in the target training image sample. In practice, the above-mentioned target training image sample can be a training image sample randomly selected from at least one training image sample set.

[0067] Step 3042: Using the above-mentioned initial image classification model, generate a label feature information group corresponding to each classification label group in the above-mentioned at least one classification label group, and obtain at least one label feature information group.

[0068] In some embodiments, the execution subject may use the initial image classification model to generate a label feature information group corresponding to each of the at least one classification label group, and obtain at least one label feature information group. The label feature information in the at least one label feature information group corresponds to the classification label in the at least one classification label group. The label feature information may represent the content feature information of the label content of the classification label.

[0069] As an example, the execution entity may input each of the at least one classification label group into the multi-layer serially connected residual neural network model included in the initial image classification model to generate a label feature information group and obtain at least one label feature information group.

[0070] In some optional implementations of some embodiments, the above-mentioned initial image classification model includes: an initial encoding model based on a first attention mechanism.

[0071] Optionally, the execution entity may input each classification label in the classification label group into the initial encoding model based on the first attention mechanism to generate the label feature information group. The initial encoding model based on the first attention mechanism may be an encoding model that has not yet been trained. The encoding model may learn the semantic relationship between the semantics corresponding to each classification label in the classification label group. Thus, through the encoding model, more accurate label feature information representing the content feature information of the label content corresponding to the classification label may be generated. There is a one-to-one correspondence between the label feature information in the label feature information group and the classification label in the classification label group. In practice, the encoding model based on the first attention mechanism may be a Transformer encoding model. That is, the corresponding first attention mechanism may be a multi-head attention mechanism.

[0072] Step 3043, using the above-mentioned initial image classification model, generates classification loss information according to the above-mentioned at least one label feature information group and the above-mentioned image feature information.

[0073] In some embodiments, the execution subject may use the initial image classification model to generate classification loss information based on the at least one label feature information group and the image feature information. The classification loss information may represent the difference between the predicted label corresponding to the image feature information and the real label of the target training image sample corresponding to the image.

[0074] As an example, first, the execution subject can use the initial image classification model to determine the similarity between each label feature information and the image feature information in at least one label feature information group to obtain a similarity information set. Then, the label category information included in the target training image sample is determined. Next, the similarity information with the numerical value in the top 5 is filtered out from the similarity information set to obtain a similarity information subset. Finally, the similarity information subset and the label category information are input into the target loss function to output the classification loss information.

[0075] In some optional implementations of some embodiments, the initial image classification model further includes: an initial decoding model based on a second attention mechanism. The initial decoding model based on the second attention mechanism is a decoding model that has not yet been trained. The decoding model can learn the semantic relationship between each classification label in at least one classification label group, and can also learn the semantic relationship between each classification label group. In practice, the decoding model based on the second attention mechanism can be a Transformer decoding model. That is, the corresponding second attention mechanism can be a multi-head attention mechanism.

[0076] Optionally, the generating of classification loss information according to the at least one label feature information group and the image feature information may include the following steps:

[0077] In the first step, the at least one label feature information group and the image feature information are input into the initial decoding model based on the second attention mechanism to output classification results for each label category. The label categories may be label categories corresponding to at least one classification label group. The classification result may be the label content of the label to which the target image corresponds to the image content.

[0078] The second step is to generate the classification loss information according to the label category information included in the target training image sample and the classification result, wherein the label category information may be the real label of the image corresponding to the target training image sample.

[0079] As an example, the execution entity may input the label category information and the classification result into a binary cross entropy loss function to generate classification loss information.

[0080] Step 3044: Determine whether the initial image classification model has been trained based on the classification loss information.

[0081] In some embodiments, the execution entity may determine whether the initial image classification model has been trained based on the classification loss information.

[0082] As an example, in response to determining that the classification loss information is less than or equal to a predetermined value, it is determined that the training of the initial image classification model is completed. In response to determining that the classification loss information is greater than a predetermined value, it is determined that the initial image classification model is not trained.

[0083] Step 3045, in response to the training being completed, determining the above initial image classification model as the image classification model.

[0084] In some embodiments, in response to the completion of training, the execution entity may determine the initial image classification model as the image classification model.

[0085] In some optional implementations of some embodiments, after step 304, the steps further include:

[0086] In response to determining that the initial image classification model has not been trained, the execution entity may adjust relevant model parameters in the initial image classification model, and reselect target training image samples from the at least one training image sample set to continue executing the training steps.

[0087] As an example, in response to determining that the initial image classification model has not been trained, the above-mentioned execution entity can perform gradient updates on relevant model parameters in the initial image classification model through a back-propagation method, and re-select training image samples that have not been previously selected from at least one training image sample set as target training image samples to continue executing the training steps.

[0088] from Figure 3 It can be seen that Figure 2 Compared with the description of some corresponding embodiments, Figure 3 In the corresponding process 300 of the image classification model training method in some embodiments, the semantic information corresponding to at least one classification label group (i.e., at least one label feature information group) is considered to perform subsequent image classification model training, so that the image classification model can learn more semantic information about the classification labels, and combined with the image feature information, a more accurate image classification model can be obtained.

[0089] Continue to refer Figure 4 , shows a process 400 of some embodiments of the image classification method according to the present disclosure. The image classification method comprises the following steps:

[0090] Step 401: Obtain at least one target label feature information group corresponding to a target image and at least one classification label group.

[0091] In some embodiments, the execution subject of the above image classification method (for example Figure 1 The electronic device 101 shown can obtain at least one target label feature information group corresponding to the target image and at least one classification label group through a wired connection method or a wireless connection method. The at least one target label feature information group is pre-generated based on an image classification model, and the image classification model is generated based on an image classification model training method. There is a one-to-one correspondence between the classification label in at least one classification label group and the target label feature information in at least one target label feature information group. The target label feature information in the target label feature information group can represent the content feature information of the label content corresponding to the classification label.

[0092] As an example, the execution subject may input each classification label in at least one classification label group into a label encoding model included in the image classification model to output target label feature information to obtain at least one target label feature information group. The label encoding model may be a model for vector encoding of labels. In practice, the label encoding model may be a multi-layer serially connected convolutional neural network model.

[0093] In some optional implementations of some embodiments, the above-mentioned image classification model includes: an encoding model based on the first attention mechanism. In practice, the encoding model based on the first attention mechanism can be a trained Transformer encoding model.

[0094] And the at least one target tag feature information group is generated by the following steps:

[0095] For each of the at least one classification label group, each classification label in the classification label group is input into the encoding model based on the first attention mechanism to generate a label feature information group as a target label feature information group.

[0096] Step 402: Generate image feature information for the target image as target image feature information.

[0097] In some embodiments, the execution subject may generate image feature information for the target image as the target image feature information, wherein the target image feature information may represent content feature information of the image content corresponding to the target image.

[0098] As an example, the execution subject may input the target image into an image feature extraction model to generate target image feature information. The image feature extraction model may be a model for extracting content feature information corresponding to the image content of the image. In practice, the image feature extraction model may be a multi-layer serially connected convolutional neural network model.

[0099] Step 403: Generate an image classification result for the target image using the image classification model based on the at least one target label feature information group and the target image feature information.

[0100] In some embodiments, the execution subject may generate an image classification result for the target image using the image classification model based on the at least one target label feature information group and the target image feature information, wherein the image classification result may be a predicted label category corresponding to the image content of the target image.

[0101] As an example, the execution subject may input at least one target tag feature information group and the target image feature information into a classification layer included in an image classification model to output an image classification result, wherein the classification layer may be a Bert (Bidirectional Encoder Representations from Transformers) decoding model.

[0102] In some optional implementations of some embodiments, the above-mentioned image classification model includes: a decoding model based on the second attention mechanism. In practice, the decoding model based on the second attention mechanism can be a trained Transformer decoding model.

[0103] Optionally, the execution entity may input the at least one target label feature information group and the target image feature information into the decoding model based on the second attention mechanism to generate the image classification result.

[0104] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the image classification method of some embodiments of the present disclosure, accurate image classification results can be generated for the target image.

[0105] Further references Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of an image classification model training device. These device embodiments are similar to Figure 2 Corresponding to the method embodiments shown, the image classification model training device can be specifically applied to various electronic devices.

[0106] like Figure 5 As shown, an image classification model training device 500 includes: a first acquisition unit 501, a semantic classification unit 502, an allocation unit 503 and a training unit 504. The first acquisition unit 501 is configured to acquire a target label set; the semantic classification unit 502 is configured to semantically classify the target label set to obtain at least one classification label group; the allocation unit 503 is configured to allocate each classification label group in the at least one classification label group to a corresponding data annotation end, so that the data annotation end generates a training image sample set corresponding to the classification label group; the training unit 504 is configured to perform model training on the initial image classification model according to the obtained at least one training image sample set to obtain an image classification model.

[0107] In some optional implementations of some embodiments, the training unit 504 can be further configured to: select training image samples from the above-mentioned at least one training image sample set as target training image samples, and perform the following training steps: input the image included in the above-mentioned target training image sample into the image feature extraction model to generate image feature information; use the above-mentioned initial image classification model to generate a label feature information group corresponding to each classification label group in the above-mentioned at least one classification label group, and obtain at least one label feature information group; use the above-mentioned initial image classification model to generate classification loss information according to the above-mentioned at least one label feature information group and the above-mentioned image feature information; determine whether the above-mentioned initial image classification model has completed model training according to the above-mentioned classification loss information; in response to the completion of training, determine the above-mentioned initial image classification model as the image classification model.

[0108] In some optional implementations of some embodiments, the training unit 504 may be further configured to: in response to determining that the initial image classification model has not been trained, adjust relevant model parameters in the initial image classification model, and reselect target training image samples from the at least one training image sample set, and continue to execute the above training steps.

[0109] In some optional implementations of some embodiments, the above-mentioned initial image classification model includes: an initial encoding model based on a first attention mechanism; and the training unit 504 can be further configured to: input each classification label in the above-mentioned classification label group into the above-mentioned initial encoding model based on the first attention mechanism to generate the above-mentioned label feature information group.

[0110] In some optional implementations of some embodiments, the above-mentioned initial image classification model also includes: an initial decoding model based on a second attention mechanism; and the training unit 504 can be further configured to: input the above-mentioned at least one label feature information group and the above-mentioned image feature information into the above-mentioned initial decoding model based on the second attention mechanism to output classification results for each label category; generate the above-mentioned classification loss information according to the label category information included in the above-mentioned target training image sample and the above-mentioned classification results.

[0111] It can be understood that the units recorded in the image classification model training device 500 are similar to those in the reference Figure 2 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the image classification model training device 500 and the units contained therein, and will not be repeated here.

[0112] Further references Figure 6 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of an image classification device. These device embodiments are similar to Figure 4 Corresponding to the method embodiments shown, the image classification device can be specifically applied to various electronic devices.

[0113] like Figure 6 As shown, an image classification device 600 includes: a second acquisition unit 601, a first generation unit 602 and a second generation unit 603. The second acquisition unit 601 is configured to acquire a target image and at least one target label feature information group corresponding to at least one classification label group, wherein the at least one target label feature information group is pre-generated based on an image classification model, and the image classification model is generated based on an image classification model training method; the first generation unit 602 is configured to: generate image feature information for the target image as target image feature information; the second generation unit 603 is configured to generate an image classification result for the target image based on the at least one target label feature information group and the target image feature information and using the image classification model.

[0114] In some optional implementations of some embodiments, the above-mentioned image classification model includes: an encoding model based on a first attention mechanism; and the above-mentioned at least one target label feature information group is generated by the following steps: for each classification label group in the above-mentioned at least one classification label group, each classification label in the above-mentioned classification label group is input into the above-mentioned encoding model based on the first attention mechanism to generate a label feature information group as the target label feature information group.

[0115] In some optional implementations of some embodiments, the above-mentioned image classification model includes: a decoding model based on a second attention mechanism; and the second generation unit 603 can be further configured to: input the above-mentioned at least one target label feature information group and the above-mentioned target image feature information into the above-mentioned decoding model based on the second attention mechanism to generate the above-mentioned image classification result.

[0116] It can be understood that the units described in the image classification device 600 are similar to those described in the reference Figure 4 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the image classification device 600 and the units included therein, and will not be described in detail here.

[0117] Reference below Figure 7 , which shows an electronic device (eg, Figure 1 Schematic diagram of the structure of the electronic device 101)700. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0118] like Figure 7 As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory 702 or a program loaded from a storage device 708 into a random access memory 703. Various programs and data required for the operation of the electronic device 700 are also stored in the random access memory 703. The processing device 701, the read-only memory 702, and the random access memory 703 are connected to each other via a bus 704. An input / output interface 705 is also connected to the bus 704.

[0119] Typically, the following devices may be connected to the input / output interface 705: an input device 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 7 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0120] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from a network through a communication device 709, or installed from a storage device 708, or installed from a read-only memory 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0121] It should be noted that the computer-readable medium in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0122] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0123] The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains a target label set; semantically classifies the target label set to obtain at least one classification label group; for each classification label group in the at least one classification label group, allocates the classification label group to a corresponding data annotation end, so that the data annotation end generates a training image sample set corresponding to the classification label group; performs model training on the initial image classification model according to the obtained at least one training image sample set to obtain an image classification model. Obtain at least one target label feature information group corresponding to the target image and at least one classification label group, wherein the at least one target label feature information group is pre-generated based on an image classification model, and the image classification model is generated based on an image classification model training method; generates image feature information for the target image as target image feature information; generates an image classification result for the target image according to the at least one target label feature information group and the target image feature information using the image classification model.

[0124] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0125] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0126] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be provided in a processor, for example, may be described as: a processor comprising a first acquisition unit, a semantic classification unit, an allocation unit, and a training unit. Among them, the names of these units do not constitute a limitation on the unit itself in certain cases, for example, the first acquisition unit may also be described as a "unit for acquiring a target label set".

[0127] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0128] Some embodiments of the present disclosure also provide a computer program product, including a computer program, which implements any of the above-mentioned image classification model training methods or image classification methods when executed by a processor.

[0129] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A method for training an image classification model, comprising: Get the target label set; Performing semantic classification on the target tag set to obtain at least one classification tag group; For each classification label group in the at least one classification label group, assigning the classification label group to a corresponding data annotation end, so that the data annotation end generates a training image sample set corresponding to the classification label group; According to the obtained at least one training image sample set, the initial image classification model is trained to obtain an image classification model.

2. The method according to claim 1, wherein: The method of performing model training on the initial image classification model according to the obtained at least one training image sample set to obtain the image classification model comprises: A training image sample is selected from the at least one training image sample set as a target training image sample, and the following training steps are performed: Inputting the image included in the target training image sample into an image feature extraction model to generate image feature information; Using the initial image classification model, generating a label feature information group corresponding to each classification label group in the at least one classification label group, to obtain at least one label feature information group; Using the initial image classification model, generating classification loss information according to the at least one label feature information group and the image feature information; Determining whether the initial image classification model has completed model training according to the classification loss information; In response to the training being completed, the initial image classification model is determined as the image classification model.

3. The method according to claim 2, wherein: The method further comprises: In response to determining that the initial image classification model has not been trained, relevant model parameters in the initial image classification model are adjusted, and target training image samples are reselected from the at least one training image sample set to continue executing the training step.

4. The method according to claim 2, wherein: The initial image classification model includes: an initial encoding model based on a first attention mechanism; and The generating of the label feature information group corresponding to each classification label group in the at least one classification label group includes: Each classification label in the classification label group is input into the initial encoding model based on the first attention mechanism to generate the label feature information group.

5. The method according to claim 4, wherein: The initial image classification model further includes: an initial decoding model based on a second attention mechanism; and The generating classification loss information according to the at least one label feature information group and the image feature information comprises: Inputting the at least one tag feature information group and the image feature information into the initial decoding model based on the second attention mechanism to output classification results for each tag category; The classification loss information is generated according to the label category information included in the target training image sample and the classification result.

6. An image classification method, comprising: Acquire at least one target label feature information group corresponding to a target image and at least one classification label group, wherein the at least one target label feature information group is pre-generated based on an image classification model, and the image classification model is generated based on the method according to any one of claims 1 to 5; generating image feature information for the target image as target image feature information; According to the at least one target label feature information group and the target image feature information, an image classification result for the target image is generated using the image classification model.

7. The method according to claim 6, wherein: The image classification model includes: an encoding model based on a first attention mechanism; and The at least one target tag feature information group is generated by the following steps: For each classification label group in the at least one classification label group, each classification label in the classification label group is input into the encoding model based on the first attention mechanism to generate a label feature information group as a target label feature information group.

8. The method according to claim 6, wherein: The image classification model includes: a decoding model based on a second attention mechanism; and The step of generating an image classification result for the target image by using the image classification model according to the at least one target label feature information group and the target image feature information comprises: The at least one target label feature information group and the target image feature information are input into the decoding model based on the second attention mechanism to generate the image classification result.

9. An image classification model training device, comprising: A first acquisition unit is configured to acquire a target tag set; A semantic classification unit, configured to perform semantic classification on the target tag set to obtain at least one classification tag group; an allocating unit configured to allocate, for each of the at least one classification label group, the classification label group to a corresponding data annotation end, so that the data annotation end generates a training image sample set corresponding to the classification label group; The training unit is configured to perform model training on the initial image classification model according to the obtained at least one training image sample set to obtain the image classification model.

10. An image classification device, comprising: A second acquisition unit is configured to acquire at least one target label feature information group corresponding to the target image and at least one classification label group, wherein the at least one target label feature information group is pre-generated based on an image classification model, and the image classification model is generated based on the method according to any one of claims 1 to 5; A first generating unit is configured to generate image feature information for the target image as target image feature information; The second generating unit is configured to generate an image classification result for the target image using the image classification model according to the at least one target label feature information group and the target image feature information.

11. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 8.

12. A computer readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

13. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images