Image tag recognition methods, media, devices and computing equipment

Abstract

This disclosure provides an image label recognition method, medium, apparatus, and computing device, relating to the field of image recognition technology. The image label recognition method includes: acquiring input content to an image label recognition model; if the input content includes a target image, obtaining, through the image label recognition model, the feature similarity between the target image and at least one label in the input content based on a first label set; determining labels with feature similarity greater than a similarity threshold as the recognition labels corresponding to the target image; and outputting the recognition labels; if the input content includes labels to be updated, updating the first text feature vector of the first label set through the image label recognition model to obtain the second text feature vector of the corresponding second label set. This disclosure can significantly improve the efficiency of the image label recognition model in outputting recognition labels corresponding to the target image when the label set is frequently updated.

Description

Technical Field

[0001] The embodiments of this disclosure relate to the field of image recognition technology, and more specifically, the embodiments of this disclosure relate to an image tag recognition method, medium, apparatus, and computing device. Background Technology

[0002] This section is intended to provide background or context for the embodiments of this disclosure as set forth in the claims. The description herein is not intended to be a prior art simply because it is included in this section.

[0003] With the increasing development and sophistication of image processing technology, the requirements for the accuracy and precision of image or video recognition are growing. For example, image label recognition models are used to identify labels on videos or images and output the corresponding labels.

[0004] Currently, existing image label recognition models are trained on labeled data, and each iteration of the training data is generally a fixed set, that is, the labels are a predefined range. For example, if there is a dataset with 500 labels, an image label recognition model that can recognize these 500 labels can be trained.

[0005] The problem this causes is that every time the label set is changed, the training data needs to be updated, and the model needs to be retrained based on the updated training data. When the label set is updated frequently, it is difficult to quickly obtain an efficient image label recognition model. Summary of the Invention

[0006] This disclosure provides an image label recognition method, medium, apparatus, and computing device to solve the problem that it is difficult to quickly obtain an efficient image label recognition model when the label set is frequently updated.

[0007] In a first aspect of this disclosure, an image tag recognition method is provided, applied to a device equipped with an image tag recognition model, the image tag recognition method comprising:

[0008] Obtain the input content for the image label recognition model;

[0009] If the input content includes a target image, the image label recognition model is used to obtain the feature similarity between the target image and at least one label in the input content based on the first label set; the label with a feature similarity greater than a similarity threshold is determined as the recognition label corresponding to the target image; and the recognition label is output.

[0010] If the input includes labels to be updated, the first text feature vector of the first label set is updated using the image label recognition model to obtain the second text feature vector of the corresponding second label set.

[0011] In one possible implementation, the image label recognition model includes a text feature extraction network. The image label recognition model updates the first text feature vector of the first label set to obtain the second text feature vector of the corresponding second label set. This includes: inputting the label to be updated into the text feature extraction network to update the first text feature vector of the first label set to obtain the second text feature vector of the corresponding second label set.

[0012] In one possible implementation, the image label recognition model further includes an image feature extraction network and a post-processing network. After obtaining the second text feature vector of the corresponding second label set, the image label recognition method further includes: acquiring new input content to the image label recognition model; inputting the target image in the input content into the image feature extraction network to extract image features and obtain the image feature vector of the target image; inputting the image feature vector and the second text feature vector into the post-processing network to calculate feature similarity and obtain the feature similarity between the target image and at least one label.

[0013] In one possible implementation, the image label recognition model includes an image feature extraction network, a text feature extraction network, and a post-processing network. Based on a first label set, it obtains the feature similarity between the target image and at least one label in the input content. This includes: inputting the target image into the image feature extraction network to extract image features, obtaining an image feature vector of the target image; inputting the image feature vector and a first text feature vector into the post-processing network to calculate feature similarity, obtaining the feature similarity between the target image and at least one label. The first text feature vector is a text feature vector obtained by inputting the first label set into the text feature extraction network for text feature extraction.

[0014] In one possible implementation, the input content includes multiple target images. Determining labels with feature similarity greater than a similarity threshold as the recognition labels corresponding to the target images includes: for the multiple target images, determining labels with feature similarity greater than a similarity threshold corresponding to each target image; sorting the labels with feature similarity greater than the similarity threshold corresponding to the multiple target images from high to low to obtain a first label sequence; filtering out duplicate labels in the first label sequence to obtain a second label sequence; and determining a preset threshold number of labels with relatively high feature similarity as the recognition labels corresponding to the multiple target images in the second label sequence.

[0015] In one possible implementation, the image label recognition method further includes: determining at least one target image contained in the video in response to the input content being a video, before inputting the input content into the image label recognition model.

[0016] In one possible implementation, outputting the identification label includes: normalizing the feature similarity between the identification label and the target image to obtain a probability value corresponding to the identification label, the probability value being used to reflect the degree of matching between the identification label and the target image; and outputting the identification label and the probability value corresponding to the identification label.

[0017] In one possible implementation, the image label recognition model includes an image feature extraction network, a text feature extraction network, and a post-processing network. The image label recognition model is trained as follows: a training set containing multiple sets of training samples is obtained, each set of training samples including a sample image, sample text, and a reference similarity between the sample image and the sample text; for each set of training samples, the following operations are performed: extracting pre-trained image feature vectors of the sample images based on the image feature extraction network; extracting pre-trained text feature vectors of the sample text based on the text feature extraction network; determining the training similarity between the pre-trained image feature vectors and the pre-trained text feature vectors based on the post-processing network; and adjusting the model parameters of the image label recognition model based on the training similarity and the reference similarity to obtain the image label recognition model.

[0018] Secondly, embodiments of this disclosure provide an image tag recognition device, applied to a device equipped with an image tag recognition model, the image tag recognition device comprising:

[0019] The acquisition module is used to acquire the input content to the image label recognition model;

[0020] The first processing module is configured to, if the input content includes a target image, obtain the feature similarity between the target image and at least one label in the input content based on a first label set using an image label recognition model; determine the label with a feature similarity greater than a similarity threshold as the recognition label corresponding to the target image; and output the recognition label.

[0021] The second processing module is used to update the first text feature vector of the first tag set through the image tag recognition model if the input content includes the tag to be updated, so as to obtain the second text feature vector of the corresponding second tag set.

[0022] In one possible implementation, the image label recognition model includes a text feature extraction network, and the second processing module is specifically used to: input the label to be updated into the text feature extraction network to update the first text feature vector of the first label set, and obtain the second text feature vector of the corresponding second label set.

[0023] In one possible implementation, the image label recognition model further includes an image feature extraction network and a post-processing network. After obtaining the second text feature vector of the corresponding second label set, the second processing module is further configured to: obtain the input content newly input to the image label recognition model; input the target image in the input content into the image feature extraction network to extract image features and obtain the image feature vector of the target image; input the image feature vector and the second text feature vector into the post-processing network to calculate the feature similarity and obtain the feature similarity between the target image and at least one label.

[0024] In one possible implementation, the image label recognition model includes an image feature extraction network, a text feature extraction network, and a post-processing network. The first processing module is specifically used to: input the target image into the image feature extraction network to extract image features and obtain an image feature vector of the target image; input the image feature vector and a first text feature vector into the post-processing network to calculate feature similarity and obtain the feature similarity between the target image and at least one label, wherein the first text feature vector is a text feature vector obtained by inputting a first set of labels into the text feature extraction network to extract text features.

[0025] In one possible implementation, the input content includes multiple target images. The first processing module is specifically used to: for the multiple target images, determine the label corresponding to each target image whose feature similarity is greater than a similarity threshold; sort the labels corresponding to the multiple target images whose feature similarity is greater than the similarity threshold from high to low to obtain a first label sequence; filter out the labels that appear repeatedly in the first label sequence to obtain a second label sequence; and in the second label sequence, determine a preset threshold of labels with high feature similarity as the recognition labels corresponding to the multiple target images.

[0026] In one possible implementation, the image tag recognition device further includes a determination module for: determining at least one target image contained in the video in response to the input content being a video, before inputting the input content into the image tag recognition model.

[0027] In one possible implementation, the first processing module is specifically used to: normalize the feature similarity between the identification label and the target image to obtain the probability value corresponding to the identification label, the probability value being used to reflect the degree of matching between the identification label and the target image; and output the identification label and the probability value corresponding to the identification label.

[0028] In one possible implementation, the image label recognition model includes an image feature extraction network, a text feature extraction network, and a post-processing network. The image label recognition model is trained as follows: a training set containing multiple sets of training samples is obtained, each set of training samples including a sample image, sample text, and a reference similarity between the sample image and the sample text; for each set of training samples, the following operations are performed: extracting pre-trained image feature vectors of the sample images based on the image feature extraction network; extracting pre-trained text feature vectors of the sample text based on the text feature extraction network; determining the training similarity between the pre-trained image feature vectors and the pre-trained text feature vectors based on the post-processing network; and adjusting the model parameters of the image label recognition model based on the training similarity and the reference similarity to obtain the image label recognition model.

[0029] Thirdly, embodiments of this disclosure provide a computing device, including: a processor, and a memory communicatively connected to the processor;

[0030] The memory stores the instructions that the computer executes;

[0031] The processor executes computer execution instructions stored in memory to implement the image tag recognition method as described in the first aspect of this disclosure.

[0032] Fourthly, embodiments of this disclosure provide a storage medium storing computer program instructions, which, when executed, implement the image tag recognition method as described in the first aspect of this disclosure.

[0033] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the image tag recognition method as described in the first aspect of this disclosure.

[0034] The image label recognition method, medium, apparatus, and computing device provided in this disclosure acquire input content to an image label recognition model; if the input content includes a target image, the image label recognition model obtains the feature similarity between the target image and at least one label in the input content based on a first label set; labels with feature similarity greater than a similarity threshold are determined as the recognition labels corresponding to the target image; and the recognition labels are output. If the input content includes labels to be updated, the image label recognition model updates the first text feature vector of the first label set to obtain the second text feature vector of the corresponding second label set. Since this disclosure can update the first text feature vector of the first label set based on the labels to be updated included in the input content without updating the image label recognition model, and obtain the second text feature vector of the corresponding second label set, and then recognize the recognition label corresponding to the target image based on the second text feature vector of the second label set, it can accurately identify the recognition label corresponding to the target image even when the label set is frequently updated, thus greatly improving the efficiency of the image label recognition model in outputting the recognition label corresponding to the target image. Attached Figure Description

[0035] The above and other objects, features, and advantages of this disclosure will become readily apparent from the following detailed description of exemplary embodiments, taken in conjunction with the accompanying drawings. Several embodiments of this disclosure are illustrated in the drawings by way of example and not limitation, in which:

[0036] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of the present disclosure;

[0037] Figure 2 A flowchart of an image tag recognition method provided in an embodiment of this disclosure;

[0038] Figure 3 A flowchart of an image tag recognition method provided in another embodiment of this disclosure;

[0039] Figure 4 This is a schematic diagram illustrating an image tag recognition model based on a first tag set, provided in an embodiment of this disclosure.

[0040] Figure 5 This is a schematic diagram illustrating an image tag recognition model based on a second tag set, provided in an embodiment of this disclosure.

[0041] Figure 6 A flowchart illustrating a training method for an image label recognition model provided in an embodiment of this disclosure;

[0042] Figure 7This is a schematic diagram illustrating the training of an image label recognition model provided in an embodiment of the present disclosure;

[0043] Figure 8 This is a schematic diagram of the structure of an image tag recognition device provided in an embodiment of the present disclosure;

[0044] Figure 9 A schematic diagram of a storage medium provided in an embodiment of this disclosure;

[0045] Figure 10 This is a schematic diagram of the structure of a computing device provided in an embodiment of the present disclosure.

[0046] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0047] The principles and spirit of this disclosure will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are given merely to enable those skilled in the art to better understand and implement this disclosure, and are not intended to limit the scope of this disclosure in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0048] Those skilled in the art will recognize that the embodiments of this disclosure can be implemented as a system, apparatus, device, method, or computer program product. Therefore, this disclosure can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software. The data involved in this disclosure can be data authorized by the user or fully authorized by all parties, and the embodiments / examples of this disclosure can be combined with each other.

[0049] According to embodiments of this disclosure, a method, medium, apparatus, and computing device for image tag recognition are proposed.

[0050] In this article, it is important to understand the following terms:

[0051] Image label recognition is the process of accurately identifying specific labels from images using image recognition technology. These labels are short, semantically defined descriptions pre-defined by humans, such as "playing basketball" or "eating." They are not limited to actions but can also include entities such as "apple" or "car."

[0052] Open set labels mean that the range of labels is not fixed and may be added or removed at any time.

[0053] Furthermore, the number of any elements in the accompanying drawings is for illustrative purposes only and not for limitation, and any naming is for distinction only and has no limiting meaning.

[0054] The principles and spirit of this disclosure will be explained in detail below with reference to several representative embodiments. Invention Overview

[0056] The inventors have discovered that, currently, for video action recognition models, such as unimodal or multimodal models, the input is a video red-green-blue (RGB) image frame or other modalities including speech, and the output is a predefined label and probability. In the above examples, regardless of whether it is a unimodal or multimodal recognition model, the labels are always predefined. It is impossible to output new labels and their probabilities without updating the model when the label set is open. That is, every time the label set is changed, the training data needs to be updated, and the model needs to be retrained based on the updated training data. When the label set is frequently updated, it is difficult to quickly obtain an efficient image label recognition model.

[0057] To address the aforementioned issues, this disclosure provides an image label recognition method, medium, apparatus, and computing device. By decoupling the label definition from the image label recognition model, the updated label can still be output without updating the image label recognition model after the label is updated. Therefore, it can significantly improve the efficiency of the image label recognition model in outputting the label corresponding to the image when the label set is frequently updated.

[0058] Application Scenarios Overview

[0059] First refer to Figure 1 Examples of application scenarios for the solutions provided in this disclosure are given. Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of the present disclosure, such as... Figure 1 As shown, in this application scenario, client 101 sends an image of a tag to be identified to server 102. Server 102 identifies the tag corresponding to the image and sends the tag back to client 101 so that client 101 can display the tag. The specific implementation process of server 102 identifying the tag corresponding to the image can be found in the solutions of the following embodiments.

[0060] It should be noted that, Figure 1 This is merely a schematic diagram illustrating one application scenario provided by an embodiment of this disclosure; this embodiment does not necessarily represent an application scenario. Figure 1 The included equipment is not limited, nor is it restricted. Figure 1 The positional relationships between devices are defined. For example, in Figure 1 The application scenario shown may also include a data storage device, which may be an external storage device relative to the client 101 or the server 102, or an internal storage device integrated into the client 101 or the server 102.

[0061] Exemplary methods

[0062] The following is combined with Figure 1 Application scenarios, refer to Figure 2 This document describes a method for image label recognition according to exemplary embodiments of the present disclosure. It should be noted that the above application scenarios are shown only to facilitate understanding of the spirit and principles of the present disclosure, and the embodiments of the present disclosure are not limited in any way. Rather, the embodiments of the present disclosure can be applied to any applicable scenario.

[0063] First, the image label recognition method is introduced through specific embodiments.

[0064] Figure 2 This is a flowchart illustrating an embodiment of an image tag recognition method provided by this disclosure. The method is applied to a device equipped with an image tag recognition model, such as a computing device, which may specifically be a server or a server cluster. Figure 2 As shown, the method in this embodiment includes:

[0065] S201. Obtain the input content to the image label recognition model.

[0066] In this embodiment, the image label recognition model is pre-trained. For details on how to train and obtain the image label recognition model, please refer to subsequent embodiments; details will not be repeated here. The input content to the image label recognition model can be input by a user to the computing device executing this method embodiment, or it can be sent by other devices to the computing device executing this method embodiment. Therefore, this input content can be obtained.

[0067] Optionally, before inputting the input content into the image label recognition model, in response to the input content being a video, at least one target image contained in the video is determined.

[0068] For example, if the input content is a video, and the video contains at least one image, then at least one target image can be identified in the video. Specifically, if the video contains, for example, three images, then it can be identified that the video contains three target images, which can be input into the image label recognition model respectively for image label recognition.

[0069] S202. If the input content includes the target image, the image label recognition model is used to obtain the feature similarity between the target image and at least one label in the input content based on the first label set; the label with a feature similarity greater than the similarity threshold is determined as the recognition label corresponding to the target image; the recognition label is output.

[0070] In this step, after obtaining the input content to the image label recognition model, if the input content includes a target image, the image label recognition model obtains the feature similarity between the target image in the input content and at least one label based on a first label set. This feature similarity is the feature similarity score. It should be noted that the input content does not contain any labels to be updated; therefore, it is not necessary to update the first label set. The feature similarity between the target image in the input content and at least one label can be obtained based on the first label set. For details on how the image label recognition model obtains the feature similarity between the target image in the input content and at least one label based on the first label set, please refer to subsequent embodiments; it will not be elaborated here.

[0071] The similarity threshold can be set as needed, and this disclosure does not impose specific limitations on it. After obtaining the feature similarity between the target image and at least one label in the input content, the labels with feature similarity greater than the similarity threshold can be determined as the recognition labels corresponding to the target image. For example, if the input content contains a target image, then according to the set similarity threshold, K labels with feature similarity greater than the similarity threshold can be obtained, and these K labels are the recognition labels corresponding to the target image.

[0072] Optionally, if the input contains multiple target images, determining the labels with feature similarity greater than a similarity threshold as the recognition labels corresponding to the target images may include: for multiple target images, determining the labels with feature similarity greater than a similarity threshold corresponding to each target image; sorting the labels with feature similarity greater than a similarity threshold corresponding to the multiple target images from high to low to obtain a first label sequence; filtering out the labels that appear repeatedly in the first label sequence to obtain a second label sequence; and determining a preset threshold number of labels with higher feature similarity as the recognition labels corresponding to the multiple target images in the second label sequence.

[0073] For example, if the input content contains N target images, based on the example of input content containing one target image, K labels with feature similarity greater than a similarity threshold can be obtained for each of the N target images, resulting in N*K recognition labels. These N*K recognition labels are sorted from high to low according to their corresponding feature similarity to obtain a first label sequence. Removing duplicate labels from the first label sequence yields a second label sequence. When removing duplicate labels from the first label sequence, for example, the label with the highest feature similarity can be retained based on the feature similarity of each duplicate label. This disclosure does not specifically limit the filtering method.

[0074] The preset threshold can be set as needed, and this disclosure does not impose specific limitations on it. After obtaining the second label sequence, a preset threshold (e.g., represented by the top M) of labels with high feature similarity can be determined as the recognition labels corresponding to multiple target images, or all labels in the second label sequence can be used as the recognition labels corresponding to multiple target images.

[0075] Optionally, the output identification label may include: normalizing the feature similarity between the identification label and the target image to obtain the probability value corresponding to the identification label, the probability value being used to reflect the degree of matching between the identification label and the target image; and outputting the identification label and the probability value corresponding to the identification label.

[0076] For example, after obtaining the recognition label corresponding to the target image, the recognition label can be output. Specifically, after obtaining the feature similarity between the recognition label and the target image, the feature similarity between the recognition label and the target image can be normalized to obtain the probability value corresponding to the recognition label, and the recognition label and the probability value corresponding to the recognition label can be output. For example, if the recognition label is "beachcombing" and the probability value corresponding to the recognition label is 0.950, then the output is: "beachcombing: 0.950".

[0077] S203. If the input content includes labels to be updated, the first text feature vector of the first label set is updated through the image label recognition model to obtain the second text feature vector of the corresponding second label set.

[0078] The second label set is used to perform image label recognition on the target images contained in the input content of the newly input image label recognition model.

[0079] In this step, the first text feature vector of the first label set is the text feature vector obtained by inputting the first label set into the image label recognition model. It can be understood that if the input content includes labels to be updated, the first text feature vector of the first label set needs to be updated using the image label recognition model to obtain the second text feature vector of the corresponding second label set. Then, based on the second text feature vector of the second label set, image label recognition is performed on the target image contained in the newly input content to the image label recognition model. The labels in the second label set include the labels in the first label set and the labels to be updated. It is not necessary to retrain the image label recognition model; that is, the image label recognition model is fixed and no longer updated. It is only necessary to update the first text feature vector of the first label set based on the labels to be updated included in the input content. For details on how to update the first text feature vector of the first label set using the image label recognition model to obtain the second text feature vector of the corresponding second label set, please refer to subsequent embodiments, which will not be repeated here. Optionally, performing image label recognition on the target image contained in the input content newly input to the image label recognition model based on the second label set may include: obtaining the feature similarity between the target image in the input content and at least one label based on the second label set using the image label recognition model; determining the label with a feature similarity greater than a similarity threshold as the recognition label corresponding to the target image; and outputting the recognition label. For details on how to obtain the feature similarity between the target image in the input content and at least one label based on the second label set, please refer to subsequent embodiments, which will not be elaborated here.

[0080] The image label recognition method provided in this disclosure involves: acquiring input content to an image label recognition model; if the input content includes a target image, obtaining the feature similarity between the target image and at least one label in the input content based on a first label set using the image label recognition model; determining labels with feature similarity greater than a similarity threshold as the recognition labels corresponding to the target image; and outputting the recognition labels; if the input content includes labels to be updated, updating the first text feature vector of the first label set using the image label recognition model to obtain the second text feature vector of the corresponding second label set. Since this disclosure can update the first text feature vector of the first label set based on the labels to be updated included in the input content without updating the image label recognition model, and then using the second text feature vector of the second label set to recognize the recognition labels corresponding to the target image through the image label recognition model, it can accurately identify the recognition labels corresponding to the target image even when the label set is frequently updated, thus greatly improving the efficiency of the image label recognition model in outputting the recognition labels corresponding to the target image.

[0081] Figure 3 This is a flowchart illustrating an image tag recognition method according to another embodiment of the present disclosure. Based on the above embodiments, this disclosure further explains how to recognize image tags. Figure 3 As shown, the method in this embodiment of the disclosure may include:

[0082] S301. Obtain the input content to the image label recognition model.

[0083] For a detailed description of this step, please refer to [link / reference]. Figure 2 The relevant description of S201 in the illustrated embodiment will not be repeated here.

[0084] The image label recognition model includes an image feature extraction network, a text feature extraction network, and a post-processing network. In this embodiment of the disclosure, Figure 2 Step S202 can further include the following two steps, S302 and S303:

[0085] S302. If the input content includes the target image, the target image is input into the image feature extraction network to extract image features and obtain the image feature vector of the target image.

[0086] In this step, exemplarily, Figure 4 This is a schematic diagram illustrating an image tag recognition model based on a first tag set, provided in an embodiment of this disclosure. Figure 4 As shown, video V1 contains n target images, including target image 1 to target image n. These n target images are input into the image feature extraction network for image feature extraction, and corresponding image feature vectors can be obtained respectively.

[0087] S303. Input the image feature vector and the first text feature vector into the post-processing network to calculate the feature similarity, and obtain the feature similarity between the target image and at least one label. The first text feature vector is the text feature vector obtained by inputting the first label set into the text feature extraction network to extract text features.

[0088] In this step, refer to Figure 4 The first label set, for example, includes labels 1 to 5. This first label set is input into a text feature extraction network to extract text features, resulting in the corresponding first text feature vector, which constitutes the label feature set. The image feature vectors and first text feature vectors corresponding to the n target images are then input into a post-processing network for feature similarity calculation, yielding the feature similarity between each of the n target images and at least one label.

[0089] S304. Determine the labels with feature similarity greater than the similarity threshold as the recognition labels corresponding to the target image; output the recognition labels.

[0090] For a detailed description of this step, please refer to [link / reference]. Figure 2 The relevant description of S202 in the illustrated embodiment. For example, refer to... Figure 4 The output identification label is, for example, label 2, and the probability value corresponding to label 2 is, for example, 0.878.

[0091] Based on the text feature extraction network included in the image label recognition model, in this embodiment of the disclosure, Figure 2 Step S203 may further include step S305 as follows:

[0092] S305. If the input content includes labels to be updated, the labels to be updated are input into the text feature extraction network to update the first text feature vector of the first label set, thereby obtaining the second text feature vector of the corresponding second label set.

[0093] In this step, if the input includes labels to be updated, then the first text feature vector of the first label set needs to be updated. Specifically, the labels to be updated are input into the text feature extraction network to update the first text feature vector of the first label set, thereby obtaining the second text feature vector of the corresponding second label set.

[0094] The image label recognition model also includes an image feature extraction network and a post-processing network. After obtaining the second text feature vector of the corresponding second label set through step S305, the following three steps S306 to S308 can be executed:

[0095] S306. Obtain the new input to the image label recognition model.

[0096] It is understandable that the input content contains the target image, and image label recognition needs to be performed on the target image based on the second label set.

[0097] S307. Input the target image from the input content into the image feature extraction network to extract image features and obtain the image feature vector of the target image.

[0098] In this step, exemplarily, Figure 5 This is a schematic diagram illustrating an image tag recognition model based on a second tag set, provided in an embodiment of this disclosure. Figure 5 As shown, based on Figure 4For example, based on the first label set, labels 6, 7, 8, 9, and 10 are added, resulting in a second label set. This second label set can be input into a text feature extraction network for text feature extraction, yielding a second text feature vector, which constitutes the label feature set. Taking video V1 containing n target images, including target image 1 to target image n, as an example, if the input to the image label recognition model contains these n target images, then inputting each of these n target images into an image feature extraction network for image feature extraction will yield corresponding image feature vectors.

[0099] S308. Input the image feature vector and the second text feature vector into the post-processing network to calculate the feature similarity, and obtain the feature similarity between the target image and at least one label.

[0100] For example, refer to Figure 5 After obtaining the image feature vector and the second text feature vector, the image feature vector and the second text feature vector can be input into the post-processing network to calculate the feature similarity between the target image and at least one label.

[0101] After completing step S308, step S304 is executed. For example, refer to... Figure 5 The output identification labels include, for example, label 2, label 6, and label 8. The probability value for label 2 is, for example, 0.878; the probability value for label 6 is, for example, 0.855; and the probability value for label 8 is, for example, 0.956. Based on... Figure 4 and Figure 5 As can be seen from the example, after the label is updated, the image label recognition model can accurately output the recognition label corresponding to the target image.

[0102] The image label recognition method provided in this embodiment obtains input content to the image label recognition model; if the input content includes a target image, the target image is input to an image feature extraction network for image feature extraction to obtain an image feature vector of the target image; the image feature vector and a first text feature vector are input to a post-processing network for feature similarity calculation to obtain the feature similarity between the target image and at least one label, where the first text feature vector is a text feature vector obtained by inputting a first label set to a text feature extraction network for text feature extraction; if the input content includes a label to be updated, the label to be updated is input to a text feature extraction network to update the first text feature vector of the first label set to obtain the second text feature vector of the corresponding second label set; new input content is obtained to the image label recognition model; the target image in the input content is input to an image feature extraction network for image feature extraction to obtain an image feature vector of the target image; the image feature vector and the second text feature vector are input to a post-processing network for feature similarity calculation to obtain the feature similarity between the target image and at least one label; labels with feature similarity greater than a similarity threshold are determined as the recognition labels corresponding to the target image; and the recognition labels are output. Since the embodiments of this disclosure can update the first text feature vector of the first tag set based on the tag to be updated included in the input content without updating the image tag recognition model, and obtain the second text feature vector of the corresponding second tag set, and then identify the recognition tag corresponding to the target image through the image tag recognition model based on the second text feature vector of the second tag set, it can greatly improve the efficiency of the image tag recognition model in outputting the recognition tag corresponding to the target image while accurately identifying the recognition tag corresponding to the target image when the tag set is frequently updated.

[0103] Figure 6 This is a flowchart illustrating a training method for an image label recognition model provided in one embodiment of this disclosure. Figure 6 As shown, the method in this embodiment of the disclosure may include:

[0104] S601. Obtain a training set containing multiple sets of training samples. Each set of training samples includes a sample image, sample text, and a reference similarity between the sample image and the sample text.

[0105] In this embodiment, the image label recognition model employs a cross-modal training model, meaning the input content to the image label recognition model can involve multiple modalities, such as text, images, or videos. Different modalities can utilize corresponding deep feature extraction networks. For example, images can use ResNet; text can use Natural Language Processing-Bidirectional Encoder Representations from Transformers (NLP-BERT); and videos can use a SlowFast network model. The data input to the training model consists of corresponding pairs of data. The training model aims to achieve high similarity between feature vectors corresponding to text descriptions and video or image content, and low similarity between feature vectors that do not correspond to text descriptions and video or image content. The training model may be an image-text model, specifically a Contrastive Language Image Pre-training (CLIP) model or a text-text model, etc., which are not specifically limited in this application.

[0106] For example, Figure 7 This is a schematic diagram illustrating the training of an image label recognition model provided in an embodiment of the present disclosure, as shown below. Figure 7 As shown, the training samples include sample images and sample text. A reference similarity between the sample images and sample text can be determined and used for comparison with the training similarity output by the image label recognition model.

[0107] For each training sample, perform the following operations:

[0108] S602. Extract pre-trained image feature vectors from sample images based on an image feature extraction network.

[0109] For example, refer to Figure 7 , can be based on Figure 7 The image feature extraction network shown extracts pre-trained image feature vectors from sample images.

[0110] S603. Extract pre-trained text feature vectors from sample texts based on a text feature extraction network.

[0111] For example, refer to Figure 7 , can be based on Figure 7 The text feature extraction network shown extracts pre-trained text feature vectors from sample texts.

[0112] S604. Determine the training similarity between the pre-trained image feature vector and the pre-trained text feature vector based on the post-processing network.

[0113] In this step, after obtaining the pre-trained image feature vector of the sample image and the pre-trained text feature vector of the sample text, the training similarity between the pre-trained image feature vector and the pre-trained text feature vector can be determined based on the post-processing network.

[0114] S605. Based on the training similarity and reference similarity, adjust the model parameters of the image label recognition model to obtain the image label recognition model.

[0115] In this step, after obtaining the training similarity between the pre-trained image feature vector and the pre-trained text feature vector, the model parameters of the image label recognition model can be adjusted based on the training similarity and the reference similarity to obtain the image label recognition model.

[0116] The image label recognition model training method provided in this disclosure can obtain an image label recognition model with good generalization ability and accurately identify the corresponding label of the image.

[0117] In summary, the technical solution provided in this disclosure has at least the following advantages:

[0118] (1) It can quickly update open set labels at low cost without updating the image label recognition model and training set, and supports label recognition of video or image.

[0119] (2) Label recognition can be performed using pre-trained models in related technologies. In related technologies, text or image feature models are generally trained on very large corpora. Even if new labels are added, the corresponding semantic corpus is usually included in the training set, so it can handle updates to open set labels;

[0120] (3) By comparing features, it is more flexible than the direct output of labels by the model in related technologies. For example, by comparing features, multiple labels corresponding to the video can be obtained based on the similarity threshold or sorting method. Since the video contains more information, multiple labels may exist at the same time. Feature similarity, i.e. feature confidence, can obtain multiple high-confidence labels at once based on the similarity threshold or sorting method, preventing the omission of some insignificant labels and making the recognition results richer.

[0121] (4) Video or text models in related technologies can be used to supplement and enrich the technical solution of this disclosure without conflict; for example, when the open set labels are updated very quickly, the image label recognition model can be updated asynchronously, and when the image label recognition model is updated slowly, its results can still be used as a supplement to the technical solution of this disclosure without missing the updated labels.

[0122] Exemplary device

[0123] Having introduced the medium of exemplary embodiments of this disclosure, the following references are made to... Figure 8 An image tag recognition apparatus according to an exemplary embodiment of this disclosure will be described. The apparatus of the exemplary embodiment of this disclosure can implement the various processes in the foregoing image tag recognition method embodiments and achieve the same functions and effects.

[0124] Figure 8 This is a schematic diagram of the structure of an image tag recognition device provided in an embodiment of the present disclosure, applied to a device equipped with an image tag recognition model. For example... Figure 8 As shown, the image tag recognition device 800 of this embodiment includes: an acquisition module 801, a first processing module 802, and a second processing module 803. Wherein:

[0125] The acquisition module 801 is used to acquire the input content to the image label recognition model.

[0126] The first processing module 802 is configured to, if the input content includes a target image, obtain the feature similarity between the target image and at least one label in the input content based on the first label set using an image label recognition model; determine the label whose feature similarity is greater than a similarity threshold as the recognition label corresponding to the target image; and output the recognition label.

[0127] The second processing module 803 is used to update the first text feature vector of the first tag set through the image tag recognition model if the input content includes the tag to be updated, so as to obtain the second text feature vector of the corresponding second tag set.

[0128] In one possible implementation, the image label recognition model includes a text feature extraction network, and the second processing module 803 can be specifically used to: input the label to be updated into the text feature extraction network to update the first text feature vector of the first label set, and obtain the second text feature vector of the corresponding second label set.

[0129] In one possible implementation, the image label recognition model further includes an image feature extraction network and a post-processing network. After obtaining the second text feature vector of the corresponding second label set, the second processing module 803 can also be used to: obtain new input content to the image label recognition model; input the target image in the input content into the image feature extraction network to extract image features and obtain the image feature vector of the target image; input the image feature vector and the second text feature vector into the post-processing network to calculate the feature similarity and obtain the feature similarity between the target image and at least one label.

[0130] In one possible implementation, the image label recognition model includes an image feature extraction network, a text feature extraction network, and a post-processing network. The first processing module 802 can be specifically used to: input the target image into the image feature extraction network to extract image features and obtain the image feature vector of the target image; input the image feature vector and the first text feature vector into the post-processing network to calculate the feature similarity and obtain the feature similarity between the target image and at least one label. The first text feature vector is the text feature vector obtained by inputting the first label set into the text feature extraction network to extract text features.

[0131] In one possible implementation, the input content includes multiple target images. The first processing module 802 can be specifically used to: for the multiple target images, determine the label corresponding to each target image whose feature similarity is greater than a similarity threshold; sort the labels corresponding to the multiple target images whose feature similarity is greater than the similarity threshold from high to low to obtain a first label sequence; filter out the labels that appear repeatedly in the first label sequence to obtain a second label sequence; and in the second label sequence, determine a preset threshold of labels with high feature similarity as the recognition labels corresponding to the multiple target images.

[0132] In one possible implementation, the image tag recognition device further includes a determination module 804, configured to: determine at least one target image contained in the video in response to the input content being a video, before inputting the input content into the image tag recognition model.

[0133] In one possible implementation, the first processing module 802 may be specifically used to: normalize the feature similarity between the identification label and the target image to obtain the probability value corresponding to the identification label, the probability value being used to reflect the degree of matching between the identification label and the target image; and output the identification label and the probability value corresponding to the identification label.

[0134] In one possible implementation, the image label recognition model includes an image feature extraction network, a text feature extraction network, and a post-processing network. The image label recognition model is trained as follows: a training set containing multiple sets of training samples is obtained, each set of training samples including a sample image, sample text, and a reference similarity between the sample image and the sample text; for each set of training samples, the following operations are performed: extracting pre-trained image feature vectors of the sample images based on the image feature extraction network; extracting pre-trained text feature vectors of the sample text based on the text feature extraction network; determining the training similarity between the pre-trained image feature vectors and the pre-trained text feature vectors based on the post-processing network; and adjusting the model parameters of the image label recognition model based on the training similarity and the reference similarity to obtain the image label recognition model.

[0135] The apparatus of this disclosure embodiment can be used to execute the image tag recognition method in any of the above method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0136] Exemplary media

[0137] After introducing the methods of exemplary embodiments of this disclosure, the following references are made. Figure 9 The storage medium of the exemplary embodiments of this disclosure will be described.

[0138] Figure 9 This is a schematic diagram of a storage medium provided according to an embodiment of the present disclosure. (See reference...) Figure 9 As shown, the storage medium 900 stores a program product for implementing the above-described method according to embodiments of the present disclosure. This program product may be a portable compact disc read-only memory (CD-ROM) and includes program code, and can run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto.

[0139] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0140] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium.

[0141] Program code for performing the operations disclosed herein can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing devices can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN).

[0142] Exemplary computing device

[0143] Having described the methods, media, and apparatus of exemplary embodiments of this disclosure, the following references... Figure 10 A computing device according to an exemplary embodiment of the present disclosure will be described.

[0144] Figure 10 The computing device 1000 shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0145] Figure 10 This is a schematic diagram of the structure of a computing device provided in an embodiment of the present disclosure, as shown below. Figure 10 As shown, the computing device 1000 is manifested as a general-purpose computing device. The components of the computing device 1000 may include, but are not limited to: at least one processing unit 1001, at least one storage unit 1002, and a bus 1003 connecting different system components (including the processing unit 1001 and the storage unit 1002). For example, the processing unit 1001 may specifically be a processor, and the storage unit 1002 stores computer execution instructions. The processing unit 1001 executes the computer execution instructions stored in the storage unit 1002 to implement the image tag recognition method described above.

[0146] Bus 1003 includes a data bus, a control bus, and an address bus.

[0147] Storage unit 1002 may include readable media in the form of volatile memory, such as random access memory (RAM) 10021 and / or cache memory 10022, and may further include readable media in the form of non-volatile memory, such as read-only memory (ROM) 10023.

[0148] Storage unit 1002 may also include a program / utility 10025 having a set (at least one) program module 10024, such program module 10024 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0149] The computing device 1000 can also communicate with one or more external devices 1004 (e.g., keyboard, pointing device, etc.). This communication can be performed via the input / output (I / O) interface 1005. Furthermore, the computing device 1000 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via a network adapter 1006. Figure 10 As shown, network adapter 1006 communicates with other modules of computing device 1000 via bus 1003. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with computing device 1000, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0150] It should be noted that although several units / modules or sub-units / modules of the image tag recognition device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0151] Furthermore, although the operations of the methods disclosed herein are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0152] While the spirit and principles of this disclosure have been described with reference to several specific embodiments, it should be understood that this disclosure is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for convenience of expression. This disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. An image label recognition method, applied to a device equipped with an image label recognition model, the image label recognition method comprising: Obtain the input content that is fed into the image label recognition model; If the input content includes a target image, then the image label recognition model is used to obtain the feature similarity between the target image and at least one label in the input content based on the first label set; the label with the feature similarity greater than a similarity threshold is determined as the recognition label corresponding to the target image; and the recognition label is output. If the input content includes tags to be updated, then the first text feature vector of the first tag set is updated through the image tag recognition model to obtain the second text feature vector of the corresponding second tag set. The image label recognition model includes a text feature extraction network. The step of updating the first text feature vector of the first label set through the image label recognition model to obtain the second text feature vector of the corresponding second label set includes: The labels to be updated are input into the text feature extraction network to update the first text feature vector of the first label set, thereby obtaining the second text feature vector of the corresponding second label set; The image label recognition model further includes an image feature extraction network and a post-processing network. After obtaining the second text feature vector of the corresponding second label set, it also includes: Obtain new input content for the image label recognition model; The target image in the input content is input into the image feature extraction network to extract image features, thereby obtaining the image feature vector of the target image; The image feature vector and the second text feature vector are input into the post-processing network to calculate feature similarity, thereby obtaining the feature similarity between the target image and at least one label.

2. The image label recognition method according to claim 1, wherein the image label recognition model includes an image feature extraction network, a text feature extraction network, and a post-processing network, and the step of obtaining the feature similarity between the target image and at least one label in the input content based on the first label set includes: The target image is input into the image feature extraction network to extract image features, thereby obtaining the image feature vector of the target image; The image feature vector and the first text feature vector are input into the post-processing network to calculate the feature similarity, thereby obtaining the feature similarity between the target image and at least one label. The first text feature vector is the text feature vector obtained by inputting the first label set into the text feature extraction network for text feature extraction.

3. The image label recognition method according to claim 1 or 2, wherein the input content includes multiple target images, and determining the label whose feature similarity is greater than a similarity threshold as the recognition label corresponding to the target image includes: For the multiple target images, determine the label corresponding to each target image whose feature similarity is greater than a similarity threshold; The labels corresponding to the multiple target images with feature similarity greater than a similarity threshold are sorted from high to low to obtain the first label sequence; Filter out the tags that appear repeatedly in the first tag sequence to obtain the second tag sequence; In the second label sequence, a preset threshold number of labels with high feature similarity are determined as the recognition labels corresponding to the multiple target images.

4. The image tag recognition method according to claim 1 or 2 further includes: Before inputting the input content into the image label recognition model, in response to the input content being a video, at least one target image contained in the video is determined.

5. The image tag recognition method according to claim 1 or 2, wherein outputting the recognized tag includes: The feature similarity between the identification label and the target image is normalized to obtain the probability value corresponding to the identification label. The probability value is used to reflect the degree of matching between the identification label and the target image. Output the identification label and the probability value corresponding to the identification label.

6. The image label recognition method according to claim 1 or 2, wherein the image label recognition model comprises an image feature extraction network, a text feature extraction network, and a post-processing network, and the image label recognition model is trained in the following manner: Obtain a training set containing multiple sets of training samples, each set of training samples including a sample image, sample text, and a reference similarity between the sample image and the sample text; For each training sample, perform the following operations: The pre-trained image feature vector of the sample image is extracted based on the image feature extraction network; The pre-trained text feature vector of the sample text is extracted based on the text feature extraction network; The training similarity between the pre-trained image feature vector and the pre-trained text feature vector is determined based on the post-processing network. Based on the training similarity and the reference similarity, the model parameters of the image label recognition model are adjusted to obtain the image label recognition model.

7. An image label recognition device, applied to a device equipped with an image label recognition model, the image label recognition device comprising: The acquisition module is used to acquire the input content to the image label recognition model, which includes a text feature extraction network, an image feature extraction network, and a post-processing network. The first processing module is configured to, if the input content includes a target image, obtain the feature similarity between the target image and at least one label in the input content based on a first label set using the image label recognition model; determine the label whose feature similarity is greater than a similarity threshold as the recognition label corresponding to the target image; and output the recognition label. The second processing module is used to update the first text feature vector of the first tag set through the image tag recognition model if the input content includes tags to be updated, so as to obtain the second text feature vector of the corresponding second tag set. The second processing module is specifically used to input the label to be updated into the text feature extraction network to update the first text feature vector of the first label set, and obtain the second text feature vector of the corresponding second label set; After obtaining the second text feature vector of the corresponding second label set, the second processing module can also be used to obtain the input content of the new input to the image label recognition model. The target image in the input content is input into the image feature extraction network to extract image features and obtain the image feature vector of the target image; the image feature vector and the second text feature vector are input into the post-processing network to calculate feature similarity and obtain the feature similarity between the target image and at least one label.

8. The image label recognition device according to claim 7, wherein the image label recognition model includes an image feature extraction network, a text feature extraction network, and a post-processing network, and the first processing module is specifically used for: The target image is input into the image feature extraction network to extract image features, thereby obtaining the image feature vector of the target image; The image feature vector and the first text feature vector are input into the post-processing network to calculate the feature similarity, thereby obtaining the feature similarity between the target image and at least one label. The first text feature vector is the text feature vector obtained by inputting the first label set into the text feature extraction network for text feature extraction.

9. The image tag recognition device according to claim 7 or 8, wherein the input content includes multiple target images, and the first processing module is specifically used for: For the multiple target images, determine the label corresponding to each target image whose feature similarity is greater than a similarity threshold; The labels corresponding to the multiple target images with feature similarity greater than a similarity threshold are sorted from high to low to obtain the first label sequence; Filter out the tags that appear repeatedly in the first tag sequence to obtain the second tag sequence; In the second label sequence, a preset threshold number of labels with high feature similarity are determined as the recognition labels corresponding to the multiple target images.

10. The image tag recognition device according to claim 7 or 8, further comprising a determining module, configured to: Before inputting the input content into the image label recognition model, in response to the input content being a video, at least one target image contained in the video is determined.

11. The image tag recognition device according to claim 7 or 8, wherein the first processing module is specifically used for: The feature similarity between the identification label and the target image is normalized to obtain the probability value corresponding to the identification label. The probability value is used to reflect the degree of matching between the identification label and the target image. Output the identification label and the probability value corresponding to the identification label.

12. The image label recognition device according to claim 7 or 8, wherein the image label recognition model comprises an image feature extraction network, a text feature extraction network, and a post-processing network, and the image label recognition model is trained in the following manner: Obtain a training set containing multiple sets of training samples, each set of training samples including a sample image, sample text, and a reference similarity between the sample image and the sample text; For each training sample, perform the following operations: The pre-trained image feature vector of the sample image is extracted based on the image feature extraction network; The pre-trained text feature vector of the sample text is extracted based on the text feature extraction network; The training similarity between the pre-trained image feature vector and the pre-trained text feature vector is determined based on the post-processing network. Based on the training similarity and the reference similarity, the model parameters of the image label recognition model are adjusted to obtain the image label recognition model.

13. A computing device, comprising: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the image tag recognition method as described in any one of claims 1 to 6.

14. A storage medium storing computer program instructions, which, when executed, implement the image tag recognition method as described in any one of claims 1 to 6.

15. A computer program product comprising a computer program that, when executed by a processor, implements the image tag recognition method as described in any one of claims 1 to 6.

Images