Image label generation method and device and computer equipment

Abstract

The invention relates to the field of image processing, and discloses an image label generation method and device and computer equipment, and the method comprises the steps: extracting image attribute features, image object features and image combination features of a to-be-labeled image; obtaining text attribute features, text object features and text combination features corresponding to different text tags in the text feature set; for each text tag, determining attribute similar data between the image attribute feature and the text attribute feature, object similar data between the image object feature and the text object feature, and combined similar data between the image combined feature and the text combined feature; and carrying out processing and screening based on the similar data, and finally generating a target label of the to-be-labeled image. Therefore, a multi-level network dominated by the attribute mapping space, the object mapping space and the combination mapping space is constructed, the features of different semantic levels in the to-be-labeled image are matched with the corresponding text features to the maximum extent, and a more accurate target label is generated for the to-be-labeled image.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a method, device and computer equipment for generating image tags. Background Art

[0002] Humans can combine objects and attributes that they have seen before to quickly recognize new categories that they have never seen before. For example, people can learn the semantic features of the attribute word "dirty" from "dirty floor" and the semantic features of the object word "rag" from "dry rag". Then, even if people have never seen "dirty rag", they can quickly imagine what this combination looks like and recognize it quickly. Similarly, we hope that computers have the same ability, so we proposed Compositional Zero-Shot Learning (CZSL). CZSL sets the training set and test set to be non-intersecting. Since the degree of interaction between attributes and objects cannot be quantified, it is very important for the network to understand and transfer the concept of combination.

[0003] However, the image recognition and classification methods based on CZSL in related technologies still have certain limitations and need to further improve their accuracy. Summary of the invention

[0004] The present application aims to solve at least one of the technical problems in the related art to a certain extent. To this end, the present application proposes a method, device and computer device for generating image tags. The main technical solutions adopted by the present application include:

[0005] In a first aspect, an embodiment of the present application provides a method for generating an image label, the method comprising: extracting features of an image to be labeled to obtain image attribute features, image object features and image combination features; wherein the image attribute features are used to describe local attribute feature data in the image to be labeled, the image object features are used to describe the paradigm feature data of the object in the image to be labeled, and the image combination features are used to describe the interactive feature data after the attribute and the object are combined in the image to be labeled; obtaining a text feature set corresponding to a set of labels to be matched; wherein the set of labels to be matched includes multiple text labels, and the text feature set includes text attribute features, text object features and text combination features corresponding to each text label; for each text label, determining attribute similarity data between the image attribute features and the text attribute features, object similarity data between the image object features and the text object features, and combination similarity data between the image combination features and the text combination features; performing weighted processing on the attribute similarity data, the object similarity data and the combination similarity data to obtain matching data between each text label and the image to be labeled; generating a target label for the image to be labeled based on the text label corresponding to the matching data that meets the preset screening conditions.

[0006] In a second aspect, an embodiment of the present application provides a device for generating image labels, the device comprising: an image feature extraction module, used to extract features of an image to be labeled, and obtain image attribute features, image object features and image combination features; wherein the image attribute features are used to describe local attribute feature data in the image to be labeled, the image object features are used to describe the paradigm feature data of the object in the image to be labeled, and the image combination features are used to describe the interactive feature data after the attribute and the object are combined in the image to be labeled; a text feature acquisition module, used to acquire a text feature set corresponding to a set of labels to be matched; wherein the set of labels to be matched includes multiple text labels, and the text feature set includes each text label. The text attribute features, text object features and text combination features corresponding to this label; a combination similarity data determination module, used to determine, for each text label, the attribute similarity data between the image attribute features and the text attribute features, the object similarity data between the image object features and the text object features, and the combination similarity data between the image combination features and the text combination features; a matching data determination module, used to perform weighted processing on the attribute similarity data, the object similarity data and the combination similarity data to obtain matching data between each text label and the image to be labeled; a target label generation module, used to generate a target label for the image to be labeled based on the text label corresponding to the matching data that meets the preset screening conditions.

[0007] In a third aspect, the present application further provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any of the above methods when executing the computer program.

[0008] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of any of the above methods when the computer program is executed by a processor.

[0009] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which implements the steps of any of the above methods when executed by a processor.

[0010] In the above embodiment, the attributes, objects and combination features in the image to be annotated are extracted and matched with the text attribute features, text object features and text combination features of the text feature set corresponding to the tag set to be matched. For each text tag, the attribute similarity data between the image attribute feature and the text attribute feature, the object similarity data between the image object feature and the text object feature, and the combination similarity data between the image combination feature and the text combination feature are determined. Further, the attribute similarity data, the object similarity data and the combination similarity data are weighted to obtain the matching data of each text tag and the image to be annotated, and finally the target label of the image to be annotated is generated based on the text label corresponding to the matching data that meets the preset screening conditions. So far, by constructing a multi-level network dominated by the three mapping spaces of attributes, objects and combinations, the features of different semantic levels in the image to be annotated are matched with the corresponding text features to the greatest extent, the ability to understand the complex semantic information in the image is enhanced, the recognition and generalization capabilities of new categories are significantly improved, and more accurate target labels are generated for the image to be annotated. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0012] Figure 1 A flowchart of a method for generating an image tag according to an embodiment of the present application;

[0013] Figure 2 A flowchart of obtaining features of an image to be annotated according to an embodiment of the present application;

[0014] Figure 3 A schematic diagram of a target label prediction model training process provided according to an embodiment of the present application;

[0015] Figure 4 A flowchart of determining a target tag prediction model according to an embodiment of the present application;

[0016] Figure 5 A flowchart for determining positive triplet loss data according to an embodiment of the present application;

[0017] Figure 6 A flowchart for determining positive classification loss data according to an embodiment of the present application;

[0018] Figure 7A flowchart for determining negative triplet loss data according to an embodiment of the present application;

[0019] Figure 8a A flowchart for determining negative classification loss data according to an embodiment of the present application;

[0020] Figure 8b A schematic diagram of prediction results using a target tag prediction model provided according to an embodiment of the present application;

[0021] Fig. 9 A structural block diagram of an image tag generating device provided according to an embodiment of the present application;

[0022] Fig.10 The figure is a diagram of the internal structure of a computer device according to one embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0024] Humans can combine objects and attributes that they have seen before to quickly recognize new categories that they have never seen before. For example, people can learn the semantic features of the attribute word "dirty" from "dirty floor" and the semantic features of the object word "rag" from "dry rag". Then, even if people have never seen "dirty rag", they can quickly imagine the appearance of this combination and quickly recognize it. Similarly, we hope that computers have the same ability, so we proposed Compositional Zero-Shot Learning. CZSL sets the training set and the test set to be non-intersecting. Since the degree of interaction between attributes and objects cannot be quantified, it is very important for the network to understand and transfer the concept of combination. However, the CZSL method in related technologies focuses on modeling objects and attributes as a combination of visual prototypes, using a large number of samples to train the classifiers of attributes and objects, and strengthening the network's ability to understand the characteristics of their respective original domains, but ignores the dependency relationship between attributes and objects. For example, the early CZSL method used a single original classifier to calculate the corresponding prediction probability as output. Some studies believe that learning the compatibility function between attributes and objects is crucial to determining composition. For example, by separating encoders and then fusing them with linear layers or multi-layer perceptrons, attributes and objects are separated from composite entities, etc. Recent high-performance work includes using graph convolutional neural networks to learn the relationship between attributes and objects, and introducing a key attention module to learn the relationship between primitive concepts through the network with information transfer.

[0025] However, these methods ignore the nature of attributes attached to objects. Attributes often play a role in changing the prototype of objects. For example, there is a clear difference between the appearance of a bottle and a melted bottle. This oversight leads to a domain gap between the visible and invisible classes. Even for the same attributes, the combination after changing the object will produce obvious changes. For example, an old house and an old dog do not have an intuitive universal commonality, even if they share the same adjective "old". The model often cannot understand this dynamic and complex change, which leads to the problem of projected domain drift and ultimately affects the performance of the model. The projected domain drift problem is due to the limited training samples and insufficient use of supervision information. Only focusing on the distance between the sample and its label may lead to network overfitting and poor robustness. Some studies introduce contrastive learning to increase robustness, in which several samples are selected as negative samples of other label samples. Although it is reasonable to introduce contrastive learning, it only requires negative samples to be far away from positive samples during training without specifying the actual range and position. This still causes the projected domain of the visual sample to drift toward the text label, affecting the transfer of knowledge from the seen class to the unseen class.

[0026] Based on this, according to an embodiment of the present application, a method, apparatus and computer device for generating image labels are provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0027] In this embodiment, a method for generating an image label is provided. Figure 1 is a flow chart of a method for generating an image tag according to an embodiment of the present application, such as Figure 1 As shown, the process includes the following steps:

[0028] S110 , extracting features from the image to be annotated to obtain image attribute features, image object features, and image combination features.

[0029] Among them, the image attribute feature can be used to describe the local attribute feature data in the image to be annotated, the image object feature can be used to describe the paradigm feature data of the object in the image to be annotated, and the image combination feature can be used to describe the interactive feature data after the combination of the attribute and the object in the image to be annotated.

[0030] Specifically, the image attribute feature is used to reflect the local features of the attributes in the image to be annotated, such as the features with strong semantics of a certain part of the details such as edge, color or texture; the image object feature can be used to reflect the features related to the main object in the image to be annotated, such as the shape, size and posture features of the object or the primitive composition features of the object; the image combination feature describes the overall characteristics of the interaction and combination of the objects and attributes in the image to be annotated, focusing on how the objects and attributes in the image interact with each other, and the overall features formed by their combination. The combination feature takes into account the semantic attributes of the main objects in the image, and also pays attention to how these attributes affect the objects, as well as the combination effect of the objects and attributes. For example, if a picture of a red apple is used as the image to be annotated, then the "red" in the picture can be regarded as the image attribute feature of the image to be annotated, and the "apple" can be regarded as the image object feature of the image to be annotated, and the "red apple" formed by the combination of the two can be used as the image combination feature, that is, the image attribute feature and the image object feature reflect common features, but the image combination feature provides a proprietary feature.

[0031] Furthermore, after the image to be annotated is input, the overall features of the image to be annotated can be obtained by using an image encoder, and then the overall features of the image to be annotated can be mapped to feature spaces representing different meanings through different mapping encoders to obtain semantically different image attribute features, image object features and image combination features.

[0032] For example, the Resnet-18 network can be used as an image encoder to obtain the overall features of the image to be annotated. Furthermore, the three input layers are designed to be 512-dimensional, the output is 300-dimensional, the fully connected layer is 2, and the MLP layer without an activation function is used as a mapping encoder for the three features. Through continuous training, adjustment and optimization of the weights and bias parameters of the three MLP layers, the overall input features are finally mapped to three feature vector spaces with the same dimensions but different parameters according to different categories, and converted into image attribute features, image object features and image combination features that can express the characteristics of different levels of the image.

[0033] S120: Obtain a text feature set corresponding to the tag set to be matched.

[0034] The tag set to be matched includes multiple text tags, and the text feature set includes text attribute features, text object features and text combination features corresponding to each text tag.

[0035] Specifically, the tag set to be matched may be a tag set storing multiple tags to be matched, and illustratively, the tag set to be matched may be a text database containing many text tags. The text tags may be keywords or phrases used to describe the image content, and may reflect the attributes, objects, and combination characteristics of the image to be annotated.

[0036] It can be understood that the text feature set can be a feature set constructed by embedding each text label in the label set to be matched into a numerical vector in advance through natural language processing technology such as word embedding technology. The text labels are mapped into vector spaces corresponding to image characteristics in different categories through word embedding technology, and converted into text attribute features, text object features and text combination features that can reflect the different characteristics of text labels, so that they can be called at any time when applied.

[0037] Among them, the text attribute feature can be a feature used to describe the attribute of a certain text label. For example, if the text label is the word "red", the text attribute feature can be a feature vector reflecting the attribute "red" obtained after feature extraction and mapping transformation for the word "red"; the text object feature can be a feature used to describe the object in a certain text label. If the text label is the word "apple", the text attribute feature can be a feature vector reflecting the object "apple" obtained after feature extraction and mapping transformation for the word "apple"; the text combination feature can be used to describe the overall feature formed by the combination of attributes and objects in a certain text label. Similarly, if the text label is the phrase "red apple", the text attribute feature can be a feature vector reflecting the whole "red apple" obtained after feature extraction and mapping transformation for the phrase "red apple".

[0038] Optionally, the text combination feature can also be obtained by concatenating the text attribute feature and the text object feature obtained by mapping, and processing them through a combination encoder. c represents the text combination feature, w a represents the text attribute feature, w o Represents the text object feature, then the method for obtaining the text combination feature can be shown as follows:

[0039] w c =C(cat(w a ,w o ))

[0040] As can be seen from the formula, the text attribute features and text object features are concatenated into vectors by means of cat, and then the concatenated vectors are processed by the linear layer C as a combined encoder to obtain the text combination feature w that can describe the entire text c . This allows the text combination feature to reflect the semantic relationship between the attribute word and the object word in the text label, and its vector dimension is also equivalent to the vector dimension of the text attribute feature and the text object feature. In the subsequent screening and matching, it can be measured with the image attribute features, image object features and image combination features of the image to be annotated through a unified standard.

[0041] S130 . For each text tag, determine attribute similarity data between image attribute features and text attribute features, object similarity data between image object features and text object features, and combination similarity data between image combination features and text combination features.

[0042] Among them, attribute similarity data can be data that measures the similarity between attribute features in an image and attribute features in a corresponding text label, and can be used to quantify the degree of match between the local attributes (color or texture, etc.) of the image to be annotated and the attributes mentioned in the text description.

[0043] Object similarity data can be data used to measure the similarity between object features in an image and object features in a corresponding text label, and is used to quantify the degree of match between the main object (such as a person or a car) in the image to be annotated and the object mentioned in the text description.

[0044] Similarly, combined similarity data can be data that measures the similarity between the overall features of the combination of attributes and objects in the image and the combined features of the attributes and objects in the corresponding text label, and is used to quantify the degree of match between the combined features formed by the interaction of attributes and objects in the image to be annotated and the combined features mentioned in the text label.

[0045] Specifically, after obtaining the image attribute features, image object features and image combination features of the image to be annotated and the text attribute features, text object features and text combination features corresponding to all text labels, the cosine similarity between the image features of the image to be annotated and the text features of the text labels can be used as a judgment criterion, and further by calculating the conditional probability with the image features of the image to be annotated as a condition, the similarity data between the image features of the image to be annotated and the text features of the text labels can be obtained.

[0046] Exemplarily, a method for calculating similar data between image features of an image to be annotated and text features of a text label using conditional probability may be shown as follows:

[0047]

[0048] Among them, k∈{a, o, c}, a represents the attribute, o represents the object, and c represents the combination; v i represents the image feature of the image to be annotated, which can be any of attributes, objects, and combinations; w k The text features corresponding to the text labels correspond to the image features to be annotated one by one; cos(v i ,w k ) is the cosine similarity between the image features of the image to be annotated and the text features of the text label; ||v i ||2 is the Euclidean norm of the image feature to be labeled, ||w k ||2 is the Euclidean norm of the text feature corresponding to the text label, and p(k|i) is the conditional probability conditioned on the image features of the image to be annotated.

[0049] Further, taking the combination category as an example, if we use v cRepresents the image combination features of the image to be labeled, using w c Represents the text combination feature corresponding to any text label, that is, both k and i are in the category of c. For each text label in the set of labels to be matched, P(c|i) can be obtained by substituting the above formula:

[0050]

[0051] The conditional probability P(c|i) can be used as the combined similarity data between the image combination features of the image to be annotated and the text combination features of any text label.

[0052] Similarly, if we use v a Represents the image attribute features of the image to be annotated, using w a represents the text attribute feature corresponding to any text label, that is, k and i are both substituted into the category a, and P(a|i) is obtained by substituting into the above formula. This conditional probability P(a|i) can be used as the attribute similarity data between the image attribute feature of the image to be annotated and the text attribute feature of any text label; if v is used o Represents the image object features of the image to be labeled, using w o Represents the text object features corresponding to any text label, that is, k and i are both substituted into the category o, and then P(o|i) is obtained by substituting it into the above formula. The conditional probability P(o|i) can be used as the object similarity data between the image object features of the image to be annotated and the text object features of any text label.

[0053] S140 , weighted processing is performed on the attribute similarity data, the object similarity data, and the combination similarity data to obtain matching data between each text label and the image to be annotated.

[0054] The matching data may be data that quantitatively describes the degree of similarity between each text label and the image to be annotated in a certain category, and may be regarded as a quantitative representation of the "distance" between the image feature vector and the text label feature vector in the feature space. For example, the matching data may be a prediction score that reflects the degree of similarity between the text attribute features, text object features, and text combination features corresponding to each text label and the image, object, and combination features contained in the image to be annotated.

[0055] Specifically, for any text tag, matching data can be obtained by weighted processing based on its attribute similarity data, object similarity data and combination similarity data. For example, as shown in the following formula:

[0056] s=αP(c∣i)+(1-α)P(a∣i)P(o∣i)

[0057] Among them, s can represent the matching data between the image to be annotated and a certain text label; P(c|i) represents the combined similarity data between the image combination features of the image to be annotated and the text combination features of the text label; P(a|i) represents the attribute similarity data between the image attribute features of the image to be annotated and the text attribute features of the text label; P(o|i) represents the object similarity data between the image object features of the image to be annotated and the text object features of the text label; α is a weight coefficient, which can be determined by prior knowledge or pre-testing.

[0058] S150: Generate a target label for the image to be annotated based on the text label corresponding to the matching data that meets the preset screening condition.

[0059] Among them, the target label can refer to the output label that is true, accurate and in line with the expectations of the given data. In the field of machine learning, the target label can be understood as the standard answer to verify the model prediction. Specifically, the target label can be the correct category or description that the image to be annotated should be classified or labeled. For example, in the image annotation task, if the image to be annotated shows a yellow cat, then "yellow cat" can be the target label of this picture.

[0060] Specifically, after obtaining the matching data corresponding to the image to be annotated and several text labels, the matching data can be screened according to the pre-set screening conditions to finally determine the target label of the image to be annotated. Exemplarily, if the matching data is a matching score, the matching scores can be ranked from high to low, and the ranking or ratio of the matching scores is taken as the preset screening condition. Further, the top N labels with the highest matching scores are selected, or the top-K labels with the highest scores are selected to determine the target label of the image to be annotated.

[0061] In the above implementation, the attributes, objects and combination features in the image to be annotated are extracted and matched with the text attribute features, text object features and text combination features of the text feature set corresponding to the tag set to be matched. For each text tag, the attribute similarity data between the image attribute feature and the text attribute feature, the object similarity data between the image object feature and the text object feature, and the combination similarity data between the image combination feature and the text combination feature are determined. Further, the attribute similarity data, the object similarity data and the combination similarity data are weighted to obtain the matching data of each text tag and the image to be annotated, and finally the target label of the image to be annotated is generated based on the text label corresponding to the matching data that meets the preset screening conditions. So far, by constructing a multi-level network dominated by the three mapping spaces of attributes, objects and combinations, the features of different semantic levels in the image to be annotated are matched with the corresponding text features to the greatest extent, the ability to understand the complex semantic information in the image is enhanced, the recognition and generalization capabilities of new categories are significantly improved, and more accurate target labels are generated for the image to be annotated.

[0062] In some embodiments, please refer to the attached Figure 2 , extract features from the input image to be annotated, and obtain image attribute features, image object features, and image combination features, including:

[0063] S210, extracting overall features of the image to be annotated to obtain overall features of the image.

[0064] The overall image features may refer to comprehensive features that include the main visual information in the image to be annotated. Specifically, a Resnet-18 network may be used as an image encoder to obtain overall image features that represent the main content and visual characteristics of the image to be annotated.

[0065] S220 , performing feature mapping on the overall features of the image through an attribute mapping encoder to obtain image attribute features.

[0066] Among them, the attribute mapping encoder can be a mapping encoder for mapping the overall features of an image to an attribute space. Exemplarily, an input layer with 512 dimensions, an output with 300 dimensions, a fully connected layer with 2, and an MLP layer without an activation function can be set as an attribute mapping encoder. By performing mapping processing on the training image in advance, the weights and bias parameters are continuously adjusted and optimized so that it can focus on capturing features related to specific attributes (such as color, texture, etc.) in the image, and converting them into image attribute features that can reflect the local characteristics of the image. Furthermore, after the attribute mapping encoder is trained, feature mapping is performed on the overall image features of the image to be annotated, and the overall image features are mapped to a specific attribute space, thereby obtaining image attribute features.

[0067] S230 , performing feature mapping on the overall features of the image through an object mapping encoder to obtain image object features.

[0068] Similarly, an MLP layer with an input layer of 512 dimensions, an output of 300 dimensions, a fully connected layer of 2, and no activation function can be set as an object mapping encoder. By pre-mapping the training image, the weight and bias parameters are continuously adjusted and optimized, so that it can focus on capturing the features related to the main objects or components in the image, and convert them into image object features that can reflect the image paradigm features. Furthermore, after the object mapping encoder is trained, the overall image features of the image to be annotated are feature mapped, and the overall image features are mapped to a specific object space to obtain the image object features.

[0069] S240, performing feature mapping on the overall features of the image through a combined mapping encoder to obtain image combined features.

[0070] Similarly, an MLP layer with an input layer of 512 dimensions, an output of 300 dimensions, a fully connected layer of 2, and no activation function can be set as a combined mapping encoder. By pre-mapping the training image, the weight and bias parameters are continuously adjusted and optimized, so that it can focus on capturing the relevant features of how the attributes and objects in the image interact and interact, and convert them into image combination features that can reflect the interactive feature data of the combination of the attributes and objects in the image to be annotated. Furthermore, after the combined mapping encoder is trained, the overall image features of the image to be annotated are feature mapped, and the overall image features are mapped to a specific combination space, thereby obtaining the image combination features.

[0071] In the above implementation, by setting three mapping encoders for different characteristics and levels of the image, each encoder focuses on capturing different aspects of the image, and the overall image features of the image to be annotated can be mapped into three primitive spaces with the same dimensions but different parameters, and semantically different image attribute features, image object features, and image combination features can be obtained. These three features can respectively reflect the characteristics of different levels and aspects in the image to be annotated, including not only the direct visual information of the image, but also deeper semantic information, which can provide basic data support for the subsequent generation of target labels for the image to be annotated.

[0072] In some embodiments, the target label is obtained by a target label prediction model.

[0073] The target label prediction model may refer to a machine learning model that analyzes the features of input data and learns how to associate these features with predefined labels to achieve automatic labeling or classification. Specifically, the target label prediction model may be a pre-trained model that can recognize and classify images to be labeled and generate correct target text labels for the images to be labeled. Figure 3 As shown, the target label prediction model 300 includes a parallel image processing branch 310 and a text processing branch 320 .

[0074] The image processing branch 310 may be a part of the target label prediction model that focuses on processing and analyzing the input image data to be labeled. Specifically, the image processing branch 310 includes an image encoder 311, and an attribute mapping encoder 313, an object mapping encoder 315, and a combination mapping encoder 317 connected to the image encoder 311.

[0075] Exemplarily, the image encoder 311 can be used to extract overall features of the image to be annotated, so as to obtain overall features of the image that represent the main content and visual characteristics of the image to be annotated. The attribute mapping encoder 313 can be a module for obtaining image attribute features that can reflect the local characteristics of the image; the object mapping encoder 315 can be a module for obtaining image object features that can reflect the image paradigm features; the combination mapping encoder 317 can be a module for obtaining image combination features that can reflect the interactive feature data after the combination of attributes and objects in the image to be annotated.

[0076] The text processing branch 320 may refer to a part of the target tag prediction model that focuses on processing and analyzing multiple text tags in the tag set to be matched. The text processing branch 320 includes a text attribute encoder 321, an object attribute encoder 323, and a combined encoder 325 connected to the text attribute encoder 321 and the object attribute encoder 323.

[0077] It is understandable that the text attribute encoder 321, the object attribute encoder 323 and the combined encoder 325 can all be modules for processing a number of text tags in the tag set to be matched to extract text features related to a certain level. Specifically, the text attribute encoder 321 can perform word embedding encoding on the text tag and map it to the attribute vector space 331 having the same dimension and parameters as the image attribute feature to obtain the text attribute feature. Similarly, the object attribute encoder 323 can perform word embedding encoding on the text tag and map it to the object vector space 333 having the same dimension and parameters as the image object feature to obtain the text object feature. It should be understood that since the combined encoder 325 is also connected to the text attribute encoder 321 and the object attribute encoder 323, the combined encoder 325 can not only perform word embedding encoding mapping on the text tag to obtain the combined object feature, but also perform vector splicing and re-embedding processing on the text attribute feature and the text object feature to obtain the combined object feature that can reflect the semantic relationship between the attribute word and the object word in the text tag, and the vector dimension is also equal to the text attribute feature and the text object feature vector dimension. Similarly, the combined object features obtained by mapping and embedding processing by the combined encoder 325 and the image combined features obtained by processing by the combined mapping encoder 317 are also in the combined vector space 335 with the same dimension and parameters.

[0078] In the above implementation, a target label prediction model is pre-trained, and the image processing branch and the text processing branch thereof work together, so that the target label prediction model can accurately identify the content contained in the image to be labeled and find the most suitable text label for the image to be labeled. By using different encoders, the image features and text features of the three categories are respectively mapped to the same vector space, ensuring the correspondence between sample pairs, taking into account both the lower semantic features of the primitive part (attributes and objects) in the combination and the higher semantic combination information of the relative changes of the attributes to the objects, retaining the characteristics of the attributes attached to the objects, and deconstructing the entanglement between the attributes and the objects, so that the target label prediction model has robust generalization performance.

[0079] In some embodiments, please refer to the attached Figure 4 , the target label prediction model is obtained by:

[0080] S410. Build an initial label prediction model based on the combined zero-shot learning algorithm.

[0081] Among them, Compositional Zero-Shot Learning refers to an algorithm whose task is to identify the combination of novel attributes and object concepts seen. Specifically, in Compositional Zero-Shot Learning, it is necessary to identify the visual initial state observed in the image to be labeled and the invisible combination relationship between objects, and generate a target label. Exemplarily, if the training set contains the observed visual original state, such as attribute features such as red or yellow, and also includes objects, such as object features such as apples or bananas, then it is necessary to identify such unlearned combination relationships between attributes and objects in the image to be labeled that contains the combination relationship of "red apple" to generate the target label "red apple". Further, an initial label prediction model is built based on the Compositional Zero-Shot Learning algorithm. It can be understood that the initial label prediction model can be a basic label prediction model that needs to be trained and improved. Although it also has parallel image processing branches and text processing branches, its recognition effect and label matching accuracy are relatively poor.

[0082] S420: Determine a first positive sample image and a first negative sample image in the training data set.

[0083] Specifically, the first negative sample image is a sample image other than the first positive sample image in the training data set. For example, if the selected first positive sample image is an image containing a "red pomegranate", then all other non-"red pomegranate" images in the training data set can be used as the first negative sample image.

[0084] Optionally, in zero-sample learning, since it is necessary to improve the matching between image and text features, the first positive sample image and the first negative sample image can be further refined. For example, although the prediction model has not learned the image sample of "red apple", after training, it can realize the combination of the text label "red apple" from the existing samples when the "red apple" picture is input. That is, if the text label "red apple" is used as the anchor point for training the initial label prediction model, when the positive and negative samples are initially constructed, in the vector space, the features of each part of the selected first positive sample image need to be as close as possible to the text label "red apple". Specifically, from the perspective of attributes, in order to enable the model to learn the attribute characteristics of "red apple" as much as possible, the selected first positive sample image can be an image containing "red pomegranate" or an image containing "red sun"; from the perspective of objects, in order to enable the model to learn the object characteristics of "red apple" as much as possible, the selected first positive sample image can be an image containing "rotten apple" or an image containing "sliced ​​apple". In contrast, the features of each part of the selected first negative sample image need to be as far away from the text label "red apple" as possible, that is, the attribute characteristics and object characteristics contained in the selected first negative sample image should be as different as possible from the text label "red apple". Therefore, the selected first negative sample image can be an image containing a "black butterfly" or an image containing a "green house".

[0085] S430: Process the first positive sample image and the first negative sample image through the initial label prediction model to obtain first loss data.

[0086] Specifically, the initial label prediction model can perform feature extraction mapping and analysis on the first positive sample image that the model hopes to correctly identify and classify and the first negative sample image that the model hopes to distinguish and reject, and predict their labels. These labels predicted by the initial label prediction model are then compared and analyzed with the actual labels of the first positive sample image and the first negative sample image to obtain the first loss data. Exemplarily, the first loss data may refer to a loss function that defines the difference between the model prediction value and the actual value, which may include cross entropy loss or mean square error loss, or may include triple loss and classification loss.

[0087] S440: Determine a second negative sample image and a second positive sample image in the training data set.

[0088] Specifically, after the model is trained for the first time with the first positive sample image as the positive sample and the first negative sample image as the negative sample and loss data is obtained, a mutual learning mechanism can be adopted to exchange the positions of the positive sample and the negative sample for learning and training. That is, when selecting the positive and negative samples for the second model training, the image that is as opposite as possible to the first positive sample image can be used as the learning object that the model mainly needs to recognize and classify during the second training, and the image similar to the first positive sample image can be used as the learning object that the model needs to reject during the second training. It can be understood that in extreme cases, the negative sample of the first training can be directly used as the positive sample input that the model needs to learn during the second training, and the positive sample of the first training can be directly used as the negative sample of the second training. That is to say, in the extreme case of high similarity (complete consistency), the second negative sample image can be the first negative sample image, and the second positive sample image can also be the first positive sample image.

[0089] For example, if the first positive sample image selected in the first training is an image containing a "red pomegranate", and the first negative sample image selected is an image containing a "black butterfly", then based on the idea of ​​mutual learning, in the second training, the positive sample that the model mainly needs to learn, that is, the second negative sample image selected in the second training should be as far away from the first positive sample image as possible in the vector space, and any image that is different from the first positive sample image of "red pomegranate" in terms of attribute characteristics and object characteristics can be selected as the second negative sample image input.

[0090] It is understandable that, since the first negative sample image was selected at that time, the image selected was also different from the first positive sample image in terms of attribute characteristics and object characteristics, so in the second training of the model, an image that is highly similar to the first negative sample image can be selected as the main positive sample for learning at this time. Even in some extreme cases, the first negative sample image can be directly used as the main positive sample for learning at this time. That is, in the second model training, the "black butterfly" image used as a negative sample in the first training can be directly used as the main positive sample for learning in the second model training.

[0091] Similarly, when the model is trained for the second time, since the second positive sample image can be a sample image other than the second negative sample image in the training data set, that is, at this time, all other non-"black butterfly" images in the training data set can be used as the first negative sample image. It can also be understood that based on the idea of ​​mutual learning, in extreme cases, the "red pomegranate" image used as a positive sample in the first training can be directly used as a negative sample to be rejected in the second model training.

[0092] S450: Process the second positive sample image and the second negative sample image through the initial label prediction model to obtain second loss data.

[0093] Similarly, when learning the second positive sample image and the second negative sample image, the initial label prediction model can perform feature extraction mapping and analysis on the second negative sample image that the desired model can correctly identify and classify and the second positive sample image that the desired model can distinguish and reject, and predict their labels, and finally obtain the second loss data based on the comparison of these test labels with the actual labels of the second positive sample image and the second negative sample image.

[0094] S460: Update the parameters of the initial label prediction model based on the first loss data and the second loss data to obtain a target label prediction model.

[0095] Specifically, in the mutual learning training process, the first loss data obtained based on the first positive sample image and the first negative sample image, and the second loss data obtained based on the second positive sample image and the second negative sample image are analyzed and processed to guide the optimization of the model parameters of the initial label prediction model. By iteratively updating the model parameters multiple times in the entire training data set, a target label prediction model that can more accurately predict the image to be labeled and generate the target label is finally obtained.

[0096] In the above implementation, through mutual learning, the model can be exposed to more diverse data representations and features, which helps to improve the model's generalization ability for unseen data. At the same time, this target label prediction model can pay attention to the dynamics of negative samples and give the same conceptual modeling, effectively utilize the information of positive and negative samples and build the relationship between elements at different levels, which not only enriches the samples of the network and enhances its learning ability, but also provides a further understanding of the different combinations that may be generated by positive and negative samples.

[0097] In some embodiments, the first positive sample image has a first positive text label, and the first loss data includes positive triplet loss data. Figure 5 , the positive triplet loss data is determined by:

[0098] S510, matching and aligning are performed according to the triplet image features of the first positive sample image, the triplet image features of the first negative sample image, and the triplet text features of the first positive text label to determine positive attribute loss data, positive object loss data, and positive combination loss data.

[0099] The triplet image feature includes image attribute feature, image object feature and image combination feature. The triplet text feature includes text attribute feature, text object feature and text combination feature.

[0100] It should be understood that the positive triplet loss data can be the loss data used to train the model to identify and distinguish different samples, and is an indicator of the performance of the model in distinguishing positive samples from negative samples.

[0101] For example, if the first positive sample image selected is an image containing "red pomegranate", then the image attribute feature in the triplet image feature of the first positive sample image is the feature data reflecting the attribute feature of "red"; the image object feature in the triplet image feature is the feature data reflecting the object feature of "pomegranate"; and the image combination feature in the triplet image feature is the feature data reflecting the combination feature of "red pomegranate". The first positive text label of the first positive sample image is the phrase "red pomegranate", the text attribute feature in the triplet text feature is the feature data reflecting the word "red"; the text object feature in the triplet text feature is the feature data reflecting the word "pomegranate"; and the text combination feature in the triplet text feature is the feature data reflecting the phrase "red pomegranate".

[0102] It should be noted that the first negative sample image is a sample image other than the first positive sample image in the training data set, and here only an image containing a "black butterfly" is used as an example. If the selected first negative sample image is an image containing a "black butterfly", then the image attribute feature in its triple image feature is the feature data reflecting the attribute feature of "black"; the image object feature in its triple image feature is the feature data reflecting the object feature of "butterfly"; and the image combination feature in its triple image feature is the feature data reflecting the combination feature of "black butterfly".

[0103] Specifically, the positive triplet loss data can be calculated by the following formula:

[0104] L tri (w k ,v k ,v′ k )=max(0,[d(w k ,v k )-d(w k ,v′ k )]+m)

[0105] Where k∈{a, o, c}, a represents the attribute, o represents the object, and c represents the combination; d(,) represents the Euclidean distance; m represents the edge value of the triple loss, which can be fixed as a constant of 0.5 in model training; w k v is the triple text feature of the first positive text label, which can be any of attributes, objects and combinations; kis the triplet image feature of the first positive sample image, which corresponds one-to-one to the triplet text feature of the first positive text label; v′ k is the triplet image feature of the first negative sample image; L tri (w k ,v k ,v′ k ) is the positive triplet loss data.

[0106] It should be understood that since the first negative sample image can be other sample images other than the first positive sample image in the training data set, when calculating the positive triplet loss data, it is not necessary to only compare the first positive sample image with a specific first negative sample image, that is, to match and align the triplet image features of the first positive sample image and the triplet text features of the first positive text label with the triplet image features corresponding to all the first negative sample images in the training data set, and finally take the maximum value as the positive attribute loss data, positive object loss data and positive combination loss data obtained by training based on the first positive sample image.

[0107] Furthermore, the image attribute feature v in the triplet image feature of the first positive sample image is a , the image attribute feature v′ in the triplet image feature of the first negative sample image a and the text attribute feature w in the triple text feature of the first positive text label a Substituting into the above formula, we can calculate the positive attribute loss data L tri (w a ,v a ,v′ a ). The image object feature v in the triplet of image features of the first positive sample image is o , the image object feature v′ in the triplet of image features of the first negative sample image o and the text object feature w in the triplet text feature of the first positive text label o Substituting into the above formula, we can calculate the positive object loss data L tri (w o ,v o ,v′ o ). Similarly, the image combination feature v in the triplet image feature of the first positive sample image is c , the image combination feature v′ in the triplet image feature of the first negative sample image c and the text combination feature w in the triple text feature of the first positive text label c Substituting into the above formula, we can get the positive combination loss data L tri (w a ,v a ,v′ a ). Among them, each characteristic parameter can continue to refer to Figure 3 shown.

[0108] S520, summing the positive attribute loss data, the positive object loss data, and the positive combination loss data to obtain positive triplet loss data.

[0109] It can be understood that, since three mapping spaces are created, the positive attribute loss data, the positive object loss data, and the positive combination loss data can be summed when the triplet loss of the prediction model is finally calculated to obtain the positive triplet loss data for model training based on the first positive sample image and the first negative sample image. Specifically, the calculation can be performed by the method shown in the following formula:

[0110] L mlp =L tri (w a ,v a ,v′ a )+L tri (w o ,v o ,v′ o )+L tri (w c ,v c ,v′ c )

[0111] Among them, L ,lp Represents the positive triplet loss data for model training based on the first positive sample image and the first negative sample image. Similarly, each feature parameter can continue to refer to Figure 3 shown.

[0112] In the above implementation, based on the features of the three levels, the triplet loss is calculated to obtain positive attribute loss data, positive object loss data, and positive combination loss data, and finally these three loss data are added together to obtain a comprehensive loss value. This comprehensive positive triplet loss data reflects the overall performance of the model in distinguishing positive samples from negative samples. Furthermore, the model parameters can be continuously optimized to reduce the positive triplet loss data, helping the model learn how to accurately identify and classify images, thereby continuously narrowing the distance between images and texts to ultimately achieve the effect of clustering, and ultimately improving the performance of the model.

[0113] In some embodiments, the first negative sample image has a first negative text label, and the first loss data also includes positive classification loss data. Figure 6 , the positive classification loss data is determined by:

[0114] S610, performing matching and alignment according to the triplet image features of the first positive sample image, the triplet text features of the first negative text label, and the triplet text features of the first positive text label to determine the positive attribute classification loss, the positive object classification loss, and the positive combination classification loss.

[0115] It should be understood that the positive classification loss data may refer to the loss calculated by the model for the classification error of the positive sample image in the classification task.

[0116] Similarly, the first negative text label is described by taking "black butterfly" as an example. If the selected first negative sample image is an image containing "black butterfly", then the first negative sample image has the first negative text label, which is the phrase "black butterfly", and the text attribute feature in its triple text feature is the feature data reflecting the word "black"; the text object feature in its triple text feature is the feature data reflecting the word "butterfly"; and the text combination feature in its triple text feature is the feature data reflecting the combination of the phrase "black butterfly".

[0117] Specifically, by setting the classification loss to integrate the object semantic information, the cosine similarity between the triple image features of the first positive sample image and the triple text features under the corresponding categories of all the first negative text labels can be calculated first, and then the cosine similarity can be applied to the classification loss to construct the semantic features of the image and text and the classification ability of the network, that is, the positive classification loss data can be determined based on the following formula:

[0118]

[0119] Where k∈{a, o, c}, a represents the attribute, o represents the object, and c represents the combination; τ is the temperature parameter, which is used to adjust the overall variance distribution of the data; w k v is the triple text feature of the first positive text label, which can be any of attributes, objects and combinations; k is the triplet image feature of the first positive sample image, which corresponds one-to-one to the triplet text feature of the first positive text label; w′ k is the triple text feature of the first negative text label; L k is the positive classification loss data.

[0120] Similarly, based on the above formula, we can get the positive attribute classification loss L a , positive object classification loss L o and the positive combination classification loss L c .

[0121] S620, performing weighted summation on the positive attribute classification loss, the positive object classification loss, and the positive combination classification loss to obtain positive classification loss data.

[0122] Similarly, since three mapping spaces are created, the positive attribute classification loss, the positive object classification loss, and the positive combination classification loss can be weighted summed to obtain the positive classification loss data. Specifically, it can be calculated by the method shown in the following formula:

[0123]

[0124] Among them, L sp Represents the positive classification loss data for model training based on the first positive sample image and the first negative sample image.

[0125] In the above implementation, based on the features at three levels, the positive attribute classification loss, the positive object classification loss, and the positive combination classification loss are calculated, and finally the three loss data are weighted and summed according to the importance or contribution of different classification losses to obtain the comprehensive positive classification loss data. This comprehensive positive classification loss data allows the model to balance the loss at different feature levels, ensuring that the model has good performance on relevant features at all levels.

[0126] In some embodiments, the second negative sample image has a second negative text label, and the second loss data includes negative triplet loss data. Figure 7 , the negative triplet loss data is determined by:

[0127] S710, performing matching and alignment according to the triplet image features of the second negative sample image, the triplet image features of the second positive sample image, and the triplet text features of the second negative text label to determine negative attribute loss data, negative object loss data, and negative combination loss data.

[0128] Among them, the triplet image feature includes image attribute feature, image object feature and image combination feature; the triplet text feature includes text attribute feature, text object feature and text combination feature.

[0129] Similarly, the negative triplet loss data can also be used to train the model to identify and distinguish different samples, and is an indicator to measure the performance of the model in distinguishing positive samples from negative samples. It can be understood that, due to the idea of ​​mutual learning, in extreme cases, an image containing a "black butterfly" can be selected as the second negative sample image as the main learning object for the training model to identify and classify, and the negative triplet loss data can be determined based on the second negative sample image.

[0130] Similarly, it can be calculated by the following formula:

[0131] L tri (w′ k ,v′ k ,vk )=max(0,[d(w′ k ,v′ k )-d(w′ k ,v k )]+m)

[0132] Where k∈{a, o, c}, a represents the attribute, o represents the object, and c represents the combination; d(,) represents the Euclidean distance; m represents the edge value of the triple loss, which can be fixed as a constant of 0.5 in model training; w′ k v′ is the triple text feature of the second negative text label, which can be any of attributes, objects and combinations; k is the triplet image feature of the second negative sample image, which corresponds one-to-one to the triplet text feature of the second negative text label; v k is the triplet image feature of the second positive sample image; L tri (w′ k ,v′ k ,v k ) is the negative triplet loss data.

[0133] Furthermore, taking the second negative sample image as a reference, the negative attribute loss data L tri (w′ a ,v′ a ,v a ), negative object loss data L tri (w′ o ,v′ o ,v o ) and negative combination loss data L tri (w′ c ,v′ c ,v c ). Among them, each characteristic parameter can continue to refer to Figure 3 shown.

[0134] S720. Summing the negative attribute loss data, the negative object loss data, and the negative combination loss data, to obtain negative triplet loss data.

[0135] Similarly, the negative triplet loss data is calculated by the method shown in the following formula:

[0136] L mln =L tri (w′ a ,v′ a ,v a )+L tri (w′ o ,v′ o ,v o )+L tri (w′ c,v′ c ,v c )

[0137] Among them, L mln Represents the negative triplet loss data for model training based on the second negative sample image and the second positive sample image. Similarly, each feature parameter can continue to refer to Figure 3 shown.

[0138] In the above implementation, the idea of ​​mutual learning is used to calculate the triplet loss based on the features of the three levels to obtain negative attribute loss data, negative object loss data, and negative combination loss data. Finally, these three loss data are added together to obtain a comprehensive loss value. This comprehensive negative triplet loss data reflects the overall performance of the model in distinguishing negative samples from positive samples when the model is trained on the second negative sample image and the second positive sample image. Furthermore, the model parameters can be continuously optimized to reduce the negative triplet loss data, helping the model learn how to accurately identify and classify images, thereby continuously narrowing the distance between the image and the text to ultimately achieve the effect of clustering, and ultimately improving the performance of the model.

[0139] In some embodiments, the second positive sample image has a second positive text label, and the second loss data also includes negative classification loss data. Figure 8a , the negative classification loss data is determined by:

[0140] S810, performing matching and alignment according to the triplet image features of the second negative sample image, the triplet text features of the second negative text label, and the triplet text features of the second positive text label to determine the negative attribute classification loss, the negative object classification loss, and the negative combination classification loss.

[0141] Similarly, the negative classification loss data may refer to the loss calculated by the model for the classification error of the positive sample image in the classification task. For example, based on the idea of ​​mutual learning, in an extreme case, the "red pomegranate" is used as an example for explanation. If the selected second positive sample image is an image containing "red pomegranate", then the second positive text label of the second positive sample image is the phrase "red pomegranate".

[0142] Furthermore, the negative classification loss data can also be determined based on the following formula:

[0143]

[0144] Where k∈{a, o, c}, a represents the attribute, o represents the object, and c represents the combination; τ is the temperature parameter used to adjust the overall variance distribution of the data; w′ k v′ is the triple text feature of the second negative text label, which can be any of attributes, objects and combinations;k is the triplet image feature of the second negative sample image, which corresponds one-to-one to the triplet text feature of the second negative text label; w k is the triple text feature of the second positive text label; L′ k is the negative classification loss data.

[0145] Similarly, based on the above formula, we can get the negative attribute classification loss L a′ , negative object classification loss L o′ and negative combination classification loss L c′ .

[0146] S820. Perform weighted summation on the negative attribute classification loss, the negative object classification loss, and the negative combination classification loss to obtain negative classification loss data.

[0147] Similarly, since three mapping spaces are created, the negative attribute classification loss, the negative object classification loss, and the negative combination classification loss can be weighted summed to obtain the negative classification loss data. Specifically, it can be calculated by the method shown in the following formula:

[0148]

[0149] Among them, L sn Represents the negative classification loss data for model training based on the second negative sample image and the second positive sample image.

[0150] Furthermore, the total loss L of the final model Sim That is, training is performed on the first positive sample image and the first negative sample image to obtain the positive triplet loss data L mlp and positive classification loss data L sp , and the negative triplet loss data L obtained by training the second negative sample image and the second positive sample image mln and negative classification loss data L sn .

[0151] Based on the interactive learning method from positive and negative samples, after repeated iterative training, the loss function of all training images is minimized, and the end-to-end training of the target label prediction model is finally achieved. The target label prediction model after updating the parameters of the initial label prediction model based on the first loss data and the second loss data predicts labels for several images to be labeled in three different datasets: UT-zap50k, C-GQA and Mit-states. Figure 8b As shown, the green label is the correct target label, and it can be seen that the prediction accuracy is high.

[0152] In the above implementation, through the mechanism of mutual learning, for different training sets, the information of different positive and negative samples is focused on learning, and the loss caused by the other end of mutual learning is supplemented by triple loss and classification loss, which not only trains the network's ability to aggregate the distance between samples, but also trains the network's ability to classify sample similarities.

[0153] It should be understood that, although the various steps in the above flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear description in this article, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above flowchart may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0154] The embodiment of this specification also provides a device 900 for generating an image tag, such as Fig. 9 As shown, it includes: an image feature extraction module 910, a text feature acquisition module 920, a combined similar data determination module 930, a matching data determination module 940 and a target label generation module 950, wherein:

[0155] The image feature extraction module 910 is used to extract features from the image to be annotated input to obtain image attribute features, image object features and image combination features; wherein the image attribute features are used to describe the local attribute feature data in the image to be annotated, the image object features are used to describe the paradigm feature data of the object in the image to be annotated, and the image combination features are used to describe the interactive feature data after the attribute and object are combined in the image to be annotated.

[0156] The text feature acquisition module 920 is used to acquire a text feature set corresponding to the tag set to be matched; wherein the tag set to be matched includes multiple text tags, and the text feature set includes text attribute features, text object features and text combination features corresponding to each text tag.

[0157] The combined similarity data determination module 930 is used to determine, for each text tag, attribute similarity data between image attribute features and text attribute features, object similarity data between image object features and text object features, and combined similarity data between image combined features and text combined features.

[0158] The matching data determination module 940 is used to perform weighted processing on the attribute similarity data, the object similarity data and the combination similarity data to obtain matching data between each text label and the image to be annotated.

[0159] The target label generation module 950 is used to generate a target label for the image to be annotated based on the text label corresponding to the matching data that meets the preset screening condition.

[0160] In some embodiments, the image feature extraction module 910 is further used to extract overall features of the annotated image to obtain overall image features; perform feature mapping on the overall image features through an attribute mapping encoder to obtain image attribute features; perform feature mapping on the overall image features through an object mapping encoder to obtain image object features; perform feature mapping on the overall image features through a combination mapping encoder to obtain image combination features.

[0161] In some embodiments, the target label is obtained through a target label prediction model; the image label generation device 900 also includes a target label prediction model construction module, which is used to construct a target label prediction model, and the target label prediction model includes a parallel image processing branch and a text processing branch; the image processing branch includes an image encoder, and an attribute mapping encoder, an object mapping encoder and a combined mapping encoder connected to the image encoder; the text processing branch includes a text attribute encoder, an object attribute encoder, and a combined encoder connected to the text attribute encoder and the object attribute encoder.

[0162] In some embodiments, a target label prediction model construction module is used to obtain a target label prediction model in the following manner: building an initial label prediction model based on a combined zero-sample learning algorithm; determining a first positive sample image and a first negative sample image in a training data set; wherein the first negative sample image is a sample image other than the first positive sample image in the training data set; processing the first positive sample image and the first negative sample image through the initial label prediction model to obtain first loss data; determining a second negative sample image and a second positive sample image in the training data set; wherein the second positive sample image is a sample image other than the second negative sample image in the training data set; processing the second positive sample image and the second negative sample image through the initial label prediction model to obtain second loss data; and updating the parameters of the initial label prediction model based on the first loss data and the second loss data to obtain a target label prediction model.

[0163] In some embodiments, the first positive sample image has a first positive text label, and the first loss data includes positive triplet loss data; the image label generation device 900 also includes a loss data determination module, which is used to determine the positive triplet loss data in the following manner: matching and aligning the triplet image features of the first positive sample image, the triplet image features of the first negative sample image, and the triplet text features of the first positive text label to determine positive attribute loss data, positive object loss data, and positive combination loss data; wherein the triplet image features include image attribute features, image object features, and image combination features; the triplet text features include text attribute features, text object features, and text combination features; the positive attribute loss data, the positive object loss data, and the positive combination loss data are summed to obtain the positive triplet loss data.

[0164] In some embodiments, the first negative sample image has a first negative text label, and the first loss data also includes positive classification loss data; the loss data determination module is further used to determine the positive classification loss data in the following manner: matching and aligning the triplet image features of the first positive sample image, the triplet text features of the first negative text label, and the triplet text features of the first positive text label to determine the positive attribute classification loss, the positive object classification loss, and the positive combination classification loss; performing weighted summation on the positive attribute classification loss, the positive object classification loss, and the positive combination classification loss to obtain the positive classification loss data.

[0165] In some embodiments, the second negative sample image has a second negative text label, and the second loss data includes negative triplet loss data; the loss data determination module is further used to determine the negative triplet loss data in the following manner: matching and aligning the triplet image features of the second negative sample image, the triplet image features of the second positive sample image, and the triplet text features of the second negative text label to determine negative attribute loss data, negative object loss data, and negative combination loss data; wherein the triplet image features include image attribute features, image object features, and image combination features; the triplet text features include text attribute features, text object features, and text combination features; and the negative attribute loss data, the negative object loss data, and the negative combination loss data are summed to obtain the negative triplet loss data.

[0166] In some embodiments, the second positive sample image has a second positive text label, and the second loss data also includes negative classification loss data; the loss data determination module is further used to determine the negative classification loss data in the following manner: matching and aligning the triplet image features of the second negative sample image, the triplet text features of the second negative text label, and the triplet text features of the second positive text label to determine the negative attribute classification loss, the negative object classification loss, and the negative combination classification loss; performing weighted summation on the negative attribute classification loss, the negative object classification loss, and the negative combination classification loss to obtain the negative classification loss data.

[0167] For the specific definition of a device for generating an image label, please refer to the definition of a method for generating an image label above, which will not be repeated here. Each module in the above-mentioned device for generating an image label can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0168] The image label generation device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0169] The embodiment of the present application also provides a computer device, which may be a terminal, and its internal structure diagram may be as follows: Fig.10 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a method for generating an image tag is implemented.

[0170] Those skilled in the art will understand that the structure shown in FIG. 8 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0171] The present application also provides a computer-readable storage medium, and the above method according to the embodiment of the present application can be implemented in hardware, firmware, or implemented as a computer code that can be recorded in a storage medium, or downloaded through a network and originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory.

[0172] The embodiment of the present application provides a computer program product, which includes computer instructions, which are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method of any embodiment of the present application.

[0173] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0174] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0175] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0176] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. Since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0177] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

[0178] Although the embodiments of the present application are described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present application, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for generating an image label, characterized in that: The method comprises: Extract features from the image to be annotated to obtain image attribute features, image object features and image combination features; wherein the image attribute features are used to describe local attribute feature data in the image to be annotated, the image object features are used to describe the paradigm feature data of the object in the image to be annotated, and the image combination features are used to describe the interactive feature data after the attribute and object are combined in the image to be annotated; Obtaining a text feature set corresponding to a tag set to be matched; wherein the tag set to be matched includes a plurality of text tags, and the text feature set includes a text attribute feature, a text object feature, and a text combination feature corresponding to each text tag; For each of the text tags, determining attribute similarity data between the image attribute feature and the text attribute feature, object similarity data between the image object feature and the text object feature, and combination similarity data between the image combination feature and the text combination feature; Performing weighted processing on the attribute similarity data, the object similarity data and the combination similarity data to obtain matching data between each text label and the image to be annotated; Based on the text labels corresponding to the matching data that meet the preset screening conditions, a target label of the image to be annotated is generated.

2. The method according to claim 1, characterized in that: The feature extraction of the image to be annotated to obtain image attribute features, image object features and image combination features includes: Extracting overall features of the image to be annotated to obtain overall features of the image; Performing feature mapping on the overall features of the image by an attribute mapping encoder to obtain the attribute features of the image; Performing feature mapping on the overall features of the image by an object mapping encoder to obtain the image object features; The overall features of the image are feature mapped by a combined mapping encoder to obtain the combined features of the image.

3. The method according to claim 1, characterized in that The target label is obtained by a target label prediction model; The target label prediction model includes a parallel image processing branch and a text processing branch; The image processing branch includes an image encoder, and an attribute mapping encoder, an object mapping encoder and a combination mapping encoder connected to the image encoder; The text processing branch includes a text attribute encoder, an object attribute encoder, and a combined encoder connected to the text attribute encoder and the object attribute encoder.

4. The method according to claim 3, characterized in that The target label prediction model is obtained by: Build an initial label prediction model based on a combined zero-shot learning algorithm; Determine a first positive sample image and a first negative sample image in a training data set; wherein the first negative sample image is another sample image other than the first positive sample image in the training data set; Processing the first positive sample image and the first negative sample image by using the initial label prediction model to obtain first loss data; Determine a second negative sample image and a second positive sample image in the training data set; wherein the second positive sample image is other sample images in the training data set other than the second negative sample image; Processing the second positive sample image and the second negative sample image by using the initial label prediction model to obtain second loss data; The parameters of the initial label prediction model are updated based on the first loss data and the second loss data to obtain the target label prediction model.

5. The method according to claim 4, characterized in that The first positive sample image has a first positive text label, and the first loss data includes positive triplet loss data; the positive triplet loss data is determined in the following manner: Matching and aligning are performed according to the triplet image features of the first positive sample image, the triplet image features of the first negative sample image, and the triplet text features of the first positive text label to determine positive attribute loss data, positive object loss data, and positive combination loss data; wherein the triplet image features include image attribute features, image object features, and image combination features; and the triplet text features include text attribute features, text object features, and text combination features; The positive triplet loss data is obtained by summing the positive attribute loss data, the positive object loss data, and the positive combination loss data.

6. The method according to claim 5, characterized in that The first negative sample image has a first negative text label, and the first loss data further includes positive classification loss data; the positive classification loss data is determined in the following manner: Performing matching and alignment according to the triplet image features of the first positive sample image, the triplet text features of the first negative text label, and the triplet text features of the first positive text label to determine a positive attribute classification loss, a positive object classification loss, and a positive combination classification loss; The positive classification loss data is obtained by performing weighted summation on the positive attribute classification loss, the positive object classification loss, and the positive combination classification loss.

7. The method according to claim 4, characterized in that The second negative sample image has a second negative text label, and the second loss data includes negative triplet loss data; the negative triplet loss data is determined in the following manner: Matching and aligning are performed according to the triplet image features of the second negative sample image, the triplet image features of the second positive sample image, and the triplet text features of the second negative text label to determine negative attribute loss data, negative object loss data, and negative combination loss data; wherein the triplet image features include image attribute features, image object features, and image combination features; and the triplet text features include text attribute features, text object features, and text combination features; The negative attribute loss data, the negative object loss data, and the negative combination loss data are summed to obtain the negative triplet loss data.

8. The method according to claim 7, characterized in that The second positive sample image has a second positive text label, and the second loss data further includes negative classification loss data; the negative classification loss data is determined in the following manner: Matching and aligning the triplet image features of the second negative sample image, the triplet text features of the second negative text label, and the triplet text features of the second positive text label to determine a negative attribute classification loss, a negative object classification loss, and a negative combination classification loss; The negative classification loss data is obtained by performing weighted summation on the negative attribute classification loss, the negative object classification loss, and the negative combination classification loss.

9. A device for generating an image label, characterized in that: The device comprises: An image feature extraction module is used to extract features from the image to be annotated, and obtain image attribute features, image object features and image combination features; wherein the image attribute features are used to describe local attribute feature data in the image to be annotated, the image object features are used to describe the paradigm feature data of the object in the image to be annotated, and the image combination features are used to describe the interactive feature data after the attribute and object are combined in the image to be annotated; A text feature acquisition module, used to acquire a text feature set corresponding to a tag set to be matched; wherein the tag set to be matched includes a plurality of text tags, and the text feature set includes a text attribute feature, a text object feature and a text combination feature corresponding to each text tag; A combined similarity data determination module, used for determining, for each text tag, attribute similarity data between the image attribute feature and the text attribute feature, object similarity data between the image object feature and the text object feature, and combined similarity data between the image combined feature and the text combined feature; A matching data determination module, used for performing weighted processing on the attribute similarity data, the object similarity data and the combination similarity data to obtain matching data between each text label and the image to be annotated; The target label generation module is used to generate the target label of the image to be annotated based on the text label corresponding to the matching data that meets the preset screening conditions.

10. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method according to any one of claims 1 to 8 by executing the computer instructions.

Images