Multi-label labeling method and device, storage medium, electronic equipment and program product

Abstract

The present disclosure provides a multi-label labeling method and device, a storage medium, an electronic device and a program product. The method comprises: obtaining a multimedia object to be labeled and a preset label library; determining a candidate label set from the preset label library based on the semantic similarity between the multimedia object and each label in the preset label library by using a first multi-modal model; obtaining a content analysis result by content analysis of the multimedia object by using a second multi-modal model, and determining at least one target label from the candidate label set as the labeling result of the multimedia object based on the associated degree between the result and each label in the candidate label set. By introducing the candidate mechanism, the label recall rate of the multimedia object is improved, and by using the two-stage collaborative labeling method, the multi-label recall accuracy and accuracy of the multimedia object are improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision technology, and in particular to a multi-label annotation method and apparatus, storage medium, electronic device and program product. Background Technology

[0002] Multi-label annotation refers to assigning multiple semantically relevant labels to a single image. By assigning multiple semantically relevant labels to the same image, users can retrieve target images based on multiple different keywords, enabling application systems to respond more naturally to users and improving user experience. This technology is particularly important in many practical applications such as photo album management, intelligent retrieval, content moderation, and remote sensing detection.

[0003] In the field of image annotation, images are typically labeled using pre-trained generative models, which can directly generate labels related to the semantics of the image. Generative models are highly dependent on sample data, but most existing open datasets are single-label image-text pair datasets. This means that while trained generative models have single-label image classification capabilities, they are significantly lacking in multi-label annotation. Summary of the Invention

[0004] In view of this, the present disclosure provides a multi-tag labeling method and apparatus, storage medium, electronic device and program product, which can effectively improve the accuracy of multi-tag labeling of multimedia objects.

[0005] According to a first aspect of the present disclosure, a multi-label annotation method is provided, the method comprising: Obtain the multimedia object to be labeled and the preset tag library; Using a first multimodal model, a candidate tag set is determined from the preset tag library based on the semantic similarity between the multimedia object and each tag in the preset tag library; The content parsing result of the multimedia object is obtained by using the second multimodal model to perform content parsing. Based on the correlation between the result and each tag in the candidate tag set, at least one target tag is determined from the candidate tag set as the annotation result of the multimedia object.

[0006] In some possible implementations, the tags in the preset tag library are organized into a hierarchical structure with semantic association, wherein the semantic association includes at least one of the following: hierarchical relationship, synonymy relationship, and near-synonym relationship between tags.

[0007] In some possible implementations, the first multimodal model is trained based on the following steps: Construct a first training set, in which training samples include multimedia objects and their corresponding descriptive text; The first multimodal model is trained using the first training set. The descriptive text corresponding to the multimedia object is obtained through at least one of the following methods: For at least one tag in the preset tag library that is associated with the content of the multimedia object, semantic expansion is performed based on the semantic relationship between the tags to obtain descriptive text; Generative multimodal large language model is used to generate descriptive text based on the content of the multimedia object.

[0008] In some possible implementations, constructing the first training set includes at least one of the following: A data filtering network is used to perform quality screening on the generated multimedia object-text pairs; Perform vertical data augmentation operations to increase the proportion of multimedia object-text pairs associated with a preset vertical domain in the training set.

[0009] In some possible implementations, the method further includes, prior to constructing a training set containing multiple multimedia object-text pairs: Based on the category distribution of the preset tag library, a target sampling quantity is set for each category; Based on the target sampling quantity, sample multimedia objects of the corresponding category from the initial set of multimedia objects.

[0010] In some possible implementations, the second multimodal model is trained based on the following steps: Obtain a second training set, in which training samples include multimedia objects, corresponding candidate label sets, and real label sets; The multimedia object and its corresponding candidate tag set are input into the second multimodal model to generate a predicted tag set; Based on the preset reward rules, reward data is calculated according to the predicted label set and the real label set; Using the reward data, the parameters of the second multimodal model are updated through reinforcement learning.

[0011] In some possible implementations, the preset reward rule includes at least one of the following: Format reward, used to reward the output format of the predicted label set for conforming to a predetermined specification; Similarity coefficient reward is used to reward the degree of overlap between the predicted label set and the real label set; Duplicate penalty is used to penalize tags that are semantically repeated in the predicted tag set.

[0012] In some possible implementations, the reinforcement learning employs a strategy based on within-group relative advantage comparison for optimization, wherein the strategy employs at least one of the following: Remove the KL divergence penalty term from the objective function; A dynamic sampling strategy is adopted; The reward value or gradient is truncated with an upper bound.

[0013] In some possible implementations, the second multimodal model is trained to output in the following format: The content parsing and reasoning process of the multimedia object is output within the first set label; The predicted label set is output within the second set of labels, wherein each label in the predicted label set is separated by a delimiter.

[0014] In some possible implementations, the training samples in the second training set are derived from at least one of the following: Manually calibrated annotation data; Automatically labeled data generated based on the first multimodal model and the second multimodal model.

[0015] In some possible implementations, the method further includes: Obtain a third training set, wherein the training samples in the third training set include multimedia objects and corresponding labeled data, and the labeled data is labeled data generated based on the first multimodal model and the second multimodal model and manually calibrated; Based on the third training set, supervised training is performed on the edge model, which is then deployed to a terminal device to perform multi-label annotation tasks.

[0016] According to a second aspect of the present disclosure, a multi-label labeling apparatus is provided, the apparatus comprising: The first acquisition unit is used to acquire the multimedia object to be labeled and the preset tag library; The first determining unit is used to determine a candidate tag set from the preset tag library based on the semantic similarity between the multimedia object and each tag in the preset tag library using a first multimodal model. The second determining unit is used to obtain the content parsing result of the multimedia object by performing content parsing on the multimedia object using the second multimodal model, and to determine at least one target tag from the candidate tag set as the annotation result of the multimedia object based on the correlation degree between the result and each tag in the candidate tag set.

[0017] In some possible implementations, the tags in the preset tag library are organized into a hierarchical structure with semantic association, wherein the semantic association includes at least one of the following: hierarchical relationship, synonymy relationship, and near-synonym relationship between tags.

[0018] In some possible implementations, the device further includes: A construction unit is used to construct a first training set, wherein the training samples in the first training set include multimedia objects and corresponding descriptive text. The first training unit is used to train the first multimodal model using the first training set. The building unit is further configured to implement at least one of the following methods: For at least one tag in the preset tag library that is associated with the content of the multimedia object, semantic expansion is performed based on the semantic relationship between the tags to obtain descriptive text; Generative multimodal large language model is used to generate descriptive text based on the content of the multimedia object.

[0019] In some possible implementations, the building unit is configured to achieve at least one of the following: A data filtering network is used to perform quality screening on the generated multimedia object-text pairs; Perform vertical data augmentation operations to increase the proportion of multimedia object-text pairs associated with a preset vertical domain in the training set.

[0020] In some possible implementations, the device further includes: The setting unit is used to set the target sampling quantity for each category according to the category distribution of the preset tag library; The sampling unit is used to sample multimedia objects of the corresponding category from the initial multimedia object set according to the target sampling quantity.

[0021] In some possible implementations, the device further includes: The second acquisition unit is used to acquire the second training set, wherein the training samples in the second training set include multimedia objects, corresponding candidate label sets and real label sets; The generation unit is used to input the multimedia object and the corresponding candidate tag set into the second multimodal model to generate a predicted tag set; A reward calculation unit is used to calculate reward data based on the predicted label set and the real label set according to a preset reward rule; An update unit is used to update the parameters of the second multimodal model through reinforcement learning using the reward data.

[0022] In some possible implementations, the preset reward rule includes at least one of the following: Format reward, used to reward the output format of the predicted label set for conforming to a predetermined specification; Similarity coefficient reward is used to reward the degree of overlap between the predicted label set and the real label set; Duplicate penalty is used to penalize tags that are semantically repeated in the predicted tag set.

[0023] In some possible implementations, the reinforcement learning employs a strategy based on within-group relative advantage comparison for optimization, wherein the strategy employs at least one of the following: Remove the KL divergence penalty term from the objective function; A dynamic sampling strategy is adopted; The reward value or gradient is truncated with an upper bound.

[0024] In some possible implementations, the second multimodal model is trained to output in the following format: The content parsing and reasoning process of the multimedia object is output within the first set label; The predicted label set is output within the second set of labels, wherein each label in the predicted label set is separated by a delimiter.

[0025] In some possible implementations, the training samples in the second training set are derived from at least one of the following: Manually calibrated annotation data; Automatically labeled data generated based on the first multimodal model and the second multimodal model.

[0026] In some possible implementations, the device further includes: The third acquisition unit is used to acquire a third training set, wherein the training samples in the third training set include multimedia objects and corresponding labeled data, and the labeled data is labeled data generated based on the first multimodal model and the second multimodal model and manually calibrated. The second training unit is used to perform supervised training on the edge model based on the third training set. The edge model is used to be deployed to a terminal device to perform multi-label annotation tasks.

[0027] According to a third aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of any of the methods described in the first aspect above.

[0028] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: processor; Memory used to store processor-executable instructions; Wherein, the processor executes the steps of any of the methods described in the first aspect above.

[0029] According to a fifth aspect of the present disclosure, a computer program product is provided, including computer-readable code or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device executes the steps of any of the methods described in the first aspect above.

[0030] The technical solutions provided in this disclosure may have the following beneficial effects: By constructing a pre-defined tag library, the problem of low semantic alignment accuracy in generating multimedia object tags based on open-source tag systems in related technologies is solved, which leads to low tag generation accuracy. This improves the semantic alignment accuracy between multimedia objects and various tags in the pre-defined tag library, thereby enhancing the tag generation accuracy of multimedia objects.

[0031] This embodiment first determines a candidate tag set for the multimedia object based on the semantic similarity between the multimedia object and tags in a preset tag library, thereby improving the tag recall rate of the multimedia object by introducing a candidate mechanism. Then, it performs content parsing on the multimedia object to obtain the content parsing results, and determines at least one target tag from the candidate tag set as the multi-tag annotation result for the multimedia object based on the correlation between the parsing results and each tag in the candidate tag set. This two-stage collaborative annotation method improves the multi-tag recall accuracy and precision of the multimedia object.

[0032] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0033] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0034] Figure 1 This disclosure is a flowchart illustrating a multi-label annotation method according to an exemplary embodiment; Figure 2 This disclosure is a flowchart illustrating a method for preprocessing a training set according to an exemplary embodiment; Figure 3 This disclosure is a flowchart illustrating a data filtering method according to an exemplary embodiment; Figure 4 This disclosure is a flowchart illustrating a method for vertical domain data augmentation according to an exemplary embodiment; Figure 5 This disclosure is a flowchart illustrating a multi-label annotation method according to an exemplary embodiment; Figure 6 This disclosure is a block diagram illustrating a multi-label labeling device according to an exemplary embodiment; Figure 7 This is a schematic diagram of the structure of an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0035] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0036] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0037] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0038] In the field of image annotation, images are typically labeled using pre-trained generative models, which can directly generate labels related to the semantics of the image. Generative models are highly dependent on sample data, but most existing open datasets are single-label image-text pair datasets, and during training, the model focuses on global features to classify images based on these features. This results in trained generative models having single-label image classification capabilities, but significant shortcomings in multi-label annotation. For example, an image may contain multiple objects of different categories, such as a dog, a bicycle, and a person. However, if the dog occupies a relatively small portion of the image, the visual model, when recognizing global features, will average out the dog's visual features, failing to obtain the dog's label information. This leads to low accuracy in multi-label image annotation.

[0039] Based on this, the present disclosure provides a multi-label annotation method, which can be applied to computer devices. For details, please refer to [link to relevant documentation]. Figure 1 The method flowchart shown includes: S101, Obtain the multimedia object to be labeled and the preset tag library.

[0040] The multimedia object can include various data objects, such as one or more of text, audio, video, and images. The preset tag library contains a large number of tag names, covering multiple categories and fields. Multimedia objects can be labeled and categorized based on the tags in the preset tag library. In an optional embodiment, different types of multimedia objects can correspond to different preset tag libraries. For example, text data can correspond to a preset text tag library, audio data to a preset audio tag library, video data to a preset video tag library, and image data to a preset image tag library. The preset text tag library can be used for multi-tag classification of text data, such as news classification, emotion recognition, and medical document classification. The preset audio tag library can be used for classification of audio data, such as environmental sound recognition and music emotion classification. The preset video tag library can be used for classification of video data, such as video content classification, behavior recognition, and event detection. Furthermore, cross-modal multi-tag annotation can be performed on multimedia objects, such as multi-tag annotation combining text and images, or multi-tag annotation combining text and images.

[0041] In some embodiments, the tags in the preset tag library are organized into a hierarchical structure with semantic relationships, wherein the semantic relationships include at least one of the following: hierarchical relationship, synonymy relationship, and near-synonymity relationship between tags. For example, the preset tag library may include multiple hierarchical tags, such as fruit and apple; it may also include synonymous tags, such as elevator and lift; and it may also include near-synonymous tags, such as refrigerator and freezer. For ease of understanding, the preset tag library will be illustrated with examples below, such as the preset image tag library shown in Table 1: Table 1 Referring to Table 1, the image tag name "ice sculpture" can also include multiple parent nodes. There is a hierarchical relationship between the parent nodes and the image tag, such as the parent node being "humanities"—"outdoors"—"scene," etc. It should be noted that Table 1 is only an exemplary embodiment. In actual applications, the number of parent nodes and tag data in the preset tag library can be expanded according to needs.

[0042] Text data can be categorized using a pre-defined text tag library, which can also include a hierarchical structure, as shown in Table 2. Table 2 As shown in Table 2, the text tag name "Learning" can also include multiple synonym nodes, such as Education, Homework, etc.

[0043] Those skilled in the art should understand that tags in video tag libraries and audio tag libraries can also be organized into a hierarchical structure with semantic relationships.

[0044] By establishing a pre-defined tag library with a semantically related hierarchical structure, multiple different tags can be automatically matched for multimedia objects during annotation, thereby improving the accuracy of multi-tag annotation. After annotating multimedia objects with multiple tags using the pre-defined tag library, it can adapt to different retrieval granularities, improving the management efficiency of multimedia objects. Furthermore, building a pre-defined tag library solves the problem of low semantic alignment accuracy in generating multimedia object tags based on open-source tag systems, which leads to low tag generation accuracy. It improves the semantic alignment accuracy between multimedia objects and the tags in the pre-defined tag library, thus enhancing the accuracy of multimedia object tag generation.

[0045] S102, using the first multimodal model, a candidate tag set is determined from the preset tag library based on the semantic similarity between the multimedia object and each tag in the preset tag library.

[0046] The first multimodal model can be a neural network model based on contrastive learning, such as a visual language model pre-trained based on contrastive learning. Using this first multimodal model, the semantic similarity between a multimedia object and each tag in a pre-defined tag library can be calculated, and a candidate tag set can be determined from the pre-defined tag library based on the similarity values. Taking an image as an example, an image code can be obtained by encoding the image, and a tag code can be obtained by encoding each tag in the tag library. Then, the cosine similarity between the image code and the tag code is calculated to determine the semantic similarity between the image and each tag in the pre-defined tag library. Finally, based on the semantic similarity values ​​between the image and each tag, tags with higher similarity are selected as the candidate tag set.

[0047] When the multimedia object is of other types, a first multimodal model corresponding to that type can be used. For example, when the multimedia object is audio, the first multimodal model can use an audio-text pre-trained model based on contrastive learning; when the multimedia object is video, the first multimodal model can use a video-text pre-trained model based on contrastive learning; when the multimedia object is text, the first multimodal model can use a semantic retrieval model to recall relevant candidate tags from a text tag library.

[0048] In some embodiments, the process of determining a candidate tag set from a preset tag library includes: calculating the semantic similarity between a multimedia object and all tags in the preset tag library using a first multimodal model; and determining the K tags with the highest semantic similarity in the preset tag library as the candidate tag set based on the semantic similarity ranking, where K is a positive integer greater than or equal to 1.

[0049] For example, in a first multimodal model that is a visual language model pre-trained based on contrastive learning, where the multimedia object is an image, the first multimodal model includes an image encoder and a text encoder. The image encoder converts the image into image vectors, and the text encoder converts each tag in a pre-defined tag library into its corresponding text vector. Then, the cosine similarity between the image vector and each tag is calculated to determine the semantic similarity between the image and all tags in the pre-defined tag library. After calculating the similarity between all tags and the image, the semantic similarities are sorted, and based on the sorting results, the K tags with the highest semantic similarity in the pre-defined tag library are selected as candidate tags in the candidate tag set.

[0050] For example, a preset tag library includes M types of tags. The semantic similarity between an image and each tag is calculated using a first multimodal model. The N tags with the highest semantic similarity in the preset tag library can be selected as tags in a candidate tag set. Here, the value of M is much larger than N; for example, N is less than 1% of M. For instance, M=3000, N=15. Those skilled in the art should understand that the number of tags in the tag library and the number of tags in the selected candidate tag set can be specifically determined based on actual circumstances, provided that the above numerical relationships are satisfied. This disclosure does not intend to impose any limitations in this regard.

[0051] S103, using the second multimodal model to perform content parsing on the multimedia object to obtain the content parsing result, and based on the correlation between the result and each tag in the candidate tag set, determining at least one target tag from the candidate tag set as the annotation result of the multimedia object.

[0052] The second multimodal model is a large model capable of parsing different multimedia objects across modalities; for example, it could be a multimodal large language model. After obtaining the candidate tag set, the multimedia objects can be further parsed to obtain content parsing results. For example, global and local semantic features can be extracted from images to obtain visual features, such as visual encoding (token) sequences. Then, based on the content parsing results, the degree of association between the multimedia object and each candidate tag in the candidate tag set can be determined. For example, the attention weight distribution and semantic consistency of the image visual features and each candidate tag can be calculated to determine the degree of association between the image and each candidate tag. After determining the degree of association, at least one target tag can be determined from the candidate tag set as the multi-tag annotation result for the multimedia object. In the process of determining the target tag, a degree of association threshold can be set, and the degree of association can be compared based on the threshold to determine the target tag from the candidate tag set.

[0053] In one possible embodiment, the second multimodal model can be a visual language model, and the multimedia object can be an image. The visual language model includes a visual encoder, a language model, and a connectivity model. The visual encoder extracts features from the image and converts the input image into a digital vector. The language model processes candidate label information from the candidate label set to obtain label vectors and performs logical reasoning. In one embodiment, the language model in the second multimodal model can be a pre-trained natural language model. The connectivity module projects the image's digital vectors into the same semantic space as the label vectors, enabling the language model to understand the image and determine the target label for the image within the candidate label set.

[0054] When inputting the image and candidate label set into the second multimodal model, a prompt can be input simultaneously. For example, a fixed prompt like "Select all labels highly relevant to the image content from the following candidate label set" can be input into the second multimodal model. The second multimodal model first extracts features from the image using a visual encoder to obtain a numerical vector of the image. Then, it converts the prompt and candidate label set into text vectors and uses an attention mechanism to match each candidate label in the candidate label set with the numerical vector of the image, determining the attention weight distribution and semantic consistency score between the candidate labels and the image to perform content parsing and determine the degree of association between the image and each label in the candidate label set. Finally, the second multimodal model determines the candidate labels whose association with the image and each label in the candidate label set is greater than a preset association threshold as the target labels.

[0055] When the multimedia object is of another type, a second multimodal model corresponding to that type can be used. For example, when the multimedia object is audio, the second multimodal model can use an audio language model; when the multimedia object is video, the second multimodal model can use a video language model; and when the multimedia object is text, the second multimodal model can use a text language model.

[0056] This embodiment first determines a candidate tag set for an image based on the semantic similarity between the multimedia object and tags in a preset tag library, thereby improving the tag recall rate of the multimedia object by introducing a candidate mechanism. Then, it performs content parsing on the multimedia object to obtain the content parsing results, and determines at least one target tag from the candidate tag set as the multi-tag annotation result for the multimedia object based on the correlation between the parsing results and each tag in the candidate tag set. This two-stage collaborative annotation method improves the multi-tag recall accuracy and precision of the multimedia object.

[0057] To improve the recall rate of the first multimodal model, this embodiment of the disclosure performs corresponding optimized training on the first multimodal model. The trained first multimodal model shows significant improvement compared to the open-source model. According to the comparison table shown in Table 3, it can be determined that constructing the first training set in the above manner and training the first multimodal model based on the first training set can significantly improve the recall rate of the first multimodal model, as detailed below: Table 3 In this context, Top100 refers to the top 100 tags in the recall test set with the highest semantic similarity to multimedia objects, Top30 refers to the top 30 tags in the recall test set with the highest semantic similarity to multimedia objects, and Top1 refers to the tag with the highest semantic similarity to multimedia objects in the recall test set. The recall rate is calculated as the ratio of the number of intersections between candidate words and true tags to the total number of true tags. As shown in Table 3, training the first multimodal model using the first training set can significantly improve the recall rate of the trained first multimodal model for image tags.

[0058] The training process of the first multimodal model will be described below. The first multimodal model is trained based on the following steps: Construct a first training set, in which training samples include multimedia objects and their corresponding descriptive text; The first multimodal model is trained using the first training set. The descriptive text corresponding to the multimedia object is obtained through at least one of the following methods: For at least one tag in the preset tag library that is associated with the content of the multimedia object, semantic expansion is performed based on the semantic relationship between the tags to obtain descriptive text; Generative multimodal large language model is used to generate descriptive text based on the content of the multimedia object.

[0059] The training samples in the first training set include multiple multimedia objects and their corresponding descriptive text. For example, the first training set includes multiple images and their corresponding descriptive text. Using the first training set, a first multimodal model can be trained based on contrastive learning. During training, the first multimodal model calculates the similarity between each multimedia object and its corresponding descriptive text, as well as the similarity between the descriptive texts of other multimedia objects in the first training set. The descriptive texts of each multimedia object are used as positive samples, and the descriptive texts of other multimedia objects are used as negative samples. By updating the parameters of the first multimodal model, the semantic similarity between each multimedia object and the positive samples is maximized, while the semantic similarity between the multimedia objects and the negative samples is reduced, thereby optimizing the first multimodal model.

[0060] The descriptive text of multimedia objects in the first training set can be obtained in several ways. One feasible approach is to first determine at least one tag associated with the content of the multimedia object in a pre-defined tag library, and then semantically expand the tags based on the semantic relationships between them to obtain the corresponding descriptive text. For example, at least one tag associated with the content of the multimedia object can be determined in the pre-defined tag library using an initial first multimodal model. Then, semantic expansion can be performed on the determined at least one tag. This semantic expansion can be based on publicly available knowledge graphs, semantic networks, or dictionaries, searching for approximate expressions of the determined at least one tag to obtain the descriptive text. Alternatively, semantic interpolation can be performed on the determined at least one tag to obtain the descriptive text. For example, the determined at least one tag can be mapped to a semantic vector, and then its neighborhood can be sampled and interpolated in the vector space. The sampled new vector can then be decoded into natural language to achieve semantic expansion and obtain the descriptive text.

[0061] Another feasible way to obtain descriptive text for multimedia objects is to use a generative multimodal large language model to generate descriptive text based on the content of the multimedia object. The following explanation uses an image as an example to illustrate the process of the generative multimodal large language model outputting descriptive text. For details, please refer to [link to relevant documentation]. Figure 2 The flowchart describes a process where an open-source image set is first acquired. Then, based on image metadata and a pre-defined cue word library, a generative multimodal language model is used to rewrite the text of the images in the set, fuse metadata, or perform zero-shot and few-shot tasks to generate descriptive text for each image. Different processing operations and tasks correspond to different cue words in the pre-defined cue word library. These cue words constrain the inference logic of the generative multimodal language model, causing it to output the corresponding descriptive text. For example, the pre-defined cue word "generate descriptive text for this image" causes the model to generate descriptive text for the image, achieving a zero-shot task—meaning the model outputs descriptive text for the images in the set without providing samples. Alternatively, the pre-defined cue word "Please refer to the following example to generate descriptive text for the image, for example, the image includes a kitten and a puppy" causes the model to generate descriptive text for the image, achieving a few-shot task—meaning the model is provided with a small number of example samples and generates descriptive text based on these examples.

[0062] Expanding the descriptive text of multimedia objects in the first training set using the methods described above can significantly improve the recall rate of the first multimodal model, as shown in Table 4: Table 4 As shown in Table 4, after constructing the first training set by semantically expanding the descriptive text of multimedia objects and training the first multimodal model, the first multimodal model can achieve a recall rate of 99.61% in the top 30 and a recall rate of 88.02% in the top 1. Furthermore, it achieves a reduction in the number of candidate labels, requiring only 17 candidate labels to be determined.

[0063] In some possible embodiments, when constructing the first training set, the training set can also be processed by filtering and vertical domain enhancement operations. The processing method of the training set can be at least one of the following: using a data filtering network to perform quality screening on the generated multimedia object-text pairs; performing vertical domain data enhancement operations to increase the proportion of multimedia object-text pairs related to a preset vertical domain in the training set.

[0064] In one embodiment, a data filtering network can be used to filter the generated multimedia object-text pairs to obtain high-quality multimedia object-text pairs. Taking images as an example, the data filtering process can be found in [reference needed]. Figure 3 As shown in the diagram, for example, the data filtering network in the first multimodal model is first trained with a high-quality image-text pair dataset to enable the data filtering network to identify noise. Then, the large-scale original image-text pair dataset is input into the trained data filtering network to obtain a large-scale, high-quality image-text dataset.

[0065] In one embodiment, vertical domain data augmentation operations can be performed on the training set using techniques such as resampling. This vertical domain augmentation increases the proportion of multimedia object-text pairs related to a preset vertical domain in the training set. The preset multimedia object-text pairs related to the preset vertical domain can be determined based on the user's actual search frequency. For example, if a user frequently searches for food in their phone's photo album, then food can be used as the preset vertical domain. For example, the implementation method of vertical domain augmentation can be found in [link to documentation]. Figure 4 The flowchart illustrates how, for both open-source single-label and multi-label image datasets, semantic generalization and resampling of the label data in the open-source datasets can be performed based on a thesaurus to generate an album-themed dataset. Then, a pre-trained generative model uses a prompt lexicon to annotate images related to a predetermined vertical domain, obtaining vertical domain text annotations. Finally, the album-themed dataset and the vertical domain text annotations are combined to obtain the first training set.

[0066] When constructing the first training set, the quality of the first training set can be improved and the amount of vertical data in the first training set can be increased by performing quality screening and vertical domain enhancement on the generated multimedia object-text pairs. This will improve the training effect of the first multimodal model, enabling the first multimodal model to accurately identify vertical domain data and improve the annotation effect of vertical domain multimedia objects.

[0067] In some optional embodiments, before constructing a training set containing multiple multimedia object-text pairs, sampling can be performed in the initial multimedia object set using a balanced sampling method. For example, based on the category distribution of the preset tag library, a target sampling quantity is set for each category; based on the target sampling quantity, multimedia objects of the corresponding category are sampled from the initial multimedia object set. For example, the tags in the preset tag library are first classified using a classifier to obtain the category distribution of tags in each category of the preset tag library, such as the quantity distribution of each tag category. Based on the category distribution, the target sampling quantity for each category can be determined; for example, a positive correlation can be established between the sampling quantity of each category and the quantity of each tag category in the preset tag library. Then, based on the target sampling quantity, multimedia objects of the corresponding category are sampled from the initial multimedia object set so that the quantity of multimedia objects is correlated with the quantity of each tag category in the preset tag library. In an optional embodiment, a category balanced sampler can also be used to sample multimedia objects of the corresponding category from the initial multimedia object set to balance the quantity of multimedia objects of different categories. By performing balanced sampling on the initial set of multimedia objects, the number of multimedia objects of each category in the training set can be balanced to obtain a training set with balanced numbers. Then, the training set is processed to obtain the first training set. In turn, when training the first multimodal model based on the first training set, the long-tail effect of the first multimodal model can be reduced and the prediction accuracy of the first multimodal model can be improved.

[0068] The training of the second multimodal large model can be based on reinforcement learning. By combining verifiable reinforcement learning with the multimodal large model, the training effect of the second multimodal large model can be significantly improved, resulting in a significant advantage in performance on different test sets. Compared with traditional supervised training and the original model, the second multimodal large model trained in this embodiment of the disclosure significantly improves the annotation accuracy of multimedia objects, as detailed in Table 5. Table 5 In this embodiment, the second multimodal large model trained based on reinforcement learning achieves an accuracy of 82.23% for labeling multimedia objects on the open-source test set and 85.88% for labeling on the vertical domain test set, which is significantly better than other models.

[0069] The training process of the second multimodal large model will be described below. The second multimodal model is trained based on the following steps: obtaining a second training set; inputting the multimedia object and the corresponding candidate label set into the second multimodal model to generate a predicted label set; calculating reward data based on the predicted label set and the real label set according to a preset reward rule; and updating the parameters of the second multimodal model through reinforcement learning using the reward data.

[0070] The training samples in the second training set include multiple multimedia objects, as well as a candidate label set and a ground truth label set for each multimedia object. The candidate label set for each multimedia object is determined by the first multimodal large model in a preset label library.

[0071] After obtaining the second training set, the multimedia objects in the second training set and the corresponding candidate label set for each multimedia object can be input into the second multimodal model. The second multimodal model performs inference, determining a preset label set for each multimedia object from the candidate label set. The predicted label set includes at least one label used to label the category of the multimedia object. After obtaining the predicted label set of the multimedia objects, the reward data for the predicted label set can be determined according to a preset reward rule. Finally, a loss function is constructed based on the reward data, and the second multimodal model is trained using reinforcement learning by optimizing the loss function, updating the parameters of the second multimodal model.

[0072] In some possible embodiments, the preset reward rules include at least one of the following: Format reward, used to reward the output format of the predicted label set for conforming to a predetermined specification; Similarity coefficient reward is used to reward the degree of overlap between the predicted label set and the real label set; Duplicate penalty is used to penalize tags that are semantically repeated in the predicted tag set.

[0073] The format reward is used to reward the output format of the predicted label set for conforming to a predetermined specification. The format reward data can be determined based on the degree of overlap between the predicted label set and the sample candidate label set. For example, it can be checked whether the predicted label in the predicted label set belongs to the candidate label set. If the predicted label belongs to the candidate label set, the format reward data is determined to be a positive score. If the predicted label does not belong to the candidate label set, the format reward data is determined to be a negative score.

[0074] The similarity coefficient reward is used to reward the degree of overlap between the predicted label set and the real label set. The reward data can be determined based on the semantic similarity between the predicted and real label sets. For example, the Jaccard similarity between the predicted and real label sets is calculated. If the predicted and real label sets are the same, the similarity reward is positive; if they are different, the similarity reward is zero.

[0075] The duplication penalty is used to penalize tags that are semantically duplicated in the predicted tag set. The penalty data can be determined based on the degree of duplication between predicted tags in the predicted tag set. For example, if there are duplicate tags in the predicted tag set, the penalty data is a negative score; if there are no duplicate tags, the penalty data is zero.

[0076] In some alternative embodiments, reinforcement learning of the initial multimodal model can be optimized using a strategy based on intra-group relative advantage comparison. The KL divergence (relative entropy) can be removed from the objective function of the intra-group relative advantage comparison strategy to simplify the objective function, improve the efficiency of model training and the robustness of model hyperparameters. An asymmetric pruning mechanism can also be added to truncate the reward value or gradient with an upper bound to prevent reward explosion.

[0077] In one embodiment, when the second multimodal model performs inference calculations on multimedia objects and determines the predicted label set for the multimedia objects, the sampling strategy can be adjusted first through a dynamic-sample mechanism. For example, different offset ranges of sampling positions can be set for different image features to avoid generating the same label for all images. Then, the reward data for each predicted label is calculated. The calculation method for the reward data can be found in the embodiment described in the previous paragraph and will not be repeated here. After determining the reward data, it can be further processed. For example, the reward of each training sample image is subtracted from the average reward to convert the reward of each training sample image into a relative advantage. Then, an asymmetric pruning strategy is used to prune the upper bound of the relative advantage. That is, when pruning the value function or advantage function, different thresholds are set for the upper and lower bounds. Finally, a loss function is constructed based on the relative advantage after upper bound pruning. When constructing the loss function, the KL divergence in the objective function of the strategy for comparing relative advantages within groups is removed. By optimizing the loss function, the parameters of the initial multimodal model are updated to obtain the second multimodal model.

[0078] The reinforcement learning strategies described above can significantly improve the training performance of the second multimodal model. The training performance of the second multimodal model under different training strategies can be seen in Table 6. Table 6 As shown in Table 6, when the preset reward rules include format reward, similarity coefficient reward, and duplication penalty, the KL divergence penalty term in the objective function is removed, and a dynamic sampling strategy and upper bound truncation of the reward value or gradient are adopted. Under this comprehensive training strategy, the second multimodal model has the highest recall rate.

[0079] In some embodiments, the output format of the second multimodal model can be constrained during training. For example, the content parsing and reasoning process of the multimedia object can be output within a first preset label; the predicted label set can be output within a second preset label, wherein the labels in the predicted label set are separated by delimiters. Here, the first preset label refers to the first label specified for use in the output format of the second multimodal model, and the second preset label refers to the second label specified for use in the output format of the second multimodal model.

[0080] In one embodiment, the output format of the second multimodal model is as follows: question=Select all tags that are highly relevant to the image content from the following list of candidate tags: [tag1, tag2, tag3,..., tag17] exist <think>< / think> The tags record the reasoning process in detail, including visual element recognition, scene analysis, and tag selection logic; <answer>< / answer> The selected tags are listed within the tags, separated by commas (,).

[0081] response= <think> Image descriptions generated by large models and the thought process behind the answers.< / think> <answer> tag1, tag2, ...< / answer> in, <think>< / think> The tag is the first set tag. <answer>< / answer> This is the second set of labels. The first set of labels outputs the content parsing and reasoning process of the second multimodal model for multimedia formations, such as visual element recognition, scene parsing, and label selection logic. The second set of labels outputs the predicted label set determined by the second multimodal model, such as tag1, tag2, ...

[0082] In some embodiments, after determining at least one target label from the candidate label set as the multi-label annotation result based on the correlation between the image and each label in the candidate label set, the predicted label set output by the second multimodal model can be manually calibrated to obtain manually calibrated annotation data. After obtaining the manually calibrated annotation data, the manually calibrated annotation data is added to the second training set to update the second training set. For example, the manually calibrated data is used as a prompt input during the training of the second multimodal model to constrain the inference logic of the second multimodal model. By updating the manually calibrated annotation data in the second training set, a human-machine collaborative annotation loop can be achieved, improving the accuracy of the second training set and thus improving the training effect of the second multimodal model.

[0083] To reduce the computational resources required by the annotation model and enable it to run smoothly on terminal devices, a lightweight edge model can be trained based on the annotation data generated by the first and second multimodal models. This edge model is then deployed to the terminal device, allowing it to perform multi-label annotation on multimedia objects. Training the edge model significantly improves its annotation accuracy for multimedia objects. In one embodiment, the annotation accuracy of the trained edge model on different test sets is shown in Table 7. Table 7 In this embodiment, the edge model trained in this disclosure achieved a top-1 recall rate of 93.63% in the multi-label test set, which is 8.49% higher than the online version, 6.76% higher than the original version, and 16.6% higher than the open-source version, demonstrating significant progress.

[0084] In one embodiment, the training process of the edge model includes: acquiring a third training set, wherein the training samples in the third training set include multimedia objects and corresponding labeled data, wherein the labeled data is labeled data generated based on the first multimodal model and the second multimodal model and manually calibrated; and performing supervised training on the edge model based on the third training set, wherein the edge model is used to deploy to a terminal device to perform a multi-label labeling task.

[0085] Based on the labeled data of multimedia objects generated by the first and second multimodal models, as well as manually calibrated labeled data, a large-scale third training set is constructed. This third training set is then used to perform supervised training on the lightweight edge model, enabling it to perform multi-label annotation of multimedia objects. After training, the edge model can be deployed on terminal devices such as mobile phones, tablets, and wearable devices to perform multi-label annotation tasks. For example, the edge model can be used to annotate images stored on electronic devices with multiple labels, allowing users to manage their photo albums by searching for tags. By deploying the lightweight edge model on terminal devices, the accuracy of labeling multimedia objects on the terminal devices can be ensured while saving computing resources and improving the efficiency of users managing multimedia data on their devices.

[0086] Taking images as an example, the flowchart of the multi-label annotation method provided in this disclosure can be found here. Figure 5 The flowchart shown includes: S501, obtain open-source image datasets; S502 performs class-balanced sampling on open-source datasets; S503, obtain a manually selected and supplemented vertical image library; S504, based on the dataset after class-balanced sampling and the vertical image library, constructs the original image pool; S505, the images in the original image pool are labeled using the first multimodal model to determine the candidate label library; S506, The candidate label library for each image in the original image pool is filtered through the second multimodal model to determine the target label of the image; S507, Obtain manual verification information for the target label; S508, based on the target labels of images and manual verification information, constructs a large-scale multi-label dataset; S509 is a multi-label model trained on the edge based on a large-scale multi-label dataset.

[0087] This embodiment employs a two-stage collaborative annotation framework. First, it determines a candidate tag set for the image based on the semantic similarity between the image and tags in a preset tag library. Then, it performs content parsing on the image to obtain the content parsing result. Based on the degree of association between the parsing result and each tag in the candidate tag set, it determines at least one target tag from the candidate tag set as the multi-tag annotation result. This accurately improves the recall rate of image tags, thereby enhancing the accuracy of multi-tag annotation for images.

[0088] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should know that this disclosure is not limited to the described order of actions, because according to this disclosure, some steps may be performed in other orders or simultaneously.

[0089] Secondly, those skilled in the art should also know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by this disclosure.

[0090] Corresponding to the aforementioned application function implementation method embodiments, this disclosure also provides embodiments of application function implementation apparatus and corresponding terminals.

[0091] Reference Figure 6 A block diagram of a multi-label labeling device according to an exemplary embodiment shows that the device may include: The first acquisition unit 601 is used to acquire the multimedia object to be labeled and the preset tag library; The first determining unit 602 is used to determine a candidate tag set from the preset tag library based on the semantic similarity between the multimedia object and each tag in the preset tag library using a first multimodal model; The second determining unit 603 is used to obtain the content parsing result of the multimedia object by performing content parsing on the multimedia object using the second multimodal model, and to determine at least one target tag from the candidate tag set as the annotation result of the multimedia object based on the correlation degree between the result and each tag in the candidate tag set.

[0092] In some possible implementations, the tags in the preset tag library are organized into a hierarchical structure with semantic association, wherein the semantic association includes at least one of the following: hierarchical relationship, synonymy relationship, and near-synonym relationship between tags.

[0093] In some possible implementations, the device further includes: A construction unit is used to construct a first training set, wherein the training samples in the first training set include multimedia objects and corresponding descriptive text. The first training unit is used to train the first multimodal model using the first training set. The building unit is further configured to implement at least one of the following methods: For at least one tag in the preset tag library that is associated with the content of the multimedia object, semantic expansion is performed based on the semantic relationship between the tags to obtain descriptive text; Generative multimodal large language model is used to generate descriptive text based on the content of the multimedia object.

[0094] In some possible implementations, the building unit is configured to achieve at least one of the following: A data filtering network is used to perform quality screening on the generated multimedia object-text pairs; Perform vertical data augmentation operations to increase the proportion of multimedia object-text pairs associated with a preset vertical domain in the training set.

[0095] In some possible implementations, the device further includes: The setting unit is used to set the target sampling quantity for each category according to the category distribution of the preset tag library; The sampling unit is used to sample multimedia objects of the corresponding category from the initial multimedia object set according to the target sampling quantity.

[0096] In some possible implementations, the device further includes: The second acquisition unit is used to acquire the second training set, wherein the training samples in the second training set include multimedia objects, corresponding candidate label sets and real label sets; The generation unit is used to input the multimedia object and the corresponding candidate tag set into the second multimodal model to generate a predicted tag set; A reward calculation unit is used to calculate reward data based on the predicted label set and the real label set according to a preset reward rule; An update unit is used to update the parameters of the second multimodal model through reinforcement learning using the reward data.

[0097] In some possible implementations, the preset reward rule includes at least one of the following: Format reward, used to reward the output format of the predicted label set for conforming to a predetermined specification; Similarity coefficient reward is used to reward the degree of overlap between the predicted label set and the real label set; Duplicate penalty is used to penalize tags that are semantically repeated in the predicted tag set.

[0098] In some possible implementations, the reinforcement learning employs a strategy based on within-group relative advantage comparison for optimization, wherein the strategy employs at least one of the following: Remove the KL divergence penalty term from the objective function; A dynamic sampling strategy is adopted; The reward value or gradient is truncated with an upper bound.

[0099] In some possible implementations, the second multimodal model is trained to output in the following format: The content parsing and reasoning process of the multimedia object is output within the first set label; The predicted label set is output within the second set of labels, wherein each label in the predicted label set is separated by a delimiter.

[0100] In some possible implementations, the training samples in the second training set are derived from at least one of the following: Manually calibrated annotation data; Automatically labeled data generated based on the first multimodal model and the second multimodal model.

[0101] In some possible implementations, the device further includes: The third acquisition unit is used to acquire a third training set, wherein the training samples in the third training set include multimedia objects and corresponding labeled data, and the labeled data is labeled data generated based on the first multimodal model and the second multimodal model and manually calibrated. The second training unit is used to perform supervised training on the edge model based on the third training set. The edge model is used to be deployed to a terminal device to perform multi-label annotation tasks.

[0102] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0103] A third aspect of this disclosure provides a computer program product including a computer program / instructions that, when executed by a processor, implement the method described in the first aspect.

[0104] For the device embodiments and computer program product embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. Furthermore, the device embodiments described above are merely illustrative; the modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, i.e., they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without any inventive effort.

[0105] Fourthly, embodiments of the multi-label labeling device provided in this disclosure can be applied to electronic devices. Please refer to [link to relevant documentation]. Figure 7 The illustration exemplifies a hardware schematic of an electronic device. For example, device 700 could be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0106] Device 700 may include one or more of the following components: processing component 702, memory 704, power supply component 706, multimedia component 708, audio component 710, input / output (I / O) interface 712, sensor component 714, and communication component 716.

[0107] Processing component 702 typically controls the overall operation of device 700, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 702 may include one or more processors 720 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 702 may include one or more modules to facilitate interaction between processing component 702 and other components. For example, processing component 702 may include a multimedia module to facilitate interaction between multimedia component 708 and processing component 702.

[0108] Memory 704 is configured to store various types of data to support the operation of device 700. Examples of this data include instructions for any application or method operating on device 700, contact data, phonebook data, messages, pictures, videos, etc. Memory 704 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0109] Power supply component 706 provides power to various components of device 700. Power supply component 706 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 700.

[0110] Multimedia component 708 includes a screen that provides an output interface between the device 700 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, swipe, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 708 includes a front-facing camera and / or a rear-facing camera. When the device 700 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0111] Audio component 710 is configured to output and / or input audio signals. For example, audio component 710 includes a microphone (MIC) configured to receive external audio signals when device 700 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 704 or transmitted via communication component 716. In some embodiments, audio component 710 also includes a speaker for outputting audio signals.

[0112] Input / output (I / O) interface 712 provides an interface between processing component 702 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0113] Sensor assembly 714 includes one or more sensors for providing state assessments of various aspects of device 700. For example, sensor assembly 714 may detect the on / off state of device 700, the relative positioning of components such as the display and keypad of device 700, changes in the position of device 700 or a component of device 700, the presence or absence of user contact with device 700, the orientation or acceleration / deceleration of device 700, and temperature changes of device 700. Sensor assembly 714 may also include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 714 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 714 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0114] Communication component 716 is configured to facilitate wired or wireless communication between device 700 and other devices. Device 700 can access wireless networks based on communication standards, such as WiFi, 2G or 3G, 4G or 5G, or combinations thereof. In one exemplary embodiment, communication component 716 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 716 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0115] In an exemplary embodiment, device 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to execute the instruction determination method of the aforementioned electronic device.

[0116] Fifthly, in exemplary embodiments, this disclosure also provides a non-transitory computer-readable storage medium including instructions, such as a memory 704 including instructions, which can be executed by a processor 720 of device 700 to complete the instruction determination method of the electronic device. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0117] The foregoing has described specific embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0118] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention applied herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not claimed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0119] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

[0120] The above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A multi-label annotation method, characterized in that, The method includes: Obtain the multimedia object to be labeled and the preset tag library; Using a first multimodal model, a candidate tag set is determined from the preset tag library based on the semantic similarity between the multimedia object and each tag in the preset tag library; The content parsing result of the multimedia object is obtained by using the second multimodal model to perform content parsing. Based on the correlation between the result and each tag in the candidate tag set, at least one target tag is determined from the candidate tag set as the annotation result of the multimedia object.

2. The method according to claim 1, characterized in that, The tags in the preset tag library are organized into a hierarchical structure with semantic association, wherein the semantic association includes at least one of the following: hierarchical relationship, synonymy relationship, and near-synonym relationship between tags.

3. The method according to claim 1, characterized in that, The first multimodal model was trained based on the following steps: Construct a first training set, in which training samples include multimedia objects and their corresponding descriptive text; The first multimodal model is trained using the first training set. The descriptive text corresponding to the multimedia object is obtained through at least one of the following methods: For at least one tag in the preset tag library that is associated with the content of the multimedia object, semantic expansion is performed based on the semantic relationship between the tags to obtain descriptive text; Generative multimodal large language model is used to generate descriptive text based on the content of the multimedia object.

4. The method according to claim 3, characterized in that, The construction of the first training set includes at least one of the following: A data filtering network is used to perform quality screening on the generated multimedia object-text pairs; Perform vertical data augmentation operations to increase the proportion of multimedia object-text pairs associated with a preset vertical domain in the training set.

5. The method according to claim 3, characterized in that, Before constructing a training set containing multiple multimedia object-text pairs, the method further includes: Based on the category distribution of the preset tag library, a target sampling quantity is set for each category; Based on the target sampling quantity, sample multimedia objects of the corresponding category from the initial set of multimedia objects.

6. The method according to claim 1, characterized in that, The second multimodal model is trained based on the following steps: Obtain a second training set, in which training samples include multimedia objects, corresponding candidate label sets, and real label sets; The multimedia object and its corresponding candidate tag set are input into the second multimodal model to generate a predicted tag set; Based on the preset reward rules, reward data is calculated according to the predicted label set and the real label set; Using the reward data, the parameters of the second multimodal model are updated through reinforcement learning.

7. The method according to claim 6, characterized in that, The preset reward rules include at least one of the following: Format reward, used to reward the output format of the predicted label set for conforming to a predetermined specification; Similarity coefficient reward is used to reward the degree of overlap between the predicted label set and the real label set; Duplicate penalty is used to penalize tags that are semantically repeated in the predicted tag set.

8. The method according to claim 6, characterized in that, The reinforcement learning employs a strategy based on within-group relative advantage comparison for optimization, wherein the strategy employs at least one of the following: Remove the KL divergence penalty term from the objective function; A dynamic sampling strategy is adopted; The reward value or gradient is truncated with an upper bound.

9. The method according to claim 6, characterized in that, The second multimodal model is trained to output in the following format: The content parsing and reasoning process of the multimedia object is output within the first set label; The predicted label set is output within the second set of labels, wherein each label in the predicted label set is separated by a delimiter.

10. The method according to claim 6, characterized in that, The training samples in the second training set are derived from at least one of the following: Manually calibrated annotation data; Automatically labeled data generated based on the first multimodal model and the second multimodal model.

11. The method according to claim 1, characterized in that, The method further includes: Obtain a third training set, wherein the training samples in the third training set include multimedia objects and corresponding labeled data, and the labeled data is labeled data generated based on the first multimodal model and the second multimodal model and manually calibrated; Based on the third training set, supervised training is performed on the edge model, which is then deployed to a terminal device to perform multi-label annotation tasks.

12. A multi-label labeling device, characterized in that, The device includes: The first acquisition unit is used to acquire the multimedia object to be labeled and the preset tag library; The first determining unit is used to determine a candidate tag set from the preset tag library based on the semantic similarity between the multimedia object and each tag in the preset tag library using a first multimodal model. The second determining unit is used to obtain the content parsing result of the multimedia object by performing content parsing on the multimedia object using the second multimodal model, and to determine at least one target tag from the candidate tag set as the annotation result of the multimedia object based on the correlation degree between the result and each tag in the candidate tag set.

13. The apparatus according to claim 12, characterized in that, The tags in the preset tag library are organized into a hierarchical structure with semantic association, wherein the semantic association includes at least one of the following: hierarchical relationship, synonymy relationship, and near-synonym relationship between tags.

14. The apparatus according to claim 12, characterized in that, The device further includes: A construction unit is used to construct a first training set, wherein the training samples in the first training set include multimedia objects and corresponding descriptive text. The first training unit is used to train the first multimodal model using the first training set. The building unit is further configured to implement at least one of the following methods: For at least one tag in the preset tag library that is associated with the content of the multimedia object, semantic expansion is performed based on the semantic relationship between the tags to obtain descriptive text; Generative multimodal large language model is used to generate descriptive text based on the content of the multimedia object.

15. The apparatus according to claim 12, characterized in that, The device further includes: The second acquisition unit is used to acquire the second training set, wherein the training samples in the second training set include multimedia objects, corresponding candidate label sets and real label sets; The generation unit is used to input the multimedia object and the corresponding candidate tag set into the second multimodal model to generate a predicted tag set; A reward calculation unit is used to calculate reward data based on the predicted label set and the real label set according to a preset reward rule; An update unit is used to update the parameters of the second multimodal model through reinforcement learning using the reward data.

16. The apparatus according to claim 15, characterized in that, The preset reward rules include at least one of the following: Format reward, used to reward the output format of the predicted label set for conforming to a predetermined specification; Similarity coefficient reward is used to reward the degree of overlap between the predicted label set and the real label set; Duplicate penalty is used to penalize tags that are semantically repeated in the predicted tag set.

17. The apparatus according to claim 12, characterized in that, The device further includes: The third acquisition unit is used to acquire a third training set, wherein the training samples in the third training set include multimedia objects and corresponding labeled data, and the labeled data is labeled data generated based on the first multimodal model and the second multimodal model and manually calibrated. The second training unit is used to perform supervised training on the edge model based on the third training set. The edge model is used to be deployed to a terminal device to perform multi-label annotation tasks.

18. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-11.

19. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The execution of the method comprises the steps of any one of claims 1-11.

20. A computer program product, comprising a computer program, characterized in that, Includes computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the steps of the method according to any one of claims 1-11.

Images