Material labeling method, model training method, multimedia content labeling method and related products

Abstract

The invention discloses a material labeling method, a model training method, a multimedia content labeling method and a related product. The method comprises the following steps: acquiring a target material and n preset labels; the preset label comprises category information of the material, and n is an integer greater than 1; searching the n preset tags by utilizing target material embedding of the target material to obtain m candidate tags matched with the target material embedding; m is a positive integer smaller than n; according to the m candidate tags and the target material, target cue words are generated, and the target cue words are used for guiding a language model to determine tags with category information matched with the target material from the m candidate tags; and under the guidance of the target cue word, determining a label with category information matched with the target material in the m candidate labels by utilizing a language model, and taking the label as a target label.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a material labeling method, a model training method, a multimedia content labeling method, and related products. Background Art

[0002] By marking the materials based on their categories, we can obtain the labels of the materials, and use the labels as a basis to determine whether the object is interested in the materials, and then recommend the materials that the object is interested in. Therefore, how to determine the labels of materials is of great significance. Summary of the invention

[0003] The present application provides a material labeling method, a model training method, a multimedia content labeling method and related products, wherein the related products include a material labeling device, a model training device, a multimedia content labeling device, an electronic device, a computer-readable storage medium and a computer program product.

[0004] In a first aspect, a material labeling method is provided, the method comprising:

[0005] Obtain a target material and n preset labels; the preset labels include category information of the material, and n is an integer greater than 1;

[0006] Using the target material embedding of the target material, the n preset tags are searched to obtain m candidate tags that match the target material embedding; wherein m is a positive integer less than n;

[0007] Generate a target prompt word according to the m candidate tags and the target material, wherein the target prompt word is used to guide the language model to determine a tag having the category information matching the target material from the m candidate tags;

[0008] Under the guidance of the target prompt word, the language model is used to determine a label having the category information matching the target material among the m candidate labels as the target label.

[0009] In combination with any embodiment of the present application, the target material embedding of the target material is used to search the n preset tags to obtain m candidate tags that match the target material embedding, including:

[0010] Extracting the embeddings of the n preset tags to obtain n preset tag embeddings, wherein the preset tags correspond one to one to the preset tag embeddings;

[0011] Determine the similarity between the embedding of the target material and the embedding of the n preset tags respectively, and obtain n reference similarities, wherein the reference similarities correspond to the preset tags one by one;

[0012] Determine m similarities with the greatest similarity from the n reference similarities as candidate similarities;

[0013] The m preset tags corresponding to the m candidate similarities are used as the m candidate tags.

[0014] In combination with any embodiment of the present application, the target prompt word includes a target example, the target example is an example of determining a second example label from m first example labels, the second example label is a label having category information matching a first reference material, the first reference material is different from the target material, and the m first example labels are obtained by searching the n preset labels using a first embedding of the first reference material to obtain m labels matching the first embedding;

[0015] The step of using the language model to determine a label having the category information matching the target material among the m candidate labels as a target label includes:

[0016] Under the prompt of the target example, the language model is used to determine a label having the category information matching the target material among the m candidate labels as the target label.

[0017] In combination with any embodiment of the present application, the target prompt word includes a target step, the target step is a step of determining a fourth example tag from m third example tags, the fourth example tag is a tag having category information matching a second reference material, the second reference material is different from the target material, and the m third example tags are obtained by searching the n preset tags using a second embedding of the second reference material to obtain m tags matching the second embedding;

[0018] The step of using the language model to determine a label having the category information matching the target material among the m candidate labels as a target label includes:

[0019] Under the prompt of the target step, the language model is used to determine a label having the category information matching the target material among the m candidate labels as the target label.

[0020] In combination with any embodiment of the present application, the respectively determining the similarity between the embedding of the target material and the embedding of the n preset tags to obtain n reference similarities includes:

[0021] Divide the n preset tag embeddings into k tag groups, where k is a positive integer less than n; there are greater than or equal to m preset tag embeddings in any of the tag groups;

[0022] Based on the k groups of label groups, determining the average value of the preset label embedding in any of the label groups to obtain k mean embeddings;

[0023] Determine the maximum similarity between the target material embedding and the k mean embeddings to obtain a first similarity;

[0024] Determine the tag group corresponding to the first similarity to obtain a target tag group;

[0025] Determine the similarity between any of the preset tags embedded in the target tag group and the target material embedded, and obtain i second similarities; i represents the number of preset tags embedded in the target tag group; i is a positive integer greater than or equal to m;

[0026] The determining m candidate similarities with the greatest similarity from the n reference similarities comprises:

[0027] The m candidate similarities having the greatest similarities are determined from the i second similarities.

[0028] In a second aspect, a model training method is provided, the method comprising:

[0029] Acquire a training prompt word and a first label; the training prompt word is used to guide the model to be trained to determine a label with category information matching the training material from m training labels; the first label is the label of the training material; the m training labels are obtained based on m retrieval labels, the m retrieval labels are determined by retrieving n preset labels, and the m labels with category information matching the embedding of the training material are determined from the n preset labels, the m training labels include the first label, and the training material embedding is the embedding of the training material; the preset label includes the category information of the material;

[0030] Under the guidance of the training prompt word, the to-be-trained model is used to determine a label having the category information matching the training material among the m training labels as a second label;

[0031] determining a target loss according to a target difference between the second label and the first label, wherein the target loss is positively correlated with the target difference;

[0032] According to the target loss, the parameters in the model to be trained are adjusted to obtain a language model.

[0033] In combination with any embodiment of the present application, when the n preset tags are retrieved using the training material embedding, and the m retrieval tags that match the training material embedding do not include the first tag, the m training tags are obtained by replacing any one of the m first retrieval tags with the first tag.

[0034] In combination with any embodiment of the present application, when the n preset tags are retrieved using the training material embedding to obtain the m retrieval tags that match the training material embedding, and the m retrieval tags include the first tag, the m training tags are obtained by replacing any tag among the m retrieval tags except the first tag with a random tag, and the random tag is any tag among the n preset tags except the m retrieval tags.

[0035] In combination with any embodiment of the present application, the model to be trained includes a decomposition branch, the decomposition branch is used to decompose the embedding of the prompt word into two sub-embeddings, the embedding obtained by performing an inner product between the two sub-embeddings represents the embedding of the prompt word, and the difference between the parameter amount of the embedding of the prompt word and the sum of the parameter amounts of the two sub-embeddings is greater than a reference threshold, and the reference threshold is a positive integer;

[0036] The method of determining, under the guidance of the training prompt word, using the to-be-trained model to determine a label having the category information matching the training material among the m training labels as a second label comprises:

[0037] Extracting embeddings in the training prompt words to obtain reference embeddings;

[0038] Extracting the reference embedding using the decomposition branch to obtain a first sub-embedding and a second sub-embedding;

[0039] Obtaining an inner product embedding by performing an inner product operation on the first sub-embedding and the second sub-embedding;

[0040] Based on the inner product embedding, using the to-be-trained model, determine a label among the m training labels that has category information matching the target material as the second label;

[0041] The step of adjusting the parameters in the model to be trained according to the target loss to obtain a language model includes:

[0042] According to the target loss, the parameters in the decomposition branch are adjusted to obtain a language model.

[0043] In a third aspect, a multimedia content annotation method is provided, the method comprising:

[0044] Acquire target multimedia content and n preset tags, wherein the preset tags include category information of the multimedia content, and n is an integer greater than 1;

[0045] Using the target multimedia content embedding of the target multimedia content, searching the n preset tags to obtain m candidate multimedia content tags that match the target multimedia content embedding; wherein m is a positive integer less than n;

[0046] Generate a multimedia content prompt word according to the m candidate multimedia content tags and the target multimedia content, wherein the multimedia content prompt word is used to guide the language model to determine a tag having category information matching the target multimedia content from the m candidate multimedia content tags;

[0047] Under the guidance of the multimedia content prompt word, the language model is used to determine a tag having the category information matching the target multimedia content among the m candidate multimedia content tags as the target multimedia content tag.

[0048] In a fourth aspect, a material labeling device is provided, the material labeling device comprising:

[0049] An acquisition unit, used to acquire a target material and n preset tags; the preset tags include category information of the material, and n is an integer greater than 1;

[0050] A retrieval unit, configured to retrieve the n preset tags using the target material embedding of the target material, and obtain m candidate tags that match the target material embedding; wherein m is a positive integer less than n;

[0051] A generating unit, configured to generate a target prompt word according to the m candidate tags and the target material, wherein the target prompt word is used to guide the language model to determine a tag having the category information matching the target material from the m candidate tags;

[0052] A determination unit is used to determine, under the guidance of the target prompt word, using the language model to determine a label having the category information matching the target material among the m candidate labels as a target label.

[0053] In conjunction with any embodiment of the present application, the above-mentioned retrieval unit is specifically used to:

[0054] Extracting the embeddings of the n preset tags to obtain n preset tag embeddings, wherein the preset tags correspond one to one to the preset tag embeddings;

[0055] Determine the similarity between the embedding of the target material and the embedding of the n preset tags respectively, and obtain n reference similarities, wherein the reference similarities correspond to the preset tags one by one;

[0056] Determine m similarities with the greatest similarity from the n reference similarities as candidate similarities;

[0057] The m preset tags corresponding to the m candidate similarities are used as the m candidate tags.

[0058] In combination with any embodiment of the present application, the target prompt word includes a target example, the target example is an example of determining a second example tag from m first example tags, the second example tag is a tag having category information matching a first reference material, the first reference material is different from the target material, and the m first example tags are obtained by searching the n preset tags using a first embedding of the first reference material to obtain m tags matching the first embedding;

[0059] The above-mentioned determination unit is specifically used for:

[0060] Under the prompt of the target example, the language model is used to determine a label having the category information matching the target material among the m candidate labels as the target label.

[0061] In combination with any embodiment of the present application, the target prompt word includes a target step, the target step is a step of determining a fourth example tag from m third example tags, the fourth example tag is a tag having category information matching a second reference material, the second reference material is different from the target material, and the m third example tags are obtained by searching the n preset tags using a second embedding of the second reference material to obtain m tags matching the second embedding;

[0062] The above-mentioned determination unit is specifically used for:

[0063] Under the prompt of the target step, the language model is used to determine a label having the category information matching the target material among the m candidate labels as the target label.

[0064] In conjunction with any embodiment of the present application, the above-mentioned retrieval unit is specifically used to:

[0065] Divide the n preset tag embeddings into k tag groups, where k is a positive integer less than n; there are greater than or equal to m preset tag embeddings in any of the tag groups;

[0066] Based on the k groups of label groups, determining the average value of the preset label embedding in any of the label groups to obtain k mean embeddings;

[0067] Determine the maximum similarity between the target material embedding and the k mean embeddings to obtain a first similarity;

[0068] Determine the tag group corresponding to the first similarity to obtain a target tag group;

[0069] Determine the similarity between any of the preset tags embedded in the target tag group and the target material embedded, and obtain i second similarities; i represents the number of preset tags embedded in the target tag group; i is a positive integer greater than or equal to m;

[0070] The m candidate similarities having the greatest similarities are determined from the i second similarities.

[0071] In a fifth aspect, a model training device is provided, the model training device comprising:

[0072] An acquisition unit is used to acquire a training prompt word and a first label; the training prompt word is used to guide the model to be trained to determine a label with category information matching the training material from m training labels; the first label is the label of the training material; the m training labels are obtained based on m retrieval labels, the m retrieval labels are m labels with category information matching the embedding of the training material determined from the n preset labels by retrieving n preset labels, the m training labels include the first label, and the training material embedding is the embedding of the training material; the preset label includes the category information of the material;

[0073] a determination unit, configured to determine, under the guidance of the training prompt word, using the to-be-trained model, a label having the category information matching the training material among the m training labels as a second label;

[0074] a loss calculation unit, configured to determine a target loss according to a target difference between the second label and the first label; the target loss is positively correlated with the target difference;

[0075] An adjustment unit is used to adjust the parameters in the model to be trained according to the target loss to obtain a language model.

[0076] In combination with any embodiment of the present application, when the n preset tags are retrieved using the training material embedding, and the m retrieval tags that match the training material embedding do not include the first tag, the m training tags are obtained by replacing any one of the m first retrieval tags with the first tag.

[0077] In combination with any embodiment of the present application, when the n preset tags are retrieved using the training material embedding to obtain the m retrieval tags that match the training material embedding, and the m retrieval tags include the first tag, the m training tags are obtained by replacing any tag among the m retrieval tags except the first tag with a random tag, and the random tag is any tag among the n preset tags except the m retrieval tags.

[0078] In combination with any embodiment of the present application, the model to be trained includes a decomposition branch, the decomposition branch is used to decompose the embedding of the prompt word into two sub-embeddings, and the embedding obtained by performing an inner product between the two sub-embeddings represents the embedding of the prompt word, and the amount of parameters in the two sub-embeddings is less than the amount of parameters of the embedding of the prompt word by a parameter threshold;

[0079] The above-mentioned determination unit is specifically used for:

[0080] Extracting embeddings in the training prompt words to obtain reference embeddings;

[0081] Extracting the reference embedding using the decomposition branch to obtain a first sub-embedding and a second sub-embedding;

[0082] Obtaining an inner product embedding by performing an inner product operation on the first sub-embedding and the second sub-embedding;

[0083] Based on the inner product embedding, using the to-be-trained model, determine a label among the m training labels that has category information matching the target material as the second label;

[0084] The above adjustment unit is specifically used for:

[0085] According to the target loss, the parameters in the decomposition branch are adjusted to obtain a language model.

[0086] In a sixth aspect, a multimedia content annotation device is provided, the multimedia content annotation device comprising:

[0087] An acquisition unit, configured to acquire target multimedia content and n preset tags, wherein the preset tags include category information of the multimedia content, and n is an integer greater than 1;

[0088] A retrieval unit, configured to retrieve the n preset tags using the target multimedia content embedding of the target multimedia content, and obtain m candidate multimedia content tags that match the target multimedia content embedding; wherein m is a positive integer less than n;

[0089] A generating unit, configured to generate a multimedia content prompt word according to the m candidate multimedia content tags and the target multimedia content, wherein the multimedia content prompt word is used to guide a language model to determine a tag having category information matching the target multimedia content from among the m candidate multimedia content tags;

[0090] A determination unit is used to determine, under the guidance of the multimedia content prompt word, using the language model to determine a tag having the category information matching the target multimedia content among the m candidate multimedia content tags as the target multimedia content tag.

[0091] In the seventh aspect, an electronic device is provided, comprising: a processor and a memory, the memory being used to store computer program code, the computer program code comprising computer instructions, and when the processor executes the computer instructions, the electronic device executes the first aspect and any embodiment thereof, or executes the second aspect and any embodiment thereof, or executes the third aspect.

[0092] In an eighth aspect, another electronic device is provided, comprising: a processor, a sending device, an input device, an output device and a memory, wherein the memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device executes the first aspect and any embodiment thereof, or executes the second aspect and any embodiment thereof, or executes the third aspect.

[0093] In the ninth aspect, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a processor, the processor is caused to execute the first aspect and any embodiment thereof, or the second aspect and any embodiment thereof, or the third aspect.

[0094] In the tenth aspect, a computer program product is provided, which includes a computer program or instructions, and when the computer program or instructions are run on a computer, the computer is caused to execute the above-mentioned first aspect and any embodiment thereof, or execute the above-mentioned second aspect and any embodiment thereof, or execute the above-mentioned third aspect.

[0095] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application.

[0096] In the embodiment of the present application, when the target material and n preset labels are obtained, the material labeling device uses the target material embedding of the target material to retrieve the n preset labels, and obtains m candidate labels with embedding matching the target material, wherein the preset labels include the category information of the material. By removing the labels with a low degree of embedding matching with the target material from the n preset labels, a small number of m labels with embedding matching with the target material are obtained. When the m candidate labels are determined, the material labeling device generates a target prompt word according to the m candidate labels and the target material. After the target prompt word is generated, the material labeling device uses the language model to determine the label with category information matching the target material from the m candidate labels as the target label under the guidance of the target prompt word. By using the language model to determine the target label, the labor cost is reduced and the speed of determining the target label is accelerated. In addition, before the material labeling device uses the language model to determine the target label, the number of labels of the candidate labels is reduced by searching, which can reduce the interference of the labels whose category information does not match the target material when determining the target label to the material labeling device, and improve the matching degree between the target label determined by the material labeling device from the m candidate labels and the target material. BRIEF DESCRIPTION OF THE DRAWINGS

[0097] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the drawings required for use in the embodiments of the present application or the background technology will be described below.

[0098] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0099] Figure 1 A schematic diagram of a material labeling method provided in an embodiment of the present application;

[0100] Figure 2 A schematic diagram of using a language model to generate a target label for a target material provided in an embodiment of the present application;

[0101] Figure 3 A flowchart of a model training method provided in an embodiment of the present application;

[0102] Figure 4 A schematic diagram of a model training provided in an embodiment of the present application;

[0103] Figure 5 A flowchart of a multimedia content annotation method provided in an embodiment of the present application;

[0104] Figure 6 A schematic diagram of generating a target multimedia content tag of a target multimedia content by using a language model provided in an embodiment of the present application;

[0105] Figure 7 A schematic diagram of extracting target multimedia content using a pre-trained embedding model provided in an embodiment of the present application;

[0106] Figure 8 A schematic diagram of the structure of a material labeling device provided in an embodiment of the present application;

[0107] Fig. 9 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0108] Fig.10 A schematic diagram of the structure of a multimedia content annotation device provided in an embodiment of the present application;

[0109] Fig.11 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0110] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0111] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0112] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0113] The execution subject of the embodiment of the present application is a material labeling device, wherein the material labeling device can be any electronic device that can execute the technical solution disclosed in the method embodiment of the present application. Optionally, the material labeling device can be one of the following: a computer, a server.

[0114] It should be understood that the method embodiment of the present application can also be implemented by a processor executing a computer program code. The following describes the embodiment of the present application in conjunction with the drawings in the embodiment of the present application. Figure 1 , Figure 1 A schematic diagram of a process flow of a material labeling method provided in an embodiment of the present application.

[0115] 101. Obtain target materials and n preset labels.

[0116] In the embodiment of the present application, the target material is the material published by the target object on the material consumption platform. The target material can be any material. Optionally, the target material includes one or more of the following: multimedia content, text, video, picture, audio, commodity. For example, the target material can be a material including text. For another example, the target material can be a material including a picture. For another example, the target material can also be a material including a video, a picture and text.

[0117] In a possible implementation scenario, when the target material includes multimedia content, the multimedia content is the content published by the target object on the material consumption platform, wherein the multimedia content includes one or more of the following: text, video, picture, audio, and commodity. For example, the multimedia content is content including text. For another example, the multimedia content is content including text and pictures. For another example, the multimedia content is content including audio, text, and video.

[0118] In the embodiment of the present application, the preset tag includes the category information of the material. The preset tag is used to classify the information carried by the material. Specifically, because different materials carry different information, the category information of different materials may be different, that is, the preset tags they belong to may be different. The number of preset tags is n, where n is a positive integer greater than 1.

[0119] In a possible implementation, there are two preset tags, including tag a1 and tag a2. The category information of the material in tag a1 is beauty makeup, and the category information of the material in tag a2 is game. 80% of the information carried by material b1 is related to makeup brushes. Since makeup brushes are a tool in beauty makeup, the category information of the information carried by material b1 is beauty makeup, and the preset tag to which material b1 belongs is tag a1. 90% of the information carried by material b2 is related to racing games. Since racing games are a type of game among games, the category information of the information carried by material b2 is games, and the preset tag to which material b2 belongs is tag a2.

[0120] In another possible implementation, there are two preset tags, including tag a1 and tag a2. The category information of the material in tag a1 is beauty and makeup, and the category information of the material in tag a2 is games. 80% of the information carried by material b1 is related to makeup brushes. Since makeup brushes are a tool in beauty and makeup, the category information of the information carried by material b1 is beauty and makeup, and the preset tag to which material b1 belongs is tag a1. 70% of the information carried by material b3 is related to foundation. Since foundation is another tool in beauty and makeup, the category information of the information carried by material b3 is also beauty and makeup, and the preset tag to which material b3 belongs is also tag a1.

[0121] In an implementation method of obtaining a target material and n preset labels, a material labeling device receives the target material input by an input component. The input component includes at least one of the following: a keyboard, a mouse, a touch screen, a touch pad, and an audio input device.

[0122] In another implementation method of obtaining the target material and n preset labels, the material labeling device receives the target material sent by the terminal. The terminal includes at least one of the following: a mobile phone, a computer, a tablet computer, and a server.

[0123] It should be understood that in the embodiment of the present application, the steps of the material labeling device acquiring the target material and n preset labels can be executed separately or simultaneously, and the present application does not limit this.

[0124] 102. Using the target material embedding of the target material, the n preset tags are searched to obtain m candidate tags that match the target material embedding.

[0125] In an embodiment of the present application, embedding characterizes the features of the data, wherein embedding includes the above-mentioned target material embedding and the preset label embedding, training material embedding, reference embedding and target embedding, and target multimedia content embedding described below. The target material embedding characterizes the features of the target material, wherein the features of the target material include the features of the data of the target data modality in the target material, and the target data modality is used to characterize the data type of the data in the target material. Optionally, the target data modality includes any one or more data modalities of image type and text type. For example, in the case where the target data modality includes an image type, the target material embedding characterizes the features of the data of the image type in the target material. For another example, in the case where the target data modality includes a text type, the target material embedding characterizes the features of the data of the text type in the target material. For another example, in the case where the target data modality includes an image type and a text type, the target material embedding characterizes the features of the data of the image type and the text type in the target material.

[0126] The data of the image type in the target material includes the images in the target material. When the target material includes a video, the cover image, key frames, and non-key frames of the video are also data of the image type.

[0127] The text-type data in the target material includes the publisher identification (identity document, ID) of the target material, the introduction text of the publisher of the target material, the title text of the target material, the body text of the target material, and any one or more of the following data: the introduction text of the publisher includes the self-description of the publisher. The title text includes the text in the target material that summarizes the content of the target material so that the object can quickly understand the content of the target material. The body text includes the text in the target material that explains the theme of the content of the target material. The comment text is the text that comments on the target material.

[0128] In the case where the target material embedding includes features of image type and text type data, the features of the data of the target data modality in the target material embedding can be made richer. In the process of the material labeling device using the target material embedding to retrieve n preset tags, the richer the features of the data of the target data modality in the target material embedding, the higher the probability that the target material embedding matches the features of the target material. Since the m candidate tags are determined by the target material embedding, the probability that the target material embedding matches the features of the target material can be increased, and the probability that the determined m candidate tags are tags that match the target material can be increased.

[0129] In an implementation method for determining the embedding of a target material, k different modalities of data in the target material are identified. Then, k embedding models that can respectively extract the embeddings of the k modalities of data are used to extract the embeddings of the k different modalities of data in the target material, and k data embeddings are obtained. The k data embeddings are then concatenated to obtain the embedding of the target material. For example, in the case where there are text-type and image-type data in the target material, the text embedding model is used to extract the embedding of the text-type data in the target material to obtain the text data embedding. The image embedding model is then used to extract the embedding of the image-type data in the target material to obtain the image data embedding. Finally, the text data embedding and the image data embedding are concatenated to obtain the target material embedding.

[0130] In another implementation of determining the embedding of the target material, an embedding of the target material is extracted using an embedding model that can extract embeddings of data containing multiple modalities to obtain the embedding of the target material.

[0131] In the embodiment of the present application, the candidate tags are m tags retrieved from n preset tags, where m is a positive integer greater than 1 and m is less than n. The material labeling device retrieves m tags that match the target material embedding from the n preset tags to obtain m candidate tags.

[0132] In a possible implementation, the material labeling device determines the embedding of n preset tags to obtain n preset tag embeddings, and then respectively determines the similarities between the n preset tag embeddings and the target material embeddings to obtain n reference similarities. From the n reference similarities, m similarities with the greatest similarity are determined as m candidate similarities. Finally, the m preset tags that have obtained the m candidate similarities are determined as m candidate tags.

[0133] In another possible implementation, the material labeling device determines the embedding of n preset labels to obtain n preset label embeddings, and then respectively determines the distances between the n preset label embeddings and the target material embedding to obtain n reference distances, and selects m distances with the smallest distances from the n reference distances as m candidate distances. Finally, the m preset labels that determine the m candidate distances are used as m candidate labels.

[0134] 103. Generate a target prompt word according to the m candidate tags and the target material.

[0135] In the embodiment of the present application, a prompt is used to guide the language model to complete the task, wherein the prompt includes the target prompt mentioned above and the multimedia content prompt below. The target prompt is used to guide the language model to determine a label with category information matching the target material from m candidate labels, wherein the target prompt includes m candidate labels and the target material.

[0136] In an embodiment of the present application, the language model has the ability of natural language processing (NLP). In a possible implementation, the language model is obtained by training with text training data, wherein the text training data includes one or more of masked text and non-masked text, and the model with NLP ability obtained by training with text training data also includes the model to be trained below. Masked text is text including a mask. When the text training data includes masked text, the language model is trained using the text training data, so that the language model can predict the content obscured by the mask in the masked text to obtain a first prediction result, and then the loss of the language model is determined based on the difference between the first prediction result and the ground truth (GT) of the mask. The parameters of the language model are updated based on the loss until the loss of the language model converges, and the training of the language model can be completed. Non-masked text is text that does not include a mask. When the text training data does not include non-masked text, using the text training data to train the language model can enable the language model to perform natural language processing tasks using the non-masked text, thereby improving the natural language processing capabilities of the language model. The natural language processing tasks include: summarizing the text, determining the order of sentences, and determining the answer to a question based on the text.

[0137] Optionally, the above language model and the model to be trained are large language models (LLM), where LLM refers to a deep learning model trained with a large amount of text data, which can generate natural language text or understand the meaning of language text. LLM can handle a variety of natural language tasks, such as text classification, question answering, dialogue, generating text summaries, and generating code.

[0138] 104. Under the guidance of the target prompt word, the language model is used to determine a label having the category information matching the target material among the m candidate labels as the target label.

[0139] In the embodiment of the present application, the target label is the label of the target material determined by the material labeling device from the m candidate labels. Specifically, the material labeling device uses a language model to determine the m category information of the m candidate labels and the information carried by the target material, and then determines the label with the category information matching the information carried by the target material from the m candidate labels as the target label.

[0140] In a possible implementation scenario, the target material is a material that does not contain any label. When the target material does not contain any label, the material labeling device determines a target label for the target material.

[0141] Generally speaking, the main method for labeling materials is to manually determine the information carried by the target material. After determining the information carried by the target material, determine the label with category information matching the information carried by the target material from the m candidate labels as the target label. Specifically, in addition to the category information of the material, any candidate label also includes the matching requirements that the information carried by the target material should meet when the information carried by the target material matches the category information. In the process of determining the target label, determine the label whose information carried by the target material meets the matching requirements from the m candidate labels as the target label. For example, the candidate labels include labels a3 and a4. The category information in label a3 is skin care products, and the matching requirements in label a3 are substances or mixtures that play a protective role. The category of label a4 is cosmetics, and the matching requirements in label a4 are substances or mixtures that play a beautifying and modifying role. When it is determined that the information carried by the target material b4 is related to the gel that protects the skin, it is determined that the information carried by the target material b4 meets the matching requirements of label a3, and label a3 is determined as the target label of the target material b4.

[0142] In order to reduce labor costs, it is impossible to manually determine the label with category information matching the information carried by the target material from the m candidate labels. Since the language model has the ability of natural language processing, the language model can be used to determine the label with category information matching the information carried by the target material from the m candidate labels as the target label.

[0143] In a possible implementation, the material labeling device determines the semantics of the information carried by the target material and the semantics of the category information in the m candidate tags using a language model. The material labeling device determines the semantic similarity between the semantics of the information carried by the target material and the semantics of the category information in the m candidate tags, and uses the tag to which the category information corresponding to the semantic similarity greater than the similarity threshold belongs as the target tag.

[0144] In another possible implementation, the material labeling device uses a language model to determine the keywords of the information carried by the target material, and determines the keywords that match the requirements in the m candidate tags. The material labeling device determines the keyword matching degree between the keywords of the information carried by the target material and the keywords that match the requirements in the m candidate tags, and uses the tag to which the keyword corresponding to the keyword matching degree greater than the matching threshold belongs as the target tag.

[0145] In another possible implementation, the material labeling device uses a language model to determine the target topic of the information carried by the target material, and then determines the probability of matching the target topic with the category information of m candidate tags, obtains m matching probabilities, and takes the label to which the category information corresponding to the matching probability greater than the matching threshold belongs as the target label.

[0146] In the case of using a language model to determine the target label of a target material, it is necessary to exclude the interference of labels whose category information does not match the target material in the m candidate labels, and then determine the target label among the m candidate labels. The more candidate labels there are (i.e., the larger m is), the more non-target labels need to be excluded, which in turn causes the labels whose category information does not match the target material to interfere with the language model in determining the target label. In the case where the interference of labels whose category information does not match the target material on the language model in determining the target label is greater, the degree of matching between the target label determined by the material labeling device using the language model and the target material can be reduced. In summary, it is necessary to reduce the number of candidate labels so that when the material labeling device determines the target label of the target material using the language model, the interference of labels whose category information does not match the target material when determining the target label is reduced, and the probability that the target label determined by the material labeling device from the m candidate labels is a label that matches the target material is increased. In order to reduce the number of candidate labels, the embodiment of the present application obtains a smaller number of m candidate labels through step 102.

[0147] In the embodiment of the present application, when the target material and n preset labels are obtained, the material labeling device uses the target material embedding of the target material to retrieve the n preset labels, and obtains m candidate labels with embedding matching the target material, wherein the preset labels include the category information of the material. By removing the labels with a low degree of embedding matching with the target material from the n preset labels, a small number of m labels with embedding matching with the target material are obtained. When the m candidate labels are determined, the material labeling device generates a target prompt word according to the m candidate labels and the target material. After the target prompt word is generated, the material labeling device uses the language model to determine the label with category information matching the target material from the m candidate labels as the target label under the guidance of the target prompt word. By using the language model to determine the target label, the labor cost is reduced and the speed of determining the target label is accelerated. In addition, before the material labeling device uses the language model to determine the target label, the number of labels of the candidate labels is reduced by searching, which can reduce the interference of the labels whose category information does not match the target material when determining the target label to the material labeling device, and improve the matching degree between the target label determined by the material labeling device from the m candidate labels and the target material.

[0148] As an optional implementation, the material labeling device performs the following steps during the execution of step 102:

[0149] 2001. Extract the embeddings of the n preset tags to obtain n preset tag embeddings.

[0150] In the embodiment of the present application, the preset tag embeds a feature representing the category information in the preset tag.

[0151] Optionally, the preset tag includes, in addition to the category information, a matching requirement for determining whether the information carried by the target material matches the category information. The preset tag embeds a feature including the matching requirement.

[0152] 2002. Determine the similarities between the target material embedding and the n preset tag embeddings respectively, and obtain n reference similarities.

[0153] In the embodiment of the present application, the reference similarity characterizes the similarity between the embedding of the target material and the embedding of the preset tag, because the reference similarity corresponds to the embedding of the preset tag one-to-one, and the embedding of the preset tag corresponds to the preset tag one-to-one, so the reference similarity corresponds to the preset tag one-to-one. The higher the reference similarity, the more similar the embedding of the target material and the embedding of the preset tag are, and the more similar the embedding of the target material and the embedding of the preset tag are, the more the target material matches the preset tag corresponding to the embedding of the preset tag, so the material labeling device can use the reference similarity as the basis for judging the match between the target material and the preset tag.

[0154] Optionally, in the case where the preset tag embedding includes embedding of category information and matching requirements, the reference similarity between the target material embedding and the preset tag embedding is determined by the degree of matching between the information carried by the target material and the category information and matching requirements in the preset tag. Compared with the implementation method of embedding that does not include matching information in the preset tag embedding, the reference similarity adds a determination condition determined by the degree of matching between the information in the target material and the matching requirements. Since the determination condition for determining the similarity is increased, the possibility of deviation in the reference similarity due to too few determination conditions is reduced, thereby improving the accuracy of the reference similarity.

[0155] In one possible implementation, the material labeling device determines the distance between the target material embedding and any preset label embedding based on n preset label embeddings to obtain n first distances, and then determines n reference similarities based on the n first distances, the reference similarities correspond to the first distances one-to-one, and the reference similarities are negatively correlated with the first distances.

[0156] In another possible implementation, the material labeling device determines the cosine value of the angle between the target material embedding and any preset label embedding based on n preset label embeddings, obtains n first cosine values, and then determines n reference similarities based on the n first cosine values, the reference similarities correspond to the first cosine values ​​one-to-one, and the reference similarities are positively correlated with the first cosine values.

[0157] 2003. Determine m similarities with the greatest similarities from the n reference similarities as candidate similarities.

[0158] In the embodiment of the present application, the m candidate similarities are the m maximum values ​​among the n reference similarities.

[0159] 2004. The m preset tags corresponding to the m candidate similarities are used as the m candidate tags.

[0160] In the embodiment of the present application, because the reference similarity is determined by embedding the candidate tags, the reference similarity corresponds to the embedding of the candidate tags one-to-one, and the embedding of the candidate tags corresponds to the candidate tags one-to-one, so the reference similarity corresponds to the candidate tags one-to-one. The reference similarity includes the candidate similarity, so there is a corresponding relationship between the reference similarity and the candidate tags.

[0161] In the embodiment of the present application, because the material labeling device uses the reference similarity as the basis for determining whether the target material matches the preset label, when the m candidate similarities are the m maximum values ​​among the n reference similarities, the m preset labels corresponding to the m candidate similarities are the labels that match the target material.

[0162] In a possible implementation scenario, the material labeling device uses an embedding model that can extract embeddings of data containing multiple data types, extracts the embedding of the target material and the embedding of n preset tags, and obtains the target material embedding and the n preset tag embedding, wherein the target material includes image-type data and text-type data, and the image-type data includes the images in the target material. In the case where the target material includes a video, the cover image, key frames, and non-key frames of the video are also image-type data. Text-type data includes the publisher identifier of the target material, the introduction text of the publisher of the target material, the title text of the target material, the body text in the target material, and the comment text of the target material. The preset tags include category information and matching requirements.

[0163] When the target material embedding and n preset labels are extracted, the material labeling device uses the cosine value of the angle between the target material embedding and the preset label embedding as the basis for judging the degree of match between the target material and the preset label. The larger the cosine value of the angle, the higher the degree of match between the target material and the preset label. The smaller the cosine value of the angle, the smaller the degree of match between the target material and the preset label. Specifically, the cosine value of the angle between the target material embedding and the n preset label embedding is determined respectively to obtain n target cosine values, and then n reference similarities are determined based on the n target cosine values, and the reference similarity is positively correlated with the target cosine value. Determine the m candidate similarities with the greatest similarity from the n reference similarities, and embed the preset label corresponding to the preset label with the calculated m candidate similarities as the candidate label. Optionally, m is 15, that is, the number of candidate labels is 15.

[0164] In this embodiment, the material labeling device extracts the embedding of n preset labels to obtain n preset label embeddings, and determines the similarity between the target material embedding and the n preset label embeddings respectively to obtain n reference similarities. The larger the reference similarity, the higher the matching degree between the target material and the preset label, and the smaller the reference similarity, the smaller the matching degree between the target material and the preset label. Therefore, the material labeling device can use the reference similarity as a basis for judging the matching degree between the target material and the preset label. In the case where the material labeling device uses the reference similarity as a basis for judging the matching degree between the target material and the preset label, the largest m similarities are determined from the n reference similarities as m candidate similarities, and the m preset labels corresponding to the m candidate similarities are used as m candidate labels. The m candidate labels determined by the material labeling device through the above method are the m labels with the highest matching degree with the target material determined by the material labeling device.

[0165] As an optional implementation, the target prompt word includes a target example, the target example is an example of determining a second example tag from m first example tags, the second example tag is a tag having category information matching the first reference material, the first reference material is different from the target material, and the m first example tags are retrieved from the n preset tags using the first embedding of the first reference material to obtain m tags matching the first embedding. The material labeling device performs the following steps in the process of executing step 104:

[0166] 3001. Under the prompt of the above target example, the above language model is used to determine a label among the above m candidate labels that has the above category information matching the above target material as the above target label.

[0167] In the embodiment of the present application, the target prompt word includes a target example, m candidate tags, and a target material. After the material labeling device determines the target example, m candidate tags, and the target material in the target prompt word, it obtains a prompt from the target example to determine the second example tag with category information matching the first reference material from the m first example tags, and determines the target tag with category information matching the target material from the m candidate tags based on the prompt. Since the target prompt word includes the target example, and the category information of the first reference material and the second example tag contained in the target example has a high degree of matching, the category information of the target tag generated by the material labeling device under the prompt of the target prompt word has a high degree of matching with the target material, and the probability that the target tag is the tag of the target material is high.

[0168] Optionally, the target prompt word includes a first position identifier for the position of the target example in the target prompt word. In this implementation, the material labeling device can accurately determine the target example in the target prompt word according to the content of the position indicated by the first position identifier, which is more accurate than determining the target example from the target prompt word through the semantics of the target prompt word.

[0169] As an optional implementation, the target prompt word includes a target step, which is a step of determining a fourth example tag from m third example tags, the fourth example tag is a tag having category information matching the second reference material, the second reference material is different from the target material, and the m third example tags are retrieved from the n preset tags using the second embedding of the second reference material to obtain m tags matching the second embedding. The material labeling device performs the following steps in the process of executing step 104:

[0170] 4001. Under the prompt of the above target step, use the above language model to determine the label with the above category information matching the above target material among the above m candidate labels as the above target label.

[0171] In the embodiment of the present application, the target prompt word includes a target step, m candidate tags and a target material. After the material labeling device determines the target step, m candidate tags and the target material in the target prompt word, it determines from the target step a tag with category information matching the reference material from the m third example tags as one or more steps of the fourth example tag, and uses the language model to determine the target tag with category information matching the target material from the m candidate tags according to the step. Optionally, the above target step is a chain of thought (CoT).

[0172] In this embodiment, in the process of determining the target label, the material labeling device determines the target label step by step according to the steps in the target step, compared with directly determining the target label. This allows the material labeling device to check the results obtained by the steps before the step when executing a step other than the first step, thereby improving the accuracy of the results obtained by the material labeling device in the process of executing the steps in the above target step, thereby increasing the probability that the target label has category information matching the target material. For example, the target step includes step 1 and step 2. The material labeling device first executes step 1 on m candidate labels and the target material to obtain result 1, and then executes step 2 on result 1 to obtain the target label. If the material labeling device determines that result 1 is an inaccurate result in the process of executing step 2 on result 1, step 1 can be re-executed on the m candidate labels and the target material to obtain result 4. When the material labeling device determines that result 4 is accurate, step 2 is executed on result 4 to obtain an accurate target label.

[0173] Optionally, the target prompt word includes a role definition text, and the role definition text is used to limit the role when the material labeling device uses the language model to determine the target label. The role definition text includes a role identifier and a target task matching the role, wherein the role identifier is used to limit the role when the material labeling device uses the language model to determine the target label. Since the target task is a task matching the role, the accuracy of generating the answer to the target task using the knowledge of the technical field to which the above role belongs is higher, and the accuracy of generating the answer to the target task using the knowledge of the technical field not corresponding to the above role is lower, and the knowledge of the technical field to which different roles belong is different.

[0174] For example, the identity in the role definition text is a multimedia content labeler. The target task performed by this role is to label the input multimedia content. In the process of labeling the input multimedia content, the language model uses the knowledge of the multimedia content labeling field corresponding to the multimedia content labeler to label and improve the accuracy of the labeled labels.

[0175] In a possible implementation, the target prompt word includes a target example, a target step, a role definition text, m candidate tags, and a target material. After the material labeling device determines the target example, the target step, the role definition text, the m candidate tags, and the target material from the target prompt word, it uses a language model to obtain a prompt from the target example to determine a second example tag with category information matching the first reference material from the m first example tags, and based on the prompt, determines a target tag with category information matching the target material from the m candidate tags. Since the target prompt word includes the target example, and the matching degree between the category information of the reference material and the second example tag contained in the target example is high, the target tag generated by the material labeling device under the prompt of the target prompt word has a high matching degree with the target material, and the probability that the target tag is the tag of the target material is high.

[0176] The material labeling device also uses a language model to determine from the target step one or more steps of determining a label with category information matching the reference material from the m first example labels, and uses the language model to determine a target label with category information matching the target material from the m candidate labels according to the step.

[0177] The material labeling device uses a language model to determine an identity identifier and a target task from a role definition text, wherein the role identifier in the role definition text is used to limit the role when the material labeling device uses the language model to determine the target label. In the embodiment of the present application, the target task in the role definition text is to determine a target label with category information that matches the target material from m candidate labels, and the target task is a task that matches the above-mentioned role. In the process of executing the target task, the material labeling device uses the knowledge of the technical field to which the role defined by the role identifier in the role definition text belongs to determine the target label with category information that matches the target material from the m candidate labels. Since the target task is a task that matches the role, the accuracy of generating the answer to the target task using the knowledge of the technical field to which the above-mentioned role belongs is higher.

[0178] The material labeling device obtains a prompt from the target example to determine a second example label with category information matching the reference material from the m first example labels, determines the role and target task defined in the identity from the role definition text, and determines one or more steps of determining a label with category information matching the reference material from the m first example labels from the target step, and then performs the target task based on the prompt, and performs the target task according to the above steps using the language model, that is, using the knowledge of the technical field corresponding to the above role to determine the target label with category information matching the target material from the m candidate labels. Since the target prompt word includes the target example, and the matching degree of the category information of the first reference material and the second example label contained in the target example is high, the target label generated by the material labeling device under the prompt of the target prompt word has a high matching degree with the target material, and the probability that the target label is the label of the target material is high.

[0179] Since the target prompt word also includes a role definition text, wherein the role identifier in the role definition text is used to limit the role when the material labeling device uses the language model to determine the target label. In the embodiment of the present application, the target task in the role definition text is to determine the target label with category information matching the target material from m candidate labels, and the target task is a task matching the above role. In the process of executing the target task, the material labeling device uses the knowledge of the technical field to which the role defined by the role identifier in the role definition text belongs to determine the target label with category information matching the target material from m candidate labels. Since the target task is a task matching the role, the accuracy of the answer to the target task generated using the knowledge of the technical field to which the above role belongs is high. Since the target prompt word also includes a target step, the target label is determined step by step according to the steps in the target step, so that the material labeling device can check the results obtained by the steps before the step when executing a step other than the first step, thereby improving the accuracy of the results obtained by the material labeling device in the process of executing the steps in the above target step, thereby improving the probability that the target label has category information matching the target material.

[0180] As an optional implementation, the target prompt word includes a target example, and the target example is characterized as an example of a second example label having category information matching a reference material determined based on m first example labels, and the reference material is different from the target material. The material labeling device performs the following steps in the process of executing step 104:

[0181] 5001. Divide the above n preset tag embeddings into k tag groups.

[0182] In the embodiment of the present application, there are greater than or equal to m preset tag embeddings in any tag group among the k tag groups.

[0183] In a possible implementation, n preset tag embeddings are randomly divided into groups to obtain k tag groups, where k is a positive integer less than n.

[0184] In another possible implementation, n preset tag embeddings are divided into k tag groups by clustering, and the similarity between any two preset tag embeddings in any tag group is greater than the similarity threshold, and the similarity between any one preset tag embedding in any two tag groups is less than the similarity threshold. For example, 1000 preset tag embeddings are divided into 2 tag groups by clustering, wherein the tag groups include tag group 1 and tag group 2, and the number of preset tag embeddings in any one tag group is 500. The similarity between any two preset tag embeddings in tag group 1 or tag group 2 is greater than the embedding threshold, and the similarity between any one preset tag embedding in tag group 1 and tag group 2 is less than the embedding threshold.

[0185] 5002. Based on the k groups of label groups, determine the average value of the preset label embeddings in any of the label groups to obtain k average embeddings.

[0186] In the embodiment of the present application, the preset label embedding is a vector used to characterize the characteristics of the preset label. After determining k groups of label groups, the material labeling device determines the average value of the preset label embedding in each group of label groups to obtain k mean embeddings. Since the preset label embedding is a vector used to characterize the characteristics of the preset label, the mean embedding characterizes the average value of the vector of the characteristics of the preset label in the label group. The mean embedding corresponds to the label group one by one.

[0187] 5003. Determine the maximum similarity between the target material embedding and the k mean embeddings to obtain a first similarity.

[0188] In the embodiment of the present application, the material labeling device determines the similarities of the target material embedding and the k mean embeddings respectively, obtains k embedding similarities, and selects the embedding similarity with the greatest similarity as the first similarity.

[0189] In a possible implementation, based on k mean embeddings, the material labeling device determines the distance between the target material embedding and any mean embedding, obtains k second distances, and then determines k embedding similarities based on the k second distances, and selects the embedding similarity with the largest similarity as the first similarity. The embedding similarity corresponds to the second distance one by one, and the embedding similarity is negatively correlated with the second distance.

[0190] In another possible implementation, based on k mean embeddings, the material labeling device determines the cosine value of the angle between the target material embedding and any mean embedding, obtains k second cosine values, and then determines k embedding similarities based on the k second cosine values, and selects the embedding similarity with the largest similarity as the first similarity. The embedding similarity corresponds to the target cosine value one by one, and the embedding similarity is positively correlated with the target cosine value.

[0191] 5004. Determine the tag group corresponding to the first similarity to obtain a target tag group.

[0192] In the embodiment of the present application, since the mean embedding corresponds to the label group one-to-one, and the mean embedding corresponds to the embedding similarity one-to-one, the embedding similarity corresponds to the label group one-to-one. The first similarity is one of the embedding similarities, and the material labeling device uses the label group corresponding to the first similarity as the target label group.

[0193] In the embodiment of the present application, mean embedding represents the average value of preset label embedding in the label group, and embedding similarity represents the similarity between mean embedding and target material embedding. The higher the embedding similarity, the more similar the target material embedding and mean embedding are. The more similar the target material embedding and mean embedding are, the more the target material embedding matches the preset label embedding in the label group to which the mean embedding belongs. Since the preset label embedding corresponds one-to-one to the preset label, it also means that the target material and the preset label corresponding to the above preset label embedding match more. In order to find a label group with a preset label that matches the target material, the material labeling device determines the maximum similarity from the embedding similarity as the first similarity, and then determines the label group corresponding to the first similarity as the target label group. At this time, the target label group has a preset label that matches the target material.

[0194] 5005. Determine the similarity between the embedding of any one of the preset tags in the target tag group and the embedding of the target material to obtain i second similarities.

[0195] In the embodiment of the present application, the target tag group includes i preset tag embeddings, i is a positive integer greater than or equal to m and less than n. The material labeling device determines the similarity between the target material embedding and any preset tag embedding based on the i preset tag embeddings in the target tag group, and obtains i second similarities.

[0196] The material labeling device performs the following steps during the target step 105:

[0197] 5006. Determine the m candidate similarities with the greatest similarity from the i second similarities.

[0198] In the embodiment of the present application, the m candidate similarities are the m maximum values ​​among the i second similarities.

[0199] It can be known from the method of step 5001 to step 5006 that the material labeling device can determine m candidate similarities by calculating (k+i) similarities, wherein k is the number of label groups, i is the number of preset label embeddings in the target label group, the number of preset label embeddings in any group of label groups is greater than or equal to m, and m is a positive integer greater than 1 and less than n. In the process of calculating the similarity, for the non-target label group, it is only necessary to calculate the similarity between the mean embedding and the target material embedding once, and it is not necessary to calculate the similarity between the preset label embedding and the target material embedding in the non-target label group. Therefore, the number of times the material labeling device calculates the similarity is reduced, thereby improving the labeling efficiency of the material labeling device.

[0200] In this embodiment, the material labeling device divides n preset label embeddings into k groups of label groups, and the number of preset label embeddings in any group of label groups is greater than or equal to m. When the material labeling device determines the average value of the preset label embeddings in any group of label groups and obtains k mean embeddings, the maximum similarity between the target material embedding and the k mean embeddings is taken as the first similarity. The material labeling device uses the similarity between the target material embedding and the mean embedding as the basis for judging the degree of match between the target material and the preset label in the label group. The higher the similarity between the target material embedding and the mean embedding, the higher the degree of match between the target material and the preset label in the label group to which the mean embedding belongs, and the first similarity is the maximum value of the similarity between the target material embedding and the k mean embeddings, so the material labeling device judges that the target material has the highest degree of match between the preset label in the label group to which the first similarity belongs, and takes the label group to which the first similarity belongs as the target label group. In the case of determining the target label group, the material labeling device determines the similarity between the target material embedding and the i preset label embeddings in the target label group respectively, and obtains i second similarities. The material labeling device uses the second similarity as a basis for judging the matching degree between the target material and the preset label in the target label group. The higher the second similarity between the target material embedding and the preset label embedding, the more the target material embedding matches the preset label embedding, and further the more the target material matches the preset label embedding corresponding to the preset label embedding. When i second similarities are determined, the material labeling device can determine the m largest similarities from the i second similarities as m reference similarities. When m reference similarities are determined, the material labeling device can determine the m preset labels with m reference similarities and use them as m candidate labels. The determined m candidate labels are the m labels with the highest matching degree with the target material determined by the material labeling device. It can be seen from the above steps that the material labeling device only needs to calculate the similarity between the mean embedding and the target material embedding once for the non-target label group, and does not need to calculate the similarity between the preset label embedding and the target material embedding in the non-target label group, thereby reducing the number of times the material labeling device calculates similarities and improving the labeling efficiency of the material labeling device.

[0201] See also Figure 2 , Figure 2A schematic diagram of a target label for a target material generated by a language model is provided for the present application. First, m candidate labels matching the target material are obtained by retrieval enhancement. Specifically, the target material is input into an embedding model, and the embedding model outputs the target material embedding of the target material, and the target material embedding is a vector representing the characteristics of the target material. After obtaining the target material embedding, m candidate labels with the highest similarity to the target material embedding are selected from the database. After obtaining m candidate labels matching the target material by retrieval enhancement, the m candidate labels and the target material are input into a language model, and the language model is used to determine the label matching the target material among the m candidate labels as the target label.

[0202] See also Figure 3 , Figure 3 A flowchart of a model training method provided in an embodiment of the present application. The executor of the model training method is a model training device, wherein the model training device can be any electronic device that can execute the technical solution disclosed in the embodiment of the method of the present application. Optionally, the model training device can be one of the following: a computer, a server.

[0203] 301. Obtain training prompt words and a first label.

[0204] In the embodiment of the present application, the first label is the label of the training material. The training prompt word is used to guide the model to be trained to determine the label with category information matching the training material from the m training labels, wherein the m training labels are obtained based on the m retrieval labels, and the m retrieval labels are determined by searching the n preset labels and having category information matching the training material embedding from the n preset labels, and the m training labels include the first label. The preset label includes the category information of the material.

[0205] In an implementation method of obtaining the training prompt word and the first label, the model training device receives the target material input by an input component. The input component includes at least one of the following: a keyboard, a mouse, a touch screen, a touch pad, and an audio input device.

[0206] In another implementation of obtaining the training prompt word and the first label, the model training device receives the target material sent by the terminal. The terminal includes at least one of the following: a mobile phone, a computer, a tablet computer, and a server.

[0207] Optionally, when searching for n preset labels using training material embedding and obtaining m search labels that match the training material embedding and the first label does not exist, any label among the m labels is replaced with the first label to obtain m training labels.

[0208] Optionally, there are random tags in the m training tags, wherein the random tags are tags other than the m retrieval tags and the first tag in the n preset tags. Specifically, by searching the n preset tags, m retrieval tags with embedding matching with the training material are obtained. When the m retrieval tags include the first tag, any tag in the m retrieval tags except the first tag is replaced with a random tag to obtain m training tags. When the m retrieval tags do not include the first tag, any tag in the m retrieval tags is replaced with the first tag, and then any tag except the first tag is replaced with a random tag to obtain m training tags. The above steps may be performed in the model training device or not. By introducing random tags in the m training tags, the difference between the m training tags in the training prompt words is improved. When the difference between the m training tags is large, the model to be trained can obtain information with large differences during the training process, thereby improving the training effect of the model to be trained. For example, in the process of training the model to be trained to become an image classification model, if the training labels are cats of different breeds, the difference in the training labels is small, the model to be trained can only obtain information about cats, and the generalization ability of the image classification model trained according to the training labels is weak. If the training labels include cats, dogs, and horses, the difference in the training labels is large, and the model to be trained can obtain diverse information about cats, dogs, and horses, thereby improving the generalization ability of the image classification model trained according to the training labels. It can be executed in the model training device or not.

[0209] It should be understood that in the embodiment of the present application, the steps of the model training device obtaining the training prompt word and the first label can be performed separately or simultaneously, and the present application does not limit this.

[0210] 302. Under the guidance of the training prompt word, the to-be-trained model is used to determine a label having the category information matching the training material from the m training labels as a second label.

[0211] In the embodiment of the present application, the second label is the label of the training material determined by the model training device from the m training labels. Specifically, the model training device uses the model to be trained to determine the category information in the m training labels in the training prompt words and the information carried by the training material, and then determines a label with category information matching the information carried by the training material from the m training labels as the second label.

[0212] 303. Determine a target loss according to a target difference between the second label and the first label.

[0213] In an embodiment of the present application, the model training device determines the target loss by the target difference between the second label and the first label, and the target loss is positively correlated with the target difference, that is, the greater the target difference, the greater the target loss.

[0214] 304. According to the target loss, adjust the parameters of the model to be trained to obtain a language model.

[0215] In the embodiment of the present application, the parameters in the model to be trained are updated according to the target loss to obtain a language model. Specifically, the model training device updates the parameters of the model to be trained to make the target loss smaller until the target loss converges to obtain a language model.

[0216] In a possible implementation, the model training device determines the gradient of the model to be trained based on the target loss, performs reverse gradient propagation on the model to be trained based on the gradient of the model to be trained, and updates the parameters of the model to be trained until the loss of the model to be trained converges to obtain a language model.

[0217] In an embodiment of the present application, when a training prompt word and a first label are obtained, the model training device, under the guidance of the training prompt word, uses the model to be trained to determine a label with category information matching the training material from m training labels to obtain a second label, wherein the training prompt word is used to guide the model to be trained to determine a label with category information matching the training material from m training labels, and the m training labels are obtained by retrieving n preset labels to obtain m labels embedded with the training material, and the preset labels include the category information of the material. The first label is the label of the training material. The target loss is determined by determining the target difference between the second label and the first label. The model training device can adjust the parameters of the model to be trained according to the target loss to obtain a language model that can be used to determine a label with category information matching the training material from the m training labels.

[0218] As an optional implementation, the model to be trained includes a decomposition branch, which is used to decompose the embedding of the training prompt word into a first sub-embedding and a second sub-embedding, and the embedding obtained by performing an inner product between the first sub-embedding and the second sub-embedding represents the embedding of the prompt word, and the difference between the parameter quantity of the embedding of the training prompt word and the sum of the parameter quantities of the first sub-embedding and the second sub-embedding is greater than a reference threshold, and the reference threshold is a positive integer. The model training device performs the following steps in the process of executing step 202:

[0219] 401. Extract the embedding in the training prompt word to obtain a reference embedding.

[0220] In the embodiment of the present application, the reference embedding represents the features of the m training labels and training materials in the target prompt word.

[0221] 402. Extract the reference embedding using the decomposition branch to obtain a first sub-embedding and a second sub-embedding.

[0222] In an embodiment of the present application, the model training device decomposes the reference embedding into a first sub-embedding and a second sub-embedding by decomposing branches, wherein the embedding obtained by performing an inner product between the first sub-embedding and the second self-embedding represents the reference embedding. The difference between the parameter amount of the reference embedding and the sum of the parameter amounts of the first sub-embedding and the second sub-embedding is greater than a reference threshold, and the reference threshold is a positive integer. Since the features of the m training labels and the training labels in the target prompt word have not changed, the larger the reference threshold, the smaller the sum of the parameter amounts of the first sub-embedding and the second sub-embedding.

[0223] 403. Obtain an inner product embedding by performing an inner product operation on the first sub-embedding and the second sub-embedding.

[0224] In an embodiment of the present application, the model training device performs an inner product on the first sub-embedding and the second sub-embedding to obtain an inner product embedding representing the reference embedding. For example, the reference embedding is a matrix with x rows and x columns, the first sub-embedding is a matrix with x rows and y columns, and the second sub-embedding is a matrix with y rows and x columns, where y and x are both positive integers greater than 1, and y is less than x. By performing an inner product on the first sub-embedding and the second sub-embedding, an inner product embedding with x rows and x columns is obtained.

[0225] 404. Based on the inner product embedding, use the to-be-trained model to determine a label among the m training labels that has category information matching the target material as the second label.

[0226] In the embodiment of the present application, the second label is the label of the training material determined by the model training device from the m training labels. Specifically, the model training device uses the model to be trained to determine the m category information in the m training labels in the inner product embedding and the information carried in the training material, and then determines the label with category information matching the information carried by the training material from the m training labels as the target label.

[0227] The model training device performs the following steps during the execution of step 304:

[0228] 405. According to the target loss, adjust the parameters in the decomposition branch to obtain a language model.

[0229] In the embodiment of the present application, the parameters of the decomposition branch in the model to be trained are updated according to the target loss to obtain a language model. Specifically, the model training device updates the parameters of the decomposition branch to make the target loss smaller until the target loss converges to obtain a language model.

[0230] In a possible implementation, the model training device determines the gradient of the decomposition branch based on the target loss, performs reverse gradient propagation on the decomposition branch based on the gradient of the decomposition branch, and updates the parameters of the decomposition branch until the loss of the decomposition branch converges to obtain a language model.

[0231] During the target step 302, the model training device needs to convert the training prompt words into reference embeddings, and then use the model to be trained to determine the m training labels and training materials in the reference embeddings. However, when the target loss has not converged, each time the parameters in the model to be trained are updated, each parameter in the reference embedding needs to be recalculated in the next training process, resulting in a large amount of calculation.

[0232] In this embodiment, by introducing a decomposition branch in the model to be trained, the decomposition branch can decompose the reference embedding into a first sub-embedding and a second sub-embedding whose parameters are smaller than the reference embedding, and the inner product embedding obtained by the inner product of the first sub-embedding and the second sub-embedding can be used to characterize the embedding of the features of the training prompt word, that is, the reference embedding. And when the target loss has not converged, only the parameters of the decomposition branch in the model to be trained need to be updated, so that in the next training process, only the parameters in the first sub-embedding and the second sub-embedding need to be recalculated, rather than the parameters in the reference embedding, thereby reducing the amount of parameter calculation of the model training device during the training process, thereby speeding up the training of the model to be trained.

[0233] See also Figure 4 , Figure 4A schematic diagram of a model training provided in the present application. Specifically, when the model training device obtains training materials and 8 training labels, the training materials and the 8 training labels are input into the model to be trained for training, wherein 4 of the 8 training labels are matching labels a determined from n preset labels that match the embedding of the training materials, 1 label is the first label b of the training material, and 3 labels are random labels c, where the random labels are labels that do not match the embedding of the training materials among the n preset labels. Specifically, when 8 matching labels a that match the embedding of the training materials are determined from n preset labels, and the first label b is not included in the matching labels a, any label is replaced with the first label b, and any three labels except the first label b are replaced with the random label c. Optionally, before the training materials and the 8 training labels are input into the model to be trained, training prompt words can be generated based on the training materials and the 8 training labels. When the model to be trained receives the training material and 8 training labels, the training material and the 8 training labels are extracted from the training material and the 8 training labels by the self-attention module to obtain a reference embedding, and the reference embedding is respectively generated by two low-rank adapters (low-rank adaptation, LoRA) to generate a first sub-embedding and a second sub-embedding. When the model training device generates the first sub-embedding and the second sub-embedding, the first sub-embedding and the second sub-embedding are inner-producted to obtain an embedding for characterizing the features of the training material and the 8 training labels, that is, the inner-product embedding. Based on the inner-product embedding, the model training device uses the model to be trained to extract the features of the training material and the 8 training labels in the inner-product embedding, and determines the label with the above-mentioned category information matching the above-mentioned training material from the above-mentioned 8 training labels as the second label. The model training device determines the target loss according to the difference between the second label and the first label. When the model training device determines the target loss, the parameters of the LoRA in the model to be trained are updated according to the target loss to obtain a language model. While updating the parameters of the LoRA, the parameters of the self-attention module or the modules other than the LoRA in the model to be trained will not be updated.

[0234] See also Figure 5 , Figure 5 A flowchart of a multimedia content annotation method provided in an embodiment of the present application. The execution subject of the multimedia content annotation method is a multimedia content annotation device, wherein the multimedia content annotation device can be any electronic device that can execute the technical solution disclosed in the embodiment of the method of the present application. Optionally, the multimedia content annotation device can be one of the following: a computer, a server.

[0235] 501. Obtain target multimedia content and n preset tags.

[0236] In the embodiment of the present application, the target multimedia content is the multimedia content published by the target object on the multimedia content consumption platform. The target multimedia content can be any multimedia content. Optionally, the target multimedia content includes one or more of the following: text, video, picture, audio. For example, the target multimedia content can be multimedia content containing text. For another example, the target multimedia content can be multimedia content containing pictures. For another example, the target multimedia content can also be multimedia content containing videos, pictures and text.

[0237] In the embodiment of the present application, the preset tag includes the category information of the multimedia content. The preset tag is used to classify the information carried by the multimedia content. Specifically, because different multimedia content carries different information, the category information to which different multimedia content belongs may be different, that is, the preset tags to which it belongs may be different. The number of preset tags is n, where n is a positive integer greater than 1.

[0238] In a possible implementation, there are two preset tags, including tag a5 and tag a6. The category information of the multimedia content in tag a5 is beauty and makeup, and the category information of the multimedia content in tag a6 is games. 80% of the information carried by multimedia content c1 is related to makeup brushes. Since makeup brushes are a tool in beauty, the category information of the information carried by multimedia content c1 is beauty, and the preset tag to which multimedia content c1 belongs is tag a5. 90% of the information carried by multimedia content c2 is related to racing games. Since racing games are a type of game among games, the category information of the information carried by multimedia content c2 is games, and the preset tag to which multimedia content c2 belongs is tag a6.

[0239] In another possible implementation, there are two preset tags, including tag a5 and tag a6. The category information of the multimedia content in tag a5 is beauty and makeup, and the category information of the multimedia content in tag a6 is games. 80% of the information carried by multimedia content c1 is related to makeup brushes. Since makeup brushes are a tool in beauty and makeup, the category information of the information carried by multimedia content c1 is beauty and makeup, and the preset tag to which multimedia content c1 belongs is tag a5. 70% of the information carried by multimedia content c3 is related to foundation. Since foundation is another tool in beauty and makeup, the category information of the information carried by multimedia content c3 is also beauty and makeup, and the preset tag to which multimedia content c3 belongs is also tag a5.

[0240] In an implementation method of acquiring target multimedia content and n preset tags, a multimedia content tagging device receives the target multimedia content input by an input component. The input component includes at least one of the following: a keyboard, a mouse, a touch screen, a touch pad, and an audio input device.

[0241] In another implementation of acquiring target multimedia content and n preset tags, the multimedia content tagging device receives the target multimedia content sent by a terminal. The terminal includes at least one of the following: a mobile phone, a computer, a tablet computer, and a server.

[0242] It should be understood that in the embodiment of the present application, the steps of the multimedia content tagging apparatus acquiring the target multimedia content and the n preset tags may be performed separately or simultaneously, and the present application does not limit this.

[0243] 502 . Using the target multimedia content embedding of the target multimedia content, search the n preset tags to obtain m candidate multimedia content tags that match the target multimedia content embedding.

[0244] The target multimedia content is embedded with features that characterize the target multimedia content, wherein the features of the target multimedia content include features of data of a target data modality in the target multimedia content, and the target data modality is used to characterize the data type of the data in the target multimedia content. Optionally, the target data modality includes any one or more data modalities of a picture type and a text type. For example, in the case where the target data modality includes a picture type, the target multimedia content is embedded with features of data that characterize the picture type in the target multimedia content. For another example, in the case where the target data modality includes a text type, the target multimedia content is embedded with features of data that characterize the text type in the target multimedia content. For another example, in the case where the target data modality includes a picture type and a text type, the target multimedia content is embedded with features of data that characterize the picture type and the text type in the target multimedia content.

[0245] The data of the picture type in the target multimedia content includes pictures in the target multimedia content. In the case where the target multimedia content includes a video, the cover image, key frames, and non-key frames of the video are also data of the picture type.

[0246] The text-type data in the target multimedia content includes the publisher identification (identity document, ID) of the target multimedia content, any one or more of the following data: the introduction text of the publisher of the target multimedia content, the title text of the target multimedia content, the body text of the target multimedia content, and the comment text of the target multimedia content. The publisher's introduction text includes the content of the publisher's self-description. The title text includes the text in the target multimedia content that summarizes the theme of the target multimedia content so that the object can quickly understand the target multimedia content. The body text includes the text in the target multimedia content that explains the theme of the target multimedia content. The comment text is the text that comments on the target multimedia content.

[0247] In the case where the target multimedia content embedding includes features of data of image type and text type, the features of the data of the target data modality in the target multimedia content embedding can be made richer. In the process of the multimedia content tagging device using the target multimedia content embedding to retrieve n preset tags, the richer the features of the data of the target data modality in the target multimedia content embedding, the higher the probability that the target multimedia content embedding matches the features of the target multimedia content. Since the m candidate multimedia content tags are determined by the target multimedia content embedding, the probability that the target multimedia content embedding matches the features of the target multimedia content can be increased, and the probability that the determined m candidate multimedia content tags are tags that match the target multimedia content can be increased.

[0248] In an implementation method for determining the embedding of target multimedia content, k different modalities of data in the target multimedia content are identified. Then, k embedding models that can respectively extract the embedding of the k modalities of data are used to extract the embedding of the k different modalities of data in the target multimedia content, and obtain k data embeddings, and then the k data embeddings are spliced ​​to obtain the target multimedia content embedding. For example, in the case where there is text type and picture type data in the target multimedia content, the text embedding model is used to extract the embedding of the text type data in the target multimedia content to obtain the text data embedding. Then, the picture embedding model is used to extract the embedding of the picture type data in the target multimedia content to obtain the picture data embedding. Finally, the text data embedding and the picture data embedding are spliced ​​to obtain the target multimedia content embedding.

[0249] In another implementation of determining the target multimedia content embedding, an embedding model capable of extracting embeddings of data containing multiple modalities is used to extract the embedding of the target multimedia content to obtain the target multimedia content embedding.

[0250] In the embodiment of the present application, the candidate multimedia content tags are m tags retrieved from n preset tags, where m is a positive integer greater than 1 and m is less than n. The multimedia content tagging device retrieves m tags that match the target multimedia content embedding from the n preset tags to obtain m candidate multimedia content tags.

[0251] In a possible implementation, the multimedia content tagging device determines the embedding of n preset tags to obtain n preset tag embeddings, and then respectively determines the similarities between the n preset tag embeddings and the target multimedia content embeddings to obtain n multimedia content similarities. From the n multimedia content similarities, m similarities with the greatest similarities are determined as m maximum similarities. Finally, the m preset tags corresponding to the m maximum similarities are used as m candidate multimedia content tags.

[0252] In another possible implementation, the multimedia content annotation device determines the embedding of n preset tags to obtain n preset tag embeddings, and then respectively determines the distances between the n preset tag embeddings and the target multimedia content embedding to obtain n multimedia content distances, and selects m distances with the smallest distances from the n multimedia content distances as m minimum distances. Finally, the m preset tags corresponding to the m candidate distances are used as m candidate multimedia content tags.

[0253] 503. Generate multimedia content prompt words according to the m candidate multimedia content tags and the target multimedia content.

[0254] The multimedia content prompt word is used to guide the language model to determine a tag having category information matching the target multimedia content from m candidate multimedia content tags, wherein the multimedia content prompt word includes the m candidate multimedia content tags and the target multimedia content.

[0255] 504 . Under the guidance of the multimedia content prompt word, use the language model to determine a label having the category information matching the target multimedia content among the m candidate multimedia content labels as the target multimedia content label.

[0256] In the embodiment of the present application, the target multimedia content tag is a tag of the target multimedia content determined by the multimedia content tagging device from the m candidate multimedia content tags. Specifically, the multimedia content tagging device uses a language model to determine the m category information in the m candidate multimedia content tags and the information carried by the target multimedia content, and then determines a tag with category information matching the information carried by the target multimedia content from the m candidate multimedia content tags as the target multimedia content tag.

[0257] In a possible implementation scenario, the target multimedia content is multimedia content that does not contain any tag. When the target multimedia content does not contain any tag, the multimedia content tagging device determines a target multimedia content tag for the target multimedia content.

[0258] Generally speaking, the main method for labeling multimedia content is to manually determine the information carried by the target multimedia content. After determining the information carried by the target multimedia content, a label with category information matching the information carried by the target multimedia content is determined from m candidate multimedia content labels as the target multimedia content label. Specifically, in addition to the category information of the multimedia content, any candidate multimedia content label also includes the matching requirements that the information carried by the target multimedia content should meet when the information carried by the target multimedia content matches the category information. In the process of determining the target multimedia content label, a label whose information carried by the target multimedia content meets the matching requirements is determined from m candidate multimedia content labels as the target multimedia content label. For example, the candidate multimedia content labels include labels a7 and a8. The category information in label a7 is skin care products, and the matching requirement in label a7 is a substance or mixture that plays a protective role. The category of label a8 is cosmetics, and the matching requirement in label a8 is a substance or mixture that plays a beautifying or modifying role. When it is determined that the information carried by the target multimedia content b4 is related to the gel for protecting the skin, it is determined that the information carried by the target multimedia content b4 meets the matching requirement of the tag a7, and the tag a7 is determined as the target multimedia content tag of the target multimedia content b4.

[0259] In order to reduce labor costs, it is impossible to manually determine the label with category information matching the information carried by the target multimedia content from the m candidate multimedia content labels. Since the language model has the ability of natural language processing, the language model can be used to determine the label with category information matching the information carried by the target multimedia content from the m candidate multimedia content labels as the target multimedia content label.

[0260] In a possible implementation, the multimedia content tagging device determines the semantics of the information carried by the target multimedia content and the semantics of the category information in the m candidate multimedia content tags using a language model. The multimedia content tagging device determines the multimedia content semantic similarity between the semantics of the information carried by the target multimedia content and the semantics of the category information in the m candidate multimedia content tags, and uses the tag to which the category information corresponding to the multimedia content semantic similarity greater than the similarity threshold belongs as the target multimedia content tag.

[0261] In another possible implementation, the multimedia content tagging device uses a language model to determine the keywords of the information carried by the target multimedia content, and determines the keywords that match the requirements in the m candidate multimedia content tags. The multimedia content tagging device determines the multimedia content keyword matching degree between the keywords of the information carried by the target multimedia content and the keywords that match the requirements in the m candidate multimedia content tags, and uses the tag to which the keyword corresponding to the multimedia content keyword matching degree greater than the matching threshold belongs as the target multimedia content tag.

[0262] In another possible implementation, the multimedia content tagging device uses a language model to determine the multimedia content theme of the information carried by the target multimedia content, and then determines the probability that the multimedia content theme matches the category information of the m candidate multimedia content tags, respectively, to obtain m multimedia content probabilities, and uses the tag to which the category information corresponding to the multimedia content probability greater than the matching threshold belongs as the target multimedia content tag.

[0263] In the case of using a language model to determine the target multimedia content tag of the target multimedia content, it is necessary to exclude the interference of the tag whose category information does not match the target multimedia content among the m candidate multimedia content tags, and then determine the target multimedia content tag among the m candidate multimedia content tags. The more the number of candidate multimedia content tags (i.e., the larger the m), the more the number of non-target multimedia content tags that need to be excluded, and thus the greater the interference of the tag whose category information does not match the target multimedia content on the language model to determine the target multimedia content tag. In the case where the interference of the tag whose category information does not match the target multimedia content on the language model to determine the target multimedia content tag is greater, the matching degree between the target multimedia content tag determined by the multimedia content tagging device using the language model and the target multimedia content can be reduced. In summary, it is necessary to reduce the number of candidate multimedia content tags, so that when the multimedia content tagging device determines the target multimedia content tag of the target multimedia content using the language model, the interference of the tag whose category information does not match the target multimedia content when determining the target multimedia content tag is reduced, and the probability that the target multimedia content tag determined by the multimedia content tagging device from the m candidate multimedia content tags is a tag that matches the target multimedia content is increased. In order to reduce the number of candidate multimedia content tags, the embodiment of the present application obtains a smaller number of m candidate multimedia content tags through step 502 .

[0264] In one possible implementation scenario, the multimedia content annotation device, when determining the target multimedia content tag of the target multimedia content, determines the consumer object who is interested in the target multimedia content tag in the multimedia content consumption platform, and pushes the target multimedia content to the consumer object, thereby providing the consumer object with the multimedia content of interest and increasing the consumer object's stickiness to the multimedia content consumption platform.

[0265] In an embodiment of the present application, after acquiring the target multimedia content and n preset tags, the multimedia content annotation device uses the target multimedia content embedding of the target multimedia content to retrieve the n preset tags, and obtains m candidate multimedia content tags that match the target multimedia content embedding, wherein the preset tags include the category information of the multimedia content. By removing the tags with a low degree of matching with the target multimedia content embedding from the n preset tags, a small number of m tags that match the target multimedia content embedding are obtained. When the m candidate multimedia content tags are determined, the multimedia content annotation device generates a target prompt word based on the m candidate multimedia content tags and the target multimedia content. After generating the target prompt word, the multimedia content annotation device, under the guidance of the target prompt word, uses a language model to determine the tag with category information matching the target multimedia content among the m candidate multimedia content tags as the target multimedia content tag. By using the language model to determine the target multimedia content tag, the labor cost is reduced and the speed of determining the target multimedia content tag is accelerated. Furthermore, before the multimedia content tagging device uses the language model to determine the target multimedia content tag, the number of candidate multimedia content tags is reduced through retrieval, which can reduce the interference of tags whose category information does not match the target multimedia content on the multimedia content tagging device when determining the target multimedia content tag, and improve the matching degree between the target multimedia content tag determined by the multimedia content tagging device from the m candidate multimedia content tags and the target multimedia content.

[0266] See also Figure 6 , Figure 6A schematic diagram of a target multimedia content label for generating target multimedia content using a language model provided in the present application. First, the embedding of the preset label is extracted and stored in a vector database. Specifically, 400 preset labels are input into a pre-trained embedding model to generate 400 preset label embeddings corresponding to the 400 preset labels, wherein the preset labels include category information of the multimedia content and the matching requirements that the information carried by the target multimedia content should meet when the information carried by the target multimedia content matches the category information, and the preset label embeddings are vectors representing the characteristics of the preset labels. After generating the 400 preset label embeddings, the preset label embeddings are stored in a vector database. Next, the embedding of the target multimedia content is extracted. Specifically, the target multimedia content is input into a pre-trained embedding model to generate a target multimedia content embedding, wherein the target multimedia content includes a multimedia content title, a multimedia content body, a multimedia content picture, and a multimedia content video, and the target multimedia content embeddings are vectors representing the characteristics of the target multimedia content. When the target multimedia content embedding is obtained, the target multimedia content embedding is used to search for 400 preset tag embeddings in the vector database, and 15 preset tag embeddings with the highest similarity to the target multimedia content embedding are obtained as candidate multimedia content tags. Next, multimedia content prompt words are generated based on the candidate multimedia content tags and the target multimedia content. Specifically, multimedia content prompt words are generated based on 15 candidate multimedia content tags and the target multimedia content, wherein the multimedia content prompt words also include role definition text, target examples, and target steps. The role definition text includes: You are a multimedia content labeler, please assist the object to label the input multimedia content. The multimedia content will be provided in the form of {multimedia content title}, {multimedia content body}, {multimedia content picture}, {multimedia content video}, and select 1 most suitable label from the candidate multimedia content tags. The target examples include: The following are a small number of examples: .... The ellipsis here represents the examples in the target example. The target steps include: Please answer according to the following ideas: .... The ellipsis here represents the steps in the target step. There is also a target text in the target prompt word, which is used to provide the target multimedia content and 15 candidate multimedia content tags, including: Now, please make your answer based on the following target multimedia content and candidate multimedia content tags: ... The ellipsis here represents the target multimedia content and the 15 candidate multimedia content tags. After determining the multimedia content prompt word, the language model is used to output the final prediction. Specifically, the multimedia content prompt word is input into the language model, and the language model is used to determine the target multimedia content tag that matches the target multimedia content from the 15 candidate multimedia content tags.

[0267] See also Figure 7 , Figure 7A schematic diagram of extracting target multimedia content using a pre-trained embedding model provided in this application. Specifically, the target multimedia content includes a multimedia content body and a multimedia content image. The multimedia content body and the multimedia content image are input into the same pre-trained embedding model to obtain a target multimedia content embedding of the target multimedia content, wherein the multimedia content body includes: #Show off your pet, and the multimedia content image includes a photo of a cat.

[0268] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0269] The method of the embodiment of the present application is described in detail above, and the device of the embodiment of the present application is provided below.

[0270] See also Figure 8 , Figure 8 This is a schematic diagram of the structure of a material labeling device provided in an embodiment of the present application. The material labeling device 1 includes: an acquisition unit 11, a retrieval unit 12, a generation unit 13, and a determination unit 14. Among them:

[0271] The acquisition unit 11 is used to acquire the target material and n preset tags; the preset tags include the category information of the material, and the n is an integer greater than 1;

[0272] A retrieval unit 12 is used to retrieve the n preset tags using the target material embedding of the target material to obtain m candidate tags that match the target material embedding; the m is a positive integer less than the n;

[0273] A generating unit 13 is used to generate a target prompt word according to the m candidate tags and the target material, wherein the target prompt word is used to guide the language model to determine a tag having the category information matching the target material from the m candidate tags;

[0274] The determination unit 14 is used to determine, under the guidance of the target prompt word, using the language model to determine a tag having the category information matching the target material among the m candidate tags as a target tag.

[0275] In conjunction with any embodiment of the present application, the retrieval unit 12 is specifically used for:

[0276] Extracting the embeddings of the n preset tags to obtain n preset tag embeddings, wherein the preset tags correspond one to one to the preset tag embeddings;

[0277] Determine the similarities between the embedding of the target material and the embedding of the n preset labels respectively, and obtain n reference similarities, wherein the reference similarities correspond to the preset labels one by one;

[0278] Determine m similarities with the greatest similarity from the n reference similarities as candidate similarities;

[0279] The m preset tags corresponding to the m candidate similarities are used as the m candidate tags.

[0280] In conjunction with any embodiment of the present application, the target prompt word includes a target example, the target example is an example of determining a second example tag from m first example tags, the second example tag is a tag having category information matching a first reference material, the first reference material is different from the target material, and the m first example tags are obtained by searching the n preset tags using a first embedding of the first reference material to obtain m tags matching the first embedding;

[0281] The determination unit 14 is specifically used for:

[0282] Prompted by the above target example, the above language model is used to determine a label among the above m candidate labels that has the above category information matching the above target material as the above target label.

[0283] In conjunction with any embodiment of the present application, the target prompt word includes a target step, the target step is a step of determining a fourth example tag from m third example tags, the fourth example tag is a tag having category information matching a second reference material, the second reference material is different from the target material, and the m third example tags are obtained by searching the n preset tags using the second embedding of the second reference material to obtain m tags matching the second embedding;

[0284] The determination unit 14 is specifically used for:

[0285] Under the prompt of the above target step, the above language model is used to determine the label having the above category information matching the above target material among the above m candidate labels as the above target label.

[0286] In conjunction with any embodiment of the present application, the retrieval unit 12 is specifically used for:

[0287] Divide the n preset tag embeddings into k tag groups, where k is a positive integer less than n; any of the tag groups contains greater than or equal to m preset tag embeddings;

[0288] Based on the k groups of labels, determine the average value of the preset label embeddings in any of the label groups to obtain k average embeddings;

[0289] Determine the maximum similarity between the target material embedding and the k mean embeddings to obtain a first similarity;

[0290] Determine the tag group corresponding to the first similarity to obtain a target tag group;

[0291] Determine the similarity between any of the preset tags embedded in the target tag group and the target material embedded, and obtain i second similarities; the i represents the number of the preset tags embedded in the target tag group; the i is a positive integer greater than or equal to m;

[0292] The m candidate similarities having the greatest similarity are determined from the i second similarities.

[0293] In the embodiment of the present application, when the target material and n preset labels are obtained, the material labeling device uses the target material embedding of the target material to retrieve the n preset labels, and obtains m candidate labels with embedding matching the target material, wherein the preset labels include the category information of the material. By removing the labels with a low degree of embedding matching with the target material from the n preset labels, a small number of m labels with embedding matching with the target material are obtained. When the m candidate labels are determined, the material labeling device generates a target prompt word according to the m candidate labels and the target material. After the target prompt word is generated, the material labeling device uses the language model to determine the label with category information matching the target material from the m candidate labels as the target label under the guidance of the target prompt word. By using the language model to determine the target label, the labor cost is reduced and the speed of determining the target label is accelerated. In addition, before the material labeling device uses the language model to determine the target label, the number of labels of the candidate labels is reduced by searching, which can reduce the interference of the labels whose category information does not match the target material when determining the target label to the material labeling device, and improve the matching degree between the target label determined by the material labeling device from the m candidate labels and the target material.

[0294] See also Fig. 9 , Fig. 9 This is a schematic diagram of the structure of a model training device provided in an embodiment of the present application. The model training device 2 includes: an acquisition unit 21, a determination unit 22, a loss calculation unit 23, and an adjustment unit 24. Among them:

[0295] The acquisition unit 21 is used to acquire a training prompt word and a first label; the training prompt word is used to guide the model to be trained to determine a label with category information matching the training material from m training labels; the first label is the label of the training material; the m training labels are obtained based on m retrieval labels, the m retrieval labels are m labels with category information matching the embedding of the training material determined from the n preset labels by retrieving n preset labels, the m training labels include the first label, the embedding of the training material is the embedding of the training material; the preset label includes the category information of the material;

[0296] A determination unit 22, used to determine, under the guidance of the training prompt word, using the to-be-trained model, a label having the category information matching the training material among the m training labels as a second label;

[0297] A loss calculation unit 23, configured to determine a target loss according to a target difference between the second label and the first label; the target loss is positively correlated with the target difference;

[0298] The adjustment unit 24 is used to adjust the parameters in the above-mentioned model to be trained according to the above-mentioned target loss to obtain a language model.

[0299] In combination with any embodiment of the present application, when the above-mentioned n preset tags are retrieved using the above-mentioned training material embedding, and the above-mentioned m retrieval tags that match the above-mentioned training material embedding do not include the above-mentioned first tag, the above-mentioned m training tags are obtained by replacing any tag among the above-mentioned m first retrieval tags with the above-mentioned first tag.

[0300] In combination with any embodiment of the present application, when the above-mentioned n preset tags are retrieved using the above-mentioned training material embedding to obtain the above-mentioned m retrieval tags that match the above-mentioned training material embedding, and the m retrieval tags include the above-mentioned first tag, the above-mentioned m training tags are obtained by replacing any tag among the above-mentioned m retrieval tags except the above-mentioned first tag with a random tag, and the above-mentioned random tag is any tag among the n preset tags except the above-mentioned m retrieval tags.

[0301] In combination with any embodiment of the present application, the model to be trained includes a decomposition branch, the decomposition branch is used to decompose the embedding of the prompt word into two sub-embeddings, the embedding obtained by performing an inner product between the two sub-embeddings represents the embedding of the prompt word, and the difference between the parameter amount of the embedding of the prompt word and the sum of the parameter amounts of the two sub-embeddings is greater than a reference threshold, and the reference threshold is a positive integer;

[0302] The determination unit 22 is specifically configured to:

[0303] Extract the embeddings in the above training prompt words to get the reference embeddings;

[0304] Extracting the reference embedding using the decomposition branch to obtain a first sub-embedding and a second sub-embedding;

[0305] An inner product embedding is obtained by performing an inner product operation on the first sub-embedding and the second sub-embedding;

[0306] Based on the inner product embedding, the to-be-trained model is used to determine a label among the m training labels that has category information matching the target material as the second label;

[0307] The adjustment unit 24 is specifically used for:

[0308] According to the above target loss, the parameters in the above decomposition branch are adjusted to obtain the language model.

[0309] In an embodiment of the present application, when a training prompt word and a first label are obtained, the model training device, under the guidance of the training prompt word, uses the model to be trained to determine a label with category information matching the training material from m training labels to obtain a second label, wherein the training prompt word is used to guide the model to be trained to determine a label with category information matching the training material from m training labels, and the m training labels are obtained by retrieving n preset labels to obtain m labels embedded with the training material, and the preset labels include the category information of the material. The first label is the label of the training material. The target loss is determined by determining the target difference between the second label and the first label. The model training device can adjust the parameters of the model to be trained according to the target loss to obtain a language model that can be used to determine a label with category information matching the training material from the m training labels.

[0310] See also Fig.10 , Fig.10 This is a schematic diagram of the structure of a multimedia content annotation device provided in an embodiment of the present application. The multimedia content annotation device 3 includes: an acquisition unit 31, a retrieval unit 32, a generation unit 33, and a determination unit 34. Among them:

[0311] An acquisition unit 31 is used to acquire target multimedia content and n preset tags, where the preset tags include category information of the multimedia content, and n is an integer greater than 1;

[0312] A retrieval unit 32 is used to retrieve the n preset tags using the target multimedia content embedding of the target multimedia content to obtain m candidate multimedia content tags that match the target multimedia content embedding; the m is a positive integer less than the n;

[0313] A generating unit 33 is used to generate a multimedia content prompt word according to the m candidate multimedia content tags and the target multimedia content, wherein the multimedia content prompt word is used to guide the language model to determine a tag having category information matching the target multimedia content from the m candidate multimedia content tags;

[0314] The determination unit 34 is used to determine, under the guidance of the multimedia content prompt word, using the language model to determine a tag having the category information matching the target multimedia content among the m candidate multimedia content tags as the target multimedia content tag.

[0315] In an embodiment of the present application, after acquiring the target multimedia content and n preset tags, the multimedia content annotation device uses the target multimedia content embedding of the target multimedia content to retrieve the n preset tags, and obtains m candidate multimedia content tags that match the target multimedia content embedding, wherein the preset tags include the category information of the multimedia content. By removing the tags with a low degree of matching with the target multimedia content embedding from the n preset tags, a small number of m tags that match the target multimedia content embedding are obtained. When the m candidate multimedia content tags are determined, the multimedia content annotation device generates a target prompt word based on the m candidate multimedia content tags and the target multimedia content. After generating the target prompt word, the multimedia content annotation device, under the guidance of the target prompt word, uses a language model to determine the tag with category information matching the target multimedia content among the m candidate multimedia content tags as the target multimedia content tag. By using the language model to determine the target multimedia content tag, the labor cost is reduced and the speed of determining the target multimedia content tag is accelerated. Furthermore, before the multimedia content tagging device uses the language model to determine the target multimedia content tag, the number of candidate multimedia content tags is reduced through retrieval, which can reduce the interference of tags whose category information does not match the target multimedia content on the multimedia content tagging device when determining the target multimedia content tag, and improve the matching degree between the target multimedia content tag determined by the multimedia content tagging device from the m candidate multimedia content tags and the target multimedia content.

[0316] In some embodiments, the functions or modules included in the device provided in the embodiments of the present application can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0317] Fig.11A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. The electronic device 4 includes a processor 41 and a memory 42. Optionally, the electronic device 4 also includes an input device 43 and an output device 44. The processor 41, the memory 42, the input device 43 and the output device 44 are coupled via a connector, and the connector includes various interfaces, transmission lines or buses, etc., which are not limited in the embodiments of the present application. It should be understood that in each embodiment of the present application, coupling refers to mutual connection in a specific manner, including direct connection or indirect connection through other devices, for example, it can be connected through various interfaces, transmission lines, buses, etc.

[0318] The processor 41 may include one or more processors, for example, one or more central processing units (CPUs). When the processor is a CPU, the CPU may be a single-core CPU or a multi-core CPU. Optionally, the processor 41 may be a processor group consisting of multiple CPUs, and the multiple processors are coupled to each other through one or more buses. Optionally, the processor may also be other types of processors, etc., which are not limited in the embodiments of the present application.

[0319] The memory 42 can be used to store computer program instructions and various computer program codes including the program code for executing the program code of the present application. Optionally, the memory includes but is not limited to random access memory (RAM), read-only memory (ROM), erasable programmable read only memory (EPROM), or portable read only memory (CD-ROM), which is used for related instructions and data.

[0320] The input device 43 is used to input data and / or signals, and the output device 44 is used to output data and / or signals. The input device 43 and the output device 44 can be independent devices or an integrated device.

[0321] It can be understood that in the embodiment of the present application, the memory 42 can be used not only to store relevant instructions, but also to store relevant data. For example, the memory 42 can be used to store target materials obtained through the input device 43, n preset labels, or the memory 42 can also be used to store target labels determined by the processor 41, etc. The embodiment of the present application does not limit the specific data stored in the memory.

[0322] Understandably, Fig.11Only a simplified design of an electronic device is shown. In practical applications, the electronic device may also include other necessary components, including but not limited to any number of input / output devices, processors, memories, etc., and all electronic devices that can implement the embodiments of the present application are within the protection scope of the present application.

[0323] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0324] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. Those skilled in the art can also clearly understand that the descriptions of the various embodiments of the present application have different focuses. For the convenience and brevity of description, the same or similar parts may not be repeated in different embodiments. Therefore, for parts not described or not described in detail in a certain embodiment, refer to the records of other embodiments.

[0325] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0326] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0327] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0328] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital versatile disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0329] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by a computer program to instruct the relevant hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned method embodiments. The aforementioned storage medium includes: a read-only memory (ROM) or a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

Claims

1. A material labeling method, characterized in that: The method comprises: Obtain a target material and n preset labels; the preset labels include category information of the material, and n is an integer greater than 1; Using the target material embedding of the target material, searching the n preset tags to obtain m candidate tags that match the target material embedding, where m is a positive integer less than n; Generate a target prompt word according to the m candidate tags and the target material, wherein the target prompt word is used to guide the language model to determine a tag having the category information matching the target material from the m candidate tags; Under the guidance of the target prompt word, the language model is used to determine a label having the category information matching the target material among the m candidate labels as the target label.

2. The method according to claim 1, characterized in that The method of using the target material embedding of the target material to search the n preset tags to obtain m candidate tags that match the target material embedding includes: Extracting the embeddings of the n preset tags to obtain n preset tag embeddings, wherein the preset tags correspond one to one to the preset tag embeddings; Determine the similarity between the embedding of the target material and the embedding of the n preset tags respectively, and obtain n reference similarities, wherein the reference similarities correspond to the preset tags one by one; Determine m similarities with the greatest similarity from the n reference similarities as candidate similarities; The m preset tags corresponding to the m candidate similarities are used as the m candidate tags.

3. The method according to claim 1 or 2, characterized in that: The target prompt word includes a target example, the target example is an example of determining a second example label from m first example labels, the second example label is a label having category information matching a first reference material, the first reference material is different from the target material, and the m first example labels are obtained by searching the n preset labels using a first embedding of the first reference material to obtain m labels matching the first embedding; The step of using the language model to determine a label having the category information matching the target material among the m candidate labels as a target label includes: Under the prompt of the target example, the language model is used to determine a label having the category information matching the target material among the m candidate labels as the target label.

4. The method according to claim 1 or 2, characterized in that: The target prompt word includes a target step, the target step is a step of determining a fourth example label from m third example labels, the fourth example label is a label having category information matching a second reference material, the second reference material is different from the target material, and the m third example labels are obtained by searching the n preset labels using a second embedding of the second reference material to obtain m labels matching the second embedding; The step of using the language model to determine a label having the category information matching the target material among the m candidate labels as a target label includes: Under the prompt of the target step, the language model is used to determine a label having the category information matching the target material among the m candidate labels as the target label.

5. The method according to claim 2, characterized in that: The respectively determining the similarities between the embedding of the target material and the embedding of the n preset tags to obtain n reference similarities comprises: Divide the n preset tag embeddings into k tag groups, where k is a positive integer less than n; there are greater than or equal to m preset tag embeddings in any of the tag groups; Based on the k groups of label groups, determining the average value of the preset label embedding in any of the label groups to obtain k mean embeddings; Determine the maximum similarity between the target material embedding and the k mean embeddings to obtain a first similarity; Determine the tag group corresponding to the first similarity to obtain a target tag group; Determine the similarity between any of the preset tags embedded in the target tag group and the target material embedded, and obtain i second similarities, where i represents the number of preset tags embedded in the target tag group, and i is a positive integer greater than or equal to m; The determining m candidate similarities with the greatest similarity from the n reference similarities comprises: The m candidate similarities having the greatest similarities are determined from the i second similarities.

6. A model training method, characterized in that: The method comprises: Acquire a training prompt word and a first label, wherein the training prompt word is used to guide the model to be trained to determine a label with category information matching the training material from m training labels, wherein the first label is the label of the training material; the m training labels are obtained based on m retrieval labels, wherein the m retrieval labels are m labels with category information matching the embedding of the training material determined from the n preset labels by retrieving n preset labels, wherein the m training labels include the first label, the embedding of the training material is the embedding of the training material, and the preset label includes the category information of the material; Under the guidance of the training prompt word, the to-be-trained model is used to determine a label having the category information matching the training material among the m training labels as a second label; determining a target loss according to a target difference between the second label and the first label, wherein the target loss is positively correlated with the target difference; According to the target loss, the parameters in the model to be trained are adjusted to obtain a language model.

7. The method according to claim 6, characterized in that When the n preset tags are retrieved using the training material embedding and the m retrieval tags matching the training material embedding do not include the first tag, the m training tags are obtained by replacing any one of the m first retrieval tags with the first tag.

8. The method according to claim 6 or 7, characterized in that: When the n preset tags are retrieved using the training material embedding to obtain the m retrieval tags that match the training material embedding, and the m retrieval tags include the first tag, the m training tags are obtained by replacing any tag among the m retrieval tags except the first tag with a random tag, and the random tag is any tag among the n preset tags except the m retrieval tags.

9. The method according to claim 6 or 7, characterized in that: The model to be trained includes a decomposition branch, the decomposition branch is used to decompose the embedding of the training prompt word into a first sub-embedding and a second sub-embedding, an embedding obtained by performing an inner product between the first sub-embedding and the second sub-embedding represents the embedding of the prompt word, and a difference between a parameter amount of the embedding of the training prompt word and a sum of parameter amounts of the first sub-embedding and the second sub-embedding is greater than a reference threshold, and the reference threshold is a positive integer; The method of determining, under the guidance of the training prompt word, using the to-be-trained model to determine a label having the category information matching the training material among the m training labels as a second label comprises: Extracting embeddings in the training prompt words to obtain reference embeddings; Extracting the reference embedding using the decomposition branch to obtain the first sub-embedding and the second sub-embedding; Obtaining an inner product embedding by performing an inner product operation on the first sub-embedding and the second sub-embedding; Based on the inner product embedding, using the to-be-trained model, determine a label among the m training labels that has category information matching the target material as the second label; The step of adjusting the parameters in the model to be trained according to the target loss to obtain a language model includes: According to the target loss, the parameters in the decomposition branch are adjusted to obtain the language model.

10. A multimedia content annotation method, characterized in that: The method comprises: Acquire target multimedia content and n preset tags, wherein the preset tags include category information of the multimedia content, and n is an integer greater than 1; Using the target multimedia content embedding of the target multimedia content, searching the n preset tags to obtain m candidate multimedia content tags that match the target multimedia content embedding, where m is a positive integer less than n; Generate a multimedia content prompt word according to the m candidate multimedia content tags and the target multimedia content, wherein the multimedia content prompt word is used to guide the language model to determine a tag having category information matching the target multimedia content from the m candidate multimedia content tags; Under the guidance of the multimedia content prompt word, the language model is used to determine a tag having the category information matching the target multimedia content among the m candidate multimedia content tags as the target multimedia content tag.

11. A material labeling device, characterized in that: The material marking device comprises: An acquisition unit, used to acquire a target material and n preset tags; the preset tags include category information of the material, and n is an integer greater than 1; A retrieval unit, configured to retrieve the n preset tags using the target material embedding of the target material, and obtain m candidate tags that match the target material embedding; wherein m is a positive integer less than n; A generating unit, configured to generate a target prompt word according to the m candidate tags and the target material, wherein the target prompt word is used to guide the language model to determine a tag having the category information matching the target material from the m candidate tags; A determination unit is used to determine, under the guidance of the target prompt word, using the language model to determine a label having the category information matching the target material among the m candidate labels as a target label.

12. A model training device, characterized in that: The model training device comprises: An acquisition unit is used to acquire a training prompt word and a first label; the training prompt word is used to guide the model to be trained to determine a label with category information matching the training material from m training labels; the first label is the label of the training material; the m training labels are obtained based on m retrieval labels, the m retrieval labels are m labels with category information matching the embedding of the training material determined from the n preset labels by retrieving n preset labels, the m training labels include the first label, and the training material embedding is the embedding of the training material; the preset label includes the category information of the material; a determination unit, configured to determine, under the guidance of the training prompt word, using the to-be-trained model, a label having the category information matching the training material among the m training labels as a second label; a loss calculation unit, configured to determine a target loss according to a target difference between the second label and the first label; the target loss is positively correlated with the target difference; An adjustment unit is used to adjust the parameters in the model to be trained according to the target loss to obtain a language model.

13. A multimedia content annotation device, characterized in that: The multimedia content annotation device comprises: An acquisition unit, configured to acquire target multimedia content and n preset tags, wherein the preset tags include category information of the multimedia content, and n is an integer greater than 1; A retrieval unit, configured to retrieve the n preset tags using the target multimedia content embedding of the target multimedia content, and obtain m candidate multimedia content tags that match the target multimedia content embedding; wherein m is a positive integer less than n; A generating unit, configured to generate a multimedia content prompt word according to the m candidate multimedia content tags and the target multimedia content, wherein the multimedia content prompt word is used to guide a language model to determine a tag having category information matching the target multimedia content from among the m candidate multimedia content tags; A determination unit is used to determine, under the guidance of the multimedia content prompt word, using the language model to determine a tag having the category information matching the target multimedia content among the m candidate multimedia content tags as the target multimedia content tag.

14. An electronic device, characterized in that: include: A processor and a memory, wherein the memory is used to store computer program code, wherein the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device executes the method as claimed in any one of claims 1 to 5, or the electronic device executes the method as claimed in any one of claims 6 to 9, or the electronic device executes the method as claimed in claim 10.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor, the processor executes the method described in any one of claims 1 to 5, or the processor executes the method described in any one of claims 6 to 9, or the processor executes the method described in claim 10.

16. A computer program product, characterized in that The computer program product includes a computer program or instructions; when the computer program or instructions are run on a computer, the computer executes the method described in any one of claims 1 to 5, or the computer executes the method described in any one of claims 6 to 9, or the computer executes the method described in claim 10.

Images