Method, device, electronic device and storage medium for generating label construction model

Abstract

The present disclosure relates to a method, device, electronic device, and storage medium for generating a label construction model. The method includes: obtaining a plurality of sample multimedia resources and a preset resource label corresponding to each sample multimedia resource, generating a target resource label corresponding to each sample multimedia resource through a first model to be trained, and based on the preset resource label, performing label detection on the target resource label through a second model to be trained to obtain a label detection result. The positive sample label information and the negative sample label information are determined through the label detection result, and based on the target resource label, the preset resource label, the sample multimedia resource, the positive sample label information, and the negative sample label information, the first model to be trained and the second model to be trained are trained to obtain a label construction model corresponding to the first model to be trained. This method can improve the label diversity of multimedia resources and improve the accuracy of label construction.

Description

Technical Field

[0001] The present disclosure relates to the field of machine learning technology, and in particular to a method, device, electronic device, and storage medium for generating a label construction model. Background Art

[0002] On short video platforms or medium and long video platforms, the amount of video data is very large. By labeling videos, labels can be used to characterize videos from multiple dimensions, making it easier to manage, recommend, and display videos. In related technologies, when determining the labels of videos, it is usually necessary to manually screen high-quality labels and classify videos based on high-quality labels. However, there is a problem that videos can only choose manually screened labels as their own labels, resulting in low diversity of multimedia resource labels. Summary of the Invention

[0003] The present disclosure provides a method, device, electronic device, and storage medium for generating a tag building model to at least address the problem of low diversity of multimedia resource tags in related technologies. The technical solutions of the present disclosure are as follows:

[0004] According to a first aspect of an embodiment of the present disclosure, a method for generating a label construction model is provided, the method comprising:

[0005] Acquire multiple sample multimedia resources and a preset resource tag corresponding to each of the sample multimedia resources;

[0006] Performing multimodal feature extraction on each of the sample multimedia resources to obtain a sample multimodal feature corresponding to each of the sample multimedia resources;

[0007] Inputting the sample multimodal features into the first to-be-trained model to construct resource labels, and obtaining target resource labels corresponding to each sample multimedia resource;

[0008] Inputting the target resource label, the preset resource label, and the sample multimodal feature into a second to-be-trained model for label detection processing to obtain a label detection result, wherein the label detection result indicates a matching status between the target resource label corresponding to each of the sample multimedia resources and each of the sample multimedia resources;

[0009] Based on the label detection result, the target resource label corresponding to the first multimedia resource and the preset resource label corresponding to the first multimedia resource are used as positive sample label information, and the target resource label corresponding to the second multimedia resource and the preset resource label corresponding to the second multimedia resource are used as negative sample label information; the first multimedia resource is a sample multimedia resource that matches the target resource label, and the second multimedia resource is a sample multimedia resource that does not match the target resource label;

[0010] Based on the target resource label, the preset resource label, the sample multimedia resource, the positive sample label information and the negative sample label information, model training is performed on the first model to be trained and the second model to be trained to obtain the label construction model, which includes a label extraction network corresponding to the first model to be trained and a comparative learning network corresponding to the second model to be trained.

[0011] As an optional embodiment, the second model to be trained includes a label feature extraction layer, a similarity determination layer, and a label detection layer. The target resource label, the preset resource label, and the sample multimodal feature are input into the second model to be trained for label detection processing. The label detection result obtained includes:

[0012] Inputting the target resource tag and the preset resource tag into the tag feature extraction layer to perform tag feature extraction respectively, to obtain a first tag feature corresponding to the target resource tag and a second tag feature corresponding to the preset resource tag;

[0013] Inputting the first label feature, the second label feature, and the sample multimodal feature into the similarity determination layer to perform similarity determination, thereby obtaining a first feature similarity between the target resource label and the sample multimedia resource, and a second feature similarity between the preset resource label and the sample multimedia resource;

[0014] The first feature similarity and the second feature similarity are input into the tag detection layer to perform tag similarity detection processing to obtain the tag detection result.

[0015] As an optional embodiment, the sample multimodal features include multiple multimodal sub-features arranged in order, the first model to be trained includes a feature decoding layer and a text generation layer, and the sample multimodal features are input into the first model to be trained to construct resource labels, and the target resource label corresponding to each sample multimedia resource is obtained, including:

[0016] When the current multimodal sub-feature is the first multimodal sub-feature among the plurality of multimodal sub-features arranged in order, inputting the first multimodal sub-feature into the feature decoding layer for feature decoding processing to obtain a feature decoding result corresponding to the first multimodal sub-feature;

[0017] When the current multimodal sub-feature is not the first multimodal sub-feature among the multiple multimodal sub-features arranged in order, obtaining a previous feature decoding result corresponding to the current multimodal sub-feature, inputting the current multimodal sub-feature and the previous feature decoding result into the feature decoding layer for feature decoding processing, and obtaining a feature decoding result corresponding to the current multimodal sub-feature, wherein the previous feature decoding result is a feature decoding result corresponding to a previous multimodal sub-feature of the current multimodal sub-feature;

[0018] The feature decoding result corresponding to the last multimodal sub-feature of the plurality of multimodal sub-features arranged in order is input into the text generation layer for text generation to obtain the target resource label.

[0019] As an optional embodiment, the performing model training on the first to-be-trained model and the second to-be-trained model based on the target resource label, the preset resource label, the sample multimedia resource, the positive sample label information, and the negative sample label information to obtain the label construction model includes:

[0020] Determining label generation loss information based on difference information between the target resource label and the preset resource label;

[0021] Determining label detection loss information based on difference information between the sample multimedia resource and the positive sample label information, and difference information between the target resource label in the negative sample label information and the preset resource label in the negative sample label information;

[0022] Based on the label generation loss information and the label detection loss information, model training is performed on the first model to be trained and the second model to be trained to obtain the label construction model.

[0023] As an optional embodiment, the extracting multimodal features of each sample multimedia resource to obtain the sample multimodal features corresponding to each sample multimedia resource includes:

[0024] Extracting image features from the resource image information corresponding to each sample multimedia resource to obtain sample image features;

[0025] Performing text feature extraction on resource text information corresponding to each sample multimedia resource to obtain sample text features;

[0026] Feature fusion is performed on the sample image features and the sample text features to obtain sample multimodal features corresponding to the sample multimedia resource.

[0027] As an optional embodiment, the resource text information includes at least two types of text information, the sample text features include a plurality of text sub-features arranged in sequence, and the text feature extraction is performed on the resource text information corresponding to each sample multimedia resource to obtain the sample text features including:

[0028] Performing text splicing on at least two types of text information corresponding to the sample multimedia resource to obtain text splicing information;

[0029] Inputting the first preset number of character information in the text splicing information into a text feature extraction model to perform text feature extraction, and obtaining text sub-features corresponding to the preset number of character information respectively.

[0030] As an optional embodiment, the resource image information includes a preset number of sequentially arranged image information, the sample image features include a plurality of sequentially arranged image sub-features, and the image feature extraction is performed on the resource image information corresponding to each sample multimedia resource to obtain the sample image features including:

[0031] The preset number of sequentially arranged image information are input into an image feature extraction model to perform image feature extraction, and image sub-features corresponding to the preset number of sequentially arranged image information are obtained.

[0032] As an optional embodiment, before obtaining the sample multimedia resource and the preset resource tag corresponding to the sample multimedia resource, the method further includes:

[0033] Performing resource classification processing on the sample multimedia resources to obtain a classification label corresponding to each of the sample multimedia resources;

[0034] The text feature extraction of the resource text information corresponding to each sample multimedia resource to obtain the sample text feature includes:

[0035] Text feature extraction is performed on the resource text information and the classification label corresponding to each of the sample multimedia resources to obtain sample text features.

[0036] According to a second aspect of an embodiment of the present disclosure, a label construction method is provided, the method comprising:

[0037] Perform multimodal feature extraction on the multimedia resources to be processed to obtain multimodal features to be processed;

[0038] Inputting the multimodal features to be processed into a label extraction network in a label construction model to construct resource labels, thereby obtaining resource labels corresponding to the multimedia resources to be processed, wherein the label construction model is generated based on the label construction model generation method according to any one of claims 1 to 8;

[0039] The resource tag corresponding to the multimedia resource to be processed and the multimedia resource to be processed are input into the tag detection network in the tag construction model to perform tag detection processing to obtain a tag detection result of the resource tag.

[0040] According to a third aspect of an embodiment of the present disclosure, there is provided a device for generating a label construction model, the device comprising:

[0041] A sample resource acquisition module is configured to acquire a plurality of sample multimedia resources and a preset resource tag corresponding to each of the sample multimedia resources;

[0042] A sample multimodal feature extraction module is configured to perform multimodal feature extraction on each of the sample multimedia resources to obtain a sample multimodal feature corresponding to each of the sample multimedia resources;

[0043] a target resource label construction module configured to execute inputting the sample multimodal features into the first to-be-trained model to construct resource labels, and obtain a target resource label corresponding to each of the sample multimedia resources;

[0044] a label detection module configured to perform label detection processing by inputting the target resource label, the preset resource label, and the sample multimodal feature into a second to-be-trained model to obtain a label detection result, wherein the label detection result indicates a matching status between the target resource label corresponding to each of the sample multimedia resources and each of the sample multimedia resources;

[0045] The sample determination module is configured to execute, based on the label detection result, the target resource label corresponding to the first multimedia resource and the preset resource label corresponding to the first multimedia resource as positive sample label information, and the target resource label corresponding to the second multimedia resource and the preset resource label corresponding to the second multimedia resource as negative sample label information; the first multimedia resource is a sample multimedia resource that matches the target resource label, and the second multimedia resource is a sample multimedia resource that does not match the target resource label;

[0046] The model training module is configured to perform model training on the first model to be trained and the second model to be trained based on the target resource label, the preset resource label, the sample multimedia resource, the positive sample label information and the negative sample label information to obtain the label construction model, and the label construction model includes a label extraction network corresponding to the first model to be trained and a comparative learning network corresponding to the second model to be trained.

[0047] As an optional embodiment, the tag detection module includes:

[0048] a tag feature extraction unit configured to input the target resource tag and the preset resource tag into the tag feature extraction layer to perform tag feature extraction respectively, thereby obtaining a first tag feature corresponding to the target resource tag and a second tag feature corresponding to the preset resource tag;

[0049] a similarity determination unit configured to input the first label feature, the second label feature, and the sample multimodal feature into the similarity determination layer to perform similarity determination, thereby obtaining a first feature similarity between the target resource label and the sample multimedia resource, and a second feature similarity between the preset resource label and the sample multimedia resource;

[0050] The tag detection processing unit is configured to input the first feature similarity and the second feature similarity into the tag detection layer to perform tag detection processing and obtain the tag detection result.

[0051] As an optional embodiment, the sample multimodal feature includes a plurality of multimodal sub-features arranged in sequence, the first to-be-trained model includes a feature decoding layer and a text generation layer, and the target resource label construction module includes:

[0052] a first decoding processing unit configured to, when a current multimodal sub-feature is a first multimodal sub-feature among the plurality of sequentially arranged multimodal sub-features, input the first multimodal sub-feature into the feature decoding layer for feature decoding processing, and obtain a feature decoding result corresponding to the first multimodal sub-feature;

[0053] The second decoding processing unit is configured to, when the current multimodal sub-feature is not the first multimodal sub-feature among the plurality of sequentially arranged multimodal sub-features, obtain a previous feature decoding result corresponding to the current multimodal sub-feature, input the current multimodal sub-feature and the previous feature decoding result into the feature decoding layer for feature decoding processing, and obtain a feature decoding result corresponding to the current multimodal sub-feature, wherein the previous feature decoding result is a feature decoding result corresponding to the multimodal sub-feature previous to the current multimodal sub-feature;

[0054] The target resource label determination unit is configured to input a feature decoding result corresponding to the last multimodal sub-feature of the plurality of multimodal sub-features arranged in order into the text generation layer for text generation to obtain the target resource label.

[0055] As an optional embodiment, the model training module includes:

[0056] a generation loss determining unit configured to determine tag generation loss information based on difference information between the target resource tag and the preset resource tag;

[0057] a detection loss determining unit configured to determine label detection loss information based on difference information between the sample multimedia resource and the positive sample label information, and difference information between the target resource label in the negative sample label information and the preset resource label in the negative sample label information;

[0058] The model training unit is configured to perform model training on the first to-be-trained model and the second to-be-trained model based on the label generation loss information and the label detection loss information to obtain the label construction model.

[0059] As an optional embodiment, the sample multimodal feature extraction module includes:

[0060] An image feature extraction unit is configured to perform image feature extraction on resource image information corresponding to each sample multimedia resource to obtain sample image features;

[0061] A first text feature extraction unit is configured to perform text feature extraction on resource text information corresponding to each sample multimedia resource to obtain sample text features;

[0062] The feature fusion unit is configured to perform feature fusion on the sample image features and the sample text features to obtain sample multimodal features corresponding to the sample multimedia resources.

[0063] As an optional embodiment, the resource text information includes at least two types of text information, the sample text feature includes a plurality of text sub-features arranged in sequence, and the first text feature extraction unit includes:

[0064] a text splicing unit configured to perform text splicing on at least two types of text information corresponding to the sample multimedia resource to obtain text splicing information;

[0065] The text sub-feature extraction unit is configured to input the first preset number of character information in the text splicing information into a text feature extraction model to perform text feature extraction, and obtain text sub-features corresponding to the preset number of character information.

[0066] As an optional embodiment, the resource image information includes a preset number of sequentially arranged image information, the sample image feature includes a plurality of sequentially arranged image sub-features, and the image feature extraction unit includes:

[0067] The image sub-feature extraction unit is configured to input the preset number of sequentially arranged image information into the image feature extraction model to perform image feature extraction, and obtain image sub-features corresponding to the preset number of sequentially arranged image information.

[0068] As an optional embodiment, the device further includes:

[0069] a resource classification module configured to perform resource classification processing on the sample multimedia resources to obtain a classification label corresponding to each of the sample multimedia resources;

[0070] The first text feature extraction unit includes:

[0071] The second text feature extraction unit is configured to perform text feature extraction on the resource text information and the classification label corresponding to each of the sample multimedia resources to obtain the sample text features.

[0072] According to a fourth aspect of the embodiments of the present disclosure, there is provided a label construction device, the device comprising:

[0073] The multimodal feature extraction module is configured to perform multimodal feature extraction on the multimedia resources to be processed to obtain multimodal features to be processed;

[0074] a resource label construction module configured to execute resource label construction by inputting the multimodal features to be processed into a label extraction network in a label construction model, thereby obtaining a resource label corresponding to the multimedia resource to be processed, wherein the label construction model is generated based on the above-mentioned label construction model generation method;

[0075] The resource tag detection module is configured to perform tag detection processing by inputting the resource tag corresponding to the multimedia resource to be processed and the multimedia resource to be processed into the tag detection network in the tag construction model to obtain a tag detection result of the resource tag.

[0076] According to a fifth aspect of an embodiment of the present disclosure, there is provided an electronic device, the electronic device including: a processor;

[0077] a memory for storing instructions executable by the processor;

[0078] The processor is configured to execute the instructions to implement the above-mentioned label construction model generation method and label construction method.

[0079] According to a fifth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the label construction model generation method and label construction method as described above.

[0080] According to a sixth aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, which implements the above-mentioned label construction model generation method and label construction method when executed by a processor.

[0081] The technical solutions provided by the embodiments of the present disclosure bring at least the following beneficial effects:

[0082] Acquire multiple sample multimedia resources and preset resource labels corresponding to each sample multimedia resource, generate target resource labels corresponding to each sample multimedia resource through the first model to be trained, and based on the preset resource labels, perform label detection on the target resource labels through the second model to be trained to obtain label detection results. Determine positive sample label information and negative sample label information through the label detection results, and perform model training on the first model to be trained and the second model to be trained based on the target resource labels, preset resource labels, sample multimedia resources, positive sample label information and negative sample label information to obtain a label construction model corresponding to the first model to be trained. This method can generate target resource labels through the first model to be trained, thereby improving the label diversity of multimedia resources, and constrain target resource labels during model training through label detection results, thereby improving the accuracy of label construction.

[0083] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0085] Figure 1 The figure is a schematic diagram of an application scenario of a method for generating a label construction model according to an exemplary embodiment.

[0086] Figure 2 The figure is a flowchart of a method for generating a label construction model according to an exemplary embodiment.

[0087] Figure 3 The present invention is a flowchart of a multimodal feature extraction method for generating a label construction model according to an exemplary embodiment.

[0088] Figure 4 The figure is a flowchart of text feature extraction in a method for generating a label building model according to an exemplary embodiment.

[0089] Figure 5 The present invention is a flowchart of determining classification labels in a method for generating a label construction model according to an exemplary embodiment.

[0090] Figure 6 The present invention is a flowchart of target resource tag construction of a tag construction model generation method according to an exemplary embodiment.

[0091] Figure 7 The figure is a flowchart of a label detection method for generating a label construction model according to an exemplary embodiment.

[0092] Figure 8 The present invention is a flowchart of a model training method for generating a label construction model according to an exemplary embodiment.

[0093] Figure 9 The figure is a flowchart of a label construction method according to an exemplary embodiment.

[0094] Figure 10 It is a structural diagram of a multimodal feature extraction model, a first model to be trained, and a second model to be trained according to a method for generating a label construction model according to an exemplary embodiment.

[0095] Figure 11 The figure is a block diagram of a device for generating a label construction model according to an exemplary embodiment.

[0096] Figure 12 The figure is a block diagram of a label generating device according to an exemplary embodiment.

[0097] Figure 13 FIG1 is a block diagram of an electronic device according to an exemplary embodiment. (General structure of a server) DETAILED DESCRIPTION

[0098] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0099] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0100] Figure 1 This is a schematic diagram of an application scenario of a method for generating a label construction model according to an exemplary embodiment. The application scenario includes a client 110 and a server 120. The server 120 obtains multiple sample multimedia resources and preset resource labels corresponding to each sample multimedia resource from the client 110. The server 120 generates a target resource label corresponding to each sample multimedia resource through a first model to be trained, and based on the preset resource label, performs label detection on the target resource label through a second model to be trained to obtain a label detection result. The server 120 determines the positive sample label information and the negative sample label information through the label detection result, and performs model training on the first model to be trained and the second model to be trained based on the target resource label, the preset resource label, the sample multimedia resource, the positive sample label information and the negative sample label information to obtain a label construction model corresponding to the first model to be trained.

[0101] In the disclosed embodiment, the client 110 includes physical devices such as smart phones, desktop computers, tablet computers, laptops, digital assistants, smart wearable devices, etc., and may also include software running in the physical devices, such as applications.

[0102] In the embodiment of the present disclosure, the server 120 may include an independent server, a distributed server, or a server cluster composed of multiple servers. The server 120 may include a network communication unit, a processor, a memory, and the like.

[0103] Figure 2FIG. 1 is a flow chart showing a method for generating a label construction model according to an exemplary embodiment. Figure 2 As shown, the method is used in a server and includes the following steps.

[0104] S210. Obtain multiple sample multimedia resources and a preset resource tag corresponding to each sample multimedia resource;

[0105] As an optional embodiment, the sample multimedia resource may be a resource having an image sequence, which may be a video or a presentation, etc. The preset resource tag may be a tag carried by the sample multimedia resource when the user publishes the sample multimedia resource or a classification tag corresponding to the sample multimedia resource.

[0106] S220. Perform multimodal feature extraction on each sample multimedia resource to obtain a sample multimodal feature corresponding to each sample multimedia resource;

[0107] As an optional embodiment, each sample multimedia resource is input into a multimodal feature extraction model for multimodal feature extraction, thereby obtaining sample multimodal features corresponding to each sample multimedia resource. The multimodal feature extraction model may include a text feature extraction model, an image feature extraction model, etc. Based on the text feature extraction model, text features can be extracted from the resource text information of the sample multimedia resource. Based on the image feature extraction model, image features can be extracted from the resource image information of the sample multimedia resource.

[0108] As an alternative embodiment, see Figure 3 , extract multimodal features from each sample multimedia resource, and obtain the sample multimodal features corresponding to each sample multimedia resource, including:

[0109] S310. Perform image feature extraction on the resource image information corresponding to each sample multimedia resource to obtain sample image features;

[0110] S320. Perform text feature extraction on the resource text information corresponding to each sample multimedia resource to obtain sample text features;

[0111] S330. Perform feature fusion on the sample image features and the sample text features to obtain sample multimodal features corresponding to the sample multimedia resources.

[0112] As an optional embodiment, the sample multimodal features may include multi-dimensional features such as images, texts, etc. Multimodal features may be extracted from the sample multimedia resources based on a multimodal feature extraction model.

[0113] The multimodal feature extraction model may include an image feature extraction layer, a text feature extraction layer, and a feature encoding fusion layer. The image feature extraction layer may be an image feature extraction model, and the text feature extraction layer may be a text feature extraction model. The image information corresponding to each sample multimedia resource is input into the image feature extraction layer for image feature extraction to obtain sample image features. The text information corresponding to each sample multimedia resource is input into the text feature extraction layer for text feature extraction to obtain sample text features.

[0114] The feature coding fusion layer may include a feature coding layer and a feature fusion layer. Sample image features and sample text features are input into the feature coding layer for feature coding, thereby obtaining sample image coding features corresponding to the sample image features and sample text coding features corresponding to the sample text features. The feature coding layer may be a network capable of extracting features from both image and text features, such as a transformer-based bidirectional coding representation model. Sample image coding features and sample text coding features are input into the feature fusion layer for feature fusion, thereby obtaining sample multimodal features.

[0115] Obtain multimodal features such as images and text corresponding to sample multimedia resources, so that the target resource labels in subsequent steps can be determined based on the multimodal features. By utilizing the complementarity of image features and text features, the accuracy of target resource labels can be improved.

[0116] As an alternative embodiment, see Figure 4 The resource text information includes at least two types of text information, and the sample text features include multiple text sub-features arranged in sequence. The text features of the resource text information corresponding to the sample multimedia resource are extracted, and the sample text features obtained include:

[0117] S410. Perform text splicing on at least two types of text information corresponding to the sample multimedia resource to obtain text splicing information;

[0118] S420. Input the first preset number of character information in the text splicing information into the text feature extraction model to perform text feature extraction, and obtain text sub-features corresponding to the preset number of character information.

[0119] As an optional embodiment, the resource text information may include at least two types of text information from a variety of different types of text information, such as resource description text information, subtitle information, image text recognition results, and speech text recognition results. The at least two types of text information are concatenated, and the first predetermined number of characters in the concatenated text information can be input into a text feature extraction model for text feature extraction, thereby obtaining text sub-features corresponding to the predetermined number of characters.

[0120] The text feature extraction model can be a model that processes natural language, such as the Text-To-Text Transfer Transformer (T5) or the Bidirectional Encoder Representations from Transformers (BERT). The text feature extraction model can output text sub-features of any preset dimension, such as 512. The dimensions of the text sub-features are the same as those of the image sub-features.

[0121] The number of character information is the same as the number of image information. In the feature fusion layer, each text sub-feature is fused with each image sub-feature to obtain multiple multi-modal sub-features arranged in order.

[0122] A preset number of character information is obtained from multiple types of text information to obtain sample text features, so that the sample text features can refer to different types of text sources, thereby improving the accuracy of the sample text features. When generating target resource tags in subsequent steps, text information from multiple sources can enrich the target resource tags, thereby improving the diversity of resource tag generation.

[0123] As an optional embodiment, the resource image information includes a preset number of sequentially arranged image information, and the sample image features include a plurality of sequentially arranged image sub-features. Image feature extraction is performed on the resource image information corresponding to the sample multimedia resource, and the obtained sample image features include:

[0124] A preset number of sequentially arranged image information is input into an image feature extraction model to extract image features, and image sub-features corresponding to the preset number of sequentially arranged image information are obtained.

[0125] As an optional embodiment, the resource image information may include a preset number of sequentially arranged image information from the resource cover information and the sampled image sampling information in the sample multimedia resource. The number of image information is the same as the number of character information. The sequence corresponding to the preset number of image information may be a resource display sequence in the sample multimedia resource, such as the video frame playback order in a video resource or the slide sequence in a presentation.

[0126] The image feature extraction model can be a model that performs feature extraction based on a self-attention mechanism, such as the Hierarchical Vision Transformer Using ShiftedWindows (Swin Transformer) or the Visual Transformer (ViT). The image feature extraction layer can also be a model that performs feature extraction based on a deep convolutional network, such as the VGG network or the ResNet. The image feature extraction model can output image sub-features of any preset dimension, such as 512 dimensions. The dimensions of the image sub-features are the same as those of the text sub-features.

[0127] Image feature extraction can be performed on a preset number of sequentially arranged image information to obtain sample image features. This allows the sample image features to carry the sequence characteristics of the image content, improving the accuracy of the sample image features. When generating target resource tags in subsequent steps, the tag generation can be constrained based on the sequence characteristics of the image content, thereby improving the accuracy of resource tag generation.

[0128] As an alternative embodiment, see Figure 5 Before obtaining the sample multimedia resource and the preset resource tag corresponding to the sample multimedia resource, the method further includes:

[0129] S510. Perform resource classification processing on the sample multimedia resources to obtain a classification label corresponding to each sample multimedia resource;

[0130] The text features of the resource text information corresponding to each sample multimedia resource are extracted, and the sample text features obtained include:

[0131] S520. Perform text feature extraction on the resource text information and classification label corresponding to each sample multimedia resource to obtain sample text features.

[0132] As an optional embodiment, the sample multimedia resources are input into a resource classification model for resource classification processing to obtain a classification label corresponding to each sample multimedia resource. The resource classification model has multiple preset classification labels. Through the resource classification model, each sample multimedia resource is associated with one or more classification labels among the multiple classification labels, thereby determining the classification label corresponding to each sample multimedia resource. Text feature extraction can be performed on the classification label and resource text information, and text splicing can be performed on the classification label and at least one type of text information included in the resource text information to obtain text splicing information. Text feature extraction can also be performed on the first preset number of characters in the text splicing information to obtain text sub-features.

[0133] The classification label can also be used as a preset resource label corresponding to the sample multimedia resource. The classification label, target resource label and sample multimedia resource are input into the second to-be-trained model for label detection processing to obtain a label detection result.

[0134] The sample multimedia resources are classified in advance to obtain classification labels that match the sample multimedia resources. Therefore, when text features are extracted based on the classification labels, the accuracy of the sample text features can be improved.

[0135] S230. Input the sample multimodal features into the first to-be-trained model to construct resource labels, and obtain the target resource label corresponding to each sample multimedia resource;

[0136] As an optional embodiment, the sample multimodal features are input into a first model to be trained, and resource labels are constructed by decoding the sample multimodal features to obtain a target resource label corresponding to each sample multimedia resource. The first model to be trained can output the target resource label word for word.

[0137] As an alternative embodiment, see Figure 6 The sample multimodal features include multiple multimodal sub-features arranged in order. The first model to be trained includes a feature decoding layer and a text generation layer. The sample multimodal features are input into the first model to be trained to construct resource labels. The target resource labels corresponding to each sample multimedia resource include:

[0138] S610. When the current multimodal sub-feature is the first multimodal sub-feature among the plurality of multimodal sub-features arranged in order, the first multimodal sub-feature is input into the feature decoding layer for feature decoding processing to obtain a feature decoding result corresponding to the first multimodal sub-feature;

[0139] S620. When the current multimodal sub-feature is not the first multimodal sub-feature among the plurality of sequentially arranged multimodal sub-features, obtaining a previous feature decoding result corresponding to the current multimodal sub-feature, inputting the current multimodal sub-feature and the previous feature decoding result into a feature decoding layer for feature decoding processing, and obtaining a feature decoding result corresponding to the current multimodal sub-feature, wherein the previous feature decoding result is a feature decoding result corresponding to the multimodal sub-feature previous to the current multimodal sub-feature;

[0140] S630. Input the feature decoding result corresponding to the last multimodal sub-feature among the multiple multimodal sub-features arranged in order into the text generation layer to generate text and obtain a target resource label.

[0141] As an optional embodiment, the feature decoding layer can be a model for feature decoding. The first multimodal sub-feature is input into the first training model for feature decoding processing to obtain a first feature decoding result. The first feature decoding result and the second multimodal sub-feature are input into the first training model for feature decoding processing to obtain a second feature decoding result, so that the second feature decoding result can incorporate the feature decoding results corresponding to the first multimodal sub-feature and the feature decoding results corresponding to the second multimodal sub-feature. By analogy, the feature decoding result corresponding to the previous multimodal sub-feature of the current multimodal sub-feature is used as the previous feature decoding result corresponding to the current multimodal sub-feature, and the previous feature decoding result is used as one of the input information of the current feature decoding, and input into the feature decoding layer together with the current multimodal sub-feature for feature decoding processing to obtain the feature decoding result corresponding to the current multimodal sub-feature. The above feature decoding process is repeated until the second-to-last feature decoding result and the last multimodal sub-feature are input into the first training model for feature decoding processing, and the feature decoding result corresponding to the last multimodal sub-feature can be obtained. The feature decoding result corresponding to the last multimodal sub-feature is the feature decoding result corresponding to the sequence that incorporates the multimodal sub-features.

[0142] The feature decoding result corresponding to the last multimodal sub-feature is input into the text generation layer for text generation to obtain the target resource label. In the text generation layer, based on the feature decoding results corresponding to the sequence of multimodal sub-features fused with the feature decoding result corresponding to the last multimodal sub-feature, text generation can be performed word by word, generating labels one by one to obtain at least one generated character information, which is then arranged to obtain the target resource label.

[0143] By iteratively performing feature decoding processing, the feature decoding result corresponding to the last multimodal sub-feature is integrated with the feature decoding results corresponding to the sequence of multimodal sub-features. In the text generation layer, text generation can be performed based on the feature decoding results corresponding to the sequence of multimodal sub-features, and then the target resource labels are obtained by combining them, thereby improving the diversity of the target resource labels.

[0144] S240. Input the target resource label, the preset resource label and the sample multimodal feature into the second to-be-trained model for label detection processing to obtain a label detection result, which indicates the matching between the target resource label corresponding to each sample multimedia resource and each sample multimedia resource;

[0145] As an optional embodiment, the target resource label, preset resource label, and sample multimodal features are input into the second model to be trained for label detection processing, and a label detection result can be obtained. During the model training process, the second model to be trained can constrain the similarity between the target resource label and the sample multimedia model based on the similarity between the preset resource label and the sample multimedia model through comparative learning, so that the label construction model corresponding to the first model to be trained can output resource labels that match the input multimedia resources.

[0146] As an alternative embodiment, see Figure 7 The second model to be trained includes a label feature extraction layer, a similarity determination layer, and a label detection layer. The target resource label, the preset resource label, and the sample multimodal features are input into the second model to be trained for label detection processing. The label detection results obtained include:

[0147] S710. Input the target resource tag and the preset resource tag into the tag feature extraction layer to extract the tag features respectively, and obtain the first tag feature corresponding to the target resource tag and the second tag feature corresponding to the preset resource tag;

[0148] S720. Input the first label feature, the second label feature, and the sample multimodal feature into the similarity determination layer to perform similarity determination, thereby obtaining a first feature similarity between the target resource label and the sample multimedia resource, and a second feature similarity between the preset resource label and the sample multimedia resource;

[0149] S730. Input the first feature similarity and the second feature similarity into the tag detection layer for tag detection processing to obtain a tag detection result.

[0150] As an optional embodiment, the label feature extraction layer may include a multi-layer transformer network and a multi-layer perceptron (MLP). The multi-layer transformer network is used to perform feature extraction on the target resource label and the preset resource label respectively to obtain multiple target resource label features and multiple preset resource label features. The multi-layer perceptron is used to perform feature fusion on the multiple target resource label features to obtain a first label feature. The multi-layer perceptron is used to perform feature fusion on the multiple preset resource label features to obtain a second label feature.

[0151] The first feature similarity between the target resource label and the sample multimedia resource can be the distance between the first label feature and the sample multimodal feature, such as cosine distance, Euclidean distance, etc. The smaller the distance between the first label feature and the sample multimodal feature, the greater the first feature similarity, and the more relevant the target resource label is to the content of the sample multimedia resource. The larger the distance between the first label feature and the sample multimodal feature, the smaller the first feature similarity, and the more irrelevant the target resource label is to the content of the sample multimedia resource.

[0152] The second feature similarity between the preset resource tag and the sample multimedia resource can be the distance between the second tag feature and the sample multimodal feature, such as cosine distance, Euclidean distance, etc. The smaller the distance between the second tag feature and the sample multimodal feature, the greater the second feature similarity, and the more relevant the preset resource tag is to the content of the sample multimedia resource. The larger the distance between the second tag feature and the sample multimodal feature, the smaller the second feature similarity, and the less relevant the preset resource tag is to the content of the sample multimedia resource.

[0153] In the label detection layer, the first feature similarity and the second feature similarity are matched to obtain a similarity matching degree. When the similarity matching degree is greater than or equal to a preset matching threshold, a label detection result is generated indicating that the target resource label matches the sample multimedia resource. When the similarity matching degree is less than the preset matching threshold, a label detection result is generated indicating that the target resource label does not match the sample multimedia resource.

[0154] By presetting the first feature similarity between the resource tag and the sample multimedia resource, the second feature similarity between the target resource tag and the sample multimedia resource is constrained so that the second feature similarity is close to the first feature similarity, thereby ensuring that the target resource tag matches the sample multimedia resource and improving the accuracy of the target resource tag.

[0155] S250. Based on the label detection result, the target resource label corresponding to the first multimedia resource and the preset resource label corresponding to the first multimedia resource are used as positive sample label information, and the target resource label corresponding to the second multimedia resource and the preset resource label corresponding to the second multimedia resource are used as negative sample label information; the first multimedia resource is a sample multimedia resource that matches the target resource label, and the second multimedia resource is a sample multimedia resource that does not match the target resource label;

[0156] As an optional embodiment, when the tag detection result indicates that the target resource tag matches the sample multimedia resource, positive sample tag information is generated. The target resource tag and the preset resource tag in the positive sample tag information correspond to the same sample multimedia resource.

[0157] When the label detection result indicates that the target resource label does not match the sample multimedia resource, negative sample label information is generated. The target resource label and the preset resource label in the negative sample label information correspond to different sample multimedia resources.

[0158] S260. Based on the target resource label, preset resource label, sample multimedia resource, positive sample label information and negative sample label information, the first model to be trained and the second model to be trained are trained to obtain a label construction model, which includes a label extraction network corresponding to the first model to be trained and a comparative learning network corresponding to the second model to be trained.

[0159] As an optional embodiment, based on the target resource label, preset resource label, sample multimedia resource, positive sample label information and negative sample label information, the first model to be trained is trained, and the second model to be trained is comparatively learned to obtain a label construction model. The label construction model may include a label extraction network corresponding to the first model to be trained and a comparative learning network corresponding to the second model to be trained. The label extraction network can extract labels from the multimodal features to be processed corresponding to the multimedia resource to be processed to obtain resource labels corresponding to the multimedia resource to be processed. The comparative learning network can determine the similarity between the resource label and the multimedia resource to be processed, and detect whether the resource label matches the multimedia resource to be processed through the similarity to obtain a label detection result of the resource label.

[0160] As an alternative embodiment, see Figure 8 Based on the target resource label, the preset resource label, the label detection result, the positive sample label information and the negative sample label information, the first to-be-trained model and the second to-be-trained model are trained to obtain a label construction model including:

[0161] S810. Based on the difference information between the target resource tag and the preset resource tag, determine the tag generation loss information;

[0162] S820. Determine label detection loss information based on the difference information between the sample multimedia resource and the positive sample label information, and the difference information between the negative sample label information;

[0163] S830. Based on the label generation loss information and the label detection loss information, the first model to be trained and the second model to be trained are trained to obtain a label construction model.

[0164] As an optional embodiment, the target resource label and the preset resource label are substituted into a preset first loss function, the difference information between the target resource label and the preset resource label is calculated, and the output result of the first loss function is determined based on the difference information, and the output result of the first loss function is used as the label generation loss information. The preset first loss function can be a cross entropy loss function (CELoss). The formula for label generation loss information is as follows:

[0165] loss_gen=CELoss(x1,x2) (Formula 1)

[0166] Among them, loss_gen is the loss information generated by the label, x1 is the preset resource label, and x2 is the target resource label.

[0167] The difference information between the sample multimedia resource and the positive sample label information can be expressed by similarity, and the difference information between the negative sample label information can also be expressed by similarity. The positive sample similarity between the sample multimedia resource and the positive sample label information is calculated, and the negative sample similarity between the preset resource label corresponding to the negative sample label information and the target resource label corresponding to the negative sample label information is calculated. The first mean square error between the positive sample similarity and 1 is determined, and the second mean square error between the negative sample similarity and 0 is determined. The sum of the first mean square error and the second mean square error is determined as the label detection loss information. The formula for label detection loss information is as follows:

[0168] loss_contrast=MSE(cosine(positive,video),1)+MSE(cosine(negative,0))(Formula 2)

[0169] Among them, loss_contrast is the label detection loss information, positive is the positive sample label information, video is the sample multimedia resource, negative is the negative sample label information, MSE is the mean square error, and cosine is the similarity calculation function, which can calculate the cosine distance.

[0170] Based on the preset loss weight information, the label generation loss information and the label detection loss information are weighted and summed to obtain the target loss information. Based on the target loss information, the first model to be trained and the second model to be trained are trained to obtain the label construction model corresponding to the first model to be trained. The formula for the target loss information is as follows:

[0171] loss_all=w1*loss_gen+w2*loss_contrast (Formula 3)

[0172] Among them, loss_all is the target loss information, w1 is the weight corresponding to the label generation loss information, w2 is the weight corresponding to the label detection loss information, loss_contrast is the label detection loss information, and loss_gen is the label generation loss information.

[0173] The label detection loss information determined by the positive and negative sample label information can be used to perform comparative learning on the second model to be trained, thereby improving the matching degree between the target resource label output by the first model to be trained and the sample multimedia resource. The label generation loss information determined by the target resource label and the preset resource label can improve the matching degree between the target resource label and the preset resource label. Therefore, model training based on label detection loss information and label generation loss information can improve the accuracy and generalization of the label construction model.

[0174] Figure 9 According to an exemplary embodiment, a flowchart of a label construction method is shown as follows: Figure 9 As shown, the method is used in a server, and the method includes:

[0175] S910. Perform multimodal feature extraction on the multimedia resources to be processed to obtain multimodal features to be processed;

[0176] S920. Input the multimodal features to be processed into the label extraction network in the label construction model to construct resource labels, and obtain resource labels corresponding to the multimedia resources to be processed. The label construction model is generated based on the above-mentioned label construction model generation method;

[0177] S930. Input the resource tag corresponding to the multimedia resource to be processed and the multimedia resource to be processed into the tag detection network in the tag construction model to perform tag detection processing, and obtain a tag detection result of the resource tag.

[0178] As an optional embodiment, image feature extraction is performed on the resource image information of the multimedia resource to be processed to obtain the image feature to be processed, and text feature extraction is performed on the resource text information of the multimedia resource to be processed to obtain the text feature to be processed. Feature fusion is performed on the image feature to be processed and the text feature to be processed to obtain the multimodal feature to be processed.

[0179] By inputting the multimodal features to be processed into the labeling model to construct resource labels, we can obtain resource labels corresponding to the multimedia resources to be processed. Because the resource labels output by the model and the sample multimedia resources are closely aligned through comparative learning during training, the resource labels output by the labeling model match the multimedia resources to be processed.

[0180] The contrastive learning network may be retained, and similarity between the resource tag and the multimedia resource to be processed may be determined through the contrastive learning network. Based on the similarity, it is determined whether the resource tag matches the multimedia resource to be processed. If the similarity is greater than a preset resource tag similarity threshold, it indicates that the resource tag matches the multimedia resource to be processed, and the resource tag may be output as a model result. If the similarity is less than the preset resource tag similarity threshold, it indicates that the resource tag does not match the multimedia resource to be processed, and the resource tag may not be output as a model result.

[0181] The tag construction model generated by the above tag construction model generation method is used to construct resource tags for the multimedia resources to be processed, which can improve the accuracy and diversity of tag construction.

[0182] As an alternative embodiment, see Figure 10 ,like Figure 10 The figure shows a schematic diagram of the structure of a multimodal feature extraction model, a first model to be trained, and a second model to be trained. This tag construction model generation method can be applied to video tagging scenarios. Multiple sample multimedia resources and a preset resource tag corresponding to each sample multimedia resource are obtained. The preset resource tag can be a tag carried by the sample multimedia resource when the user publishes the sample multimedia resource, or a classification tag corresponding to the sample multimedia resource. The sample multimedia resource can be a sample video.

[0183] The multimodal feature extraction model includes an image feature extraction layer, a text feature extraction layer, and a feature encoding fusion layer. The resource image information of each sample multimedia resource can be a preset number of sequentially arranged image information, such as video frame information obtained after video sampling. The preset number of sequentially arranged image information is input into the image feature extraction layer for image feature extraction, and image sub-features corresponding to the preset number of sequentially arranged image information can be obtained. These image sub-features are sample image features. The resource text information of each sample multimedia resource can be at least two types of text information, such as video introduction, video title, video subtitles, and text information recognized in the video. After splicing the at least two types of text information, a preset number of character information is obtained, and the preset number of character information is input into the text feature extraction layer for text feature extraction. Text sub-features corresponding to the preset number of character information can be obtained. These text sub-features are sample text features. When splicing text information, the text information with the largest weight can be determined from at least two types of text information, and the text information can be used as the first text information so that the text information can be selected. For example, the video introduction can be used as the first text information, and then the video title, video subtitles and text information recognized in the video can be spliced.

[0184] The sample image features and sample text features are input into the feature encoding fusion layer, where the sample image features are feature encoded, and the sample text features are feature encoded. The feature encoding results are then fused to obtain the sample multimodal features. The first model to be trained includes a feature decoding layer and a text generation layer. The sample multimodal features can be input into the feature decoding layer for feature decoding processing to obtain feature decoding results. The feature decoding results are then input into the text generation layer for text generation to obtain the target resource label corresponding to each sample multimedia resource.

[0185] The second model to be trained includes a label feature extraction layer, a similarity determination layer, and a label detection layer. The target resource label and the preset resource label are input into the label feature extraction layer for label feature extraction to obtain a first label feature corresponding to the target resource label and a second label feature corresponding to the preset resource label. The first label feature, the second label feature, and the sample multimodal feature are input into the similarity determination layer for similarity determination to obtain a first feature similarity between the target resource label and the sample multimedia resource, and a second feature similarity between the preset resource label and the sample multimedia resource. The first feature similarity and the second feature similarity are input into the label detection layer for similarity detection to obtain a label detection result. The label detection result can indicate the matching between the target resource label and the sample video. When the target resource label matches the sample video, the target resource label and the corresponding preset resource label are used as positive sample label information. When the target resource label does not match the sample video, the target resource label and the corresponding preset resource label are used as negative sample label information.

[0186] Based on the target resource label, preset resource label, sample multimedia resource, positive sample label information and negative sample label information, model training is performed on the first model to be trained and the second model to be trained to obtain a label construction model corresponding to the first model to be trained.

[0187] The disclosed embodiment proposes a method for generating a label construction model, which includes: obtaining a plurality of sample multimedia resources and a preset resource label corresponding to each sample multimedia resource, generating a target resource label corresponding to each sample multimedia resource through a first model to be trained, and based on the preset resource label, performing label detection on the target resource label through a second model to be trained to obtain a label detection result. The positive sample label information and the negative sample label information are determined through the label detection result, and the first model to be trained and the second model to be trained are trained based on the target resource label, the preset resource label, the sample multimedia resource, the positive sample label information and the negative sample label information to obtain a label construction model corresponding to the first model to be trained. The method can generate the target resource label through the first model to be trained, thereby improving the label diversity of the multimedia resources, and constrain the target resource label during model training through the label detection result, thereby improving the accuracy of label construction.

[0188] Figure 11 FIG. 1 is a block diagram of a device for generating a label building model according to an exemplary embodiment. Figure 11 , the device comprises:

[0189] The sample resource acquisition module 1110 is configured to acquire a plurality of sample multimedia resources and a preset resource tag corresponding to each sample multimedia resource;

[0190] The sample multimodal feature extraction module 1120 is configured to perform multimodal feature extraction on each sample multimedia resource to obtain a sample multimodal feature corresponding to each sample multimedia resource;

[0191] The target resource label construction module 1130 is configured to execute inputting the sample multimodal features into the first to-be-trained model to construct resource labels, and obtain a target resource label corresponding to each sample multimedia resource;

[0192] The label detection module 1140 is configured to perform label detection processing by inputting the target resource label, the preset resource label, and the sample multimodal features into the second to-be-trained model to obtain a label detection result, wherein the label detection result indicates a match between the target resource label corresponding to each sample multimedia resource and each sample multimedia resource;

[0193] The sample determination module 1150 is configured to perform, based on the label detection result, using the target resource label corresponding to the first multimedia resource and the preset resource label corresponding to the first multimedia resource as positive sample label information, and using the target resource label corresponding to the second multimedia resource and the preset resource label corresponding to the second multimedia resource as negative sample label information; the first multimedia resource is a sample multimedia resource that matches the target resource label, and the second multimedia resource is a sample multimedia resource that does not match the target resource label;

[0194] The model training module 1160 is configured to perform model training on the first model to be trained and the second model to be trained based on the target resource label, preset resource label, sample multimedia resource, positive sample label information and negative sample label information to obtain a label construction model, which includes a label extraction network corresponding to the first model to be trained and a comparative learning network corresponding to the second model to be trained.

[0195] As an optional embodiment, the tag detection module includes:

[0196] The tag feature extraction unit is configured to input the target resource tag and the preset resource tag into the tag feature extraction layer to perform tag feature extraction respectively, and obtain a first tag feature corresponding to the target resource tag and a second tag feature corresponding to the preset resource tag;

[0197] a similarity determination unit configured to input the first label feature, the second label feature, and the sample multimodal feature into the similarity determination layer to perform similarity determination, thereby obtaining a first feature similarity between the target resource label and the sample multimedia resource, and a second feature similarity between the preset resource label and the sample multimedia resource;

[0198] The tag detection processing unit is configured to input the first feature similarity and the second feature similarity into the tag detection layer to perform tag detection processing and obtain a tag detection result.

[0199] As an optional embodiment, the sample multimodal feature includes a plurality of multimodal sub-features arranged in order, the first model to be trained includes a feature decoding layer and a text generation layer, and the target resource label construction module includes:

[0200] The first decoding processing unit is configured to, when the current multimodal sub-feature is the first multimodal sub-feature among the plurality of sequentially arranged multimodal sub-features, input the first multimodal sub-feature into the feature decoding layer for feature decoding processing, and obtain a feature decoding result corresponding to the first multimodal sub-feature;

[0201] The second decoding processing unit is configured to, when the current multimodal sub-feature is not the first multimodal sub-feature among the plurality of sequentially arranged multimodal sub-features, obtain a previous feature decoding result corresponding to the current multimodal sub-feature, input the current multimodal sub-feature and the previous feature decoding result into a feature decoding layer for feature decoding processing, and obtain a feature decoding result corresponding to the current multimodal sub-feature, wherein the previous feature decoding result is a feature decoding result corresponding to the multimodal sub-feature previous to the current multimodal sub-feature;

[0202] The target resource label determination unit is configured to input the feature decoding result corresponding to the last multimodal sub-feature of the plurality of multimodal sub-features arranged in order into the text generation layer for text generation to obtain the target resource label.

[0203] As an optional embodiment, the model training module includes:

[0204] a generation loss determining unit configured to determine tag generation loss information based on difference information between the target resource tag and the preset resource tag;

[0205] a detection loss determining unit configured to determine label detection loss information based on difference information between the sample multimedia resource and the positive sample label information, and difference information between the negative sample label information;

[0206] The model training unit is configured to perform model training on the first model to be trained and the second model to be trained based on the label generation loss information and the label detection loss information, and obtain a label construction model corresponding to the first model to be trained.

[0207] As an optional embodiment, the sample multimodal feature extraction module includes:

[0208] An image feature extraction unit is configured to perform image feature extraction on resource image information corresponding to each sample multimedia resource to obtain sample image features;

[0209] A first text feature extraction unit is configured to perform text feature extraction on resource text information corresponding to each sample multimedia resource to obtain sample text features;

[0210] The feature fusion unit is configured to perform feature fusion on the sample image features and the sample text features to obtain sample multimodal features corresponding to the sample multimedia resources.

[0211] As an optional embodiment, the resource text information includes at least two types of text information, the sample text feature includes a plurality of sequentially arranged text sub-features, and the first text feature extraction unit includes:

[0212] a text splicing unit configured to perform text splicing on at least two types of text information corresponding to the sample multimedia resource to obtain text splicing information;

[0213] The text sub-feature extraction unit is configured to input the first preset number of character information in the text splicing information into the text feature extraction model to perform text feature extraction, and obtain text sub-features corresponding to the preset number of character information.

[0214] As an optional embodiment, the resource image information includes a preset number of sequentially arranged image information, the sample image feature includes a plurality of sequentially arranged image sub-features, and the image feature extraction unit includes:

[0215] The image sub-feature extraction unit is configured to input a preset number of sequentially arranged image information into an image feature extraction model to perform image feature extraction, and obtain image sub-features corresponding to the preset number of sequentially arranged image information.

[0216] As an optional embodiment, the device further includes:

[0217] A resource classification module is configured to perform resource classification processing on sample multimedia resources and obtain a classification label corresponding to each sample multimedia resource;

[0218] The first text feature extraction unit includes:

[0219] The second text feature extraction unit is configured to perform text feature extraction on the resource text information and classification label corresponding to each sample multimedia resource to obtain sample text features.

[0220] Figure 12 A label construction device according to an exemplary embodiment includes:

[0221] The multimodal feature extraction module 1210 is configured to perform multimodal feature extraction on the multimedia resource to be processed to obtain the multimodal features to be processed;

[0222] The resource label construction module 1220 is configured to execute the label extraction network of the label construction model to construct resource labels by inputting the multimodal features to be processed into the label construction model to obtain resource labels corresponding to the multimedia resources to be processed. The label construction model is generated based on the above-mentioned label construction model generation method;

[0223] The resource tag detection module 1230 is configured to perform tag detection processing by inputting the resource tag corresponding to the multimedia resource to be processed and the multimedia resource to be processed into the tag detection network in the tag construction model to obtain a tag detection result of the resource tag.

[0224] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0225] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory including instructions, wherein the instructions can be executed by a processor of an electronic device to perform the above method. Alternatively, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0226] Figure 13 This is a block diagram of an electronic device for ... according to an exemplary embodiment. The electronic device may be a server, and its internal structure diagram may be as shown in FIG. Figure 13 As shown. The electronic device includes a processor, a memory and a network interface connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a method for generating a label construction model or a label construction method is implemented.

[0227] Those skilled in the art will understand that Figure 13 The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present disclosure, and does not constitute a limitation on the electronic device to which the scheme of the present disclosure is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0228] In an exemplary embodiment, an electronic device is also provided, including: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the method for generating a label construction model or the label construction method in the embodiment of the present disclosure.

[0229] In an exemplary embodiment, a storage medium is further provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the method for generating a label construction model or the label construction method in the embodiment of the present disclosure.

[0230] In an exemplary embodiment, a computer program product including instructions is further provided. When the computer program product is run on a computer, the computer is caused to execute the method for generating a label building model or the label building method in the embodiments of the present disclosure.

[0231] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, which can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0232] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0233] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for generating a label construction model, characterized in that: The method comprises: Acquire multiple sample multimedia resources and a preset resource tag corresponding to each of the sample multimedia resources; Performing multimodal feature extraction on each of the sample multimedia resources to obtain a sample multimodal feature corresponding to each of the sample multimedia resources; Inputting the sample multimodal features into the first to-be-trained model to construct resource labels, and obtaining target resource labels corresponding to each sample multimedia resource; Inputting the target resource label, the preset resource label, and the sample multimodal feature into a second to-be-trained model for label detection processing to obtain a label detection result, wherein the label detection result indicates a matching status between the target resource label corresponding to each of the sample multimedia resources and each of the sample multimedia resources; Based on the label detection result, the target resource label corresponding to the first multimedia resource and the preset resource label corresponding to the first multimedia resource are used as positive sample label information, and the target resource label corresponding to the second multimedia resource and the preset resource label corresponding to the second multimedia resource are used as negative sample label information; the first multimedia resource is a sample multimedia resource that matches the target resource label, and the second multimedia resource is a sample multimedia resource that does not match the target resource label; Determining label generation loss information based on difference information between the target resource label and the preset resource label; Determining label detection loss information based on difference information between the sample multimedia resource and the positive sample label information, and difference information between the target resource label in the negative sample label information and the preset resource label in the negative sample label information; Based on the label generation loss information and the label detection loss information, model training is performed on the first model to be trained and the second model to be trained to obtain the label construction model, which includes a label extraction network corresponding to the first model to be trained and a contrastive learning network corresponding to the second model to be trained.

2. The method for generating a label construction model according to claim 1, wherein: The second model to be trained includes a label feature extraction layer, a similarity determination layer, and a label detection layer. The target resource label, the preset resource label, and the sample multimodal feature are input into the second model to be trained for label detection processing. The label detection result obtained includes: Inputting the target resource tag and the preset resource tag into the tag feature extraction layer to perform tag feature extraction respectively, to obtain a first tag feature corresponding to the target resource tag and a second tag feature corresponding to the preset resource tag; Inputting the first label feature, the second label feature, and the sample multimodal feature into the similarity determination layer to perform similarity determination, thereby obtaining a first feature similarity between the target resource label and the sample multimedia resource, and a second feature similarity between the preset resource label and the sample multimedia resource; The first feature similarity and the second feature similarity are input into the tag detection layer to perform tag similarity detection to obtain the tag detection result.

3. The method for generating a label construction model according to claim 1, wherein: The sample multimodal features include a plurality of multimodal sub-features arranged in sequence, the first model to be trained includes a feature decoding layer and a text generation layer, and the step of inputting the sample multimodal features into the first model to be trained to construct resource labels to obtain a target resource label corresponding to each sample multimedia resource includes: When the current multimodal sub-feature is the first multimodal sub-feature among the plurality of multimodal sub-features arranged in order, inputting the first multimodal sub-feature into the feature decoding layer for feature decoding processing to obtain a feature decoding result corresponding to the first multimodal sub-feature; When the current multimodal sub-feature is not the first multimodal sub-feature among the multiple multimodal sub-features arranged in order, obtaining a previous feature decoding result corresponding to the current multimodal sub-feature, inputting the current multimodal sub-feature and the previous feature decoding result into the feature decoding layer for feature decoding processing, and obtaining a feature decoding result corresponding to the current multimodal sub-feature, wherein the previous feature decoding result is a feature decoding result corresponding to a previous multimodal sub-feature of the current multimodal sub-feature; The feature decoding result corresponding to the last multimodal sub-feature of the plurality of multimodal sub-features arranged in order is input into the text generation layer for text generation to obtain the target resource label.

4. The method for generating a label construction model according to claim 1, wherein: The extracting multimodal features of each of the sample multimedia resources to obtain the sample multimodal features corresponding to each of the sample multimedia resources includes: Extracting image features from the resource image information corresponding to each sample multimedia resource to obtain sample image features; Performing text feature extraction on resource text information corresponding to each sample multimedia resource to obtain sample text features; Feature fusion is performed on the sample image features and the sample text features to obtain sample multimodal features corresponding to the sample multimedia resource.

5. The method for generating a label construction model according to claim 4, wherein: The resource text information includes at least two types of text information, and the sample text features include a plurality of text sub-features arranged in sequence; the text feature extraction is performed on the resource text information corresponding to each sample multimedia resource to obtain the sample text features including: Performing text splicing on at least two types of text information corresponding to the sample multimedia resource to obtain text splicing information; Inputting the first preset number of character information in the text splicing information into a text feature extraction model to perform text feature extraction, and obtaining text sub-features corresponding to the preset number of character information respectively.

6. The method for generating a label construction model according to claim 4, wherein: The resource image information includes a preset number of sequentially arranged image information, and the sample image feature includes a plurality of sequentially arranged image sub-features; The extracting image features of the resource image information corresponding to each sample multimedia resource to obtain the sample image features includes: The preset number of sequentially arranged image information are input into an image feature extraction model to perform image feature extraction, and image sub-features corresponding to the preset number of sequentially arranged image information are obtained.

7. The method for generating a label construction model according to claim 4, wherein: Before obtaining the sample multimedia resource and the preset resource tag corresponding to the sample multimedia resource, the method further includes: Performing resource classification processing on the sample multimedia resources to obtain a classification label corresponding to each of the sample multimedia resources; The text feature extraction of the resource text information corresponding to each sample multimedia resource to obtain the sample text feature includes: Text feature extraction is performed on the resource text information and the classification label corresponding to each of the sample multimedia resources to obtain the sample text feature.

8. A label construction method, characterized in that: The method comprises: Perform multimodal feature extraction on the multimedia resources to be processed to obtain multimodal features to be processed; Inputting the multimodal features to be processed into a label extraction network in a label construction model to construct resource labels, thereby obtaining resource labels corresponding to the multimedia resources to be processed, wherein the label construction model is generated based on the label construction model generation method according to any one of claims 1 to 7; The resource tag corresponding to the multimedia resource to be processed and the multimedia resource to be processed are input into the tag detection network in the tag construction model to perform tag detection processing to obtain a tag detection result of the resource tag.

9. A device for generating a label construction model, characterized in that: The device comprises: A sample resource acquisition module is configured to acquire a plurality of sample multimedia resources and a preset resource tag corresponding to each of the sample multimedia resources; A sample multimodal feature extraction module is configured to perform multimodal feature extraction on each of the sample multimedia resources to obtain a sample multimodal feature corresponding to each of the sample multimedia resources; a target resource label construction module configured to execute inputting the sample multimodal features into the first to-be-trained model to construct resource labels, and obtain a target resource label corresponding to each of the sample multimedia resources; a label detection module configured to perform label detection processing by inputting the target resource label, the preset resource label, and the sample multimodal feature into a second to-be-trained model to obtain a label detection result, wherein the label detection result indicates a matching status between the target resource label corresponding to each of the sample multimedia resources and each of the sample multimedia resources; The sample determination module is configured to execute, based on the label detection result, the target resource label corresponding to the first multimedia resource and the preset resource label corresponding to the first multimedia resource as positive sample label information, and the target resource label corresponding to the second multimedia resource and the preset resource label corresponding to the second multimedia resource as negative sample label information; the first multimedia resource is a sample multimedia resource that matches the target resource label, and the second multimedia resource is a sample multimedia resource that does not match the target resource label; The model training module includes: a generation loss determining unit configured to determine tag generation loss information based on difference information between the target resource tag and the preset resource tag; a detection loss determining unit configured to determine label detection loss information based on difference information between the sample multimedia resource and the positive sample label information, and difference information between the target resource label in the negative sample label information and the preset resource label in the negative sample label information; The model training unit is configured to perform model training on the first model to be trained and the second model to be trained based on the label generation loss information and the label detection loss information to obtain the label construction model, wherein the label construction model includes a label extraction network corresponding to the first model to be trained and a contrastive learning network corresponding to the second model to be trained.

10. A label construction device, characterized in that: The device comprises: The multimodal feature extraction module is configured to perform multimodal feature extraction on the multimedia resources to be processed to obtain multimodal features to be processed; The resource label construction module is configured to execute resource label construction by inputting the multimodal features to be processed into the label construction model generated by the generation device of the label construction model according to claim 9, so as to obtain the resource label corresponding to the multimedia resource to be processed.

11. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method for generating a label construction model according to any one of claims 1 to 7 or the label construction method according to claim 8.

12. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method for generating a label construction model according to any one of claims 1 to 7 or the label construction method according to claim 8.

Images