A label configuration method, device, equipment and storage medium

Abstract

The embodiment of the present application discloses a label configuration method, device, equipment and storage medium in the field of artificial intelligence, wherein the method includes: obtaining a target resource to be configured with a label; performing feature extraction processing on the target reference data of the target resource through a feature extraction model to obtain a feature vector corresponding to the target reference data; determining the probability that the target resource belongs to each candidate classification label according to the feature vector corresponding to the target reference data through a label classification model, wherein the label classification model and the feature extraction model are independent of each other; determining the target classification label corresponding to the target resource according to the probability that the target resource belongs to each candidate classification label. This method can use large-scale training samples to train the model used, and the iterative training time consumed is relatively short, which can meet the actual application needs of the network platform.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence (AI) technology, and in particular to a tag configuration method, apparatus, device, and storage medium. Background Art

[0002] In the era of big data, many online platforms need to assign tags to their resources so they can provide personalized services like resource recommendations. For example, video platforms need to assign tags to their video resources so they can provide personalized recommendations based on the tags.

[0003] At present, a mainstream label configuration method is to use the label configuration model trained in an end-to-end (End2End) manner to determine the corresponding label for the resource to be configured with the label. However, this label configuration method is usually difficult to meet the business needs in actual product applications. The reason is that the label configuration model trained in an End2End manner is usually more complex, which contains coupled feature extraction structures and classification structures. For such complex models, end-to-end full-link learning, it is usually impossible to use large-scale training samples, because large-scale training samples will make the iterative training time of the model very long; and in actual product applications, only by iteratively training the model with large-scale training samples can the model have better generalization capabilities. It can be seen that the above-mentioned label configuration method is currently difficult to meet the actual application needs of the network platform. Summary of the Invention

[0004] The embodiments of the present application provide a label configuration method, apparatus, device and storage medium, which can use large-scale training samples to train the model used, and the iterative training time consumed is relatively short, which can meet the actual application needs of the network platform.

[0005] In view of this, the first aspect of the present application provides a tag configuration method, the method comprising:

[0006] Get the target resource to be configured with the label;

[0007] Performing feature extraction processing on the target reference data of the target resource through a feature extraction model to obtain a feature vector corresponding to the target reference data;

[0008] Determine the probability that the target resource belongs to each candidate classification label based on the feature vector corresponding to the target reference data through a label classification model; the label classification model and the feature extraction model are independent of each other;

[0009] The target classification label corresponding to the target resource is determined according to the probability that the target resource belongs to each candidate classification label.

[0010] A second aspect of the present application provides a label configuration device, the device comprising:

[0011] Resource acquisition module, used to obtain the target resource to be configured with tags;

[0012] A feature extraction module is used to perform feature extraction processing on the target reference data of the target resource through a feature extraction model to obtain a feature vector corresponding to the target reference data;

[0013] a label classification module, configured to determine the probability that the target resource belongs to each candidate classification label according to the feature vector corresponding to the target reference data through a label classification model; the label classification model is independent of the feature extraction model;

[0014] The label determination module is used to determine the target classification label corresponding to the target resource according to the probability that the target resource belongs to each candidate classification label.

[0015] A third aspect of the present application provides a device, comprising a processor and a memory:

[0016] The memory is used to store computer programs;

[0017] The processor is configured to execute the steps of the tag configuration method described in the first aspect according to the computer program.

[0018] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and the computer program is used to execute the steps of the tag configuration method described in the first aspect.

[0019] In a fifth aspect, the present application provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of the tag configuration method described in the first aspect.

[0020] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0021] The embodiment of the present application provides a label configuration method, which innovatively performs a label configuration task for a target resource through two independent stages. Specifically, in the first stage, the target reference data of the target resource is first subjected to feature extraction processing by a feature extraction model to obtain a feature vector corresponding to the target reference data; in the second stage, the probability that the target resource belongs to each candidate classification label is determined by a label classification model based on the feature vector corresponding to the target reference data; finally, the target classification label corresponding to the target resource is determined based on the probability that the target resource belongs to each candidate classification label. The above method decouples the feature extraction structure and label classification structure required for performing the label configuration task, and uses independent feature extraction models and label classification models to perform feature extraction operations and label classification respectively. Accordingly, when iteratively training the model used when performing the label configuration task, the feature extraction model and the label classification model can be trained independently; for a feature extraction model with a large number of parameters, it is usually relatively fixed and the iteration frequency is low; for a label classification model with a small number of parameters, the iteration frequency is high, but it is iteratively trained using large-scale training samples, without consuming a long iteration training time. In this way, the two structures with different iteration frequencies are decoupled. In actual applications, the label classification model used in the second stage is mainly iterated. Large-scale training samples can be used to complete the iterative training of the model in a shorter iterative training time, meeting the business needs of the network platform in actual product applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A schematic diagram of an application scenario of the tag configuration method provided in an embodiment of the present application;

[0023] Figure 2 A flowchart of a tag configuration method provided in an embodiment of the present application;

[0024] Figure 3 A schematic diagram of the principle of the anti-disturbance design provided in an embodiment of the present application;

[0025] Figure 4 A schematic diagram of the principle of aggregating multimodal features based on a gating mechanism provided in an embodiment of the present application;

[0026] Figure 5 A flowchart of a method for training a label classification model according to an embodiment of the present application;

[0027] Figure 6 A schematic diagram of the training principle of the label classification model provided in an embodiment of the present application;

[0028] Figure 7 A schematic diagram of an exemplary multi-level tag structure provided in an embodiment of the present application;

[0029] Figure 8 A schematic diagram of the training principle when comprehensive consideration is given to global classification labels and local classification labels provided in an embodiment of the present application;

[0030] Figure 9 A schematic diagram of the structure of the tag configuration model provided in an embodiment of the present application;

[0031] Figure 10 A schematic diagram of the structure of the classification model training module provided in an embodiment of the present application;

[0032] Figure 11 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application;

[0033] Figure 12 A schematic diagram of the structure of the server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to help those skilled in the art better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0035] The terms "first," "second," "third," "fourth," etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential sequence. It should be understood that the numbers used in this way are interchangeable where appropriate so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions, e.g., a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or apparatus.

[0036] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0037] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0038] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0039] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0040] The solutions provided in the embodiments of this application involve artificial intelligence machine learning technology, which is specifically illustrated by the following embodiments:

[0041] In related technologies, the models used for label configuration tasks are typically trained iteratively using an end-to-end approach. These models are complex, consisting of coupled feature extraction and label classification structures. When training these models iteratively using an end-to-end approach, both the coupled feature extraction and label classification structures must be trained simultaneously. Using large-scale training samples to train these models would result in lengthy iterations, making it difficult to meet the business needs of network platforms in real-world product applications.

[0042] In response to the problems existing in the above-mentioned related technologies, an embodiment of the present application provides a label configuration method, which can use large-scale training samples to iteratively train the model used in the method, and the iterative training time is short, which can meet the business needs of the network platform in actual product applications.

[0043] Specifically, in the label configuration method provided in the embodiment of the present application, the target resource to be configured with a label is first obtained; then, feature extraction processing is performed on the target reference data of the target resource through a feature extraction model to obtain a feature vector corresponding to the target reference data; then, through a label classification model, the probability that the target resource belongs to each candidate classification label is determined according to the feature vector corresponding to the target reference data; finally, according to the probability that the target resource belongs to each candidate classification label, the target classification label corresponding to the target resource is determined.

[0044] The above-mentioned label configuration method innovatively performs the label configuration task for the target resource through two independent stages. In the first stage, the target reference data of the target resource is first subjected to feature extraction processing by a feature extraction model to obtain the feature vector corresponding to the target reference data. In the second stage, the probability of the target resource belonging to each candidate classification label is determined by the label classification model based on the feature vector corresponding to the target reference data. Compared with the model obtained by training through the End2End method in the related art, the above-mentioned method decouples the feature extraction structure and label classification structure required for performing the label configuration task, and uses independent feature extraction models and label classification models to perform feature extraction operations and label classification respectively; accordingly, when iteratively training the model used to perform the label configuration task, the feature extraction model and label classification model can be trained independently; for the feature extraction model with a large number of parameters, it is usually relatively fixed and the iteration frequency is low; for the label classification model with a small number of parameters, the iteration frequency is high, but it is iteratively trained using large-scale training samples, without consuming a long iteration training time. In this way, the two structures with different iteration frequencies are decoupled. In actual applications, the label classification model used in the second stage is mainly iterated. Large-scale training samples can be used to complete the iterative training of the model in a shorter iterative training time, meeting the business needs of the network platform in actual product applications.

[0045] Experimental studies have shown that when using the End2End method to train a video multi-label classification model (i.e., a model for multi-level label configuration for video resources), it takes more than one week of iterative training time to use millions of video training samples, and the graphics processing unit (GPU) video memory requirements are high. The batch size (Batch_size) cannot be too large and is difficult to exceed 100. After adopting the two-stage design in the label configuration method provided in the embodiment of the present application, when training the label classification model therein, the model iteration can converge within two days using millions of video training samples, and can support tens of millions of training samples, and the Batch_size can be parallelized to more than 200.

[0046] It should be understood that the tag configuration method provided in the embodiments of the present application can be applied to devices that support model operation, such as terminal devices or servers. Specifically, the terminal device can be a computer, smartphone, tablet computer, personal digital assistant (PDA), etc.; the server can be an application server or a web server. In actual deployment, it can be a standalone server, a cluster server, or a cloud server.

[0047] To facilitate understanding of the tag configuration method provided in the embodiment of the present application, the following uses an example in which the execution subject of the tag configuration method is a server and the tag configuration method is used to configure tags for video resources to exemplify the application scenarios to which the tag configuration method is applicable.

[0048] See also Figure 1 , Figure 1 Schematic diagram of the application scenario of the tag configuration method provided in the embodiment of this application. Figure 1 As shown, this application scenario includes a server 110 and a terminal device 120, which can communicate with each other via a network. The server 110 is used to execute the tag configuration method provided in the embodiment of the present application, in which a mutually independent feature extraction model 111 and tag classification model 112 are running; the terminal device 120 is used to upload the target resource to be configured with a tag to the server 110.

[0049] In actual applications, the user can upload the target video resource to the server 110 through the application client corresponding to the video network platform running in the terminal device 120.

[0050] After receiving the target video resource, the server 110 may first perform feature extraction processing on the target reference data of the target video resource through the feature extraction model 111 to obtain a feature vector corresponding to the target reference data. Exemplarily, the feature extraction model 111 may perform feature extraction processing on target reference data of multiple modalities related to the target video resource. For example, feature extraction processing may be performed on target reference data of three modalities, namely, video frames, audio, and reference text (such as title text, content introduction text, etc.) of the target video resource, to obtain feature vectors corresponding to the target reference data of the three modalities.

[0051] Furthermore, the server 110 can determine the probability that the target video resource belongs to each candidate classification label based on the feature vector corresponding to the target reference data through the label classification model 112. For example, if the server 110 extracts the feature vectors corresponding to the target reference data of multiple modalities through the feature extraction model 111, the label classification model 112 needs to aggregate the feature vectors corresponding to the target reference data of these multiple modalities, and then determine the probability that the target video resource belongs to each candidate classification label based on the aggregated feature vectors; the candidate classification labels here may include classification labels at multiple levels, and the label classification model 112 needs to determine the probability that the target video resource belongs to each classification label at each level accordingly.

[0052] Ultimately, server 110 can determine the target classification label corresponding to the target video resource based on the probability that the target video resource belongs to each candidate classification label. Subsequently, the relevant service server can accumulate user profiles based on the classification labels determined by server 110 for each video resource to make personalized video resource recommendations. Furthermore, classification labels can be used to perform tasks such as expanding recall queues, reducing diversity, and sorting recommended resources. This application does not impose any restrictions on the subsequent use of classification labels.

[0053] It should be understood that Figure 1 The application scenarios shown are only examples. In actual applications, in addition to obtaining the video resources uploaded by the terminal device 120 as the target video resources to be configured with tags, the target video resources to be configured with tags can also be obtained from a database or other server. This application does not impose any restrictions on the method of obtaining the target video resources. In addition, in addition to being used to configure classification tags for video resources, the tag configuration method provided in the embodiment of the present application can also be used to configure classification tags for other types of resources, for example, configuring classification tags for audio resources, text resources, etc. This application does not impose any restrictions on the processing objects of the tag configuration method provided in the embodiment of the present application.

[0054] The tag configuration method provided by this application is described in detail below through a method embodiment.

[0055] See also Figure 2 , Figure 2 This is a flow chart of the tag configuration method provided in the embodiment of the present application. For the sake of convenience, the following embodiment is still introduced by taking the execution subject of the tag configuration method as the server as an example. Figure 2 As shown, the tag configuration method includes the following steps:

[0056] Step 201: Obtain the target resource to be configured with a tag.

[0057] In actual applications, the server can receive resources uploaded by users through terminal devices as target resources to be configured with tags, or it can obtain resources without tags from the database or other servers as target resources to be configured with tags. This application does not impose any restrictions on the implementation method of the server obtaining target resources.

[0058] It should be noted that the label configuration method provided in the embodiment of the present application can be applied to different application scenarios and configure labels for different types of resources. For example, it can be used to configure labels for video resources on a video network platform, it can also be used to configure labels for audio resources on an audio network platform, and it can also be used to configure labels for news resources on a news network platform. No limitation is imposed on the application scenarios to which the label configuration method provided in the embodiment of the present application is applicable, and on the types of target resources to be configured with labels.

[0059] Step 202: Perform feature extraction processing on the target reference data of the target resource through a feature extraction model to obtain a feature vector corresponding to the target reference data.

[0060] After the server obtains the target resource to be tagged, it can use the feature extraction model to perform feature extraction processing on the target reference data of the target resource, thereby obtaining a feature vector corresponding to the target reference data. The target reference data here is data related to the target resource and can provide effective reference information for configuring the tag. For example, the target resource's own content data, attribute data, and data associated with the target resource can all serve as the target reference data of the target resource. This application does not impose any restrictions on the type of target reference data.

[0061] It should be noted that the feature extraction model is a model specifically used to mine the characteristics of the data itself. It includes a huge number of parameters, usually in the tens to hundreds of millions, but the feature extraction model is relatively fixed and does not need to be iteratively trained frequently. Usually, once the training of the feature extraction model is completed, there is no need to iteratively train the feature extraction model for a long time.

[0062] It should be understood that for different types of target reference data, the feature extraction models used will also be different. Exemplarily, for target reference data in image form, a residual network (Residual Network, ResNet) model or an efficient (Efficient) network model can be used as a feature extraction model, for target reference data in audio form, a VGG (Visual Geometry Group) model or a convolutional neural network (Convolutional Neural Networks, CNN) model can be used as a feature extraction model, and for target reference data in text form, a BERT (Bidirectional Encoder Representations from Transformer) model or a text convolutional neural network (Text Convolutional Neural Networks, TextCNN) model can be used as a feature extraction model. This application does not specifically limit the feature extraction model used.

[0063] The following takes the target resource as the target video resource as an example to introduce in detail the implementation process of the server using the feature extraction model to perform feature extraction processing on the target reference data of the target resource.

[0064] When the target resource acquired by the server is a target video resource, the target reference data of the target video resource may include at least one of the following: multiple target video frames in the target video resource, target audio in the target video resource, and target reference text of the target video resource. The target reference text may include at least one of a title text and a content introduction text. Accordingly, when the target reference data of the target video resource includes multiple target video frames in the target video resource, the server may perform feature extraction processing on the multiple target video frames through an image feature extraction model to obtain feature vectors corresponding to each of the multiple target video frames; when the target reference data of the target video resource includes target audio in the target video resource, the server may perform feature extraction processing on the multiple mel-spectra corresponding to the target audio through an audio feature extraction model to obtain feature vectors corresponding to each of the multiple mel-spectra; when the target reference data of the target video resource includes the target reference text of the target video resource, the server may perform feature extraction processing on the text segmentation in the target reference text through a text feature extraction model to obtain feature vectors corresponding to the text segmentation.

[0065] Specifically, the server can extract several key video frames from the target video resource as target video frames, or the server can also collect target video frames from the target video resource at a fixed frequency, and then use a pre-trained image feature extraction model, such as a ResNet model or an Efficient model, to perform feature extraction processing on each target video frame to obtain a feature vector corresponding to each target video frame. The server can convert the target audio in the target video resource into multiple Mel spectrum segments, and then use a pre-trained audio feature extraction model, such as a VGGish model or a stacked CNN model, to perform feature extraction processing on each Mel spectrum segment to obtain a feature vector corresponding to each Mel spectrum segment. The server can perform word segmentation processing on the title text and / or content introduction text of the target video resource to obtain text segmentation, and then use a pre-trained text feature extraction model, such as a BERT model or a TextCNN model, to perform feature extraction processing on the text segmentation to obtain feature vectors corresponding to the text segmentation.

[0066] It should be understood that in actual applications, when the target reference data simultaneously includes multiple target video frames in the target video resource, target audio in the target video resource, and target reference text of the target video resource, the server can perform feature extraction processing on the multiple target video frames, multiple Mel-spectrogram segments corresponding to the target audio, and text segmentation in the target reference text one by one in a preset order, or it can perform feature extraction processing on the multiple target video frames, multiple Mel-spectrogram segments corresponding to the target audio, and text segmentation in the target reference text in parallel. This application does not impose any restrictions on the order of feature extraction for various target reference data.

[0067] In addition, when the target resource is a target video resource, the target reference data of the target video resource may also include only any one or two of the target video frame, target audio and target reference text. Of course, it may also include other types of data in addition to the target video frame, target audio and target reference text, such as attribute data of the target video resource, etc. This application does not impose any restrictions on the target reference data used when configuring labels for the target video resource.

[0068] It should be noted that in the method provided in the embodiment of the present application, the target resource obtained by the server can be not only a target video resource, but also other types of resources such as a target audio resource. When the target resource obtained by the server is a target audio resource, the audio of the target audio resource itself, the lyrics text and / or the song title text of the target audio resource can be used as target reference data, and the audio feature extraction model is used to perform feature extraction processing on the multiple Mel spectrum segments corresponding to the audio of the target audio resource itself to obtain feature vectors corresponding to the multiple Mel spectrum segments, and the text feature extraction model is used to perform feature extraction processing on the text segmentation in the lyrics text and / or song title text of the target audio resource to obtain feature vectors corresponding to the text segmentation. The present application does not make any restrictions on the type of target resource and the feature extraction processing method adopted for the target resource.

[0069] Step 203: Determine the probability that the target resource belongs to each candidate classification label according to the feature vector corresponding to the target reference data through a label classification model; the label classification model and the feature extraction model are independent of each other.

[0070] The server completes the feature extraction processing of the target reference data of the target resource through the feature extraction model. After obtaining the feature vector corresponding to the target reference data, the feature vector corresponding to the target reference data can be further input into a pre-trained label classification model to analyze and process the feature vector corresponding to the target reference data through the label classification model to determine the probability that the target resource belongs to each candidate classification label.

[0071] It should be noted that the label classification model used in step 203 and the feature extraction model used in step 202 are two independent model structures, and they can be iteratively trained separately. The label classification model includes fewer parameters and usually requires frequent iterative training. However, due to the small number of parameters included in the label classification model, even if a large number of training samples are used to train it, it does not take a long time for iterative training.

[0072] In addition, the inventors of this application have found that for an overfitting neural network model, adding a certain disturbance to its input will greatly change the output of the model. For example, Figure 3 As shown, the prediction result of the model based on the input feature a is panda, and a small disturbance is added to the input feature a, and the prediction result of the model based on the input feature with the disturbance becomes gibbon. It can be seen that it is easy to overfit the label classification model when training the label classification model at present, and the generalization ability of the trained label classification model is low. To address this problem, the method provided in the embodiment of the present application adds a design to counteract disturbances in the process of training the label classification model, so as to enhance the anti-interference ability of the label classification model, alleviate the overfitting phenomenon of the label classification model, and improve the generalization ability of the label classification model. The training process of the label classification model will be introduced in detail below through another method embodiment.

[0073] Below, taking the target resource as the target video resource, and the target reference data including multiple target video frames in the target video resource, the target audio in the target video resource, and the target reference text of the target video resource as an example, the server uses the label classification model to determine the probability that the target resource belongs to each candidate classification label according to the feature vector corresponding to the target reference data. A detailed introduction is given to the implementation method.

[0074] The label classification model in the embodiment of the present application may include a unimodal feature aggregation module, a multimodal feature aggregation module and a label classification model, wherein the unimodal feature aggregation module includes a video feature aggregation layer, an audio feature aggregation layer and a text feature aggregation layer. Accordingly, the server can aggregate the feature vectors corresponding to each of the multiple target video frames through the video feature aggregation layer to obtain a video aggregate feature vector, aggregate the feature vectors corresponding to each of the multiple mel spectra through the audio feature aggregation layer to obtain an audio aggregate feature vector, and aggregate the feature vectors corresponding to each text word through the text feature aggregation layer to obtain a text aggregate feature vector; then, the multimodal feature aggregation module aggregates the video aggregate feature vector, the audio aggregate feature vector and the text aggregate feature vector to obtain a target feature vector; and then, the label classification module determines the probability that the target resource belongs to each candidate classification label according to the target feature vector.

[0075] Exemplarily, assume that the server performs feature extraction processing on N1 (N1 is an integer greater than 1) target video frames through the image feature extraction model to obtain N1*1536 (1536 is the dimension of the dense vector obtained after each target video frame is processed by the image feature extraction model) feature vectors, performs feature extraction processing on N2 (N2 is an integer greater than 1) Mel spectrum segments through the audio feature extraction model to obtain N2*128 (128 is the dimension of the dense vector obtained after each Mel spectrum segment is processed by the audio feature extraction model) feature vectors, and performs text feature extraction processing on N3 (N3 is a positive integer) text segmentation words through the text feature extraction model to obtain N3*256 (256 is the dimension of the dense vector obtained after each text segmentation is processed by the text feature extraction model) feature vectors.

[0076] The tag classification model can use a video feature aggregation layer, such as the Netvlad or Nextvlad architecture, to aggregate the N1*1536 feature vector to obtain a video aggregate feature vector. The server can use an audio feature aggregation layer, such as the Netvlad or Nextvlad architecture, to aggregate the N2*128 feature vector to obtain an audio aggregate feature vector. The server can use the TextCNN architecture to aggregate the N3*256 feature vector to obtain a text aggregate feature vector. The tag classification model can then use a multimodal feature aggregation module to map the video, audio, and text aggregate feature vectors to a fixed dimension and aggregate the fixed-dimensional video, audio, and text aggregate feature vectors to obtain the target feature vector. Furthermore, the label classification model can use a label classification module, such as a Softmax structure, to process the target feature vector to output the probability that the target resource belongs to each candidate classification label; the candidate classification labels here can include various classification labels belonging to different levels. For example, assuming that the classification labels for video resources include three levels, the candidate classification labels here include various classification labels under these three levels.

[0077] It should be noted that in the related art, the aggregation of multimodal feature vectors is generally achieved by splicing feature vectors. However, considering that the feature vectors of certain modalities may have a certain proportion of missing items, the reference value of feature vectors with missing items will be reduced, and the degree of attention paid by classification labels of different categories to feature vectors of different modalities is also different. For example, the classification labels of sports categories should pay more attention to video frame features, and the classification labels of life categories should comprehensively refer to audio frame features, audio features and text features. Based on this, the label configuration method provided in the embodiment of the present application proposes a multimodal feature aggregation mode based on a gate mechanism and a multimodal feature aggregation method based on an attention mechanism. The following introduces these two multimodal feature aggregation methods respectively.

[0078] When aggregating multimodal features based on the gate mechanism, the label classification model needs to first map the video aggregate feature vector, audio aggregate feature vector and text aggregate feature vector to the target dimension through the fully connected layer in the multimodal feature aggregation module to obtain the target video feature vector, target audio feature vector and target text feature vector; and, perform linear mapping processing on the video aggregate feature vector, audio aggregate feature vector and text feature aggregation vector respectively, and determine the gate units corresponding to the target video feature vector, target audio feature vector and target text feature vector respectively based on the video aggregate feature vector, audio aggregate feature vector and text aggregate feature vector after linear mapping processing; and then, based on the gate units corresponding to the target video feature vector, target audio feature vector and target text feature vector respectively, perform weighted aggregation processing on the target video feature vector, target audio feature vector and target text feature vector to obtain the target feature vector.

[0079] Figure 4 This is a schematic diagram of the implementation principle of aggregating multimodal features based on a gate mechanism provided in an embodiment of the present application. Figure 4As shown, the label classification model can first use the fully connected layer in the multimodal feature aggregation module to map the video aggregate feature vector, audio aggregate feature vector, and text aggregate feature vector to the target dimension, thereby obtaining the target video feature vector, target audio feature vector, and target text feature vector. The fully connected layer here can include a nonlinear mapping layer, a batch normalization layer, and a residual connection layer. Furthermore, the video aggregate feature vector, audio aggregate feature vector, and text aggregate feature vector are linearly mapped and mapped to the interval [0, 1] using the sigmoid function, respectively, to obtain three gate units corresponding to the target video feature vector, target audio feature vector, and target text feature vector, respectively. Furthermore, the target video feature vector, target audio feature vector, and target text feature vector are weightedly aggregated using the gate units corresponding to the target video feature vector, target audio feature vector, and target text feature vector, thereby obtaining a high-level feature vector after aggregating the multimodal features, namely the target feature vector.

[0080] When aggregating multimodal features based on the Attention mechanism, the label classification model needs to first map the video aggregate feature vector, audio aggregate feature vector, and text aggregate feature vector to the target dimension through the fully connected layer in the multimodal feature aggregation module to obtain the target video feature vector, target audio feature vector, and target text feature vector; then, determine the similarity between the target video feature vector and the target latent vector as the weight corresponding to the target video feature vector, determine the similarity between the target audio feature vector and the target latent vector as the weight corresponding to the target audio feature vector, and determine the similarity between the target text feature vector and the target latent vector as the weight corresponding to the target text feature vector; then, based on the weights corresponding to the target video feature vector, target audio feature vector, and target text feature vector, perform weighted aggregation processing on the target video feature vector, target audio feature vector, and target text feature vector to obtain the target feature vector.

[0081] Specifically, the fully connected layer in the multimodal feature aggregation module is first used to map the video aggregate feature vector, audio aggregate feature vector, and text aggregate feature vector to the target dimension to obtain the target video feature vector, target audio feature vector, and target text feature vector. The fully connected layer here can include a nonlinear mapping layer, a BatchNormalization layer, and a residual connection layer. Then, the target latent vector of the target dimension is determined, and the similarity between the target video feature vector and the target latent vector is calculated as the weight corresponding to the target video feature vector, the similarity between the target audio feature vector and the target latent vector is calculated as the weight corresponding to the target audio feature vector, and the similarity between the target text feature vector and the target latent vector is calculated as the weight corresponding to the target text feature vector. Furthermore, the target video feature vector, target audio feature vector, and target text feature vector are multiplied by their respective corresponding weights to obtain a high-level feature vector after aggregating the multimodal features, namely the target feature vector.

[0082] It should be understood that in actual applications, the label classification model can also adopt other methods to aggregate multimodal feature vectors, such as directly splicing the multimodal feature vectors together. This application does not impose any restrictions on the implementation method of the label classification model aggregating multimodal feature vectors.

[0083] It should be understood that when the target resource obtained by the server is a target audio resource, the implementation method of the label classification model determining the probability that the target audio resource belongs to each candidate classification label based on the feature vector corresponding to the target reference data is similar to the implementation method of the label classification model determining the probability that the target video resource belongs to each candidate classification label in the above text; the only difference is that the unimodal feature aggregation module in the label classification model includes an audio feature aggregation layer and a text feature aggregation layer, which are respectively used to aggregate the feature vectors corresponding to multiple Mel-spectra of the target audio resource and the feature vectors corresponding to each text segmentation, and the multimodal feature aggregation module in the label classification model only needs to perform aggregation processing on the audio aggregation feature vector and the text aggregation feature vector.

[0084] Step 204: Determine the target classification label corresponding to the target resource based on the probability that the target resource belongs to each candidate classification label.

[0085] After the server determines the probability that the target resource belongs to each candidate classification label through the label classification model, it can further determine the target classification label corresponding to the target resource based on the probability that the target resource belongs to each candidate classification label.

[0086] Exemplarily, the server may pre-set a target threshold value such as 0.5, and then, for each candidate classification label, determine whether the probability that the target resource belongs to the candidate classification label exceeds the target threshold value. If so, the candidate classification label may be used as the target classification label corresponding to the target resource. Alternatively, the server may also sort the probabilities of the target resource belonging to each candidate label at each level in descending order, and finally use a preset number of candidate labels with the highest ranking as the target classification labels for the target resource at that level. Of course, in actual applications, the server may also adopt other methods to determine the target classification label corresponding to the target resource based on the probability that the target resource belongs to each candidate classification label. This application does not impose any limitation on the implementation method of determining the target classification label corresponding to the target resource.

[0087] The above-mentioned label configuration method innovatively performs the label configuration task for the target resource through two independent stages. In the first stage, the target reference data of the target resource is first subjected to feature extraction processing by a feature extraction model to obtain the feature vector corresponding to the target reference data. In the second stage, the probability of the target resource belonging to each candidate classification label is determined by the label classification model based on the feature vector corresponding to the target reference data. Compared with the model obtained by training through the End2End method in the related art, the above-mentioned method decouples the feature extraction structure and label classification structure required for performing the label configuration task, and uses independent feature extraction models and label classification models to perform feature extraction operations and label classification respectively; accordingly, when iteratively training the model used to perform the label configuration task, the feature extraction model and label classification model can be trained independently; for the feature extraction model with a large number of parameters, it is usually relatively fixed and the iteration frequency is low; for the label classification model with a small number of parameters, the iteration frequency is high, but it is iteratively trained using large-scale training samples, without consuming a long iteration training time. In this way, the two structures with different iteration frequencies are decoupled. In actual applications, the label classification model used in the second stage is mainly iterated. Large-scale training samples can be used to complete the iterative training of the model in a shorter iterative training time, meeting the business needs of the network platform in actual product applications.

[0088] As mentioned above, in order to alleviate the overfitting phenomenon of the label classification model and enable the label classification model to have better generalization ability, the method provided in the embodiment of the present application introduces an anti-disturbance design in the process of training the label classification model. The training method of the label classification model provided in the embodiment of the present application is introduced in detail below through a method embodiment.

[0089] See also Figure 5 , Figure 5The flowchart of the training method of the label classification model provided in the embodiment of the present application is as follows. For the convenience of description, the following embodiment is introduced by taking the execution subject of the training method of the label classification model as the server as an example. Figure 5 As shown, the training method of the label classification model includes the following steps:

[0090] Step 501: Acquire training samples; the training samples include training resources and their corresponding annotated classification labels.

[0091] Before the server trains the label classification model, it usually needs to obtain a large number of training samples. Each training sample includes a training resource and the annotation classification label corresponding to the training resource.

[0092] It should be understood that the type of training resources included in the training sample depends on the application scenario of the label classification model to be trained. If the label classification model is used to configure labels for video resources, the training resources included in the training sample should be training video resources. If the label classification model is used to configure labels for audio resources, the training resources included in the training sample should be training audio resources. This application does not impose any restrictions on the application scenarios of the label classification model or the types of training resources included in the training sample.

[0093] Step 502: Perform feature extraction processing on the training reference data of the training resource using the feature extraction model to obtain a first feature vector corresponding to the training reference data.

[0094] After obtaining the training samples, the trained feature extraction model can be used to perform feature extraction processing on the training reference data of the training resource, thereby obtaining the first feature vector corresponding to the training reference data. The training reference data here is data related to the training resource and can provide effective reference information for configuring the label. For example, the content data, attribute data of the training resource itself, and data associated with the training resource can all be used as training reference data for the training resource. This application does not impose any restrictions on the type of training reference data.

[0095] It should be understood that the feature extraction models used for different types of training reference data may also vary. For example, for training reference data in the form of images, the ResNet model or the Efficient network model can be used as the feature extraction model; for training reference data in the form of audio, the VGGish model or the CNN model can be used as the feature extraction model; for training reference data in the form of text, the BERT model or the TextCNN model can be used as the feature extraction model. This application does not specifically limit the feature extraction model used.

[0096] The following takes the training resource as a training video resource as an example to introduce in detail the implementation process of the server using the feature extraction model to perform feature extraction processing on the training reference data of the training resource.

[0097] When the training resource included in the training sample is a training video resource, the training reference data of the training video resource may include at least one of the following: multiple training video frames in the training video resource, training audio in the training video resource, and training reference text of the training video resource, where the training reference text may include at least one of a title text and a content introduction text. Accordingly, when the training reference data includes multiple training video frames in the training video resource, the server may perform feature extraction processing on the multiple training video frames using an image feature extraction model to obtain first feature vectors corresponding to each of the multiple training video frames; when the training reference data includes training audio in the training video resource, the server may perform feature extraction processing on multiple mel-spectra corresponding to the training audio using an audio feature extraction model to obtain first feature vectors corresponding to each of the multiple mel-spectra; when the training reference data includes training reference text in the training video resource, the server may perform feature extraction processing on text segmentation in the training reference text using a text feature extraction model to obtain first feature vectors corresponding to the text segmentation.

[0098] Specifically, the server can extract several key video frames from the training video resources as training video frames, or the server can also collect training video frames from the training video resources at a fixed frequency, and then use a pre-trained image feature extraction model, such as a ResNet model or an Efficient model, to perform feature extraction processing on each training video frame to obtain the first feature vector corresponding to each training video frame. The server can convert the training audio in the training video resources into multiple mel spectrum segments, and then use a pre-trained audio feature extraction model, such as a VGGish model or a stacked CNN model, to perform feature extraction processing on each mel spectrum segment to obtain the first feature vector corresponding to each mel spectrum segment. The server can perform word segmentation processing on the title text and / or content introduction text of the training video resources to obtain text segmentation, and then use a pre-trained text feature extraction model, such as a BERT model or a TextCNN model, to perform feature extraction processing on the text segmentation to obtain the first feature vector corresponding to the text segmentation.

[0099] It should be understood that in actual applications, the training reference data of the training video resources may also include only any one or two of the training video frames, training audio and training reference text. Of course, it may also include other types of data in addition to training video frames, training audio and training reference text, such as attribute data of the training video resources, etc. This application does not impose any limitations on the training reference data of the training video resources.

[0100] It should be understood that when the label classification model to be trained is used to configure labels for other types of resources, the server can obtain training samples containing other types of training resources, and then use the feature extraction model to perform feature extraction processing on the training reference data of such training resources. Taking the training resources included in the training sample as training audio resources as an example, the audio of the training audio resource itself, the lyrics text and / or song title text of the training audio resource can be used as training reference data, and the audio feature extraction model is used to perform feature extraction processing on the multiple Mel spectra corresponding to the audio of the training audio resource itself, and obtain the first feature vector corresponding to each of the multiple Mel spectra. The text feature extraction model is used to perform feature extraction processing on the text segmentation in the lyrics text and / or song title text of the training audio resource, and obtain the first feature vector corresponding to the text segmentation. This application does not make any restrictions on the type of training resources and the feature extraction processing method used for the training reference data of the training resources.

[0101] Step 503: Determine the first prediction probability that the training resource belongs to each candidate classification label according to the first eigenvector corresponding to the training reference data through the basic label classification model to be trained; and determine the first loss function according to the first prediction probability and the labeled classification label.

[0102] The server uses the feature extraction model to perform feature extraction processing on the training reference data of the training resources in the training sample. After obtaining a first feature vector corresponding to the training reference data, the server can further input the first feature vector corresponding to the training reference data into the basic label classification model currently to be trained, so that the basic label classification model analyzes and processes the first feature vector corresponding to the training reference data and outputs a first predicted probability that the training resource belongs to each candidate classification label. Furthermore, the server can construct a first loss function based on the first predicted probability that the training resource belongs to each candidate classification label and the annotated classification label corresponding to the training resource included in the training sample.

[0103] The following takes the training resource as a training video resource, and the training reference data includes multiple training video frames in the training video resource, training audio in the training video resource, and training reference text of the training video resource as an example. Figure 6 The training principle diagram of the label classification model shown is an exemplary introduction to the implementation method of the server using the basic label classification model to determine the first prediction probability of the training resource belonging to each candidate classification label according to the first eigenvector corresponding to the training reference data.

[0104] like Figure 6As shown, the server can use the image feature extraction model, the audio feature extraction model and the text feature extraction model to perform feature extraction processing on multiple training video frames in the training video resources, multiple Mel spectrum segments in the training audio and text segmentation in the training reference text, and obtain the first feature vectors corresponding to each of the multiple training video frames, the first feature vectors corresponding to each of the multiple Mel spectrum segments and the first feature vector corresponding to the text segmentation.

[0105] Furthermore, the server may utilize the basic label classification model to process the first eigenvectors corresponding to the training reference data of various modalities. Figure 6 The process corresponding to the white square in the middle is the processing process of the basic label classification model for the first feature vector. Figure 6 As shown, a video feature aggregation layer, such as the Nextvlad structure, can be used to aggregate the first feature vectors corresponding to each training video frame to obtain a first aggregated training video feature vector. An audio feature aggregation layer, such as the Nextvlad structure, can be used to aggregate the first feature vectors corresponding to each mel-spectrogram to obtain a first aggregated training audio feature vector. A text feature aggregation layer, such as the TextCNN structure, can be used to aggregate the first feature vectors corresponding to each text word segmentation to obtain a first aggregated training text feature vector. Then, a multimodal feature aggregation module in the basic label classification model aggregates the first aggregated training video feature vector, the first aggregated training audio feature vector, and the first aggregated training text feature vector to obtain a first training feature vector. Furthermore, a label classification module in the basic label classification model determines, based on the first training feature vector, the first predicted probability that the training video resource belongs to each candidate classification label. Finally, a first loss function is determined based on the first predicted probability that the training video resource belongs to each candidate classification label and the labeled classification label corresponding to the training video resource.

[0106] Assume that the first eigenvector of the input basic label classification model is x. After a series of linear and nonlinear mapping processes of the basic label classification model, the first predicted probability of the training resource belonging to each candidate classification label is obtained. Then, the first loss function is minimized by sigmoid cross entropy using formula (1), which is recorded as Normal_loss.

[0107] Normal_loss=-min log p(y|x,θ) (1)

[0108] Among them, y is the first predicted probability of the training resource belonging to each candidate classification label output by the basic label classification model, x is the first eigenvector input to the basic label classification model, and θ is the model parameter in the basic label classification model.

[0109] In practical applications, classification labels generally have a hierarchical structure. Figure 7 What is shown is an exemplary label structure including three levels. Based on this, in the process of training the basic label classification model, the global classification labels and local classification labels can be comprehensively considered to improve the accuracy of the trained label classification model.

[0110] That is, the server can use the basic label classification model to determine, based on the first eigenvector corresponding to the training reference data, a third predicted probability that the training resource belongs to each global classification label, and a fourth predicted probability that the training resource belongs to each local classification label; the global classification label here is obtained by combining local classification labels at different levels. A third loss function can then be determined based on the third predicted probability and the labeled classification label corresponding to the training resource, and a fourth loss function can be determined based on the fourth predicted probability and the labeled classification label corresponding to the training resource. Furthermore, the first loss function described above is determined based on the third and fourth loss functions.

[0111] The following takes the three-level tag structure as an example, combined with Figure 8 The implementation principle diagram shown in FIG is used to exemplify the above implementation method. Figure 8 As shown, the basic label classification model can determine the probability Tag of the training resource belonging to each global classification label for the first input feature vector x through branch 801. G The global classification label here is the whole obtained by combining the first-level label, the second-level label and the third-level label; the basic label classification model can determine the probability Tag of the training resource belonging to each first-level label for the first eigenvector x of the input through branches 802, 803 and 804. L 1 , the probability of belonging to each secondary tag Tag L 2 and the probability of belonging to each third-level label Tag L 3 . Then, based on the labeled classification labels and probability tags corresponding to the training resources G Construct the third loss function based on the labeled classification labels and probability tags corresponding to the training resources L 1 、Tag L 2 and Tags L 3 Construct a fourth loss function. Assign a weight β to the third loss function and a weight 1-β to the fourth loss function. Based on the weights corresponding to the third and fourth loss functions, perform a weighted summation of the third and fourth loss functions to obtain the first loss function.

[0112] It should be understood that in practical applications, the label classification model can be used not only to perform three-level label classification tasks, but also to perform more or fewer levels of label classification tasks. This application does not impose any limitations on the hierarchical structure of labels.

[0113] Step 504: Add a disturbance vector to the first eigenvector corresponding to the training reference data to obtain a second eigenvector corresponding to the training reference data.

[0114] Adding adversarial perturbation design during the training process of the label classification model is essentially adding a perturbation vector to the input data of the basic label classification model to be trained, that is, adding the perturbation vector to the first eigenvector corresponding to the training reference data, thereby obtaining the second eigenvector corresponding to the training reference data.

[0115] The reason why the adversarial perturbation design is introduced in the training process of the label classification model is that after confirming that the label classification model has converged in the process of training the label classification model, when the converged label classification model is tested using a test sample set with the same distribution as the training sample set, it is found that the F1 index of the test sample set is 5 to 10 points lower than the F1 index of the training sample set. It can be seen that the trained label classification model is overfitting and has poor generalization ability. In response to this situation, the method provided in the embodiment of the present application introduces an adversarial perturbation design in the training process of the label classification model to enhance the anti-interference ability of the label classification model, which plays an effect similar to regularization, thereby alleviating the overfitting phenomenon of the label classification model and improving the generalization ability of the model.

[0116] In actual applications, the server can determine the perturbation vector to be added to the first feature vector in any of the following ways: The first way is to determine the perturbation vector based on the upward direction of the gradient of the first loss function determined by step 503. For the basic label classification model, the most feared perturbation vector during its training process is the perturbation vector added in the upward direction of the gradient of the first loss function, because the training goal of the basic label classification model is to make the first loss function descend along the direction of the gradient. Adding a perturbation vector in the upward direction of the gradient of the first loss function will affect the model convergence during training. The second way is to determine the perturbation vector based on the downward direction of the gradient of the first loss function determined by step 503. For the basic label classification model, the simplest perturbation vector is the perturbation vector added in the downward direction of the gradient of the first loss function. The third way is to randomly generate a noise vector that conforms to the normal distribution as the perturbation vector.

[0117] It should be understood that no matter which of the above methods is used to determine the disturbance vector, the added disturbance vector needs to be limited to a certain amplitude to prevent the added disturbance vector from being too large and having a negative impact on model training.

[0118] Experimental research has found that adding a perturbation vector using the first method (i.e., adding the perturbation vector along the upward gradient of the first loss function) leads to the best convergence for the trained label classification model. Specifically, to determine the perturbation vector using this method, the partial derivative of the first loss function with respect to the first eigenvector is calculated, and the resulting gradient is normalized. This normalized result is then corrected using a preset perturbation coefficient to obtain the perturbation vector.

[0119] Specifically, the perturbation vector r that needs to be added to the first eigenvector can be calculated using formula (2):

[0120]

[0121] Wherein, γ is a disturbance coefficient, which is used to control the amplitude of the disturbance vector. In practical applications, the disturbance coefficient can be set according to actual needs, for example, it can be set to 0.5. represents the partial derivative of the first loss function loss with respect to the first eigenvector x. L2 is a normalization function that normalizes the gradient after the partial derivative to achieve gradient scaling and prevent excessive gradient fluctuations. This prevents adding a perturbation vector with excessive fluctuations to the first eigenvector, which would cause the second eigenvector corresponding to the obtained training reference data to fluctuate too much.

[0122] It should be noted that in practical applications, adding perturbation vectors to continuous values ​​is meaningful. Based on this, the server can add perturbation vectors in multiple locations, for example, adding a perturbation vector to the first eigenvector of the input, adding a perturbation vector to the dense vector obtained after aggregation processing by the feature aggregation layer Nextvlad in the unimodal feature aggregation module, adding a perturbation vector to the dense vector obtained after aggregation processing by the multimodal feature aggregation module, and so on. Experimental research has found that adding a perturbation vector to the first eigenvector of the original input often achieves the best model training results. This is because the feature aggregation layer in the unimodal feature aggregation module, such as Nextvlad, contains a relatively large number of parameters, which is the main reason for the model's easy overfitting. Adding a perturbation vector to the input of the unimodal feature aggregation module can improve the generalization of the unimodal feature aggregation module.

[0123] The following takes the training resource as a training video resource, and the training reference data includes multiple training video frames in the training video resource, training audio in the training video resource, and training reference text of the training video resource. After being processed by the feature extraction module, the first feature vectors corresponding to the multiple training video frames, the first feature vectors corresponding to the multiple Mel spectrum segments, and the first feature vector corresponding to the text segmentation are obtained as an example. Figure 6The diagram of the training principle of the label classification model shown in FIG. 1 exemplifies the implementation method of the server adding a perturbation vector to the first feature vector.

[0124] Specifically, the server can add a video perturbation vector to the first feature vectors corresponding to each of the multiple training video frames to obtain a second feature vector corresponding to each of the multiple training video frames; it can add an audio perturbation vector to the first feature vector corresponding to each of the multiple Mel-spectra segments to obtain a second feature vector corresponding to each of the multiple Mel-spectra segments; it can add a text perturbation vector to the first feature vector corresponding to the text word segmentation to obtain a second feature vector corresponding to the text word segmentation.

[0125] like Figure 6 As shown, the video perturbation vector, audio perturbation vector and text perturbation vector can be determined respectively based on the upward direction of the gradient of the first loss function, and then, the video perturbation vector, audio perturbation vector and text perturbation vector are added respectively to the first feature vectors corresponding to each of the multiple training video frames, the first feature vectors corresponding to each of the multiple Mel spectrum segments and the first feature vector corresponding to the text segmentation, thereby obtaining the second feature vectors corresponding to each of the multiple training video frames, the second feature vectors corresponding to each of the multiple Mel spectrum segments and the second feature vector corresponding to the text segmentation.

[0126] It should be understood that in actual applications, the server may also add the corresponding disturbance vector to any one or two of the first feature vectors corresponding to each of multiple training video frames, the first feature vectors corresponding to each of multiple mel-spectrogram segments, and the first feature vector corresponding to the text segmentation to obtain the corresponding second feature vector. This application does not impose any restrictions on the objects to which the disturbance vectors are added.

[0127] Step 505: Determine the second predicted probability that the training resource belongs to each candidate classification label according to the second eigenvector corresponding to the training reference data through the basic label classification model; determine the second loss function according to the second predicted probability and the labeled classification label.

[0128] After the server adds the perturbation vector to the first eigenvector corresponding to the training reference data to obtain a second eigenvector corresponding to the training reference data, it can further input the second eigenvector corresponding to the training reference data into the current basic label classification model to be trained, so that the basic label classification model analyzes and processes the second eigenvector corresponding to the training reference data and outputs a second predicted probability that the training resource belongs to each candidate classification label. Furthermore, the server can construct a second loss function based on the second predicted probabilities of the training resource belonging to each candidate classification label and the annotated classification label corresponding to the training resource included in the training sample.

[0129] The following takes the training resource as a training video resource, and adds a perturbation vector to the first feature vectors corresponding to each of the multiple training video frames in the training video resource, the first feature vectors corresponding to each of the multiple Mel spectrums corresponding to the training audio in the training video resource, and the first feature vector corresponding to the text segmentation in the training reference text of the training video resource as an example. Figure 6 The training principle diagram of the label classification model shown is an exemplary introduction to the implementation method of the server using the basic label classification model to determine the second prediction probability of the training resource belonging to each candidate classification label according to the second eigenvector corresponding to the training reference data.

[0130] like Figure 6 As shown, the server can use the basic label classification model to process the second feature vectors corresponding to each of the multiple training video frames, the second feature vectors corresponding to each of the multiple Mel spectrum segments, and the second feature vectors corresponding to the text segmentation. Figure 6 The process corresponding to the gray square is the process for processing the second eigenvector. Figure 6 As shown, a video feature aggregation layer, such as the Nextvlad structure, can be used to aggregate the second feature vectors corresponding to each training video frame to obtain a second aggregated training video feature vector. An audio feature aggregation layer, such as the Nextvlad structure, can be used to aggregate the second feature vectors corresponding to each mel-spectrogram to obtain a second aggregated training audio feature vector. A text feature aggregation layer, such as the TextCNN structure, can be used to aggregate the second feature vectors corresponding to each text word to obtain a second aggregated training text feature vector. Then, the modal feature aggregation module in the basic label classification model aggregates the second aggregated training video feature vector, the second aggregated training audio feature vector, and the second aggregated training text feature vector to obtain a second training feature vector. Furthermore, the label classification module in the basic label classification model determines, based on the second training feature vector, the second predicted probability that the training video resource belongs to each candidate classification label. Finally, a second loss function is determined based on the second predicted probability that the training video resource belongs to each candidate classification label and the annotated classification label corresponding to the training video resource.

[0131] Assuming the first eigenvector is x, add the perturbation vector r to the first eigenvector x to obtain the second eigenvector x+r that is input into the basic label classification model. The second eigenvector x+r is input into the basic label classification model. After a series of linear and nonlinear mapping processes by the basic label classification model, the second predicted probability of the training resource belonging to each candidate classification label is obtained. Next, the second loss function is minimized by sigmoid cross entropy using Equation (3), denoted as Adv_loss.

[0132] Adv_loss=-min log p(y|x+r,θ) (3)

[0133] Among them, y is the second predicted probability of the training resource belonging to each candidate classification label output by the basic label classification model, x+r is the second eigenvector input to the basic label classification model, and θ is the model parameter in the basic label classification model.

[0134] Similarly, in the process of training the basic label classification model based on the second eigenvector, the global classification label and the local classification label can also be comprehensively considered to improve the accuracy of the trained label classification model.

[0135] That is, the server can use the basic label classification model to determine the fifth predicted probability of the training resource belonging to each global classification label and the sixth predicted probability of the training resource belonging to each local classification label based on the second eigenvector corresponding to the training reference data. The global classification label here is obtained by combining local classification labels of different levels. Then, a fifth loss function can be determined based on the fifth predicted probability and the labeled classification label corresponding to the training resource, and a sixth loss function can be determined based on the sixth predicted probability and the labeled classification label corresponding to the training resource. Furthermore, the second loss function described above is determined based on the fifth and sixth loss functions.

[0136] The following uses a three-level label structure as an example to illustrate the above implementation method. The basic label classification model can use the first branch to determine the probability of a training resource belonging to each global classification label based on the input second eigenvector x+r. Here, the global classification label is the overall combination of the first-level label, the second-level label, and the third-level label. The basic label classification model can use the second branch, the third branch, and the fourth branch to determine the probability of a training resource belonging to each first-level label, the probability of belonging to each second-level label, and the probability of belonging to each third-level label, respectively, based on the input second eigenvector x+r. Furthermore, a fifth loss function is constructed based on the labeled classification label corresponding to the training resource and the probability of the training resource belonging to each global classification label. A sixth loss function is constructed based on the labeled classification label corresponding to the training resource and the probability of the training resource belonging to each local classification label. A weight β is assigned to the fifth loss function, and a weight 1-β is assigned to the sixth loss function. Based on the weights corresponding to the fifth and sixth loss functions, the fifth and sixth loss functions are weighted summed to obtain the second loss function.

[0137] It should be understood that in practical applications, the label classification model can be used not only to perform three-level label classification tasks, but also to perform more or fewer levels of label classification tasks. This application does not impose any limitations on the hierarchical structure of labels.

[0138] Step 506: Determine a target loss function based on the first loss function and the second loss function.

[0139] After the server obtains the first loss function through step 503 and the second loss function through step 505, it can further determine the target loss function for training the basic label classification model based on the first loss function and the second loss function.

[0140] Exemplarily, the server may add the first loss function Normal_loss to the second loss function Adv_loss to obtain the target loss function; alternatively, the server may configure corresponding weights for the first loss function Normal_loss and the second loss function Adv_loss, respectively, and perform weighted summation of the first loss function Normal_loss and the second loss function Adv_loss based on the weights corresponding to the first loss function Normal_loss and the second loss function Adv_loss, to obtain the target loss function. This application does not impose any limitation on the implementation method of determining the target loss function based on the first loss function and the second loss function.

[0141] Step 507: Based on the target loss function, the basic label classification model is trained.

[0142] After the server obtains the target loss function, it can train the basic label classification model based on the target loss function. If the target loss function can converge, the first loss function and the second loss function can also converge accordingly. In this way, the trained label classification model has a certain redundancy capability and the generalization ability of the network model is improved.

[0143] It should be understood that in actual applications, when the number of iterative training of the basic label classification model reaches a preset iteration number threshold, the training of the basic label classification model can be considered to have been completed, and the basic label classification model at this time can be used as a label classification model that can be put into practical application. Alternatively, when the performance of the trained basic label classification model meets the preset requirements, the training of the basic label classification model can be considered to have been completed, and the basic label classification model at this time can be used as a label classification model that can be put into practical application. This application does not impose any restrictions on the training completion conditions of the basic label classification model.

[0144] The above-mentioned training method of the label classification model introduces adversarial perturbation design during the training process of the label classification model, thereby alleviating the overfitting phenomenon of the label classification model and enabling the label classification model to have better generalization ability.

[0145] Experiments have shown that the baseline indicators of the video resource label classification model, which implements feature extraction and label classification in two stages, are as follows:

[0146] p=79.6%, R=65.4%, F1-score=71.8%

[0147] After introducing the adversarial perturbation design during the training process of the label classification model, the benefits of the trained label classification model are as follows:

[0148] p=81.4%, R=66.7%, F1-score=73.3%

[0149] After introducing the gate mechanism in the multimodal feature aggregation stage of the label classification model, the benefits of the label classification model are as follows:

[0150] p=81.9%, R=67.0%, F1-score=73.7%

[0151] After comprehensively considering the global classification labels and local classification labels during the training process of the label classification model, the benefits of the trained label classification model are as follows:

[0152] p=82.9%, R=67.7%, F1-score=74.5%

[0153] Among them, p is precision, that is, accuracy; r is recall, that is, recall rate; F1 is an indicator used in statistics to measure the accuracy of classification models, which takes into account both the accuracy and recall rate of the model.

[0154] With respect to the label configuration method described above, the present application also provides a corresponding label configuration device to enable the above label configuration method to be applied and implemented in practice.

[0155] See also Figure 9 , Figure 9 It is the above Figure 2 The structure diagram of the tag configuration device 900 corresponding to the tag configuration method shown in FIG. Figure 9 As shown, the label configuration device 900 includes:

[0156] Resource acquisition module 901, used to acquire the target resource to be configured with a tag;

[0157] A feature extraction module 902 is configured to perform feature extraction processing on the target reference data of the target resource using a feature extraction model to obtain a feature vector corresponding to the target reference data;

[0158] The label classification module 903 is used to determine the probability that the target resource belongs to each candidate classification label according to the feature vector corresponding to the target reference data through a label classification model; the label classification model is independent of the feature extraction model;

[0159] The label determination module 904 is configured to determine a target classification label corresponding to the target resource according to the probability that the target resource belongs to each candidate classification label.

[0160] Optional, in Figure 9 On the basis of the label configuration device shown, the target resource is a target video resource, and the target reference data includes at least one of the following: multiple target video frames in the target video resource, target audio in the target video resource, and target reference text of the target video resource, wherein the target reference text includes at least one of a title text and a content introduction text;

[0161] The feature extraction module 902 is specifically used to:

[0162] When the target reference data includes multiple target video frames in the target video resource, performing feature extraction processing on the multiple target video frames respectively through an image feature extraction model to obtain feature vectors corresponding to each of the multiple target video frames;

[0163] When the target reference data includes target audio in the target video resource, performing feature extraction processing on multiple Mel-spectrogram segments corresponding to the target audio using an audio feature extraction model to obtain feature vectors corresponding to each of the multiple Mel-spectrogram segments;

[0164] When the target reference data includes the target reference text of the target video resource, feature extraction processing is performed on the text segmentation in the target reference text through a text feature extraction model to obtain a feature vector corresponding to the text segmentation.

[0165] Optional, in Figure 9 On the basis of the label configuration device shown, when the target reference data includes multiple target video frames in the target video resource, target audio in the target video resource, and target reference text of the target video resource, the label classification model includes a unimodal feature aggregation module, a multimodal feature aggregation module and a label classification module, and the unimodal feature aggregation module includes a video feature aggregation layer, an audio feature aggregation layer and a text feature aggregation layer;

[0166] The tag classification module 903 is specifically used to:

[0167] Aggregating the feature vectors corresponding to the multiple target video frames through the video feature aggregation layer to obtain a video aggregate feature vector; aggregating the feature vectors corresponding to the multiple mel-spectrograms through the audio feature aggregation layer to obtain an audio aggregate feature vector; aggregating the feature vectors corresponding to the text segmentation through the text feature aggregation layer to obtain a text aggregate feature vector;

[0168] Aggregating the video aggregate feature vector, the audio aggregate feature vector, and the text aggregate feature vector through the multimodal feature aggregation module to obtain a target feature vector;

[0169] The label classification module determines the probability that the target resource belongs to each candidate classification label according to the target feature vector.

[0170] Optionally, the tag classification module 903 aggregates the video aggregate feature vector, the audio aggregate feature vector, and the text aggregate feature vector to obtain a target feature vector in the following manner:

[0171] Mapping the video aggregate feature vector, the audio aggregate feature vector, and the text aggregate feature vector to a target dimension through a fully connected layer in the multimodal feature aggregation module to obtain a target video feature vector, a target audio feature vector, and a target text feature vector;

[0172] Performing linear mapping processing on the video aggregate feature vector, the audio aggregate feature vector, and the text aggregate feature vector, respectively, and determining gate units corresponding to the target video feature vector, the target audio feature vector, and the target text feature vector, respectively, based on the video aggregate feature vector, the audio aggregate feature vector, and the text aggregate feature vector after the linear mapping processing;

[0173] Based on the gate units corresponding to the target video feature vector, the target audio feature vector and the target text feature vector, weighted aggregation processing is performed on the target video feature vector, the target audio feature vector and the target text feature vector to obtain the target feature vector.

[0174] Optionally, the tag classification module 903 aggregates the video aggregate feature vector, the audio aggregate feature vector, and the text aggregate feature vector to obtain a target feature vector in the following manner:

[0175] Mapping the video aggregate feature vector, the audio aggregate feature vector, and the text aggregate feature vector to a target dimension through a fully connected layer in the multimodal feature aggregation module to obtain a target video feature vector, a target audio feature vector, and a target text feature vector;

[0176] Determine the similarity between the target video feature vector and the target latent vector as the weight corresponding to the target video feature vector; determine the similarity between the target audio feature vector and the target latent vector as the weight corresponding to the target audio feature vector; determine the similarity between the target text feature vector and the target latent vector as the weight corresponding to the target text feature vector;

[0177] Based on the weights corresponding to the target video feature vector, the target audio feature vector and the target text feature vector, weighted aggregation processing is performed on the target video feature vector, the target audio feature vector and the target text feature vector to obtain the target feature vector.

[0178] Optional, in Figure 9 On the basis of the label configuration device shown, the label configuration device 900 further includes: a classification model training module 1000. Figure 10 Schematic diagram of the structure of the classification model training module 1000 provided in the embodiment of the present application. Figure 10 As shown, the classification model training module 1000 includes:

[0179] The training sample acquisition submodule 1001 is used to acquire training samples; the training samples include training resources and their corresponding annotated classification labels;

[0180] A feature extraction submodule 1002 is configured to perform feature extraction processing on the training reference data of the training resource using the feature extraction model to obtain a first feature vector corresponding to the training reference data;

[0181] A first loss function determination submodule 1003 is configured to determine, using a basic label classification model to be trained and based on a first eigenvector corresponding to the training reference data, a first prediction probability that the training resource belongs to each candidate classification label; and determine a first loss function based on the first prediction probability and the annotated classification label.

[0182] a disturbance vector adding submodule 1004, configured to add a disturbance vector to the first eigenvector corresponding to the training reference data to obtain a second eigenvector corresponding to the training reference data;

[0183] A second loss function determination submodule 1005 is configured to determine, using the basic label classification model and based on the second eigenvector corresponding to the training reference data, a second predicted probability that the training resource belongs to each candidate classification label; and determine a second loss function based on the second predicted probability and the annotated classification label.

[0184] A target loss function determination submodule 1006 is configured to determine a target loss function based on the first loss function and the second loss function;

[0185] The model training submodule 1007 is used to train the basic label classification model based on the target loss function.

[0186] Optional, in Figure 10 On the basis of the classification model training module shown, the training resource is a training video resource, and the training reference data includes at least one of the following: multiple training video frames in the training video resource, training audio in the training video resource, and training reference text of the training video resource; the training reference text includes at least one of a title text and a content introduction text;

[0187] The feature extraction submodule 1002 and the disturbance vector addition submodule 1004 are specifically used for:

[0188] When the training reference data includes multiple training video frames in the training video resource, performing feature extraction processing on the multiple training video frames respectively using an image feature extraction model to obtain first feature vectors corresponding to each of the multiple training video frames; adding a video perturbation vector to the first feature vectors corresponding to each of the multiple training video frames to obtain second feature vectors corresponding to each of the multiple training video frames;

[0189] When the training reference data includes training audio in the training video resource, performing feature extraction processing on multiple Mel-spectrogram segments corresponding to the training audio using an audio feature extraction model to obtain first feature vectors corresponding to each of the multiple Mel-spectrogram segments; adding an audio perturbation vector to the first feature vectors corresponding to each of the multiple Mel-spectrogram segments to obtain second feature vectors corresponding to each of the multiple Mel-spectrogram segments;

[0190] When the training reference data includes the training reference text of the training video resource, feature extraction processing is performed on the text segmentation in the training reference text through a text feature extraction model to obtain a first feature vector corresponding to the text segmentation; and a text perturbation vector is added to the first feature vector corresponding to the text segmentation to obtain a second feature vector corresponding to the text segmentation.

[0191] Optional, in Figure 10 On the basis of the classification model training module shown, the disturbance vector adding submodule 1004 is specifically configured to determine the disturbance vector by any of the following methods:

[0192] Determining the disturbance vector based on the upward direction of the gradient of the first loss function;

[0193] Determining the disturbance vector based on the downward direction of the gradient of the first loss function;

[0194] Generate a noise vector that conforms to the normal distribution as the disturbance vector.

[0195] Optional, in Figure 10 On the basis of the classification model training module shown, the disturbance vector adding submodule 1004 determines the disturbance vector based on the upward direction of the gradient of the first loss function in the following manner:

[0196] Calculating the partial derivative of the first loss function with respect to the first eigenvector;

[0197] Normalize the gradient after partial derivative;

[0198] The disturbance vector is obtained by correcting the result of the normalization process using a preset disturbance coefficient.

[0199] Optional, in Figure 10 Based on the classification model training module shown, the first loss function determination submodule 1003 is specifically used to:

[0200] Determining, by the basic label classification model, a third predicted probability that the training resource belongs to each global classification label and a fourth predicted probability that the training resource belongs to each local classification label based on the first eigenvector corresponding to the training reference data; the global classification label is obtained by combining the local classification labels at different levels;

[0201] Determining a third loss function based on the third predicted probability and the labeled classification label; determining a fourth loss function based on the fourth predicted probability and the labeled classification label;

[0202] Determine the first loss function according to the third loss function and the fourth loss function;

[0203] The second loss function determination submodule 1005 is specifically used to:

[0204] Determining, by the basic label classification model, a fifth predicted probability that the training resource belongs to each of the global classification labels and a sixth predicted probability that the training resource belongs to each of the local classification labels based on the second eigenvector corresponding to the training reference data;

[0205] Determining a fifth loss function based on the fifth predicted probability and the labeled classification label; determining a sixth loss function based on the sixth predicted probability and the labeled classification label;

[0206] The second loss function is determined according to the fifth loss function and the sixth loss function.

[0207] The label configuration device innovatively performs the label configuration task for the target resource through two independent stages. In the first stage, the target reference data of the target resource is first subjected to feature extraction processing by a feature extraction model to obtain a feature vector corresponding to the target reference data. In the second stage, the probability of the target resource belonging to each candidate classification label is determined by a label classification model based on the feature vector corresponding to the target reference data. Compared with the model obtained by training through the End2End method in the related art, the device decouples the feature extraction structure and label classification structure required for performing the label configuration task, and uses independent feature extraction models and label classification models to perform feature extraction operations and label classification respectively; accordingly, when iteratively training the model used in performing the label configuration task, the feature extraction model and the label classification model can be trained independently; for the feature extraction model with a large number of parameters, it is usually relatively fixed and the iteration frequency is low; for the label classification model with a small number of parameters, the iteration frequency is high, but it is iteratively trained using large-scale training samples, without consuming a long iteration training time. In this way, the two structures with different iteration frequencies are decoupled. In actual applications, the label classification model used in the second stage is mainly iterated. Large-scale training samples can be used to complete the iterative training of the model in a shorter iterative training time, meeting the business needs of the network platform in actual product applications.

[0208] The embodiment of the present application also provides a device for configuring tags, which may specifically be a terminal device or a server. The terminal device and server provided in the embodiment of the present application will be introduced below from the perspective of hardware entity.

[0209] See also Figure 11 , Figure 11 This is a schematic diagram of the structure of the terminal device provided in the embodiment of the present application. Figure 11 For ease of explanation, only the parts related to the embodiments of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiments of the present application. The terminal can be any terminal device including a mobile phone, a tablet computer, a personal digital assistant (PDA), a point of sales (POS), a car computer, etc. For example, the terminal is a computer:

[0210] Figure 11 FIG2 is a block diagram showing a partial structure of a computer related to a terminal provided in an embodiment of the present application. Figure 11The computer includes components such as a radio frequency (RF) circuit 1110, a memory 1120, an input unit 1130, a display unit 1140, a sensor 1150, an audio circuit 1160, a wireless fidelity (WiFi) module 1170, a processor 1180, and a power supply 1190. Those skilled in the art will appreciate that Figure 11 The computer structure shown in the figure does not constitute a limitation of the computer, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0211] Memory 1120 can be used to store software programs and modules. Processor 1180 executes the various computer functions and applications and processes data by running the software programs and modules stored in memory 1120. Memory 1120 may primarily include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function (such as a sound playback function or an image playback function); the data storage area may store data created based on the use of the computer (such as audio data, a phone book, etc.). In addition, memory 1120 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0212] Processor 1180 is the computer's control center, connecting all computer components using various interfaces and circuits. It executes software programs and / or modules stored in memory 1120 and accesses data stored in memory 1120 to perform various computer functions and process data. Optionally, processor 1180 may include one or more processing units. Preferably, processor 1180 integrates an application processor and a modem processor. The application processor primarily handles the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into processor 1180.

[0213] In the embodiment of the present application, the processor 1180 included in the terminal further has the following functions:

[0214] Get the target resource to be configured with the label;

[0215] Performing feature extraction processing on the target reference data of the target resource through a feature extraction model to obtain a feature vector corresponding to the target reference data;

[0216] Determine the probability that the target resource belongs to each candidate classification label based on the feature vector corresponding to the target reference data through a label classification model; the label classification model and the feature extraction model are independent of each other;

[0217] The target classification label corresponding to the target resource is determined according to the probability that the target resource belongs to each candidate classification label.

[0218] Optionally, the processor 1180 is further configured to execute steps of any implementation of the tag configuration method provided in the embodiments of the present application.

[0219] See also Figure 12 , Figure 12 A structural diagram of a server 1200 provided for an embodiment of the present application. The server 1200 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1222 (for example, one or more processors) and memory 1232, and one or more storage media 1230 (for example, one or more mass storage devices) for storing application programs 1242 or data 1244. Among them, the memory 1232 and the storage medium 1230 can be temporary storage or permanent storage. The program stored in the storage medium 1230 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 1222 can be configured to communicate with the storage medium 1230 to execute a series of instruction operations in the storage medium 1230 on the server 1200.

[0220] The server 1200 may also include one or more power supplies 1226, one or more wired or wireless network interfaces 1250, one or more input and output interfaces 1258, and / or one or more operating systems, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0221] The steps performed by the server in the above embodiment can be based on the Figure 12 The server structure shown.

[0222] The CPU 1222 is configured to execute the following steps:

[0223] Get the target resource to be configured with the label;

[0224] Performing feature extraction processing on the target reference data of the target resource through a feature extraction model to obtain a feature vector corresponding to the target reference data;

[0225] Determine the probability that the target resource belongs to each candidate classification label based on the feature vector corresponding to the target reference data through a label classification model; the label classification model and the feature extraction model are independent of each other;

[0226] The target classification label corresponding to the target resource is determined according to the probability that the target resource belongs to each candidate classification label.

[0227] Optionally, the CPU 1222 may also be configured to execute steps of any implementation of the tag configuration method provided in the embodiments of the present application.

[0228] An embodiment of the present application further provides a computer-readable storage medium for storing a computer program, wherein the computer program is used to execute any one of the implementation methods of a tag configuration method described in the aforementioned embodiments.

[0229] The present application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any one of the tag configuration methods described in the aforementioned embodiments.

[0230] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0231] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0232] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0233] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0234] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (English full name: Read-Only Memory, English abbreviation: ROM), random access memory (English full name: Random Access Memory, English abbreviation: RAM), disk or optical disk and other media that can store computer programs.

[0235] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0236] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

[0237] It should be noted that the user information (including but not limited to user device information, user personal information, user portrait information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

Claims

1. A label configuration method, characterized in that: The method comprises: Obtain a target resource to be configured with a tag; the target resource is a target video resource; Performing feature extraction processing on target reference data of the target resource using a feature extraction model to obtain a feature vector corresponding to the target reference data; the target reference data includes at least one of the following: multiple target video frames in the target video resource, target audio in the target video resource, and target reference text in the target video resource; Determine the probability that the target resource belongs to each candidate classification label based on the feature vector corresponding to the target reference data through a label classification model; the label classification model and the feature extraction model are independent of each other; Determining a target classification label corresponding to the target resource based on the probability that the target resource belongs to each candidate classification label; The label classification model is trained as follows: Obtain training resources and their corresponding annotation classification labels; Extracting a first feature vector corresponding to the training reference data of the training resource through the feature extraction model; determining a first prediction probability of the training resource belonging to each candidate classification label based on the first feature vector through the basic label classification model to be trained; determining a first loss function based on the first prediction probability and the annotated classification label; the candidate classification labels include a global classification label and a local classification label; the global classification label is obtained by combining the local classification labels of different levels; Adding a perturbation vector to the first eigenvector to obtain a second eigenvector corresponding to the training reference data; determining, using the basic label classification model, a second predicted probability that the training resource belongs to each of the candidate classification labels based on the second eigenvector; and determining a second loss function based on the second predicted probability and the annotated classification label; A target loss function is determined according to the first loss function and the second loss function; and the basic label classification model is trained based on the target loss function.

2. The method according to claim 1, characterized in that The target reference text includes at least one of a title text and a content introduction text; The feature extraction model is used to perform feature extraction processing on the target reference data of the target resource to obtain a feature vector corresponding to the target reference data, including: When the target reference data includes multiple target video frames in the target video resource, performing feature extraction processing on the multiple target video frames respectively through an image feature extraction model to obtain feature vectors corresponding to each of the multiple target video frames; When the target reference data includes target audio in the target video resource, performing feature extraction processing on multiple Mel-spectrogram segments corresponding to the target audio using an audio feature extraction model to obtain feature vectors corresponding to each of the multiple Mel-spectrogram segments; When the target reference data includes the target reference text of the target video resource, feature extraction processing is performed on the text segmentation in the target reference text through a text feature extraction model to obtain a feature vector corresponding to the text segmentation.

3. The method according to claim 2, characterized in that When the target reference data includes multiple target video frames in the target video resource, target audio in the target video resource, and target reference text of the target video resource, the label classification model includes a unimodal feature aggregation module, a multimodal feature aggregation module and a label classification module, and the unimodal feature aggregation module includes a video feature aggregation layer, an audio feature aggregation layer and a text feature aggregation layer; The determining, using a label classification model and based on a feature vector corresponding to the target reference data, the probability that the target resource belongs to each candidate classification label includes: Aggregating the feature vectors corresponding to the multiple target video frames through the video feature aggregation layer to obtain a video aggregate feature vector; aggregating the feature vectors corresponding to the multiple mel-spectrograms through the audio feature aggregation layer to obtain an audio aggregate feature vector; aggregating the feature vectors corresponding to the text segmentation through the text feature aggregation layer to obtain a text aggregate feature vector; Aggregating the video aggregate feature vector, the audio aggregate feature vector, and the text aggregate feature vector through the multimodal feature aggregation module to obtain a target feature vector; The label classification module determines the probability that the target resource belongs to each candidate classification label according to the target feature vector.

4. The method according to claim 3, characterized in that The step of aggregating the video aggregate feature vector, the audio aggregate feature vector, and the text aggregate feature vector through the multimodal feature aggregation module to obtain a target feature vector includes: Mapping the video aggregate feature vector, the audio aggregate feature vector, and the text aggregate feature vector to a target dimension through a fully connected layer in the multimodal feature aggregation module to obtain a target video feature vector, a target audio feature vector, and a target text feature vector; Performing linear mapping processing on the video aggregate feature vector, the audio aggregate feature vector, and the text aggregate feature vector, respectively, and determining gate units corresponding to the target video feature vector, the target audio feature vector, and the target text feature vector, respectively, based on the video aggregate feature vector, the audio aggregate feature vector, and the text aggregate feature vector after the linear mapping processing; Based on the gate units corresponding to the target video feature vector, the target audio feature vector and the target text feature vector, weighted aggregation processing is performed on the target video feature vector, the target audio feature vector and the target text feature vector to obtain the target feature vector.

5. The method according to claim 3, characterized in that The step of aggregating the video aggregate feature vector, the audio aggregate feature vector, and the text aggregate feature vector through the multimodal feature aggregation module to obtain a target feature vector includes: Mapping the video aggregate feature vector, the audio aggregate feature vector, and the text aggregate feature vector to a target dimension through a fully connected layer in the multimodal feature aggregation module to obtain a target video feature vector, a target audio feature vector, and a target text feature vector; Determine the similarity between the target video feature vector and the target latent vector as the weight corresponding to the target video feature vector; determine the similarity between the target audio feature vector and the target latent vector as the weight corresponding to the target audio feature vector; determine the similarity between the target text feature vector and the target latent vector as the weight corresponding to the target text feature vector; Based on the weights corresponding to the target video feature vector, the target audio feature vector and the target text feature vector, weighted aggregation processing is performed on the target video feature vector, the target audio feature vector and the target text feature vector to obtain the target feature vector.

6. The method according to claim 1, characterized in that The training resource is a training video resource, and the training reference data includes at least one of the following: a plurality of training video frames in the training video resource, training audio in the training video resource, and training reference text of the training video resource; The training reference text includes at least one of a title text and a content introduction text; The extracting, by the feature extraction model, a first feature vector corresponding to the training reference data of the training resource, and adding a perturbation vector to the first feature vector to obtain a second feature vector corresponding to the training reference data include: When the training reference data includes multiple training video frames in the training video resource, performing feature extraction processing on the multiple training video frames respectively using an image feature extraction model to obtain first feature vectors corresponding to each of the multiple training video frames; adding a video perturbation vector to the first feature vectors corresponding to each of the multiple training video frames to obtain second feature vectors corresponding to each of the multiple training video frames; When the training reference data includes training audio in the training video resource, performing feature extraction processing on multiple Mel-spectrogram segments corresponding to the training audio using an audio feature extraction model to obtain first feature vectors corresponding to each of the multiple Mel-spectrogram segments; adding an audio perturbation vector to the first feature vectors corresponding to each of the multiple Mel-spectrogram segments to obtain second feature vectors corresponding to each of the multiple Mel-spectrogram segments; When the training reference data includes the training reference text of the training video resource, feature extraction processing is performed on the text segmentation in the training reference text through a text feature extraction model to obtain a first feature vector corresponding to the text segmentation; and a text perturbation vector is added to the first feature vector corresponding to the text segmentation to obtain a second feature vector corresponding to the text segmentation.

7. The method according to claim 1, characterized in that The perturbation vector is determined by any of the following methods: Determining the disturbance vector based on the upward direction of the gradient of the first loss function; Determining the disturbance vector based on the downward direction of the gradient of the first loss function; Generate a noise vector that conforms to the normal distribution as the disturbance vector.

8. The method according to claim 7, characterized in that The determining the disturbance vector based on the upward direction of the gradient of the first loss function includes: Calculating the partial derivative of the first loss function with respect to the first eigenvector; Normalize the gradient after partial derivative; The disturbance vector is obtained by correcting the result of the normalization process using a preset disturbance coefficient.

9. The method according to claim 1, characterized in that The method of determining, by using the basic label classification model to be trained, a first prediction probability that the training resource belongs to each candidate classification label according to the first feature vector; and determining a first loss function according to the first prediction probability and the labeled classification label, includes: Determining, by the basic label classification model and based on the first eigenvector corresponding to the training reference data, a third predicted probability that the training resource belongs to each global classification label and a fourth predicted probability that the training resource belongs to each local classification label; Determining a third loss function based on the third predicted probability and the labeled classification label; determining a fourth loss function based on the fourth predicted probability and the labeled classification label; Determine the first loss function according to the third loss function and the fourth loss function; The method of determining, by the basic label classification model, a second prediction probability that the training resource belongs to each candidate classification label according to the second feature vector; and determining a second loss function according to the second prediction probability and the annotated classification label, includes: Determining, by the basic label classification model, a fifth predicted probability that the training resource belongs to each of the global classification labels and a sixth predicted probability that the training resource belongs to each of the local classification labels based on the second eigenvector corresponding to the training reference data; Determining a fifth loss function based on the fifth predicted probability and the labeled classification label; determining a sixth loss function based on the sixth predicted probability and the labeled classification label; The second loss function is determined according to the fifth loss function and the sixth loss function.

10. A label configuration device, characterized in that: The device comprises: A resource acquisition module is used to acquire a target resource to be configured with a tag; the target resource is a target video resource; a feature extraction module, configured to perform feature extraction processing on target reference data of the target resource using a feature extraction model to obtain a feature vector corresponding to the target reference data; the target reference data comprising at least one of the following: multiple target video frames in the target video resource, target audio in the target video resource, and target reference text in the target video resource; a label classification module, configured to determine the probability that the target resource belongs to each candidate classification label according to the feature vector corresponding to the target reference data through a label classification model; the label classification model is independent of the feature extraction model; A label determination module, configured to determine a target classification label corresponding to the target resource based on the probability that the target resource belongs to each candidate classification label; A classification model training module, wherein the classification model training module includes: The training sample acquisition submodule is used to obtain training resources and their corresponding annotation classification labels; a feature extraction submodule, configured to extract a first feature vector corresponding to the training reference data of the training resource through the feature extraction model; A first loss function determination submodule is configured to determine, using a basic label classification model to be trained, a first prediction probability that the training resource belongs to each candidate classification label according to the first feature vector; and determine a first loss function according to the first prediction probability and the annotated classification label; the candidate classification labels include global classification labels and local classification labels; the global classification label is obtained by combining the local classification labels at different levels; a disturbance vector adding submodule, configured to add a disturbance vector to the first eigenvector to obtain a second eigenvector corresponding to the training reference data; A second loss function determination submodule is configured to determine, by using the basic label classification model, a second prediction probability that the training resource belongs to each of the candidate classification labels according to the second feature vector; and determine a second loss function according to the second prediction probability and the annotated classification label; A target loss function determination submodule, configured to determine a target loss function based on the first loss function and the second loss function; The model training submodule is used to train the basic label classification model based on the target loss function.

11. The device according to claim 10, characterized in that The target reference text includes at least one of a title text and a content introduction text; The feature extraction module is specifically used for: When the target reference data includes multiple target video frames in the target video resource, performing feature extraction processing on the multiple target video frames respectively through an image feature extraction model to obtain feature vectors corresponding to each of the multiple target video frames; When the target reference data includes target audio in the target video resource, performing feature extraction processing on multiple Mel-spectrogram segments corresponding to the target audio using an audio feature extraction model to obtain feature vectors corresponding to each of the multiple Mel-spectrogram segments; When the target reference data includes the target reference text of the target video resource, feature extraction processing is performed on the text segmentation in the target reference text through a text feature extraction model to obtain a feature vector corresponding to the text segmentation.

12. The device according to claim 11, characterized in that When the target reference data includes multiple target video frames in the target video resource, target audio in the target video resource, and target reference text of the target video resource, the label classification model includes a unimodal feature aggregation module, a multimodal feature aggregation module and a label classification module, and the unimodal feature aggregation module includes a video feature aggregation layer, an audio feature aggregation layer and a text feature aggregation layer; The label classification module is specifically used for: Aggregating the feature vectors corresponding to the multiple target video frames through the video feature aggregation layer to obtain a video aggregate feature vector; aggregating the feature vectors corresponding to the multiple mel-spectrograms through the audio feature aggregation layer to obtain an audio aggregate feature vector; aggregating the feature vectors corresponding to the text segmentation through the text feature aggregation layer to obtain a text aggregate feature vector; Aggregating the video aggregate feature vector, the audio aggregate feature vector, and the text aggregate feature vector through the multimodal feature aggregation module to obtain a target feature vector; The label classification module determines the probability that the target resource belongs to each candidate classification label according to the target feature vector.

13. The device according to claim 12, characterized in that The label classification module is specifically used for: Mapping the video aggregate feature vector, the audio aggregate feature vector, and the text aggregate feature vector to a target dimension through a fully connected layer in the multimodal feature aggregation module to obtain a target video feature vector, a target audio feature vector, and a target text feature vector; Performing linear mapping processing on the video aggregate feature vector, the audio aggregate feature vector, and the text aggregate feature vector, respectively, and determining gate units corresponding to the target video feature vector, the target audio feature vector, and the target text feature vector, respectively, based on the video aggregate feature vector, the audio aggregate feature vector, and the text aggregate feature vector after the linear mapping processing; Based on the gate units corresponding to the target video feature vector, the target audio feature vector and the target text feature vector, weighted aggregation processing is performed on the target video feature vector, the target audio feature vector and the target text feature vector to obtain the target feature vector.

14. The device according to claim 12, characterized in that The label classification module is specifically used for: Mapping the video aggregate feature vector, the audio aggregate feature vector, and the text aggregate feature vector to a target dimension through a fully connected layer in the multimodal feature aggregation module to obtain a target video feature vector, a target audio feature vector, and a target text feature vector; Determine the similarity between the target video feature vector and the target latent vector as the weight corresponding to the target video feature vector; determine the similarity between the target audio feature vector and the target latent vector as the weight corresponding to the target audio feature vector; determine the similarity between the target text feature vector and the target latent vector as the weight corresponding to the target text feature vector; Based on the weights corresponding to the target video feature vector, the target audio feature vector and the target text feature vector, weighted aggregation processing is performed on the target video feature vector, the target audio feature vector and the target text feature vector to obtain the target feature vector.

15. The device according to claim 10, characterized in that The training resource is a training video resource, and the training reference data includes at least one of the following: a plurality of training video frames in the training video resource, training audio in the training video resource, and training reference text of the training video resource; the training reference text includes at least one of a title text and a content introduction text; The feature extraction submodule and the disturbance vector addition submodule are specifically used to: When the training reference data includes multiple training video frames in the training video resource, performing feature extraction processing on the multiple training video frames respectively using an image feature extraction model to obtain first feature vectors corresponding to each of the multiple training video frames; adding a video perturbation vector to the first feature vectors corresponding to each of the multiple training video frames to obtain second feature vectors corresponding to each of the multiple training video frames; When the training reference data includes training audio in the training video resource, performing feature extraction processing on multiple Mel-spectrogram segments corresponding to the training audio using an audio feature extraction model to obtain first feature vectors corresponding to each of the multiple Mel-spectrogram segments; adding an audio perturbation vector to the first feature vectors corresponding to each of the multiple Mel-spectrogram segments to obtain second feature vectors corresponding to each of the multiple Mel-spectrogram segments; When the training reference data includes the training reference text of the training video resource, feature extraction processing is performed on the text segmentation in the training reference text through a text feature extraction model to obtain a first feature vector corresponding to the text segmentation; and a text perturbation vector is added to the first feature vector corresponding to the text segmentation to obtain a second feature vector corresponding to the text segmentation.

16. The device according to claim 10, characterized in that The disturbance vector adding submodule is specifically configured to determine the disturbance vector by any of the following methods: Determining the disturbance vector based on the upward direction of the gradient of the first loss function; Determining the disturbance vector based on the downward direction of the gradient of the first loss function; Generate a noise vector that conforms to the normal distribution as the disturbance vector.

17. The device according to claim 16, characterized in that The disturbance vector adding submodule determines the disturbance vector based on the upward direction of the gradient of the first loss function in the following manner: Calculating the partial derivative of the first loss function with respect to the first eigenvector; Normalize the gradient after partial derivative; The disturbance vector is obtained by correcting the result of the normalization process using a preset disturbance coefficient.

18. The device according to claim 10, characterized in that The first loss function determination submodule is specifically used to: Determining, by the basic label classification model and based on the first eigenvector corresponding to the training reference data, a third predicted probability that the training resource belongs to each global classification label and a fourth predicted probability that the training resource belongs to each local classification label; Determining a third loss function based on the third predicted probability and the labeled classification label; determining a fourth loss function based on the fourth predicted probability and the labeled classification label; Determine the first loss function according to the third loss function and the fourth loss function; The second loss function determination submodule is specifically used to: Determining, by the basic label classification model, a fifth predicted probability that the training resource belongs to each of the global classification labels and a sixth predicted probability that the training resource belongs to each of the local classification labels based on the second eigenvector corresponding to the training reference data; Determining a fifth loss function based on the fifth predicted probability and the labeled classification label; determining a sixth loss function based on the sixth predicted probability and the labeled classification label; The second loss function is determined according to the fifth loss function and the sixth loss function.

19. A device, characterized in that The device includes a processor and a memory; The memory is used to store computer programs; The processor is configured to execute the tag configuration method according to any one of claims 1 to 9 according to the computer program.

20. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store a computer program, and the computer program is used to execute the tag configuration method according to any one of claims 1 to 9.

21. A computer program product, characterized in that The method comprises computer instructions, which are stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the label configuration method according to any one of claims 1 to 9.

Images