Media data processing method and device, equipment and storage medium

Abstract

The embodiment of the invention discloses a media data processing method and device, equipment and a storage medium, and is applied to the artificial intelligence technology, and the method comprises the steps: carrying out the feature fusion of a sample media feature and tag text features corresponding to N candidate tags, and obtaining a sample fusion feature; according to the sample fusion features and label labeling probabilities corresponding to the N candidate labels respectively, generating association relationship features corresponding to the N candidate labels respectively; and according to the association relationship features and the sample fusion features corresponding to the N candidate tags, predicting predicted tag probabilities corresponding to the N candidate tags, and according to the label labeling probabilities and the predicted tag probabilities corresponding to the N candidate tags, performing iterative training on the initial tag generation model to obtain a target tag generation model. According to the method and the device, the interaction relationship and the coexistence statistical rule between different candidate tags are explicitly considered, and the accuracy of model training and the accuracy of media tag generation can be improved.

Description

Technical Field

[0001] The present application relates to fields such as artificial intelligence technology, and in particular to a media data processing method, device, equipment and storage medium. Background Art

[0002] Generating media tags is a crucial component of media data processing. Automatically generating media tags for massive amounts of media data can provide downstream content distribution chains (such as recommendation systems, content operations, and content review) with media content features of varying granularity, improving content distribution efficiency while significantly reducing the cost of manual content review. Due to the diversity of media data, the number of media tags in tag libraries typically used in business scenarios can reach hundreds of thousands or even millions, making it challenging to assign appropriate media tags to each piece of media data.

[0003] Currently, the main approach to generating media labels for media data is to train a classification model. Specifically, a batch of media data is labeled with corresponding media labels, a training set is constructed, and a classification model is trained using the training set. The classification model independently predicts the presence of each candidate media label, and the predictions are then trained based on the predictions and the labeled media labels. Training a classification model essentially involves having the model perform an independent binary classification task for each candidate label. In practice, this model training approach has been found to be relatively inaccurate, leading to low accuracy in generating media labels for media data. Summary of the Invention

[0004] The embodiments of the present application provide a media data processing method, apparatus, device, and storage medium, which can improve the accuracy of model training and the accuracy of media tag generation.

[0005] An embodiment of the present application provides a method for processing media data, including:

[0006] Obtaining sample media features of the sample media data, as well as label description text and labeling probabilities corresponding to N candidate labels of the sample media data, where N is an integer greater than 1, and the labeling probabilities reflect the probability that the sample media data has the corresponding candidate label;

[0007] Using the initial label generation model, feature extraction is performed on the label description texts corresponding to the N candidate labels to obtain label text features corresponding to the N candidate labels. Feature fusion is performed on the sample media features and the label text features corresponding to the N candidate labels to obtain sample fusion features corresponding to the N candidate labels.

[0008] Generate association features corresponding to the N candidate labels based on the sample fusion features corresponding to the N candidate labels and the labeling probabilities; the association features reflect the association relationship between the corresponding candidate label and other candidate labels;

[0009] According to the association relationship features corresponding to the above N candidate labels and the above sample fusion features, the predicted label probabilities corresponding to the above N candidate labels are predicted. According to the labeled label probabilities corresponding to the above N candidate labels and the above predicted label probabilities, the above initial label generation model is iteratively trained to obtain the target label generation model.

[0010] An embodiment of the present application provides a method for processing media data, including:

[0011] When receiving target media data to be processed, obtaining target media features of the target media data;

[0012] The target label generation model is used to extract features from the label description texts corresponding to the N candidate labels to obtain label text features corresponding to the N candidate labels. Feature fusion is performed on the target media features and the label text features corresponding to the N candidate labels to obtain target fusion features corresponding to the N candidate labels. The target label generation model is obtained by iteratively training the initial label generation model using the method described above.

[0013] Generate updated association relationship features corresponding to the N candidate labels respectively according to the target fusion features corresponding to the N candidate labels and the annotated label probabilities;

[0014] According to the target fusion feature and the update association relationship feature, a media label of the target media data is determined from the N candidate labels.

[0015] An embodiment of the present application provides, on one hand, a media data processing device, including:

[0016] An acquisition module is configured to acquire sample media features of sample media data, as well as label description text and labeling probabilities corresponding to N candidate labels of the sample media data, where N is an integer greater than 1, and the labeling probabilities reflect the probability that the sample media data has the corresponding candidate label;

[0017] An extraction module is configured to extract features from the label description texts corresponding to the N candidate labels using an initial label generation model to obtain label text features corresponding to the N candidate labels, and to fuse the sample media features with the label text features corresponding to the N candidate labels to obtain sample fusion features corresponding to the N candidate labels.

[0018] A generation module is used to generate association relationship features corresponding to the N candidate tags respectively based on the sample fusion features corresponding to the N candidate tags and the annotated tag probabilities; the association relationship features reflect the association relationship between the corresponding candidate tag and other candidate tags;

[0019] The training module is used to predict the predicted label probabilities corresponding to the above N candidate labels based on the association relationship features corresponding to the above N candidate labels and the above sample fusion features, and iteratively train the above initial label generation model based on the labeled label probabilities corresponding to the above N candidate labels and the above predicted label probabilities to obtain the target label generation model.

[0020] An embodiment of the present application provides, on one hand, a media data processing device, including:

[0021] an acquisition module, configured to acquire target media features of the target media data upon receiving the target media data to be processed;

[0022] An extraction module is configured to extract features from the label description texts corresponding to the N candidate labels using a target label generation model to obtain label text features corresponding to the N candidate labels, and to fuse the target media features with the label text features corresponding to the N candidate labels to obtain target fusion features corresponding to the N candidate labels. The target label generation model is obtained by iteratively training the initial label generation model using the aforementioned method.

[0023] A generation module is used to generate updated association relationship features corresponding to the N candidate tags respectively according to the target fusion features corresponding to the N candidate tags and the annotated tag probabilities;

[0024] A determination module is configured to determine a media tag of the target media data from the N candidate tags based on the target fusion feature and the updated association relationship feature.

[0025] On one hand, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0026] On one hand, an embodiment of the present application provides a computer program product, including a computer program, which implements the steps of the above method when executed by a processor.

[0027] In this application, the association relationship feature reflects the association relationship between different candidate tags, that is, the association relationship feature reflects the interaction relationship and coexistence statistical laws between different candidate tags. In other words, in the process of training the initial tag generation model, the interaction relationship and coexistence statistical laws between different candidate tags are explicitly considered, which is conducive to providing more fine-grained information for the training process of the initial tag generation model and improving the training accuracy of the initial tag generation model. The target tag generation model obtained through training can solve the problem of low accuracy in the generation of media tags due to independent prediction of each candidate tag, thereby improving the generation accuracy of media tags. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0029] Figure 1 This is a schematic diagram of a media data processing system provided by this application;

[0030] Figure 2 This is a schematic diagram of an application scenario of a media data processing method provided by this application;

[0031] Figure 3 This is a schematic diagram of an application scenario of a media data processing method provided by this application;

[0032] Figure 4 This is a flowchart of a media data processing method provided by this application;

[0033] Figure 5 This is a schematic diagram of a scenario for obtaining video multimodal features of sample video data provided by this application;

[0034] Figure 6 is a schematic diagram of sample video data provided by this application;

[0035] Figure 7 This is a flowchart of another media data processing method provided by this application;

[0036] Figure 8 This is a flowchart of another media data processing method provided by this application;

[0037] Figure 9 This is a schematic structural diagram of a media data processing device provided in an embodiment of the present application;

[0038] Figure 10This is a schematic structural diagram of a media data processing device provided in an embodiment of the present application;

[0039] Figure 11 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0041] The embodiments of the present application may involve artificial intelligence technology and fields such as autonomous driving and smart transportation. Artificial intelligence (AC) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is the study of the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0042] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0043] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0044] This application primarily involves machine learning technology within artificial intelligence technology. It utilizes machine learning technology to construct an initial label generation model. During the training process of the initial label generation model, the interaction and coexistence statistical patterns between different candidate labels are considered. This facilitates providing more fine-grained information for the training process of the initial label generation model, thereby improving the training accuracy of the initial label generation model. The target label generation model obtained through training can address the issue of low media label generation accuracy caused by independently predicting each candidate label, thereby improving the accuracy of media label generation.

[0045] In order to facilitate a clearer understanding of the present application, the media data processing system for implementing the present application is first introduced. Figure 1 As shown, the media data processing system includes a server 10 and a terminal cluster. The terminal cluster may include one or more terminals. The number of terminals is not limited here. Figure 1 As shown, taking the terminal cluster including 4 terminals as an example, the terminal cluster can specifically include terminal 11a, terminal 12a, terminal 13a, and terminal 14a; it can be understood that terminal 11a, terminal 12a, terminal 13a, and terminal 14a can all be connected to the server 10 through the network, so that each terminal can exchange data with the server 10 through the network connection.

[0046] It is understandable that the server can be an independent physical server, or a server cluster or distributed system composed of at least two physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN), and basic cloud computing services such as big data and artificial intelligence platforms. The terminal can specifically refer to a vehicle-mounted terminal, a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a screen speaker, a smart TV, a smart watch, etc., but is not limited to this. Each terminal and server can be directly or indirectly connected through wired or wireless communication. At the same time, the number of terminals and servers can be one or at least two, and this application does not limit this.

[0047] Each of the above terminals may have one or more target applications installed. The target applications herein may refer to applications that have media data processing capabilities (e.g., generating and retrieving media data). For example, target applications may include independent applications, web applications, and mini-programs within a host application. Specifically, target applications may include browser applications, video playback applications, social networking applications, and content publishing applications.

[0048] Among them, the server 10 refers to a device that provides back-end services for the target application in the terminal. In one embodiment, the server 10 can train the initial label generation model based on the sample media data to obtain the target label generation model. The target label generation model in the server 10 can be called by the target application in the terminal to generate media labels for the target media data.

[0049] In one embodiment, after obtaining the target tag generation model, the server 10 can send the target tag generation model to each terminal (taking terminal 11a as an example). Terminal 11a can store the target tag generation model locally and call the target tag generation model from the local through the target application to generate media tags for the target media data.

[0050] In one embodiment, after obtaining the target tag generation model, the server 10 can send the target tag generation model to the cloud server, and each terminal (taking terminal 11a as an example) can call the target tag generation model in the cloud server through the target application to generate media tags for the target media data.

[0051] The sample media data of this application may refer to media data used to train the initial label generation model, the target media data may refer to media data for which labels are to be generated, and the media data may include video data, image data, audio data, etc. Candidate labels may refer to media labels that media data may have; when the media data is video data or image data, the media labels of the media data may refer to object categories corresponding to objects contained in the media data, subject categories of the media data, etc. Object types may include one or more of animals, plants, buildings, people, etc.; subject categories may include life, technology, knowledge, etc. When the media data is audio data, the candidate labels of the media data may be used to reflect attributes such as the audio style of the media data, and audio styles may include rock, folk, light music, etc.

[0052] It should be noted that dedicated target label generation models corresponding to video data, audio data, and image data can be trained. For example, the sample media data may include sample video data, and the initial label generation model is iteratively trained using the sample video data to obtain a target label generation model for generating media labels for video data. Alternatively, the sample media data may include sample image data, and the initial label generation model is iteratively trained using the sample image data to obtain a target label generation model for generating media labels for image data. Alternatively, the sample media data may include sample audio data, and the initial label generation model is iteratively trained using the sample audio data to obtain a target label generation model for generating media labels for audio data.

[0053] Alternatively, the sample media data may include sample image data, sample video data, and sample audio data. The sample media data is used to iteratively train the initial label generation model to obtain a target label generation model for generating media labels for target media data (i.e., video data, audio data, and image data). In this case, the target label generation model can be called a universal label generation model.

[0054] Among them, as shown in Table 1, the label description text of the candidate label may include the candidate label, the label meaning of the candidate label, and the label type to which the candidate label belongs. The label type may include one or more of the first-level classification and the second-level classification; the second-level classification is and the subcategory of the first-level classification. The first-level classification and the second-level classification can be determined based on the interest points of the media data. The label meaning can be called label description information, which is used to indicate the main content reflected by the media data with the corresponding candidate label, or the label meaning is used to reflect the reason why the media data has the corresponding candidate label.

[0055] Table 1

[0056]

[0057]

[0058] In practical applications, the label description text of candidate labels can be constructed according to the following label template: [Candidate Label Chinese Name]: Belongs to the [Secondary Category] type under the [First Category] category, and the label meaning is: [Label Description Information]. For example, the label description text for "Fun Experiments" in Table 1 can be: Fun Experiments: Belongs to the Highlights - Fun type under the Life category, and the label meaning is: Fun little experiments, challenging content, non-scientific experimental content, as long as it is somewhat interesting (normal scientific experiments are not suitable for this label). If the video experiment project is both interesting and has a strong curiosity and curiosity, it can be marked together with the "Wonderful Experiments".

[0059] The initial label generation model in this application may refer to the label generation model to be trained, and the target label generation model may be the one trained by the initial label generation model. Figure 2 As shown, the initial tag generation model may include a text feature extraction network 21b, a visual feature extraction network 22b, an audio feature extraction network 23b, a text feature extraction network 24b, a feature fusion network 25b, a semantic decoupling network 26b, and a graph network 27b.

[0060] The text feature extraction network 21b can be used to extract text features from the tag description text of the candidate tag to obtain the tag text features of the candidate tag; the text feature extraction network 24b can be used to extract text features from the sample media data to obtain the text features of the sample media data. The text features can be called text tokens and are used to reflect the text features (words or characters) of the tag description text. The text feature extraction network 21b and the text feature extraction network 24b can be referred to as text encoders.

[0061] The visual feature extraction network 22b can be used to extract visual features of the sample media data. These visual features can be called visual tokens and can be used to reflect the visual features (color, texture, etc.) of the video frames (or image data) in the sample media data. The visual feature extraction network 22b can be a visual feature encoder. The audio feature extraction network 23b can be used to extract audio features from the sample media data. These audio features can be called audio tokens and can be used to reflect the audio features (frequency, volume, timbre, etc.) of the audio frames in the sample media data. The audio feature extraction network 23b can be an audio feature encoder.

[0062] Among them, the feature fusion network 25b can be used to fuse the audio features, text features, and visual features of the sample media data to obtain the media features of the sample media data, which are marked as sample media features; for example, the feature fusion network 25b can be a transformer. The semantic decoupling network 26b can be used to perform feature fusion (semantic decoupling or attention pooling feature fusion) on the sample media features and the label text features to obtain sample fusion features corresponding to the candidate labels. The sample fusion features may include semantic information reflecting the label description text and the sample media data, so that the initial label generation model can better understand the label description text and the sample media data. The semantic decoupling network 26b can be a network based on attention fusion, such as a cross-attention network, a multi-head attention network, etc.

[0063] Among them, the graph network 27b can be a graph structure used to generate the corresponding candidate labels, and the predicted label probability corresponding to the candidate labels is determined according to the graph structure and the sample fusion features corresponding to the candidate labels. The graph structure can be used to reflect the association relationship between different candidate labels; the graph network 27b can refer to a gated neural network (Graph Gated Neural Network, GGNN), a recurrent graph neural network (Recurrent Graph Neural Networks), a graph convolution neural network (Graph Convolution Networks), a graph attention network (Graph Attention Networks), etc.; among them, the gated neural network is a variant of the graph neural network (Graph Neural Network, GNN).

[0064] In practical applications, such as Figure 2 As shown, the media tag generation scenario of video data is taken as an example for explanation, that is, the sample media data may refer to sample video data, and the number of candidate tags for the sample video data is 9 as an example for explanation, and the 9 candidate tags are respectively recorded as candidate tag 21a, candidate tag 22a, candidate tag 23a, candidate tag 24a, candidate tag 25a, candidate tag 26a, candidate tag 27a, candidate tag 28a, and candidate tag 29a.

[0065] During the training process of the initial label generation model, the server 10 can construct label description texts corresponding to the nine candidate labels according to the template described in Table 1 above, and perform feature extraction on the label description texts corresponding to the nine candidate labels through the text feature extraction network 21b in the initial label generation model to obtain label text features corresponding to the nine candidate labels. The server 10 can obtain text information of the sample video data, which can include text content obtained by performing text recognition on the video frames of the sample video data, and descriptive information of the sample video data. The descriptive information of the sample video data can include one or more of the theme, story summary, and character introduction of the sample video data. The text feature extraction network 24b in the initial label generation model extracts (i.e., encodes) the text information in the sample video data to obtain text features of the sample video data; the visual feature extraction network 22b in the initial label generation model extracts (encodes) the video frames in the sample video data to obtain visual features of the sample video data; and the audio feature extraction network 23b in the initial label generation model extracts features on the audio frames in the sample video data to obtain audio features of the sample video data.

[0066] After obtaining the label text features, audio features, visual features and text features, the audio features, visual features and text features can be fused through the feature fusion network 25b in the initial label generation model to obtain the sample media features of the sample video data. Through the semantic decoupling network 26b in the initial label generation model, the sample media features of the sample video data and the label text features of the candidate label 21a are fused to obtain the sample fusion features corresponding to the candidate label 21a. The sample fusion features corresponding to the candidate label 21a can be used to reflect the semantic information of the candidate label 21a and the sample video data. Through the semantic decoupling network 26b in the initial label generation model, the sample media features of the sample video data and the candidate label 22a are fused to obtain the sample fusion features of the candidate label 22a. The sample fusion features of the candidate label 22a can be used to reflect the semantic information of the candidate label 21a and the sample video data. By analogy, repeating the above steps can obtain the sample fusion features corresponding to the 9 candidate labels.

[0067] The graph network of the initial label generation model can determine the coexistence relationship between different candidate labels based on the labeling probabilities of the 9 candidate labels in multiple sample video data, and construct a network structure based on the coexistence relationship. The network structure may include 9 nodes, one node corresponds to one candidate label, and the edges between the nodes reflect the coexistence relationship between the candidate labels. The labeling probability is used to reflect the labeling probability of the sample video data having the corresponding candidate label, that is, the labeling probability can be obtained by manually labeling the sample video data, and the labeling probability can be 0 or 1. When it is determined that candidate label 21a exists based on the labeling probabilities of candidate label 21a and candidate label 22a in multiple sample video data, and candidate label 22a also exists, it is determined that candidate label 21a and candidate label 22a have a coexistence relationship, and the node corresponding to candidate label 21a is connected to the node corresponding to candidate label 22a. Similarly, if the labeling probabilities of candidate labels 22a and 25a in multiple sample video data indicate that candidate label 25a exists, and candidate label 22a also exists, a coexistence relationship is determined between candidate labels 22a and 25a, and the node corresponding to candidate label 22a is connected to the node corresponding to candidate label 25a. Similarly, the coexistence relationships between candidate labels can be obtained, and a graph structure can be constructed based on the coexistence relationships between candidate labels.

[0068] After obtaining the graph structure, the graph network of the initial label generation model can be used to perform feature learning between the candidate labels with coexistence relationships in the graph structure based on the labeling probabilities and sample fusion features of the 9 candidate labels for any sample video data, thereby obtaining label results for the 9 candidate labels for the sample video data. The label results may include the predicted label probabilities of the 9 candidate labels for the sample video data. Similarly, by repeating the above steps, the label results corresponding to all sample video data can be obtained. The server 10 can iteratively train the initial label generation model based on the labeling probabilities and predicted label probabilities of the 9 candidate labels for each sample video data to obtain the target label generation model.

[0069] After obtaining the target label generation model, the target label generation model can be applied to the video data publishing scenario, video data review scenario, etc. Taking the video data publishing scenario as an example, Figure 3 As shown, the video data publishing process may include steps such as content generation, content processing, and content distribution. Content generation may refer to any of the aforementioned terminals (using terminal 11a as an example) generating video data, referred to as target video data. The target video data may be captured by terminal 11a, or may be obtained by terminal 11a editing previously published video data.

[0070] The content processing process can include both machine tagging and manual tagging. Machine tagging can involve the terminal 11a, through a target application, invoking a target tag generation model in the server 10 to generate media tags for the target video data. Manual tagging can involve manually reviewing and correcting the media tags of the target video data, or manually adding new media tags to the target video data. The content distribution process can involve publishing the target video data and its media tags to the network, thereby achieving the target video data's distribution.

[0071] In summary, this application explicitly considers the interactive relationship and coexistence statistical laws between different candidate labels, which is conducive to improving the training accuracy of the initial label generation model and improving the generation accuracy of media labels; and the trained target label generation model can be directly called by the terminal or server, which can realize end-to-end calling of the target label generation model and improve the flexibility of the application.

[0072] Further, see Figure 4 , is a flow chart of a media data processing method provided by an embodiment of the present application. Figure 4 As shown, this method can be Figure 1 It can be executed by any terminal in the terminal cluster in Figure 1It can also be executed by the server in Figure 1 The terminals and servers in the terminal cluster are collaboratively executed. In this application, the devices used to execute the media data processing method can be collectively referred to as computer devices. The method may include the following steps:

[0073] S101. Obtain sample media features of sample media data, as well as label description text and labeling probabilities corresponding to N candidate labels of the sample media data, where N is an integer greater than 1, and the labeling probabilities reflect the labeling probabilities of the sample media data having the corresponding candidate labels.

[0074] The computer device may perform feature extraction on the sample media data to obtain sample media features of the sample media data, obtain label description text corresponding to each of N candidate labels for the sample media data, and obtain labeling probabilities. The labeling probabilities may be obtained by manually labeling the sample media data. The labeling probabilities reflect the probability that the sample media data has the corresponding candidate label. The labeling probabilities may be 0 or 1. When the labeling probability is 0, it indicates that the sample media data does not have the corresponding candidate label, and when the labeling probability is 1, it indicates that the sample media data has the corresponding candidate label.

[0075] The number of sample media data can be multiple, and the labeling probability of the same candidate label for different sample media data can be different. For example, if the candidate label is fun experiment, and the sample media data includes fun experiment videos and remakes of TV series, the labeling probability of the candidate label for fun experiment videos is 1, indicating that fun experiment videos are fun experiments; the labeling probability of fun experiments for remakes of TV series is 0, indicating that remakes of TV series are not fun experiments. In particular, a sample media data can have one or more candidate labels, that is, the labeling probability of the sample media data for one or more candidate labels is 1.

[0076] The sample media data may include one or more of sample video data, sample audio data, and sample image data. When the sample media data includes sample video data, the sample media features of the sample video data may include audio features, visual features, and text features of the sample video data. When the sample media data includes sample image data, the sample media features of the sample image data may include visual features and text features of the sample image data. When the sample media data includes sample audio data, the sample media features of the sample audio data may include audio features and text features of the sample audio data.

[0077] In one embodiment, the above-mentioned acquisition of the sample media features of the sample media data and the label description texts corresponding to the N candidate labels of the sample media data may include the following steps S11 to S14:

[0078] S11. Encode the text information in the sample media data to obtain text features of the sample media data.

[0079] Specifically, the computer device can encode the text information in the sample media data (i.e., feature extraction) through the text feature extraction network in the initial label generation model to obtain the text features of the sample media data. The text features of the sample media data are used to reflect the semantic information corresponding to the text information in the sample media data.

[0080] In which, when the sample media data includes sample image data, the text information in the sample image data may include descriptive information for the sample image data and at least one of the first text contents; the first text content is text content obtained by text extraction from each frame image in the sample image data, and the descriptive information of the sample image data may reflect at least one or more of the size, subject, resolution, source, etc. of the sample image data.

[0081] In which, when the sample media data includes sample audio data, the text information of the sample audio data may include a second text content and description information of the sample audio data. The second text content may be the text content obtained by text conversion of the audio frames in the sample audio data. The description information of the sample audio data may reflect the publisher, length, name, etc. of the sample audio data.

[0082] In which, when the sample media data includes sample video data, the text information of the sample video data may include a second text content and descriptive information of the sample video data. The descriptive information of the sample video data may include one or more of the theme, story summary, character introduction, etc. of the sample video data. The second text content may be text content obtained by text recognition of the video frames of the sample video data.

[0083] S12: Encode the non-text information in the sample media data to obtain non-text features of the sample media data.

[0084] When the sample media data includes sample image data, the non-text information in the sample image data may include each frame of the sample image data. The visual feature extraction network in the initial label generation model can be used to encode each frame of the sample image data to obtain visual features of the sample image data. The visual features of the sample image data are determined as non-text features of the sample image data. The visual features of the sample image data reflect information such as color and texture of the image in the sample image data.

[0085] Among them, when the sample media data includes sample audio data, the non-text information of the sample audio data may include audio frames in the sample audio data. The audio frames in the sample audio data can be encoded through the audio feature extraction network in the initial label generation model to obtain the audio features of the sample audio data, and the audio features of the sample audio data are determined as the non-text features of the sample audio data; the audio features of the sample audio data reflect the frequency, volume, timbre and other information of the sample audio data.

[0086] In which, when the sample media data includes sample video data, the non-text information of the sample video data may include at least one of the video frames and audio frames in the sample video data. The audio frame in the sample video data is encoded through the audio feature extraction network in the initial label generation model to obtain the audio features of the sample video data. The video frame in the sample video data is encoded through the visual feature extraction network in the initial label generation model to obtain the visual features of the sample video data. The audio features and visual features of the sample video data are determined as the non-text features of the sample video data.

[0087] S13: Fusing the text features and non-text features of the sample media data to obtain sample media features of the sample media data.

[0088] When the sample media data includes sample image data, the visual features and text features of the sample image data are fused to obtain sample media features of the sample image data.

[0089] When the sample media data includes sample audio data, the audio features and text features of the sample audio data are fused to obtain sample media features of the sample audio data.

[0090] When the sample media data includes sample video data, the visual features, audio features, and text features of the sample video data are fused to obtain sample media features of the sample video data.

[0091] S14: Generate a tag description text corresponding to each candidate tag according to each candidate tag, the tag type to which each candidate tag belongs, and the tag meaning.

[0092] Specifically, the computer device can add each candidate tag, the tag type to which each candidate tag belongs, and the tag meaning to the tag template to obtain the tag description text corresponding to each candidate tag. The tag type to which the candidate tag belongs here can include a primary category and a secondary category.

[0093] For example, the sample video data includes information in multiple modalities (such as video frames, text information, audio frames, etc.), and the methods for generating sample media features of the sample video data may include: (1) based on a multimodal network method (late-fusion form), the multimodal network may include an encoder corresponding to each modal information, and each encoder extracts features and fuses them to obtain sample media features of the sample video data. (2) Non-text modal information such as video frames and audio frames can be converted into corresponding tokens through an encoder, and the text tokens corresponding to the text information are used together through a transformer for feature fusion and prediction (early-fusion form) to obtain sample media features of the sample video data. Taking the sample video data 50a as an example, the above two methods can be abstracted as follows Figure 5 As shown, the computer device can extract audio frames from the sample video data 50a using Automatic Speech Recognition (ASR) technology; and perform character recognition on the video frames in the sample video data 50a using Optical Character Recognition (OCR) technology to obtain the text content in the sample video data 50a. The text content and description information of the sample video data 50a are determined as the text information of the sample video data 50a. The text information, video frames, and audio frames of the sample video data 50a are input into the multimodal network 51a. The multimodal network 51a performs feature extraction and feature fusion on the text information, video frames, and audio frames of the sample video data 50a to obtain video multimodal features of the sample video data 50a. The video multimodal features of the sample video data 50a are determined as sample media features of the sample video data 50a. Multimodality here can refer to multiple dimensions, and video multimodal features can include visual features, text features, and audio features.

[0094] S102. Through the initial label generation model, feature extraction is performed on the label description texts corresponding to the above N candidate labels to obtain the label text features corresponding to the above N candidate labels, and feature fusion is performed on the above sample media features and the label text features corresponding to the above N candidate labels to obtain the sample fusion features corresponding to the above N candidate labels.

[0095] In the present application, a computer device may input the label description texts corresponding to the N candidate labels into an initial label generation model, and perform feature extraction on the label description texts corresponding to the N candidate labels through the initial label generation model to obtain label text features corresponding to the N candidate labels. Label text features may refer to semantic information used to reflect the corresponding candidate labels. Feature fusion is performed on the sample media features and the label text features corresponding to the N candidate labels to obtain sample fusion features corresponding to the N candidate labels, that is, the sample fusion features corresponding to each candidate label are used to reflect the semantic information of the corresponding candidate label and the sample media data.

[0096] In one embodiment, the initial tag generation model may include a text feature extraction network and a semantic decoupling network; the above step S102 may include the following steps S21 to S23:

[0097] S21. Through the above-mentioned text feature extraction network, feature extraction is performed on the label description text corresponding to the c-th candidate label to obtain the label text feature corresponding to the above-mentioned c-th candidate label; c is an integer less than or equal to N.

[0098] Specifically, the computer device can perform feature extraction on the label description texts corresponding to the N candidate labels respectively through a text extraction network in a parallel manner or a serial manner to obtain label text features corresponding to the N candidate labels respectively.

[0099] The working principle of the text feature extraction network here is as follows: feature extraction is performed on the label description text of the corresponding candidate label to obtain the text token corresponding to the candidate label, and the text token is converted into label text features, that is, text embedding features.

[0100] S22. Perform a linear transformation on the label text features corresponding to the c-th candidate label through the semantic decoupling network to obtain the linear label text features corresponding to the c-th candidate label.

[0101] Specifically, the computer device can perform linear transformation on the label text features corresponding to the N candidate labels in parallel or serial manner through the semantic decoupling network to obtain the linear label text features corresponding to the N candidate labels.

[0102] The parallel method may refer to processing features corresponding to N candidate tags (such as tag text features) at the same time, and the serial method may refer to processing features corresponding to only one candidate tag at a time.

[0103] For example, the label text features corresponding to N candidate labels can be expressed as: X = {x1, x2, x3, ..., x N}; X is a text feature set, which includes the label text features corresponding to N candidate labels. The following formula (1) can be used to perform a linear transformation on the label text features corresponding to each candidate label:

[0104] Q c =W Q *x c (1)

[0105] Among them, Q c It can refer to the linear label text feature corresponding to the c-th candidate label, Q c The size is d k *1, namely Q c d k A matrix with one row and one column, W Q It can refer to the learnable parameters for linear transformation of label text features in the semantic decoupling network. The learnable parameters can refer to the model parameters that can be adjusted during the training process of the initial label generation model. c Refers to the label text feature corresponding to the cth candidate label.

[0106] S23. Through the semantic decoupling network, the linear label text features corresponding to the c-th candidate label and the sample media features are fused to obtain the sample fusion features corresponding to the c-th candidate label.

[0107] Specifically, through the semantic decoupling network, the linear label text features corresponding to the N candidate labels are fused with the sample media features in a parallel or serial manner to obtain sample fusion features corresponding to the N candidate labels.

[0108] Optionally, the semantic decoupling network includes a media feature transformation layer and a feature fusion layer, and the media feature transformation layer includes key parameters and value parameters;

[0109] The above step S23 may include the following steps S231 to S233:

[0110] S231 . Perform a linear transformation on the sample media feature according to the key parameter and value pair parameter through the media feature transformation layer to obtain a first linear media feature corresponding to the key parameter and a second linear media feature corresponding to the value parameter.

[0111] Specifically, the computer device can perform a linear transformation on the sample media feature through the key parameter in the media feature transformation layer to obtain a first linear media feature corresponding to the key parameter, and perform a linear transformation on the sample media feature through the value parameter in the media feature transformation layer to obtain a second linear media feature corresponding to the value parameter.

[0112] For example, the computer device may use the following formula (2) to calculate the first linear media feature and the second linear media feature:

[0113] K=W k *F, V=W v *F (2)

[0114] In formula (2), W k Refers to the key parameters in the media feature transformation layer, W v Represents the value parameter in the media feature transformation layer, W k 、W v Both refer to the learnable parameters in the media feature transformation layer. F is the sample media feature, K is the first linear media feature, and V is the second linear media feature. The size of K and V is N*d k .

[0115] S232: Perform feature fusion on the first linear media feature and the linear tag text feature corresponding to the c-th candidate tag to obtain an initial fused feature corresponding to the c-th candidate tag.

[0116] Specifically, the computer device may perform feature fusion on the first linear media feature and the linear tag text features corresponding to the N candidate tags in a parallel manner or a serial manner to obtain initial fused features corresponding to the N candidate tags.

[0117] S233 . According to the feature dimension corresponding to the linear label text feature, the initial fusion feature corresponding to the c-th candidate label and the second linear media feature are fused to obtain a sample fusion feature corresponding to the c-th candidate label.

[0118] Specifically, the computer device can fuse the initial fusion features corresponding to the N candidate labels and the above-mentioned second linear media features in a serial or parallel manner according to the feature dimension corresponding to the linear label text feature to obtain the sample fusion features corresponding to the N candidate labels.

[0119] For example, the computer device can use the following formula (3) to calculate the sample fusion features corresponding to each candidate label:

[0120]

[0121] Among them, in formula (3) is the sample fusion feature corresponding to the cth candidate label, softmax refers to the normalization function, sqrt refers to the square root operation, d k It refers to Q c feature dimension.

[0122] S103. Generate association relationship features corresponding to the N candidate tags respectively based on the sample fusion features corresponding to the N candidate tags and the labeling probabilities; the association relationship features reflect the association relationship between the corresponding candidate tag and other candidate tags.

[0123] In this application, a computer device can generate association relationship features corresponding to N candidate labels respectively according to the sample fusion features and labeling probabilities corresponding to the N candidate labels through an initial label generation model. By obtaining the association relationship features corresponding to each candidate label, it is beneficial to explore the interaction relationship and coexistence statistical laws between different candidate labels, that is, to explore finer-grained semantic information of the candidate labels, provide more accurate and finer-grained information for the training process of the initial label generation model, and help improve the training accuracy of the initial label generation model.

[0124] The association features corresponding to each candidate tag reflect the interaction and coexistence statistics between the corresponding candidate tag and other candidate tags. The interaction can be called a coexistence relationship. Two candidate tags with a coexistence relationship mean that if one candidate tag exists, the other candidate tag also exists. The coexistence statistics can be defined as the probability that one candidate tag exists and the other candidate tag also exists.

[0125] For example, Figure 6 As shown, the sample media data includes sample image data 60a, and the sample image data 50a is an image of an indoor scene. The sample image data 60a includes a dog, a sofa, a table, a chair, a person, and a computer, that is, the labeling probability of the sample image data regarding the candidate tags: "dog", "sofa", "person", "computer", "chair", and "table" is all 1. In actual applications, even if the sample image data 60a is not provided, and only the candidate tags: "dog", "sofa", "person", "computer", and "chair" are given, it can be roughly inferred that the candidate tag "table" also exists in the sample image data 60a based on the association relationship between the candidate tags. For an image of an indoor scene, candidate tags such as "ship" and "glacier" cannot appear together with the above-mentioned candidate tags ("dog", "sofa", "person", "computer", "chair", and "table"). Therefore, by utilizing the association relationship between candidate tags, rather than simply predicting each candidate tag independently, the training accuracy of the initial tag generation model can be improved, and thus the generation accuracy of media tags can be improved.

[0126] S104. Predict the predicted label probabilities corresponding to the N candidate labels based on the association relationship features corresponding to the N candidate labels and the sample fusion features. Iteratively train the initial label generation model based on the labeled label probabilities corresponding to the N candidate labels and the predicted label probabilities to obtain a target label generation model.

[0127] In the present application, a computer device can use the initial label generation model to predict the predicted label probabilities corresponding to the N candidate labels based on the association relationship features corresponding to the N candidate labels and the sample fusion features. The predicted label probability is the predicted probability that the corresponding sample media data has the corresponding candidate label, and the predicted label probability can fall within the interval [0, 1]. Furthermore, the initial label generation model can be iteratively trained based on the labeled label probabilities corresponding to the N candidate labels and the predicted label probabilities to obtain a target label generation model. The target label generation model can be used to generate media labels for target media data. By introducing the association relationship features corresponding to the N candidate labels, the training accuracy of the initial label generation model can be improved.

[0128] In this application, the association relationship feature reflects the association relationship between different candidate tags, that is, the association relationship feature reflects the interaction relationship and coexistence statistical laws between different candidate tags. In other words, in the process of training the initial tag generation model, the interaction relationship and coexistence statistical laws between different candidate tags are taken into account, which is conducive to providing more fine-grained information for the training process of the initial tag generation model and improving the training accuracy of the initial tag generation model. The target tag generation model obtained through training can solve the problem of low accuracy in the generation of media tags due to independent prediction of each candidate tag, thereby improving the accuracy of media tag generation.

[0129] Further, see Figure 7 , is a flow chart of a media data processing method provided by an embodiment of the present application. Figure 7 As shown, this method can be Figure 1 It can be executed by any terminal in the terminal cluster in Figure 1 It can also be executed by the server in Figure 1 The terminals and servers in the terminal cluster are collaboratively executed. In this application, the devices used to execute the media data processing method can be collectively referred to as computer devices. The method may include the following steps:

[0130] S201. Obtain sample media features of sample media data, as well as label description text and labeling probabilities corresponding to N candidate labels of the sample media data, where N is an integer greater than 1, and the labeling probabilities reflect the labeling probabilities of the sample media data having the corresponding candidate labels.

[0131] S202. Through the initial label generation model, feature extraction is performed on the label description texts corresponding to the above N candidate labels to obtain the label text features corresponding to the above N candidate labels, and feature fusion is performed on the above sample media features and the label text features corresponding to the above N candidate labels to obtain the sample fusion features corresponding to the above N candidate labels.

[0132] S203 , determining the coexistence relationship between every two candidate tags according to the labeling probabilities corresponding to the N candidate tags through the graph network of the initial tag generation model.

[0133] In this application, a computer device can use the graph network of the initial label generation model to obtain a ratio between the number of media and the total number of media based on the labeling probabilities corresponding to each of the N candidate labels, the number of media corresponding to each two candidate labels appearing in the same sample media data, and the total number of sample media data. When the ratio is greater than a ratio threshold, it is determined that the two candidate labels have a coexistence relationship. The ratio threshold can be pre-set, such as 0, 0.5, etc., and 0.5 is used as an example in the following description.

[0134] For example, the sample media data includes 10 sample video data, such as Figure 2 The probability that candidate tag 21a and candidate tag 22a are labeled for 6 sample video data at the same time is 1, that is, candidate tag 21a and candidate tag 22a will appear in 6 sample video data at the same time, that is, the number of media corresponding to candidate tag 21a and candidate tag 22a is 6. 6 / 10=0.6 is greater than the ratio threshold (0.5), so it can be determined that there is a coexistence relationship (i.e., an interactive relationship) between candidate tag 21a and candidate tag 22a. Candidate tag 21a and candidate tag 25a do not appear in the same sample video data, the number of media corresponding to candidate tag 21a and candidate tag 25a is 0, and the ratio between the number of media and the total number is: 0 / 10=0 is less than the ratio threshold, so it can be determined that there is no coexistence relationship between candidate tag 21a and candidate tag 25a.

[0135] S204 , constructing a graph structure based on the coexistence relationship between each two candidate tags; a node of the graph structure corresponds to a candidate tag, and an edge of the graph structure reflects the coexistence relationship between each two candidate tags.

[0136] In the present application, a computer device can construct a graph structure based on the coexistence relationship between each two candidate labels. The graph structure may include N nodes, one node corresponding to one candidate label, and the edges of the graph structure reflect the coexistence relationship between each two candidate labels, that is, the nodes corresponding to the two candidate labels with a coexistence relationship are connected to form an edge.

[0137] S205. Based on the sample fusion features corresponding to the above N candidate labels and the above labeled label probabilities, perform T feature learning between the candidate labels with coexistence relationships in the above graph structure to obtain the association relationship features corresponding to the above N candidate labels; T is a positive integer.

[0138] In this application, the computer device can perform T times of feature learning between candidate labels with coexistence relationships in the graph structure based on the sample fusion features and labeling probabilities corresponding to the N candidate labels, and obtain the association relationship features corresponding to the N candidate labels.

[0139] In one embodiment, the above step S205 may include the following steps S31 to S33:

[0140] S31. Determine a first coexistence probability based on the above-mentioned label probabilities corresponding to the above-mentioned N candidate labels and the above-mentioned graph structure; the above-mentioned first coexistence probability reflects the probability that an adjacent candidate label exists when the c-th candidate label exists; c is an integer less than or equal to N, and the above-mentioned adjacent candidate label is a candidate label corresponding to a node in the above-mentioned graph structure that has a coexistence relationship with the above-mentioned c-th candidate label.

[0141] Specifically, the number of first media corresponding to the sample media data with the cth candidate label can be determined based on the probability of labeling multiple sample media data with the cth candidate label; the number of second media corresponding to the sample media data with both the cth candidate label and the adjacent candidate label can be determined based on the probability of labeling multiple sample media data with the cth candidate label and the adjacent candidate label, and the ratio between the number of second media and the number of first media is determined as the first co-existence probability corresponding to the cth candidate label, that is, the first co-existence probability is obtained by dividing the number of second media by the number of first media.

[0142] S32. Determine a second co-existence probability based on the labeled label probabilities corresponding to the N candidate labels and the graph structure; the second co-existence probability reflects the probability that the c-th candidate label exists when the adjacent candidate label exists.

[0143] Specifically, the number of third media corresponding to the sample media data with adjacent candidate tags can be determined based on the probability of labeling multiple sample media data with adjacent candidate tags; the number of second media corresponding to the sample media data with both the cth candidate tag and the adjacent candidate tag can be determined based on the probability of labeling multiple sample media data with the cth candidate tag and the adjacent candidate tag, and the ratio between the number of second media and the number of third media is determined as the second co-existence probability corresponding to the cth candidate tag, that is, the second co-existence probability is obtained by dividing the second media number by the third media number.

[0144] For example, the number of sample media data is 100. Figure 2 For example, let's take candidate tag 21a in the example above as an example. Candidate tag 22a is an adjacent candidate tag of candidate tag 21a. Assume that the probability of candidate tag 21a labeling 50 sample media data is 1, the probability of candidate tag 22a labeling 30 sample media data is 1, and the probability of candidate tags 22a and 21a labeling 10 sample media data is 1. In other words, the number of first media corresponding to sample media data with candidate tag 21a is 50, the number of third media corresponding to sample media data with candidate tag 22a is 30, and the number of second media corresponding to sample media data with both candidate tags 21a and 22a is 10. The first co-occurrence probability between candidate tags 21a and 22a is 10 / 50 = 1 / 5; the second co-occurrence probability between candidate tags 21a and 22a is 10 / 30 = 1 / 3.

[0145] S33. Based on the first coexistence probability, the second coexistence probability, the sample fusion features corresponding to the c-th candidate label and the adjacent candidate labels, perform T times of feature learning between the c-th candidate label and the adjacent candidate labels to obtain the association relationship features corresponding to the c-th candidate label.

[0146] Specifically, the computer device can perform T times of feature learning between the cth candidate label and the adjacent candidate labels based on the first co-existence probability, the second co-existence probability, and the sample fusion features corresponding to the cth candidate label and the adjacent candidate labels, to obtain the association relationship features corresponding to the cth candidate label.

[0147] In one embodiment, the above step S33 may include the following steps S331 to S335:

[0148] S331, obtain the candidate association relationship feature output for the c-th candidate label in the t-1th feature learning t is a positive integer less than or equal to T. When t is 1, the candidate association feature output for the c-th candidate label in the 0th feature learning is is the sample fusion feature corresponding to the c-th candidate label above.

[0149] Specifically, when t=1, in the 0th feature student, the candidate association feature is output for the cth candidate label. The sample fusion feature corresponding to the cth candidate label; when t≠1, the candidate association feature output for the cth candidate label in the t-1th feature learning It can be obtained by referring to the following steps S332 to S335, where the output refers to the output from the graph network.

[0150] S332, obtain the candidate association relationship features output for the adjacent candidate tags in the above t-1th feature learning In the above 0th feature learning, the candidate association relationship features output for the above adjacent candidate labels are is the sample fusion feature corresponding to the above adjacent candidate labels.

[0151] Specifically, when t=1, in the 0th feature student, the candidate association relationship features are output for adjacent candidate labels. Fusion features for samples corresponding to adjacent candidate labels; when t≠1, the candidate association features output for the adjacent candidate labels in the t-1th feature learning Refer to the following steps S333 to S335 to obtain. The c′th candidate tag is the adjacent candidate tag of the c′th candidate tag, c′≠c, and c′ can be a positive integer less than or equal to N. The c′th candidate tag corresponds to the candidate association relationship feature It can be obtained according to the first co-existence probability and the second co-existence probability corresponding to the c′th candidate label and the sample fusion features corresponding to the N candidate labels.

[0152] S333: Based on the first coexistence probability, the second coexistence probability and the candidate association relationship characteristics Determine the interaction features between the cth candidate label and the adjacent candidate labels

[0153] Specifically, the computer device can calculate the candidate association relationship characteristics based on the first coexistence probability, the second coexistence probability and the candidate association relationship characteristics. Determine the interaction features between the cth candidate label and the adjacent candidate labels The interactive feature Reflects the coexistence statistical law (i.e., coexistence probability) between the cth candidate label and its adjacent candidate labels.

[0154] Optionally, the above step S333 may include the following steps S3331 to S3333:

[0155] S3331, the first coexistence probability and the candidate association relationship feature A product operation is performed to obtain the first interaction feature of the c-th candidate tag with respect to the adjacent candidate tags.

[0156] Specifically, the candidate association relationship feature corresponding to the first co-existence probability and each adjacent candidate tag can be Perform a product operation to obtain the first interaction feature of the cth candidate label for each adjacent candidate label. The first interaction feature reflects the existence of the cth candidate label, the probability of the adjacent candidate label, and the sample fusion feature corresponding to the adjacent candidate label.

[0157] S3332: the second coexistence probability and the candidate association relationship feature A product operation is performed to obtain the second mutual features of the adjacent candidate tags for the c-th candidate tag.

[0158] Specifically, the computer device may calculate the candidate association relationship feature corresponding to the second coexistence probability and each adjacent candidate tag. A product operation is performed to obtain the second mutual feature of each adjacent candidate label for the above c-th candidate label; the second mutual feature reflects the probability of the c-th candidate label existing when the adjacent candidate label exists, as well as the sample fusion feature corresponding to the adjacent candidate label.

[0159] S3333: Determine the interaction feature between the cth candidate tag and the adjacent candidate tag based on the first interaction feature and the second interaction feature.

[0160] Specifically, the computer device can determine the interaction feature between the cth candidate tag and the adjacent candidate tag according to the first interaction feature and the second interaction feature.

[0161] Optionally, the above step S3333 may include the following steps S33331 to S33332:

[0162] S33331. When the number of adjacent candidate tags of the c-th candidate tag is one, the first interaction feature and the second interaction feature are determined as the interaction features between the c-th candidate tag and the adjacent candidate tag.

[0163] Specifically, when the number of adjacent candidate tags of the cth candidate tag is one, the computer device can directly determine the first interaction feature and the second interaction feature as the interaction feature between the cth candidate tag and the adjacent candidate tag. Interaction features is a matrix with one row and two columns, the interaction feature It includes a first interaction feature and a second interaction feature.

[0164] S33332. When there are multiple adjacent candidate tags for the c-th candidate tag, sum the first interaction features corresponding to the multiple adjacent candidate tags to obtain a first total interaction feature, sum the second interaction features corresponding to the multiple adjacent candidate tags to obtain a second total interaction feature, and determine the first total interaction feature and the second total interaction feature as the interaction feature between the c-th candidate tag and the adjacent candidate tag.

[0165] Specifically, when there are multiple adjacent candidate tags for the c-th candidate tag, the first interaction features corresponding to the multiple adjacent candidate tags are summed to obtain a first total interaction feature, and the second interaction features corresponding to the multiple adjacent candidate tags are summed to obtain a second total interaction feature. The first total interaction feature and the second total interaction feature are determined as the interaction feature between the c-th candidate tag and the adjacent candidate tag.

[0166] For example, when the number of adjacent candidate tags of the c-th candidate tag is multiple, the computer device can use the following formula (4) to calculate the interaction feature between the c-th candidate tag and the adjacent candidate tags:

[0167]

[0168] In formula (4), a cc′ is the first co-existence probability corresponding to the c-th candidate label, that is, a cc′ is the probability (i.e. conditional probability) that the c′th candidate label exists when the cth candidate label exists; a c′c is the second co-existence probability corresponding to the c-th candidate label, that is, a c′c is the probability (i.e. conditional probability) that the cth candidate label exists when the c′th candidate label exists.

[0169] S334. According to the above interaction characteristics For the above candidate association features Perform feature learning to obtain the candidate association feature output for the c-th candidate label in the t-th feature learning

[0170] Specifically, computer devices can use graph networks to For the above candidate association features Perform feature learning to obtain the candidate association feature output for the c-th candidate label in the t-th feature learning Candidate relationship features Including the association relationship features learned by the t-th feature learning and the association relationship features learned before the t-th feature learning.

[0171] Optionally, the above step S334 may include the following steps S3341 to S3343:

[0172] S3341, based on the above interaction characteristics and the above candidate association features Determine forget control parameters and newly generated control parameters

[0173] Among them, the forgetting control parameter Can be used to control the forgetting of candidate association features Which information in the new control parameters It can be used to control what new information is generated in this feature learning.

[0174] For example, the computer device can calculate the forgetting control parameter by the following formula (5):

[0175]

[0176] Where σ(x) in formula (5) refers to the sigmoid function, W z 、U z Both refer to the parameters used to control the generation of forgetting control in graph networks. Model parameters, further, the computer equipment can calculate the new control parameters by the following formula (6):

[0177]

[0178] Among them, W in formula (6) r 、U r Both refer to the control parameters used to control the generation of new control parameters in the graph network. The model parameters.

[0179] S3342, based on the above interaction characteristics and the newly generated control parameters For the above candidate association features Perform feature learning to obtain the newly generated association relationship features corresponding to the c-th candidate label above

[0180] Among them, the newly generated correlation characteristics is the newly generated association feature for the cth candidate label during the tth feature learning process.

[0181] Specifically, the computer device can use the following formula (7) to calculate the newly generated correlation characteristics:

[0182]

[0183] Among them, in formula (7) Used to control the features of candidate association relationships Which information generates new information? W and U can refer to the model parameters in the graph network used to control the generation of new association relationship features.

[0184] S3343, according to the above forgetting control parameters For the above new correlation characteristics and the above candidate association features Perform feature learning to obtain the candidate association feature output for the c-th candidate label in the t-th feature learning

[0185] Among them, the candidate association relationship features It may include the newly learned association relationship features and the association relationship features learned before the t-th feature learning.

[0186] Specifically, the computer device can use the following formula (8) to calculate the candidate association relationship features:

[0187]

[0188] Among them, the formula (8) Used to control the forgetting of candidate association features What information in Used to control the forgetting of newly generated association features What information in it.

[0189] S335, the candidate association feature outputted by the T-th feature learning for the c-th candidate label Determine the association relationship feature corresponding to the c-th candidate label.

[0190] Specifically, repeat the above steps S331 to S334 until T feature learning is completed between the c-th candidate tag and the adjacent candidate tags, and the candidate association relationship feature output by the T-th feature learning for the c-th candidate tag can be The association feature corresponding to the c-th candidate label is determined, and the association feature corresponding to each candidate label is obtained through multiple feature learning, which is conducive to improving the accuracy of obtaining the association feature.

[0191] S206. Predict the predicted label probabilities corresponding to the N candidate labels based on the association relationship features corresponding to the N candidate labels and the sample fusion features. Iteratively train the initial label generation model based on the labeled label probabilities corresponding to the N candidate labels and the predicted label probabilities to obtain a target label generation model.

[0192] In this application, the computer device can perform feature splicing on the association relationship features and sample fusion features corresponding to the cth candidate label to obtain the spliced ​​features corresponding to the cth candidate label, and identify the spliced ​​features corresponding to the cth candidate label through the initial label generation model to obtain the predicted label probability corresponding to the cth candidate label. This process is repeated until the predicted label probabilities corresponding to N candidate labels are obtained. Based on the labeled label probabilities corresponding to the N candidate labels and the predicted label probabilities, the initial label generation model is iteratively trained to obtain the target label generation model.

[0193] For example, the computer device can use the following formula (9) to calculate the predicted label probability corresponding to the c-th candidate label:

[0194]

[0195] Among them, s in formula (9) c is the predicted label probability corresponding to the c-th candidate label, is the association feature corresponding to the c-th candidate label, is the sample fusion feature corresponding to the c-th candidate label.

[0196] Optionally, there are multiple sample media data. The iterative training of the initial label generation model based on the annotated label probabilities corresponding to the N candidate labels and the predicted label probabilities to obtain a target label generation model may include the following steps S41 to S43:

[0197] S41 : Determine a label generation loss for each of the sample media data based on the labeled label probabilities corresponding to the N candidate labels and the predicted label probabilities for each of the sample media data.

[0198] The computer device may output the labeled label probabilities and the predicted label probabilities corresponding to the N candidate labels for each sample media data item into the loss function of the initial label generation model, thereby obtaining a label generation loss for each sample media data item. The label generation loss reflects the label generation accuracy of the initial label generation model for each sample media data item. Specifically, a larger label generation loss indicates a lower label generation accuracy for the sample media data item by the initial label generation model; a smaller label generation loss indicates a higher label generation accuracy for the sample media data item by the initial label generation model.

[0199] Among them, the loss function here can refer to the cross entropy loss function (Binary Cross Entropy Loss, BCE Loss), mean squared error (MSE), mean absolute error (MAE), etc. This application takes the cross entropy loss function as an example for explanation.

[0200] S42: Sum the label generation losses corresponding to each of the sample media data to obtain the total label generation loss of the initial label generation model.

[0201] The total label generation loss reflects the label generation loss of the initial label generation model for multiple sample media data. For example, the computer device can use the following formula (10) to calculate the total label generation loss of the initial label generation model:

[0202]

[0203] Where P in formula (10) is the number of sample media data, N is the number of candidate labels, and y n,c It refers to the probability of the c-th candidate label for the n-th sample media data. L is the predicted label probability of the cth candidate label for the nth sample media data. BCE The total label generation loss for the initial label generation model.

[0204] S43. Iteratively train the initial label generation model according to the total label generation loss to obtain a target label generation model.

[0205] Specifically, the computer device can determine the convergence state of the initial label generation model based on the total loss of label generation. The convergence state includes a converged state or an unconverged state. The converged state may refer to that the total loss of label generation of the initial label generation model is less than or equal to a preset loss value, and the unconverged state may refer to that the total loss of label generation of the initial label generation model is greater than a preset loss value. When the convergence state of the initial label generation model is a converged state, the initial label generation model can be determined as the target label generation model. When the convergence state of the initial label generation model is an unconverged state, the model parameters of the initial label generation model can be adjusted based on the total loss of label generation, and the adjusted initial label generation model can be continuously trained using sample media data until the convergence state of the adjusted initial label generation model is a converged state, or the number of iterative training of the initial label generation model reaches a threshold number, and the adjusted initial label generation model can be determined as the target label generation model.

[0206] The preset loss value may be set according to the application scenario of the target label generation model, or the preset loss value may be the minimum value of the loss function of the initial label generation model.

[0207] In this application, multiple feature learning is performed on candidate tags with co-existence relationships to explore the co-existence relationships (i.e., interactive relationships) and co-existence statistical laws between different candidate tags. This is conducive to accurately obtaining the association relationship features corresponding to each candidate tag. The initial tag generation model is iteratively trained through information such as the association relationship features corresponding to each candidate tag, which is conducive to improving the training accuracy of the initial tag generation model.

[0208] Further, see Figure 8 , is a flow chart of a media data processing method provided by an embodiment of the present application. Figure 8 As shown, this method can be Figure 1 It can be executed by any terminal in the terminal cluster in Figure 1 It can also be executed by the server in Figure 1 The terminals and servers in the terminal cluster are collaboratively executed. In this application, the devices used to execute the media data processing method can be collectively referred to as computer devices. The method may include the following steps:

[0209] S301: When target media data to be processed is received, target media features of the target media data are obtained.

[0210] In the present application, the target media data may be obtained by a computer device from other devices, or the target media data may be generated by a computer device. The computer device may perform feature extraction on the target media data to obtain target media features, which may include multimodal features of the target media data, such as multiple audio features, visual features, and text features.

[0211] S302. Through the target label generation model, feature extraction is performed on the label description texts corresponding to the above N candidate labels to obtain the label text features corresponding to the above N candidate labels, and feature fusion is performed on the target media features and the label text features corresponding to the above N candidate labels to obtain the target fusion features corresponding to the above N candidate labels; the target label generation model is obtained by iteratively training the above initial label generation model in the previous article.

[0212] In the present application, a computer device may use the target tag generation model to perform feature extraction on the tag description text corresponding to each of the N candidate tags to obtain tag text features corresponding to each of the N candidate tags, and perform feature fusion on the target media features and the tag text features corresponding to each of the N candidate tags to obtain target fusion features corresponding to each of the N candidate tags. The target fusion features corresponding to each candidate tag may include the tag text features of the corresponding candidate tag and the target media features.

[0213] S303 : Generate updated association relationship features corresponding to the N candidate tags respectively according to the target fusion features corresponding to the N candidate tags respectively and the annotated tag probabilities.

[0214] In this application, the computer device can use the target label generation model to generate updated association relationship features corresponding to N candidate labels according to the target fusion features and labeling label probabilities corresponding to the N candidate labels. The updated association relationship features corresponding to the N candidate labels reflect the interactive relationship between different candidate labels under the target media data, as well as the coexistence statistical laws.

[0215] S304: Determine a media tag of the target media data from the N candidate tags based on the target fusion feature and the update association relationship feature.

[0216] In the present application, a computer device may determine, using a target label generation model, a predicted label probability for each candidate label relative to the target media data based on the target fusion feature and the updated association relationship feature; and select, from among the N candidate labels, a candidate label whose predicted label probability is greater than a probability threshold as the media label for the target media data. The probability threshold may be set based on the application scenario of the target media data, or may be a default value.

[0217] It should be noted that, for the description of step S301 in this application, reference can be made to the description of step S101 in the previous text; for the description of step S302, reference can be made to the description of step S102 in the previous text; for the description of step S303, reference can be made to the description of step S103 and steps S203 to S205 in the previous text; and for the description of step S304, reference can be made to the description of step S104 in the previous text.

[0218] In this application, after the target label generation model is trained, the target label generation model can be used to identify the interaction relationship and co-existence statistical laws between N candidate labels under the target media data. Based on the interaction relationship and co-existence statistical laws between N candidate labels under the target media data, the media label of the target media data is generated, providing more fine-grained information for the media label generation process and improving the accuracy of media label generation.

[0219] See Figure 9 , is a structural diagram of a media data processing device provided by an embodiment of the present application. Figure 9 As shown, the media data processing device may include:

[0220] Acquisition module 911 is configured to acquire sample media features of sample media data, as well as label description text and labeling probabilities corresponding to N candidate labels of the sample media data, where N is an integer greater than 1, and the labeling probabilities reflect the probability that the sample media data has the corresponding candidate label.

[0221] Extraction module 912 is configured to perform feature extraction on the label description texts corresponding to the N candidate labels using an initial label generation model to obtain label text features corresponding to the N candidate labels, and to perform feature fusion on the sample media features and the label text features corresponding to the N candidate labels to obtain sample fusion features corresponding to the N candidate labels.

[0222] The generating module 913 is configured to generate association features corresponding to the N candidate tags respectively based on the sample fusion features corresponding to the N candidate tags and the labeling probabilities; the association features reflect the association relationship between the corresponding candidate tag and other candidate tags;

[0223] The training module 914 is used to predict the predicted label probabilities corresponding to the above-mentioned N candidate labels based on the association relationship features corresponding to the above-mentioned N candidate labels and the above-mentioned sample fusion features, and iteratively train the above-mentioned initial label generation model based on the labeled label probabilities corresponding to the above-mentioned N candidate labels and the above-mentioned predicted label probabilities to obtain the target label generation model.

[0224] Optionally, the initial label generation model includes a text feature extraction network and a semantic decoupling network;

[0225] Optionally, the extraction module 912 is specifically configured to perform feature extraction on the label description text corresponding to the c-th candidate label through the text feature extraction network to obtain the label text feature corresponding to the c-th candidate label; c is an integer less than or equal to N;

[0226] Through the semantic decoupling network, the label text features corresponding to the c-th candidate label are linearly transformed to obtain the linear label text features corresponding to the c-th candidate label;

[0227] Through the semantic decoupling network, the linear label text features corresponding to the c-th candidate label and the sample media features are fused to obtain the sample fusion features corresponding to the c-th candidate label.

[0228] Optionally, the semantic decoupling network includes a media feature transformation layer and a feature fusion layer, and the media feature transformation layer includes key parameters and value parameters;

[0229] Optionally, an extraction module 912 is specifically configured to perform a linear transformation on the sample media feature according to the key parameter and value pair parameter through the media feature transformation layer to obtain a first linear media feature corresponding to the key parameter and a second linear media feature corresponding to the value parameter;

[0230] Performing feature fusion on the first linear media feature and the linear label text feature corresponding to the c-th candidate label to obtain an initial fused feature corresponding to the c-th candidate label;

[0231] According to the feature dimension corresponding to the linear label text feature, the initial fusion feature corresponding to the c-th candidate label and the second linear media feature are fused to obtain the sample fusion feature corresponding to the c-th candidate label.

[0232] Optionally, a generation module 913 is specifically configured to determine the coexistence relationship between each two candidate tags according to the labeling probabilities corresponding to the N candidate tags through the graph network of the initial tag generation model;

[0233] A graph structure is constructed based on the coexistence relationship between each two candidate labels; a node of the graph structure corresponds to a candidate label, and an edge of the graph structure reflects the coexistence relationship between each two candidate labels;

[0234] According to the sample fusion features corresponding to the above N candidate labels and the above labeled label probabilities, T feature learnings are performed between the candidate labels with coexistence relationships in the above graph structure to obtain the association relationship features corresponding to the above N candidate labels; T is a positive integer.

[0235] Optionally, a generation module 913 is specifically configured to determine a first coexistence probability based on the labeled label probabilities corresponding to the N candidate labels and the graph structure; the first coexistence probability reflects the probability that an adjacent candidate label exists when the c-th candidate label exists; c is an integer less than or equal to N, and the adjacent candidate label is a candidate label corresponding to a node in the graph structure that has a coexistence relationship with the c-th candidate label;

[0236] Determine a second co-existence probability based on the labeled label probabilities corresponding to the N candidate labels and the graph structure; the second co-existence probability reflects the probability that the c-th candidate label exists when the adjacent candidate label exists;

[0237] According to the first coexistence probability, the second coexistence probability, and the sample fusion features corresponding to the N candidate labels, feature learning is performed T times between the c-th candidate label and the adjacent candidate labels to obtain the association relationship feature corresponding to the c-th candidate label.

[0238] Optionally, a generation module 913 is specifically used to obtain the candidate association relationship feature output for the c-th candidate label in the t-1th feature learning t is a positive integer less than or equal to T. When t is 1, the candidate association feature output for the c-th candidate label in the 0th feature learning is is the sample fusion feature corresponding to the c-th candidate label above;

[0239] Get the candidate association relationship features output by the above adjacent candidate labels in the above t-1th feature learning In the above 0th feature learning, the candidate association relationship features output for the above adjacent candidate labels are The sample fusion features corresponding to the above adjacent candidate labels;

[0240] According to the first co-existence probability, the second co-existence probability and the candidate association relationship characteristics Determine the interaction features between the cth candidate label and the adjacent candidate labels

[0241] According to the above interaction characteristics For the above candidate association features Perform feature learning to obtain the candidate association feature output for the c-th candidate label in the t-th feature learning

[0242] The candidate association feature output for the c-th candidate label in the T-th feature learning is Determine the association relationship feature corresponding to the c-th candidate label.

[0243] Optionally, a generating module 913 is specifically configured to generate the first coexistence probability and the candidate association relationship feature. Perform a product operation to obtain the first interaction feature of the c-th candidate label with respect to the adjacent candidate label;

[0244] The second coexistence probability and the candidate association relationship feature Perform a product operation to obtain the second mutual feature of the adjacent candidate tags for the c-th candidate tag;

[0245] According to the first interaction feature and the second interaction feature, the interaction feature between the cth candidate tag and the adjacent candidate tag is determined.

[0246] Optionally, the generating module 913 is specifically configured to determine the first interaction feature and the second interaction feature as the interaction feature between the c-th candidate tag and the adjacent candidate tag when the number of adjacent candidate tags of the c-th candidate tag is one.

[0247] When there are multiple adjacent candidate tags for the c-th candidate tag, the first interaction features corresponding to the multiple adjacent candidate tags are summed to obtain a first total interaction feature, and the second interaction features corresponding to the multiple adjacent candidate tags are summed to obtain a second total interaction feature. The first total interaction feature and the second total interaction feature are determined as the interaction feature between the c-th candidate tag and the adjacent candidate tag.

[0248] Optionally, a generation module 913 is specifically configured to generate and the above candidate association features Determine forget control parameters and newly generated control parameters

[0249] According to the above interaction characteristics and the newly generated control parameters For the above candidate association features Perform feature learning to obtain the newly generated association relationship features corresponding to the c-th candidate label above

[0250] According to the above forget control parameters For the above new correlation characteristics and the above candidate association features Perform feature learning to obtain the candidate association feature output for the c-th candidate label in the t-th feature learning

[0251] Optionally, there are multiple sample media data.

[0252] Optionally, a training module 914 is specifically configured to determine a label generation loss for each of the sample media data based on the labeled label probabilities corresponding to the N candidate labels and the predicted label probabilities for each of the sample media data;

[0253] Sum the label generation losses corresponding to each of the above sample media data to obtain the total label generation loss of the above initial label generation model;

[0254] The above initial label generation model is iteratively trained according to the above total label generation loss to obtain the target label generation model.

[0255] Optionally, an acquisition module 911 is specifically configured to encode text information in the sample media data to obtain text features of the sample media data;

[0256] Encoding the non-text information in the sample media data to obtain non-text features of the sample media data;

[0257] Fusing the text features and non-text features of the sample media data to obtain sample media features of the sample media data;

[0258] According to each candidate tag, the tag type to which each candidate tag belongs, and the tag meaning, a tag description text corresponding to each candidate tag is generated.

[0259] In this application, the association relationship feature reflects the association relationship between different candidate tags, that is, the association relationship feature reflects the interaction relationship and coexistence statistical laws between different candidate tags. In other words, in the process of training the initial tag generation model, the interaction relationship and coexistence statistical laws between different candidate tags are taken into account, which is conducive to providing more fine-grained information for the training process of the initial tag generation model and improving the training accuracy of the initial tag generation model. The target tag generation model obtained through training can solve the problem of low accuracy in the generation of media tags due to independent prediction of each candidate tag, thereby improving the accuracy of media tag generation.

[0260] See Figure 10, is a structural diagram of a media data processing device provided by an embodiment of the present application. Figure 10 As shown, the media data processing device may include:

[0261] An acquisition module 1011 is configured to acquire target media features of the target media data upon receiving the target media data to be processed;

[0262] Extraction module 1012 is configured to perform feature extraction on the label description texts corresponding to the N candidate labels using a target label generation model to obtain label text features corresponding to the N candidate labels, and to perform feature fusion on the target media features and the label text features corresponding to the N candidate labels to obtain target fusion features corresponding to the N candidate labels. The target label generation model is obtained by iteratively training the initial label generation model using the above method.

[0263] A generating module 1013 is configured to generate updated association relationship features corresponding to the N candidate tags respectively according to the target fusion features corresponding to the N candidate tags respectively and the annotated tag probabilities;

[0264] The determination module 1014 is configured to determine a media tag of the target media data from the N candidate tags based on the target fusion feature and the updated association relationship feature.

[0265] In this application, after the target label generation model is trained, the target label generation model can be used to identify the interaction relationship and co-existence statistical laws between N candidate labels under the target media data. Based on the interaction relationship and co-existence statistical laws between N candidate labels under the target media data, the media label of the target media data is generated, providing more fine-grained information for the media label generation process and improving the accuracy of media label generation.

[0266] See Figure 11 , is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. Figure 11As shown, the above-mentioned computer device 1000 may refer to a terminal or a server, including: a processor 1001, a network interface 1004 and a memory 1005. In addition, the above-mentioned computer device 1000 may also include: a user interface 1003, and at least one communication bus 1002. The communication bus 1002 is used to realize the connection and communication between these components. In some embodiments, the user interface 1003 may include a display screen (DcSPlay), a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WC-FC interface). The memory 1005 may be a high-speed RAM memory, or a non-volatile memory (non-volatileMeMory), such as at least one disk memory. The memory 1005 may optionally also be at least one storage device away from the aforementioned processor 1001. As Figure 11 As shown, the memory 1005 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module, and a computer program.

[0267] exist Figure 11 In the computer device 1000 shown, the network interface 1004 can provide network communication functions; the user interface 1003 is mainly used to provide an input interface; and the processor 1001 can be used to call the computer program stored in the memory 1005 to implement the steps in the various method embodiments of the present application.

[0268] In this application, the association relationship feature reflects the association relationship between different candidate tags, that is, the association relationship feature reflects the interaction relationship and coexistence statistical laws between different candidate tags. In other words, in the process of training the initial tag generation model, the interaction relationship and coexistence statistical laws between different candidate tags are taken into account, which is conducive to providing more fine-grained information for the training process of the initial tag generation model and improving the training accuracy of the initial tag generation model. The target tag generation model obtained through training can solve the problem of low accuracy in the generation of media tags due to independent prediction of each candidate tag, thereby improving the accuracy of media tag generation.

[0269] It should be understood that the computer device described in the embodiments of the present application can execute the description of the above-mentioned media data processing method in the corresponding embodiments above, and can also execute the description of the above-mentioned media data processing device in the corresponding embodiments above, and the description thereof will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated here.

[0270] The collection and processing of the relevant data in this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in practice, and the informed consent or separate consent of the personal information subject should be obtained. Subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0271] In addition, it should be pointed out here that: the embodiment of the present application also provides a computer-readable storage medium, and the above-mentioned computer-readable storage medium stores a computer program executed by the media data processing device mentioned above, and the above-mentioned computer program includes program instructions. When the above-mentioned processor executes the above-mentioned program instructions, it can execute the description of the above-mentioned media data processing method in the corresponding embodiment above, so it will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application.

[0272] As an example, the above program instructions may be deployed on a computer device for execution, or deployed on at least two computer devices at one location for execution, or executed on at least two computer devices distributed at at least two locations and interconnected via a communication network. The at least two computer devices distributed at at least two locations and interconnected via a communication network may constitute a blockchain network.

[0273] The computer-readable storage medium can be the media data processing device provided in any of the aforementioned embodiments or the central storage unit of the computer device, such as the hard disk or memory of the computer device. The computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SMART Memory Card, SMC), a secure digital (SD) card, a flash memory card, etc. equipped with the computer device. Furthermore, the computer-readable storage medium can also include both the central storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium can also be used to temporarily store data that has been output or is about to be output.

[0274] The terms "first," "second," and the like in the description, claims, and drawings of the embodiments of the present application are used to distinguish between contents in different media, rather than to describe a specific order. In addition, the terms "including" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, apparatus, product, or device that includes a series of steps or units is not limited to the listed steps or modules, but may optionally include steps or modules that are not listed, or may optionally include other steps and units inherent to these processes, methods, apparatuses, products, or devices.

[0275] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0276] The present application also provides a computer program product, including a computer program. When executed by a processor, the computer program implements the description of the media data processing method and decoding method described in the corresponding embodiments above. Therefore, they will not be repeated here. In addition, the description of the beneficial effects of using the same method will not be repeated. For technical details not disclosed in the embodiments of the computer program product involved in this application, please refer to the description of the method embodiments of this application.

[0277] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0278] The methods and related devices provided by the embodiments of the present application are described with reference to the method flow charts and / or structural diagrams provided by the embodiments of the present application. Specifically, each process and / or block in the method flow charts and / or structural diagrams, as well as the combination of processes and / or blocks in the flow charts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable network-connected device to generate a machine, so that the instructions executed by the processor of the computer or other programmable network-connected device generate instructions for implementing the process. Figure 1 Schematic diagram of one or more processes and / or structures Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable network-connected device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the function specified in the process. Figure 1 Schematic diagram of one or more processes and / or structures Figure 1 These computer program instructions can also be loaded onto a computer or other programmable network connected device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide the functions for implementing the process. Figure 1 The flow or flows and / or structures illustrate the steps of the functions specified in one block or multiple blocks.

[0279] The above disclosure is only a preferred embodiment of the present application, and certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A media data processing method, characterized in that: include: Obtaining sample media features of the sample media data, as well as label description texts and labeling probabilities corresponding to N candidate labels of the sample media data; N is an integer greater than 1, and the labeling probability reflects the labeling probability of the sample media data having the corresponding candidate label; Using the initial label generation model, feature extraction is performed on the label description texts corresponding to the N candidate labels to obtain label text features corresponding to the N candidate labels, and feature fusion is performed on the sample media features and the label text features corresponding to the N candidate labels to obtain sample fusion features corresponding to the N candidate labels; Generate association relationship features corresponding to the N candidate labels respectively according to the sample fusion features corresponding to the N candidate labels and the labeling label probability; The association relationship feature reflects the association relationship between the corresponding candidate tag and other candidate tags; According to the association relationship features corresponding to the N candidate labels and the sample fusion features, the predicted label probabilities corresponding to the N candidate labels are predicted. According to the labeled label probabilities corresponding to the N candidate labels and the predicted label probabilities, the initial label generation model is iteratively trained to obtain a target label generation model.

2. The method according to claim 1, wherein The initial label generation model includes a text feature extraction network and a semantic decoupling network; The initial label generation model is used to extract features from the label description texts corresponding to the N candidate labels to obtain label text features corresponding to the N candidate labels, and feature fusion is performed on the sample media features and the label text features corresponding to the N candidate labels to obtain sample fusion features corresponding to the N candidate labels, including: Through the text feature extraction network, feature extraction is performed on the label description text corresponding to the c-th candidate label to obtain the label text feature corresponding to the c-th candidate label; c is an integer less than or equal to N; Performing a linear transformation on the label text features corresponding to the c-th candidate label through the semantic decoupling network to obtain the linear label text features corresponding to the c-th candidate label; Through the semantic decoupling network, feature fusion is performed on the linear label text features corresponding to the c-th candidate label and the sample media features to obtain the sample fusion features corresponding to the c-th candidate label.

3. The method according to claim 2, wherein The semantic decoupling network includes a media feature transformation layer and a feature fusion layer, and the media feature transformation layer includes key parameters and value parameters; The step of fusing the linear label text features corresponding to the c-th candidate label with the sample media features through the semantic decoupling network to obtain the sample fusion features corresponding to the c-th candidate label includes: Performing a linear transformation on the sample media feature according to the key parameter and the value pair parameter through the media feature transformation layer to obtain a first linear media feature corresponding to the key parameter and a second linear media feature corresponding to the value parameter; Performing feature fusion on the first linear media feature and the linear label text feature corresponding to the c-th candidate label to obtain an initial fused feature corresponding to the c-th candidate label; According to the feature dimension corresponding to the linear label text feature, the initial fusion feature corresponding to the c-th candidate label and the second linear media feature are fused to obtain a sample fusion feature corresponding to the c-th candidate label.

4. The method according to claim 1, wherein Generating the association relationship features corresponding to the N candidate tags respectively according to the sample fusion features corresponding to the N candidate tags respectively and the labeling label probability includes: Determining the coexistence relationship between each two candidate tags based on the labeled tag probabilities corresponding to the N candidate tags through the graph network of the initial tag generation model; Constructing a graph structure according to the coexistence relationship between each two candidate tags; a node of the graph structure corresponds to a candidate tag, and an edge of the graph structure reflects the coexistence relationship between each two candidate tags; According to the sample fusion features corresponding to the N candidate labels and the labeling probability, T feature learning is performed between the candidate labels with coexistence relationships in the graph structure to obtain the association relationship features corresponding to the N candidate labels; T is a positive integer.

5. The method according to claim 4, wherein The method of performing T feature learning on candidate labels having a coexistence relationship in the graph structure based on the sample fusion features corresponding to the N candidate labels and the labeling probability to obtain association relationship features corresponding to the N candidate labels includes: Determining a first coexistence probability based on the labeled label probabilities corresponding to the N candidate labels and the graph structure; the first coexistence probability reflects the probability that an adjacent candidate label exists when the cth candidate label exists; c is an integer less than or equal to N, and the adjacent candidate label is a candidate label corresponding to a node in the graph structure that has a coexistence relationship with the cth candidate label; Determining a second co-existence probability based on the labeled label probabilities corresponding to the N candidate labels and the graph structure; the second co-existence probability reflects the probability that the c-th candidate label exists when the adjacent candidate label exists; According to the first co-existence probability, the second co-existence probability, and the sample fusion features corresponding to the N candidate tags, feature learning is performed T times between the c-th candidate tag and the adjacent candidate tags to obtain the association relationship feature corresponding to the c-th candidate tag.

6. The method according to claim 5, wherein The step of performing T feature learning on the c-th candidate tag and the adjacent candidate tags based on the first co-existence probability, the second co-existence probability, and the sample fusion features corresponding to the N candidate tags to obtain the association relationship feature corresponding to the c-th candidate tag includes: Get the candidate association relationship features output for the cth candidate label in the t-1th feature learning t is a positive integer less than or equal to T. When t is 1, the candidate association feature output for the cth candidate label in the 0th feature learning is is the sample fusion feature corresponding to the c-th candidate label; Obtain the candidate association relationship features output for the adjacent candidate tags in the t-1th feature learning The candidate association relationship features outputted for the adjacent candidate tags in the 0th feature learning Fusion features of samples corresponding to the adjacent candidate labels; According to the first co-existence probability, the second co-existence probability and the candidate association relationship feature Determine the interaction feature between the cth candidate tag and the adjacent candidate tags According to the interaction characteristics For the candidate association relationship features Perform feature learning to obtain the candidate association feature output for the cth candidate label in the tth feature learning The candidate association feature output for the c-th candidate label in the T-th feature learning is Determine the association relationship feature corresponding to the c-th candidate tag.

7. The method according to claim 6, wherein The method according to the first co-existence probability, the second co-existence probability and the candidate association relationship feature Determine the interaction feature between the cth candidate tag and the adjacent candidate tags include: The first coexistence probability and the candidate association relationship feature Perform a product operation to obtain a first interaction feature of the c-th candidate tag with respect to the adjacent candidate tag; The second coexistence probability and the candidate association relationship feature Perform a product operation to obtain a second mutual feature of the adjacent candidate tags for the c-th candidate tag; Determine the interaction feature between the cth candidate tag and the adjacent candidate tag based on the first interaction feature and the second interaction feature 8. The method according to claim 7, wherein The interaction feature between the cth candidate tag and the adjacent candidate tag is determined based on the first interaction feature and the second interaction feature. include: When the number of adjacent candidate tags of the cth candidate tag is one, the first interaction feature and the second interaction feature are determined as the interaction features between the cth candidate tag and the adjacent candidate tag. When there are multiple adjacent candidate tags for the c-th candidate tag, the first interaction features corresponding to the multiple adjacent candidate tags are summed to obtain a first total interaction feature, and the second interaction features corresponding to the multiple adjacent candidate tags are summed to obtain a second total interaction feature. The first total interaction feature and the second total interaction feature are determined as the interaction feature between the c-th candidate tag and the adjacent candidate tag.

9. The method according to claim 6, wherein According to the interaction characteristics For the candidate association relationship features Perform feature learning to obtain the candidate association feature output for the cth candidate label in the tth feature learning include: According to the interaction characteristics and the candidate association relationship features Determine forget control parameters and newly generated control parameters According to the interaction characteristics and the newly generated control parameters For the candidate association relationship features Perform feature learning to obtain the newly generated association relationship feature corresponding to the cth candidate label According to the forgetting control parameter The newly generated association relationship feature and the candidate association relationship features Perform feature learning to obtain the candidate association feature output for the cth candidate label in the tth feature learning 10. The method according to claim 1, wherein There are multiple sample media data; The iterative training of the initial label generation model according to the labeled label probabilities and the predicted label probabilities respectively corresponding to the N candidate labels to obtain a target label generation model includes: Determining a label generation loss for each of the sample media data based on the labeled label probabilities and the predicted label probabilities corresponding to the N candidate labels for each of the sample media data; Summing the label generation losses corresponding to each sample media data to obtain the total label generation loss of the initial label generation model; The initial label generation model is iteratively trained according to the total label generation loss to obtain a target label generation model.

11. The method according to claim 1, wherein The acquiring of the sample media features of the sample media data and the label description texts corresponding to the N candidate labels of the sample media data respectively include: Encoding text information in the sample media data to obtain text features of the sample media data; Encoding the non-text information in the sample media data to obtain non-text features of the sample media data; fusing the text features and non-text features of the sample media data to obtain sample media features of the sample media data; According to each candidate tag, the tag type to which each candidate tag belongs, and the tag meaning, a tag description text corresponding to each candidate tag is generated.

12. A media data processing method, characterized in that: include: When receiving target media data to be processed, obtaining target media features of the target media data; Using a target label generation model, feature extraction is performed on the label description texts corresponding to the N candidate labels to obtain label text features corresponding to the N candidate labels, and feature fusion is performed on the target media features and the label text features corresponding to the N candidate labels to obtain target fusion features corresponding to the N candidate labels; The target label generation model is obtained by iteratively training the initial label generation model using the method described in any one of claims 1 to 11 above; Generate updated association relationship features corresponding to the N candidate tags respectively according to the target fusion features and the annotated tag probabilities corresponding to the N candidate tags respectively; The media tag of the target media data is determined from the N candidate tags according to the target fusion feature and the updated association relationship feature.

13. A media data processing device, characterized in that: include: an acquisition module configured to acquire sample media features of sample media data, as well as label description text and labeling probabilities corresponding to N candidate labels of the sample media data, wherein N is an integer greater than 1, and the labeling probabilities reflect the probability that the sample media data has the corresponding candidate label; An extraction module is configured to perform feature extraction on the label description texts corresponding to the N candidate labels respectively through an initial label generation model to obtain label text features corresponding to the N candidate labels respectively, and to perform feature fusion on the sample media features and the label text features corresponding to the N candidate labels respectively to obtain sample fusion features corresponding to the N candidate labels respectively; A generation module, configured to generate association relationship features corresponding to the N candidate tags respectively according to the sample fusion features corresponding to the N candidate tags respectively and the labeling label probability; The association relationship feature reflects the association relationship between the corresponding candidate tag and other candidate tags; The training module is used to predict the predicted label probabilities corresponding to the N candidate labels according to the association relationship features corresponding to the N candidate labels and the sample fusion features, and iteratively train the initial label generation model according to the marked label probabilities corresponding to the N candidate labels and the predicted label probabilities to obtain a target label generation model.

14. A media data processing device, characterized in that: include: an acquisition module, configured to acquire target media features of the target media data when receiving target media data to be processed; An extraction module is configured to perform feature extraction on the label description texts corresponding to the N candidate labels using a target label generation model to obtain label text features corresponding to the N candidate labels, and to perform feature fusion on the target media features and the label text features corresponding to the N candidate labels to obtain target fusion features corresponding to the N candidate labels; The target label generation model is obtained by iteratively training the initial label generation model using the method described in any one of claims 1 to 11 above; A generation module, configured to generate updated association relationship features corresponding to the N candidate tags respectively according to the target fusion features and the labeling label probabilities corresponding to the N candidate tags respectively; A determination module is configured to determine a media tag of the target media data from the N candidate tags based on the target fusion feature and the updated association relationship feature.

15. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 12 are implemented.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

Images