Method for classifying multimedia data, method for training classification model, and related apparatuses

Abstract

Embodiments of the present application provide a multimedia data classification method and related device, which relate to the fields of artificial intelligence and video classification. The method comprises: obtaining multimedia data containing at least two labels; extracting a first data vector corresponding to the multimedia data, and a first label vector corresponding to each label; performing classification retrieval on the first data vector in a preset sample database to determine a second data vector of the multimedia data; performing co-occurrence relationship analysis between the labels based on the first label vector to determine a second label vector of each label; and determining a classification result of the multimedia data based on the similarity between the second data vector and the second label vector. The embodiments of the present application are beneficial to improving the accuracy of multimedia data classification.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and data classification technology. Specifically, this application relates to a method for classifying multimedia data, a method for training a classification model, and related apparatus. Background Technology

[0002] In multimedia data classification, such as classifying videos or images, graph convolution techniques or multi-label data are generally used. However, when using graph convolution for classification, the available information is limited to the attributes of the video itself, and the information between videos cannot be used for classification, resulting in low accuracy. On the other hand, when using multi-label data for classification, it is limited to manually designed features, has low scalability, and cannot utilize the information between labels for classification, resulting in low accuracy for classification based on multi-label data.

[0003] Therefore, existing technologies suffer from limitations in the information used for classification and low classification accuracy when classifying multimedia data. Summary of the Invention

[0004] This application provides a multimedia data classification method and related apparatus, which can solve the problem of low classification accuracy of multimedia data. The technical solution is as follows:

[0005] According to one aspect of the embodiments of this application, a method for classifying multimedia data is provided, the method comprising:

[0006] Retrieve multimedia data containing at least two tags;

[0007] Extract the first data vector corresponding to the multimedia data, and the first tag vector corresponding to each tag respectively;

[0008] Based on the first data vector, a classification retrieval is performed in a preset sample database to determine the second data vector of the multimedia data;

[0009] Based on the first tag vector, perform co-occurrence relationship analysis between tags to determine the second tag vector for each tag;

[0010] The classification result of the multimedia data is determined based on the similarity between the second data vector and the second label vector.

[0011] In one feasible embodiment, the step of performing a classification search in a preset sample database based on the first data vector to determine the second data vector of the multimedia data includes:

[0012] For the multimedia data, a classification retrieval is performed in a preset sample database based on the first data vector to obtain the first sample data;

[0013] The sample database is classified and searched for each first sample data to obtain the second sample data corresponding to each first sample data.

[0014] By aggregating the information corresponding to the multimedia data, the first sample data, and the second sample data, a second data vector of the multimedia data is obtained.

[0015] In a feasible embodiment, the step of aggregating the information corresponding to the multimedia data, the first sample data, and the second sample data to obtain the second data vector of the multimedia data includes:

[0016] For each first sample data, a first directed edge is established with the multimedia data, and the first sample data is determined as a first-order neighbor of the multimedia data;

[0017] For each second sample data, a second directed edge is established with respect to the corresponding first sample data, and the second sample data is determined as a second-order neighbor of the multimedia data;

[0018] Based on the multimedia data, the first directed edge, the first-order neighbor, the second directed edge, and the second-order neighbor, a data directed graph is generated;

[0019] Information is aggregated based on the directed graph of the data to obtain the second data vector of the multimedia data.

[0020] In one feasible embodiment, the information aggregation based on the directed graph of the data to obtain a second data vector of the multimedia data includes:

[0021] Based on the weight coefficients between the multimedia data and the first-order neighbors, the vector weighted sum of the first-order neighbors is calculated to obtain the intermediate data vector.

[0022] Based on the weight coefficients between the nodes represented by the intermediate data vector and the second-order neighbors, the vector weighted sum of the second-order neighbors is calculated to obtain the second data vector.

[0023] In a feasible embodiment, the step of performing co-occurrence relationship analysis between tags based on the first tag vector to determine the second tag vector of each tag includes:

[0024] Perform the following co-occurrence analysis steps for each tag:

[0025] Based on the first tag vector, a co-occurrence relationship analysis is performed in a preset sample tag library to obtain the first sample tag;

[0026] Co-occurrence relationship analysis is performed on each first sample label in the sample label library to obtain the second sample label corresponding to each first sample label.

[0027] By aggregating the information corresponding to the label, the first sample label, and the second sample label, a second label vector for the label is obtained.

[0028] In one feasible embodiment, the co-occurrence analysis includes:

[0029] Determine the first frequency of occurrence of the tag to be analyzed in the sample tag library;

[0030] In the sample label library, determine the second frequency of co-occurrence of each sample label with the label to be analyzed;

[0031] The sample tags whose quotient of the second frequency and the first frequency is greater than a preset value are identified as tags that have a co-occurrence relationship with the tags to be analyzed.

[0032] In a feasible embodiment, the step of aggregating the information corresponding to the tag, the first sample tag, and the second sample tag to obtain the second tag vector includes:

[0033] For each first sample label, establish a first association edge with that label, and determine the first sample label as a first-order neighbor of that label;

[0034] For each second sample label, establish a second association edge with its corresponding first sample label, and determine the second sample label as a second-order neighbor of that label;

[0035] Based on the label, the first associated edge, the first-order neighbor of the label, the second associated edge, and the second-order neighbor of the label, generate an undirected label graph;

[0036] Information is aggregated based on the undirected graph of the label to obtain the second label vector of the label.

[0037] In one feasible embodiment, the step of aggregating information based on the undirected graph of the label to obtain the second label vector includes:

[0038] Based on the weight coefficients between the label and its first-order neighbors, calculate the vector weighted sum of the first-order neighbors of the label to obtain the intermediate label vector;

[0039] Based on the weight coefficients between the node represented by the intermediate label vector and the second-order neighbors of the label, the vector weighted sum of the second-order neighbors of the label is calculated to obtain the second label vector.

[0040] In one feasible embodiment, determining the classification result of the multimedia data based on the similarity between the second data vector and the second label vector includes at least one of the following:

[0041] Calculate the similarity score between the second data vector and each of the second label vectors, and determine at least one label with the highest similarity score and greater than a first preset threshold as the classification result of the multimedia data;

[0042] Calculate the similarity score between the second data vector and each of the second tag vectors, and determine at least one tag with a similarity score lower than a second preset threshold as a tag with incorrect multimedia data labeling;

[0043] Calculate the similarity score between the second data vector and each of the second tag vectors. If any similarity score is greater than a third preset threshold, it is determined that the multimedia data needs to be supplemented with additional tags.

[0044] Another aspect of this application embodiment also provides a method for training a classification model, including:

[0045] Acquire training data, which includes sample multimedia data, as well as sample labels and real labels corresponding to the sample multimedia data;

[0046] The sample multimedia data and sample labels are input into the classification model to obtain a predicted data vector corresponding to the sample multimedia data and a predicted label vector corresponding to the sample label; a loss value is determined based on the similarity between the predicted data vector and the predicted label vector, and the true label; and the network parameters of the classification model are adjusted based on the loss value.

[0047] The trained classification model is used to perform the multimedia data classification method described above.

[0048] According to another aspect of the embodiments of this application, a multimedia data classification apparatus is provided, the apparatus comprising:

[0049] The acquisition module is used to acquire multimedia data containing at least two tags;

[0050] The extraction module is used to extract the first data vector corresponding to the multimedia data and the first tag vector corresponding to each tag.

[0051] The retrieval module is used to perform classification retrieval in a preset sample database based on the first data vector to determine the second data vector of the multimedia data;

[0052] The analysis module is used to perform co-occurrence relationship analysis between tags based on the first tag vector, and determine the second tag vector of each tag;

[0053] The determination module is used to determine the classification result of the multimedia data based on the similarity between the second data vector and the second label vector.

[0054] According to another aspect of the embodiments of this application, a training apparatus for a classification model is provided, the apparatus comprising:

[0055] The training data acquisition module is used to acquire training data, which includes sample multimedia data, as well as sample labels and real labels corresponding to the sample multimedia data.

[0056] The training module is used to input the sample multimedia data and sample labels into the classification model to obtain a predicted data vector corresponding to the sample multimedia data and a predicted label vector corresponding to the sample labels; determine a loss value based on the similarity between the predicted data vector and the predicted label vector and the real label; and adjust the network parameters of the classification model based on the loss value.

[0057] The trained classification model is used to perform the multimedia data classification method described above.

[0058] According to another aspect of the present application, an electronic device is provided, the electronic device including a memory, a processor and a computer program stored in the memory, the processor executing the computer program to implement the steps of the multimedia data classification method described above.

[0059] According to another aspect of the embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the steps of the above-described multimedia data classification method.

[0060] According to one aspect of the embodiments of this application, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the multimedia data classification method described above.

[0061] The beneficial effects of the technical solutions provided in this application are:

[0062] This application provides a method and related apparatus for classifying multimedia data. Specifically, when classifying multimedia data with at least two tags, firstly, the multimedia data and tags are treated as independent data and vectors are extracted. Specifically, for each multimedia data, a first data vector is extracted, and then a classification retrieval is performed in a preset sample database based on the first data vector to determine a second data vector for the multimedia data. This classification retrieval allows the use of not only the attribute information of the multimedia data itself but also the similarity information between multimedia data during classification. For each tag, a first tag vector is extracted, and then a co-occurrence relationship analysis is performed based on the first tag vector to determine a second tag vector. This co-occurrence relationship analysis allows the simultaneous use of the relationship information between tags during multimedia data classification. Based on this, since the second data vector reflects the similarity between multimedia data, and the second tag vector reflects the co-occurrence relationship between tags used to identify multimedia data, the classification result of the multimedia data determined based on the similarity between the second data vector and the second tag vector is beneficial to improving the classification accuracy of multimedia data. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.

[0064] Figure 1 A flowchart illustrating a multimedia data classification method provided in an embodiment of this application;

[0065] Figure 2 A flowchart illustrating a training method for a classification model provided in an embodiment of this application;

[0066] Figure 3 This application provides a schematic diagram of the execution logic of a classification model.

[0067] Figure 4a A schematic diagram of a directed data graph provided in an embodiment of this application;

[0068] Figure 4b A schematic diagram of a label undirected graph provided in an embodiment of this application;

[0069] Figure 5 A schematic diagram of the architecture of a classification model provided in an embodiment of this application;

[0070] Figure 6 A schematic diagram of the structure of a multimedia data classification device provided in an embodiment of this application;

[0071] Figure 7A schematic diagram of the structure of a training device for a classification model provided in an embodiment of this application;

[0072] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0073] Figure 9 This application provides an embodiment of an optional structure diagram of a distributed system applied to a blockchain system.

[0074] Figure 10 An optional schematic diagram of the block structure provided in the embodiments of this application. Detailed Implementation

[0075] The embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the embodiments described below with reference to the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions of the embodiments of this application.

[0076] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the terms “comprising” and “including” as used in embodiments of this application mean that the corresponding feature can be implemented as the presented feature, information, data, step, operation, element, and / or component, but do not exclude implementation as other features, information, data, step, operation, element, component, and / or combinations thereof supported by the art. It should be understood that when we say that an element is “connected” or “coupled” to another element, the one element can be directly connected or coupled to the other element, or it can mean that the one element and the other element establish a connection relationship through an intermediate element. Furthermore, “connected” or “coupled” as used herein can include wireless connection or wireless coupling. The term “and / or” as used herein indicates at least one of the items defined by the term; for example, “A and / or B” can be implemented as “A,” or as “B,” or as “A and B.”

[0077] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0078] The following explains the relevant technologies for multimedia data classification:

[0079] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0080] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and intelligent transportation.

[0081] This application relates to the field of machine learning, which is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0082] In the classification and processing of multimedia data in related technologies, such as classifying videos or images, graph convolution techniques or multi-label data are generally used. However, when classifying using graph convolution, the available information is limited to the attributes of the video itself, and the information between videos cannot be used for classification, resulting in low accuracy. On the other hand, when classifying using multi-label data, it is limited to manually designed features, has low scalability, and cannot utilize the information between labels for classification, resulting in low accuracy for classification based on multi-label data.

[0083] Therefore, when performing multimedia data classification, the relevant technologies suffer from limitations in the information used for classification and low classification accuracy.

[0084] In view of at least one of the above-mentioned technical problems or areas for improvement in the related technologies, this application proposes a multimedia data classification method, apparatus, electronic device, storage medium, and program product. Specifically, this application classifies multimedia data by combining multimedia data with corresponding tags, which can simultaneously model video similarity relationships and potential consistency relationships of tags, thereby improving the accuracy of multimedia data classification in multi-tag tasks.

[0085] The technical solutions of this application and their effects are described below through several exemplary embodiments. It should be noted that the following embodiments can be referenced, borrowed from, or combined with each other. Identical terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0086] This application provides a method for classifying multimedia data, such as... Figure 1 As shown, the method includes the following steps S101-S105:

[0087] Step S101: Obtain multimedia data containing at least two tags.

[0088] Specifically, the embodiments of this application can be applied to multi-label classification tasks, and can process data to be classified consisting of multimedia data and a set of labels.

[0089] Multimedia data can include audio data, video data, image data, and other audio-visual data. The acquired labels can be labels from a dataset used for multimedia data classification.

[0090] Optionally, this application can be implemented using a pre-built classification model, and the data obtained in step S101 can be used as input data for the classification model.

[0091] Step S102: Extract the first data vector corresponding to the multimedia data and the first tag vector corresponding to each tag.

[0092] Specifically, different network models can be used to extract vector information for different objects. For multimedia data, networks such as CNN (Convolutional Neural Network), TSM (Temporal Shift Module for Efficient Video Understanding), and ViT (Vision Transformer) can be used to extract vector information; for tags, BERT (Bidirectional Encoder Representation from Transformers) can be used to extract vector information.

[0093] Among them, such as Figure 3 As shown, for the input multimedia data, vectors can be extracted using a CNN model to obtain the corresponding first data vector (which can be 64-dimensional). Optionally, in the embodiments of this application, a multi-label classification method can be used to train the CNN model when training the classification model, so as to improve the representation ability of the extracted vectors, making similar vectors not only similar in the content they represent (such as video images), but also tending to belong to the same category.

[0094] Among them, such as Figure 3 As shown, for each input label (e.g., the label can be input as plaintext), a vector can be extracted using the BERT model to obtain the corresponding 768-dimensional first label vector. Then, the data dimensionality is reduced using the PCA (Principal Component Analysis) module to obtain a 64-dimensional first label vector. Optionally, in training the classification model, this embodiment can collect a large amount of multimedia data attribute information (such as video titles) and topic tag information (such as hashtag #) to fine-tune the BERT model (fine-tuning the model, i.e., retraining the pre-trained BERT model), making the vectors extracted by the model more suitable for the application scenario, such as video classification.

[0095] Optionally, such as Figure 5 As shown, the first data vector is extracted through the module for extracting vector 1 in the classification model, and the first label vector is extracted through the module for extracting vector 2 in the classification model.

[0096] Step S103: Based on the first data vector, perform classification retrieval in the preset sample database to determine the second data vector of the multimedia data.

[0097] Specifically, classification retrieval can be implemented using the KNN algorithm. The principle of KNN (K-Nearest Neighbor) is that when predicting a new value x, the category to which x belongs is determined based on the K nearest neighbors to x. In the application of this embodiment, KNN retrieval can be used to retrieve K sample data similar to the multimedia data from a preset sample database, and then the second data vector of the multimedia data can be determined based on these K sample data. Optionally, classification retrieval can also be implemented using other algorithms suitable for classification, such as the K-means clustering algorithm.

[0098] Step S104: Analyze the co-occurrence relationship between tags based on the first tag vector to determine the second tag vector for each tag.

[0099] Specifically, co-occurrence relationships are determined based on the frequency with which a label co-occurs with other labels in the dataset. In other words, this embodiment defines the correlation between labels through co-occurrence frequency. For example, in the dataset, a higher frequency of co-occurrence compared to the frequency of co-occurrence indicates that the two labels are semantically similar or have an inclusion relationship. Furthermore, if two labels co-occur frequently with the same label, it also indicates that the two labels have a certain commonality. Based on the analysis principle of co-occurrence relationships proposed in this embodiment, for each label, the co-occurrence relationship between that label and other labels in the dataset can be analyzed based on the first label vector. Then, the second label vector of that label is determined based on all labels that have a co-occurrence relationship with it.

[0100] Step S105: Determine the classification result of the multimedia data based on the similarity between the second data vector and the second label vector.

[0101] Specifically, the similarity between the multimedia data and each label can be determined by calculating the similarity between the second data vector and each second label vector, and the classification result of the multimedia data can be finally determined based on the similarity.

[0102] In this embodiment, the similarity relationship between multimedia data and the potential consistency relationship between tags are modeled simultaneously. This can make full use of the global information represented by multimedia data and tags to assist multimedia data in multi-label classification, which can effectively improve the classification accuracy of multimedia data.

[0103] The data involved in the multimedia data classification method provided in this application embodiment can be stored on the blockchain; such as multimedia data, tags, vectors, classification results, etc., can be stored on the blockchain.

[0104] The following describes the specific content of determining the second data vector of multimedia data in the embodiments of this application.

[0105] In a feasible embodiment, step S103 involves performing a classification search in a preset sample database based on the first data vector to determine the second data vector of the multimedia data, including the following steps A1-A3:

[0106] Step A1: For multimedia data, perform classification retrieval in a preset sample database based on the first data vector to obtain the first sample data.

[0107] Specifically, assuming the multimedia data is video A, after extracting the first data vector represented by video A, a KNN search can be performed in the corresponding video library of the application scenario. For example, if the current application scenario is implemented for a short video creation platform and sharing community (hereinafter referred to as application B), the video library can be the database of application B, which stores the video data (also called sample data) involved in the operation of the application. That is, the first data vector of video A can be used to retrieve K1 similar videos (first sample data) similar to video A in the database of application B through KNN search. Among them, with video A as the center, the distance C between each video in the database of application B and video A is calculated. Then, the K1 videos with a distance C less than a first preset distance can be used as the first sample data D.

[0108] The distance calculated in the KNN algorithm can be Manhattan distance, Euclidean distance, Minkowski distance, etc., and this application does not limit this.

[0109] Step A2: Perform a classification search on each first sample data in the sample database to obtain the second sample data corresponding to each first sample data.

[0110] Specifically, after executing step A1, K1 first sample data can be obtained. In order to better learn the correlation between multimedia data, this embodiment of the application uses each first sample data as the object to be predicted and performs KNN search again in the preset sample database, based on the example of step A1. Assuming that there are currently K1 videos D (such as D1, D2...DK), KNN search is performed for each video D. Taking video D1 as an example: with video D1 as the center, the distance E between each video in the database of application B and video D1 is calculated. Then, K2 videos with a distance E less than a second preset distance can be used as second sample data F.

[0111] In order to improve the accuracy of the retrieved second sample data, a second preset distance can be set to be smaller than the first preset distance (the closer to the center, the higher the similarity).

[0112] Optionally, the first preset distance and the second preset distance can be adjusted according to the actual situation, and this application does not impose any restrictions on this.

[0113] Step A3: Aggregate the information corresponding to the multimedia data, the first sample data, and the second sample data to obtain the second data vector of the multimedia data.

[0114] Specifically, the objects processed when aggregating information include: multimedia data, first sample data that is directly related to the multimedia data, and second sample data that is indirectly related to the multimedia data; that is, step A3 can aggregate information of sample data that is related to the multimedia data based on the multimedia data, so as to classify the multimedia data by utilizing the correlation between the multimedia data and the sample data.

[0115] In a feasible embodiment, step A3 involves aggregating the information corresponding to the multimedia data, the first sample data, and the second sample data to obtain the second data vector of the multimedia data, including the following steps A31-A34:

[0116] Step A31: Establish a first directed edge between each first sample data and the multimedia data, and determine the first sample data as a first-order neighbor of the multimedia data.

[0117] Step A32: For each second sample data, establish a second directed edge with the corresponding first sample data, and determine the second sample data as a second-order neighbor of the multimedia data.

[0118] Step A33: Generate a directed graph of data based on multimedia data, first directed edges, first-order neighbors, second directed edges, and second-order neighbors.

[0119] Step A34: Aggregate information based on the directed graph of the data to obtain the second data vector of the multimedia data.

[0120] Specifically, such as Figure 4a As shown in the diagram, circles represent multimedia data, squares represent the first sample data, and triangles represent the second sample data. The first directed edge is the edge between the circle and the square, indicated by an arrow; the second directed edge is the edge between the triangle and the square, also indicated by an arrow. From... Figure 4a It is evident that the first sample data is directly related to the multimedia data, that is... Figure 4a The square in the middle represents the first-order neighbor of the circle; the second sample data has an indirect relationship with the multimedia data, that is... Figure 4a The middle triangle is a second-order neighbor of the circle.

[0121] Optionally, such as Figure 3 and 5 As shown, in the overall network structure of the classification model, it can be constructed using... Figure 1 The network module constructs a directed graph of data. Optionally, such as Figure 4a As shown, the directed data graph constructed in this application is a second-order neighbor graph.

[0122] In a feasible embodiment, step A34 involves information aggregation based on a directed data graph to obtain a second data vector of multimedia data, including the following steps A341-A342:

[0123] Step A341: Based on the weight coefficients between the multimedia data and the first-order neighbors, calculate the vector weighted sum of the first-order neighbors to obtain the intermediate data vector.

[0124] Step A342: Based on the weight coefficients between the node and its second-order neighbors represented by the intermediate data vector, calculate the vector weighted sum of the second-order neighbors to obtain the second data vector.

[0125] Specifically, such as Figure 3 and Figure 5 As shown, information aggregation can be performed using the graph convolution module in the classification model (as shown in Figure Convolution 1). Specifically, the graph convolution module used in this embodiment can be a combination of GraphSage (Inductive Representation Learning on Large Graphs) and GAT (Graph Attention Networks). GraphSage aggregates the second-order neighbor graph of a node to obtain the node's target vector (such as the second data vector), while the core idea of ​​GAT is to self-learn the weights between the node and its neighbors, and then use the self-learned weight coefficients to weightedly fuse the features from the first-order neighbors to obtain the node's target features. This application combines the processing principles of GraphSage and GAT to perform second-order weighted aggregation on each node.

[0126] The weighted aggregation process includes the following steps:

[0127] Step 1: Calculate the weighted sum of the convolutions of the first-layer graph and the nodes (e.g., ...). Figure 4a The neighbor vectors connected to the circular node shown are calculated by weighting the first data vector, resulting in the intermediate data vector of the node; wherein the intermediate data vector contains first-order neighbor nodes (such as...). Figure 4a Information about the square node shown;

[0128] Step 2: Calculate the weighted sum of the convolutions in the second-layer graph and the sum of the nodes (e.g., ...). Figure 4a The neighbor vectors connected to the circular node (as shown) are calculated by weighting the intermediate data vectors to obtain the node's final second data vector; where the second data vector contains second-order neighbor nodes (such as...). Figure 4aInformation about the triangular nodes shown.

[0129] Optionally, in practical applications, 25 nodes can be sampled for first-order neighbors (i.e., K1 = 25), and 10 nodes can be sampled for second-order neighbors (i.e., K2 = 10) to improve the accuracy of the extracted second data vector by limiting the number of neighbor nodes at each order.

[0130] Optionally, such as Figure 3 As shown, a second data vector can also be generated by deploying a GCN (Graph Convolutional Network) to aggregate second-order neighbor information.

[0131] The following describes the specific details of determining the second tag vector for each tag in the embodiments of this application.

[0132] In a feasible embodiment, step S104 involves performing co-occurrence relationship analysis between tags based on the first tag vector to determine the second tag vector for each tag, including the following step B1:

[0133] Step B1: Perform the following co-occurrence analysis steps B11-B13 for each tag:

[0134] Specifically, since multi-label tasks typically involve multiple labels that need to be predicted, this application's embodiments employ a method of extracting vector information separately for each label. For example... Figure 3 As shown, assuming there are N tags from tag 1 to tag N, steps B11-B13 need to be performed on each of the N tags. The following explanation uses tag 1 as an example.

[0135] Step B11: Based on the first label vector, perform co-occurrence relationship analysis in the preset sample label library to obtain the first sample label.

[0136] Specifically, after extracting the first label vector represented by label 1, co-occurrence analysis can be performed on the dataset containing label 1. If the current application scenario is implemented for a short video creation platform and sharing community (hereinafter referred to as Application B), the dataset can be the label dataset (sample label library) used by Application B for video classification, aggregation, and other processing. This dataset can be existing or set by the developers based on the actual application scenario; this application does not limit this. It is understood that the dataset includes labels 1 to N. The first sample label (which may include multiple labels) that co-occurs with label 1 can be obtained from the dataset of Application B through co-occurrence analysis using the first label vector of label 1.

[0137] Step B12: Perform co-occurrence relationship analysis on each first sample label in the sample label library to obtain the second sample label corresponding to each first sample label.

[0138] Specifically, after executing step B1, multiple first sample labels can be obtained. In order to better learn the correlation between labels, this embodiment of the application uses each first sample label as the object to be predicted and performs co-occurrence relationship analysis in the sample label library, based on the example of step B1. Assuming that there are currently 20 first sample labels (such as first sample label 1, first sample label 2, ..., first sample label 20), co-occurrence relationship analysis is performed for each first sample label. Taking first sample label 1 as an example: through the first label vector of first sample label 1, second sample labels (which may include multiple ones) that have a co-occurrence relationship with first sample label 1 can be obtained from the dataset of application B through co-occurrence relationship analysis.

[0139] Step B13: Aggregate the information corresponding to the label, the first sample label, and the second sample label to obtain the second label vector of the label.

[0140] Specifically, the objects processed when aggregating information include: the tag, a first sample tag that has a direct co-occurrence relationship with the tag, and a second sample tag that has an indirect co-occurrence relationship with the tag; that is, step B13 can aggregate the information of sample tags that are related to the tag based on the tag obtained in step S101, so as to use the correlation between tags to classify multimedia data.

[0141] In a feasible embodiment, the co-occurrence analysis involved in steps B12 and B11 includes the following steps C1-C3:

[0142] Step C1: Determine the first frequency of occurrence of the label to be analyzed in the sample label library.

[0143] Step C2: Determine the second frequency of co-occurrence of each sample label and the label to be analyzed in the sample label library.

[0144] Step C3: The sample labels whose quotient of the second frequency and the first frequency is greater than a preset value are identified as labels that have a co-occurrence relationship with the labels to be analyzed.

[0145] Specifically, when performing co-occurrence analysis on a certain label (the label to be analyzed) obtained in step S101, the first frequency of the label appearing in the sample label library is first determined; then, the second frequency of co-occurrence between the label and each sample label in the sample label library is determined; then, the co-occurrence degree value (characterizing the degree of connection between two labels) = second frequency / first frequency is compared with a first preset value. If the co-occurrence degree value is greater than the first preset value, it is determined that there is a co-occurrence relationship between the two labels, and the sample label can be identified as the first sample label.

[0146] Specifically, when performing co-occurrence analysis on the first sample label, the first frequency of the first sample label appearing in the sample label library is first determined; then, the second frequency of co-occurrence between the first sample label and each sample label in the sample label library is determined; then, the co-occurrence value is compared with a second preset value. If the co-occurrence value is greater than the second preset value, it is determined that there is a co-occurrence relationship between the two labels, and the sample label can be identified as the second sample label.

[0147] The first frequency of the tag to be analyzed in the sample tag library can be the frequency determined by statistics; that is, the frequency determined when the tag to be analyzed is used to identify multimedia data in the sample tag library. For example, under the same conditions, if N classifications of multimedia data are performed (either using a single tag or multiple tags), the number of times the tag to be analyzed is used to identify multimedia data in these N classifications, M, is the frequency of the tag to be analyzed. At this time, the first frequency corresponding to the tag to be analyzed can be determined based on the ratio of the frequency of the tag to be analyzed (also called frequency) to the number of classifications N.

[0148] The second frequency of co-occurrence of each sample label and the label to be analyzed in the sample label library can be determined statistically; that is, the frequency determined when other sample labels and the label to be analyzed are used to identify the same multimedia data in the sample label library. Specifically, under the same conditions, for N multimedia data classifications performed to determine the first frequency as described above, the number of times the label to be analyzed co-occurred with other sample labels in these N classification tasks is counted (e.g., if the sample label library includes Q sample labels, Q-1 times L can be obtained). At this time, the second frequency corresponding to the label to be analyzed can be determined based on the ratio of the number of co-occurrences L to the number of classifications N.

[0149] Optionally, the co-occurrence relationship between labels can be determined by identifying the first frequency of the label to be analyzed in the sample label library and the second frequency of co-occurrence between each sample label and the label to be analyzed in the sample label library. The co-occurrence value can be the ratio of the second frequency to the first frequency.

[0150] In order to improve the accuracy of the second sample labels obtained from the analysis, a second preset value can be set to be greater than a first preset value. Optionally, the first and second preset values ​​can also be adjusted according to actual conditions, and this application does not impose any restrictions on this.

[0151] In a feasible embodiment, step B13 aggregates the information corresponding to the label, the first sample label, and the second sample label to obtain the second label vector of the label, including the following steps B131-B134:

[0152] Step B131: Establish a first association edge with each first sample label and determine the first sample label as a first-order neighbor of that label.

[0153] Step B132: Establish a second association edge between each second sample label and its corresponding first sample label, and determine the second sample label as a second-order neighbor of that label.

[0154] Step B133: Generate an undirected label graph based on the label, the first associated edge, the first-order neighbors of the label, the second associated edge, and the second-order neighbors of the label.

[0155] Step B134: Aggregate information based on the undirected label graph to obtain the second label vector of the label.

[0156] Specifically, such as Figure 4b As shown in the figure, circles represent the labels obtained in step S101, squares represent the first sample labels, and triangles represent the second sample labels. The first associated edge is an undirected edge between the circle and the square; the second associated edge is an undirected edge between the triangle and the square. Figure 4b It is evident that the first sample label is directly related to the label obtained in step S101, that is... Figure 4b The square in the middle is a first-order neighbor of the circle; the label indirectly related to the label obtained in step S101 is the label of the second sample, that is... Figure 4b The middle triangle is a second-order neighbor of the circle.

[0157] Optionally, such as Figure 3 and Figure 5 As shown, in the overall network structure of the classification model, it is possible to construct... Figure 2 The network modules construct a labeled undirected graph. Optionally, such as Figure 4b As shown, the undirected label graph constructed in this embodiment is a second-order neighbor graph.

[0158] In an undirected labeled graph, the labels connected by edges have potential relationships. A Generative Network (GCN) can be used to learn these latent relationships, thereby aiding in the label classification of multimedia data. During network training, depending on the label size, either a network graph containing all labels can be used directly for training, or a subgraph can be constructed by sampling second-order neighbors for each label.

[0159] In a feasible embodiment, step B134 involves information aggregation based on the undirected graph of the label to obtain the second label vector, including the following steps B1341-B1342:

[0160] Step B1341: Based on the weight coefficients between the label and its first-order neighbors, calculate the vector weighted sum of the first-order neighbors of the label to obtain the intermediate label vector.

[0161] Step B1342: Based on the weight coefficients between the node represented by the intermediate label vector and the second-order neighbors of the label, calculate the vector weighted sum of the second-order neighbors of the label to obtain the second label vector.

[0162] Specifically, such as Figure 3 and Figure 5 As shown, information aggregation can be performed using the graph convolution module in the classification model (as shown in Figure Convolution 2). Specifically, the graph convolution module used in this embodiment can be a combination of GraphSage (Inductive Representation Learning on Large Graphs) and GAT (Graph Attention Networks). GraphSage aggregates the second-order neighbor graph of a node to obtain the node's target vector (such as the second label vector), while the core idea of ​​GAT is to self-learn the weights between the node and its neighbors, and then use the self-learned weight coefficients to weightedly fuse the features from the first-order neighbors to obtain the node's target features. This application combines the processing principles of GraphSage and GAT to perform second-order weighted aggregation on each node.

[0163] The weighted aggregation process includes the following steps:

[0164] Step 1: Calculate the weighted sum of the convolutions of the first-layer graph and the nodes (e.g., ...). Figure 4b The neighbor vectors connected to the circular node shown are calculated by weighting the first label vector, resulting in the node's intermediate label vector; where the intermediate label vector contains the first-order neighbor nodes (such as...). Figure 4b Information about the square node shown;

[0165] Step 2: Calculate the weighted sum of the convolutions in the second-layer graph and the sum of the nodes (e.g., ...). Figure 4bThe neighbor vectors connected to the circular node (shown) are calculated by weighting the intermediate label vectors, resulting in the node's final second label vector; where the second label vector contains the second-order neighbor nodes (such as...). Figure 4b Information about the triangular nodes shown.

[0166] Optionally, in practical applications, 25 nodes can be sampled for the first-order neighbors of the label (i.e., K1 = 25), and 10 nodes can be sampled for the second-order neighbors (i.e., K2 = 10) to improve the accuracy of the extracted second label vector by limiting the number of neighbor nodes at each order.

[0167] Optionally, such as Figure 3 As shown, second-order neighbor information can also be aggregated by deploying GCN (Graph Convolutional Network) to generate a second label vector.

[0168] The following is a detailed explanation of the classification results of multimedia data determined based on the second data vector and the second label vector in this application.

[0169] In a feasible embodiment, step S105, based on the similarity between the second data vector and the second label vector, determines the classification result of the multimedia data, including at least one of the following steps D1-D3:

[0170] Step D1: Calculate the similarity score between the second data vector and each second label vector, and determine at least one label with the highest similarity score and greater than the first preset threshold as the classification result of the multimedia data.

[0171] Step D2: Calculate the similarity score between the second data vector and each second label vector, and identify at least one label with a similarity score lower than the second preset threshold as a label with incorrect multimedia data labeling.

[0172] Step D3: Calculate the similarity score between the second data vector and each second label vector. If any similarity score is greater than the third preset threshold, it is determined that the multimedia data needs to be supplemented with additional labels.

[0173] Specifically, in the process of responding to online instances, the complete label graph network (used to generate the undirected label graph) can be processed through the GCN model to obtain the second label vector, which is then stored. For multimedia data, a second-order neighbor subnetwork can be constructed in real time, and the second data vector can be obtained through the GCN model. The cosine similarity between the second data vector and all the stored second label vectors is calculated, and the K results with the highest scores and greater than a certain threshold are the classification results of the multimedia data.

[0174] Optionally, for data with existing labels, if the classification model outputs results showing that the scores of multimedia data and corresponding labels are too low, it indicates that the current multimedia data may be mislabeled.

[0175] Optionally, if the multimedia data scores highly with a certain label, it indicates that the current multimedia data may have been missed in labeling.

[0176] The embodiments of this application, through the implementation of the above steps, can be applied to the real-time classification of online multimedia data, and can also be used to clean up mislabeled and missing labels in the data.

[0177] Regarding the classification model used in the above embodiments, this application also provides a method for training the classification model, such as... Figure 2 As shown, the process includes the following steps S201-S202:

[0178] Step S201: Obtain training data, which includes sample multimedia data, as well as sample labels and real labels corresponding to the sample multimedia data.

[0179] Step S202: Input the sample multimedia data and sample labels into the classification model to obtain the predicted data vector corresponding to the sample multimedia data and the predicted label vector corresponding to the sample labels; determine the loss value based on the similarity between the predicted data vector and the predicted label vector, and the true label; adjust the network parameters of the classification model based on the loss value.

[0180] Specifically, the real labels are the labels of the positive samples corresponding to each sample multimedia data. The sample labels corresponding to the sample multimedia data belong to the sample label library described in the above embodiments.

[0181] During training, the goal is to achieve higher vector similarity between multimedia data and real label pairs for the same sample, and lower vector similarity between multimedia data and real label pairs for different samples.

[0182] Considering that multi-label tasks may involve different label sizes in different scenarios, two feasible methods for training classification models with different label sizes are presented below:

[0183] The first approach is for multi-label tasks with a small number of labels, such as tasks with a total number of labels < 100. Under the condition that the memory and training speed are acceptable, a complete graph network containing all labels is used during training. The second sample vector of all labels is calculated in one training cycle. Then, the cosine similarity between the second data vector and all second sample vectors is calculated. The loss is calculated using the multi-label BCE (Binary Cross Entropy, used to calculate the binary cross entropy between the predicted value and the true value) loss as the loss function, and gradient backpropagation is performed.

[0184] The second approach is for multi-label tasks with a large number of labels, such as tasks with a total number of labels > 10,000, where it is impossible to load a complete label graph network. Therefore, during a training process, the labels and second-order neighbors of the sample multimedia data sub-network (i.e., the network containing the sample multimedia data and its second-order neighbors) are extracted in real time based on the sample multimedia data of the current training batch, and a label sub-network is constructed. This method can effectively solve the problem of training on large-scale data, and when the number of sample multimedia data in the batch is large enough, the accuracy and recall of the model can be effectively guaranteed.

[0185] The classification model trained by the above method in this application embodiment can clean up more than 20,000 invalid labels; compared with the CNN classification model, when the number of labels reaches 30,000, the accuracy is improved by about 5%, and the classification results after training perform better in terms of label aggregation.

[0186] In addition, embodiments of this application can use trained classifier vectors to represent the similarity between labels, to characterize semantically similar labels, and to jointly model the graph network of labels based on co-occurrence relationships and semantic similarity rows, so as to more accurately model the association between labels, and can also perform iterative training.

[0187] Alternatively, statistical information can be used to construct a multimedia data-label heterogeneity graph to further link the relationship between multimedia data and labels.

[0188] In this embodiment, both the second data vector and the second label vector can be obtained by aggregating second-order neighbor information using GCN. That is, GCN can be used to automatically extract the most suitable high-order information for multimedia data classification. Specifically, in implementation, this application can simultaneously model the similarity relationships between multimedia data (e.g., video data) nodes and the relationships between multimedia data labels (e.g., video labels), constructing a network graph of relationships between multimedia data and multiple labels. By fully utilizing the global information represented by multimedia data and labels, it assists in multi-label classification of multimedia data, effectively improving the classification accuracy of multimedia data. Furthermore, this embodiment can also perform data cleaning based on the global information represented by multimedia data and labels to effectively remove impurity data.

[0189] Below is a feasible application example.

[0190] In a short video social sharing platform, to better manage the video data stored on the platform, the stored video data can be categorized, and each video data can be stored both structurally and physically based on its category. At this time, the videos to be categorized can include videos recently uploaded or created by users on the platform, as well as videos that need to be recategorized (e.g., backend administrators may periodically update video tags; after the tag update, videos related to that tag need to be recategorized synchronously). Therefore, it is understandable that a single data classification task may have multiple processing instances (each processing instance consists of a video and a tag set). In this application, the classification task can be achieved through a trained classification model (also known as a classifier). By simultaneously modeling the similarity relationship between videos and the potential consistency between tags, the similarity between videos and tags is determined based on this information, and finally, the classification result of the video is determined based on this similarity.

[0191] The following explains the possible classification results:

[0192] (1) A video corresponds to multiple categories: the similarity score of the video with several tags is higher than the first preset threshold. In this case, data can be stored directly based on the classification results.

[0193] (2) Videos are mislabeled: A video's similarity score with one or more labels is lower than the second preset threshold. In this case, an alarm can be issued to indicate that the current label of the video is incorrect and needs to be reprocessed; alternatively, the classification or storage processing of the video can be directly interrupted.

[0194] (3) Video tags are missing: A video's similarity score with a certain tag is greater than the third preset threshold, or a video's similarity score with a certain tag is much greater than its similarity score with other tags. In this case, a prompt message can be issued, indicating that the current tag corresponding to the video is too singular and cannot effectively improve the video's promotion and adaptability, and that the corresponding tag for the video needs to be supplemented; a prompt message can also be issued simultaneously asking whether to interrupt the classification or storage processing of the video.

[0195] In a feasible embodiment, the multimedia data classification method provided above can be executed on a terminal or on a server. Specifically, when executed on a terminal, task information (multimedia data containing at least two tags) can be directly obtained and input into the classification model. The classification model then predicts the category of the multimedia data and outputs the classification result. When executed on a server, upon receiving a classification task instruction from the terminal, the server can obtain task information from the terminal or obtain task-related information from the database and input it into the classification model. The classification model then predicts the category of the multimedia data and feeds back the classification result to the terminal.

[0196] This application provides a multimedia data classification device, such as... Figure 6 As shown, the multimedia data classification device 600 may include: an acquisition module 601, an extraction module 602, a retrieval module 603, an analysis module 604, and a determination module 605.

[0197] The acquisition module 601 is used to acquire multimedia data containing at least two tags; the extraction module 602 is used to extract the first data vector corresponding to the multimedia data and the first tag vector corresponding to each tag; the retrieval module 603 is used to perform classification retrieval in a preset sample database based on the first data vector to determine the second data vector of the multimedia data; the analysis module 604 is used to perform co-occurrence relationship analysis between tags based on the first tag vector to determine the second tag vector of each tag; and the determination module 605 is used to determine the classification result of the multimedia data based on the similarity between the second data vector and the second tag vector.

[0198] In a feasible embodiment, when the retrieval module 603 performs a classification retrieval based on the first data vector in a preset sample database to determine the second data vector of the multimedia data, it is specifically used for:

[0199] For multimedia data, a classification retrieval is performed in a preset sample database based on the first data vector to obtain the first sample data;

[0200] For each first sample data, a classification search is performed in the sample database to obtain the second sample data corresponding to each first sample data.

[0201] By aggregating the information corresponding to the multimedia data, the first sample data, and the second sample data, a second data vector of the multimedia data is obtained.

[0202] Specifically, by aggregating the information corresponding to the multimedia data, the first sample data, and the second sample data, a second data vector of the multimedia data is obtained, including:

[0203] For each first sample data, establish a first directed edge with the multimedia data, and determine the first sample data as a first-order neighbor of the multimedia data;

[0204] For each second sample data, a second directed edge is established with respect to the corresponding first sample data, and the second sample data is determined as a second-order neighbor of the multimedia data.

[0205] Generate a directed graph based on multimedia data, first directed edges, first-order neighbors, second directed edges, and second-order neighbors;

[0206] Information aggregation is performed based on the directed graph of the data to obtain the second data vector of the multimedia data;

[0207] Among them, information aggregation based on the directed graph of data yields the second data vector of multimedia data, including:

[0208] Based on the weight coefficients between multimedia data and first-order neighbors, the vector weighted sum of first-order neighbors is calculated to obtain the intermediate data vector.

[0209] Based on the weight coefficients between the node and its second-order neighbors represented by the intermediate data vector, the weighted sum of the second-order neighbors' vectors is calculated to obtain the second data vector.

[0210] In a feasible embodiment, when the analysis module 604 performs co-occurrence relationship analysis between tags based on the first tag vector to determine the second tag vector of each tag, it is specifically used for:

[0211] Perform the following co-occurrence analysis steps for each tag:

[0212] Based on the first label vector, co-occurrence relationship analysis is performed in a preset sample label library to obtain the first sample label;

[0213] Co-occurrence analysis is performed on each first sample label in the sample label library to obtain the second sample label corresponding to each first sample label.

[0214] By aggregating the information corresponding to the label, the first sample label, and the second sample label, a second label vector for the label is obtained.

[0215] Co-occurrence analysis includes:

[0216] Determine the first frequency of occurrence of the label to be analyzed in the sample label library;

[0217] In the sample label library, determine the second frequency of co-occurrence between each sample label and the label to be analyzed;

[0218] The sample labels whose quotient of the second frequency and the first frequency is greater than a preset value are identified as labels that have a co-occurrence relationship with the labels to be analyzed.

[0219] Specifically, by aggregating the information corresponding to the label, the first sample label, and the second sample label, a second label vector for the label is obtained, including:

[0220] For each first sample label, establish a first association edge with that label, and determine the first sample label as a first-order neighbor of that label;

[0221] For each second sample label, establish a second association edge with its corresponding first sample label, and determine the second sample label as a second-order neighbor of that label;

[0222] Based on the label, the first associated edge, the first-order neighbor of the label, the second associated edge, and the second-order neighbor of the label, generate an undirected label graph;

[0223] Information aggregation is performed based on the undirected labeled graph to obtain the second label vector of the label.

[0224] Specifically, information aggregation based on the undirected label graph yields the second label vector for that label, including:

[0225] Based on the weight coefficients between the label and its first-order neighbors, calculate the vector weighted sum of the first-order neighbors of the label to obtain the intermediate label vector;

[0226] Based on the weight coefficients between the intermediate label vector and the second-order neighbors of the label, the weighted sum of the vectors of the second-order neighbors of the label is calculated to obtain the second label vector.

[0227] In one feasible embodiment, when determining the classification result of multimedia data based on the similarity between the second data vector and the second label vector, the determining module 605 is specifically used for at least one of the following:

[0228] Calculate the similarity score between the second data vector and each second label vector, and determine at least one label with the highest similarity score and greater than the first preset threshold as the classification result of the multimedia data;

[0229] Calculate the similarity score between the second data vector and each second label vector, and identify at least one label with a similarity score lower than a second preset threshold as a label with incorrect multimedia data labeling;

[0230] Calculate the similarity score between the second data vector and each second label vector. If any similarity score is greater than a third preset threshold, it is determined that the multimedia data needs to be supplemented with additional labels.

[0231] This application also provides a training device for a classification model, such as... Figure 7 As shown, the device 700 may include a training data acquisition module 701 and a training module 702.

[0232] The training data acquisition module 701 is used to acquire training data, which includes sample multimedia data, sample labels, and real labels corresponding to the sample multimedia data. The training module 702 is used to input the sample multimedia data and sample labels into the classification model to obtain the predicted data vector corresponding to the sample multimedia data and the predicted label vector corresponding to the sample label. Based on the similarity between the predicted data vector and the predicted label vector, and the real label, a loss value is determined, and the network parameters of the classification model are adjusted based on the loss value. The trained classification model is used to execute the multimedia data classification method provided in the above embodiment.

[0233] The apparatus in this application embodiment can execute the method provided in this application embodiment, and the implementation principle is similar. The actions performed by each module in the apparatus of each embodiment of this application correspond to the steps in the method of each embodiment of this application. For detailed functional descriptions of each module of the apparatus, please refer to the descriptions in the corresponding methods shown above, which will not be repeated here.

[0234] The blockchain referred to in this application is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains a certain amount of processed data used to verify the validity of its information (anti-counterfeiting) and to generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0235] This application provides an electronic device, including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps of a multimedia data classification method. Compared with related technologies, this method can achieve the following: When classifying multimedia data with at least two tags, firstly, the multimedia data and tags are treated as independent data, and vector extraction is performed. Specifically, for the multimedia data, a first data vector is extracted, and then a classification retrieval is performed in a preset sample database based on the first data vector to determine the second data vector of the multimedia data. The classification retrieval allows the use of not only the attribute information of the multimedia data itself but also the similarity information between multimedia data during multimedia data classification. For each tag, a first tag vector corresponding to the tag is extracted, and then a co-occurrence relationship analysis is performed based on the first tag vector to determine the second tag vector. The co-occurrence relationship analysis allows the use of the relationship information between tags simultaneously during multimedia data classification. Based on this, the classification result of the multimedia data is determined based on the similarity between the second data vector and the second tag vector, which helps to improve the classification accuracy of multimedia data.

[0236] In one alternative embodiment, an electronic device is provided, such as Figure 8 As shown, Figure 8 The illustrated electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of this application.

[0237] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0238] Bus 4002 may include a pathway for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 8 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0239] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium capable of carrying or storing computer programs and capable of being read by a computer, without limitation herein.

[0240] The memory 4003 stores computer programs that execute embodiments of this application, and its execution is controlled by the processor 4001. The processor 4001 executes the computer programs stored in the memory 4003 to implement the steps shown in the foregoing method embodiments.

[0241] Electronic devices include, but are not limited to: servers, user terminals, and vehicle terminals.

[0242] The system involved in this application embodiment can be a distributed system formed by connecting a client and multiple nodes (any form of computing device accessing the network, such as a server, user terminal, or vehicle terminal) through network communication. Taking a blockchain system as an example of a distributed system, see [link to relevant documentation]. Figure 9 , Figure 9 This is an optional structural diagram of the distributed system 100 provided in this application embodiment applied to a blockchain system. It consists of multiple nodes (any form of computing device in the network, such as servers or user terminals) and clients, forming a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on top of the Transmission Control Protocol (TCP). In the distributed system, any machine, such as a server or terminal, can join and become a node. A node includes a hardware layer, a middleware layer, an operating system layer, and an application layer.

[0243] See Figure 9 The functions of each node in the blockchain system shown may include:

[0244] 1) Routing: A basic function of nodes used to support communication between nodes.

[0245] In addition to routing capabilities, nodes can also have the following functions:

[0246] 2) Applications are deployed in the blockchain to implement specific business needs. They record data related to the implementation of functions to form record data, carry digital signatures in the record data to indicate the source of the task data, and send the record data to other nodes in the blockchain system. When other nodes successfully verify the source and integrity of the record data, they add the record data to a temporary block.

[0247] 3) A blockchain consists of a series of blocks that are sequentially generated. Once a new block is added to the blockchain, it will not be removed. The blocks contain the data submitted by the nodes in the blockchain system.

[0248] See Figure 10 , Figure 10This is an optional schematic diagram of the block structure provided in this application embodiment. Each block includes the hash value of the transaction records stored in this block (the hash value of this block) and the hash value of the previous block. The blocks are connected through their hash values ​​to form a blockchain. Additionally, the block may include information such as a timestamp when it was generated. A blockchain is essentially a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains relevant information used to verify the validity of the information (anti-counterfeiting) and to generate the next block.

[0249] This application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement the steps and corresponding content of the aforementioned method embodiments.

[0250] This application also provides a computer program product, including a computer program that, when executed by a processor, can implement the steps and corresponding content of the aforementioned method embodiments.

[0251] The terms "first," "second," "third," "fourth," "1," "2," etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than that shown in the figures or text.

[0252] It should be understood that although arrows indicate various operation steps in the flowcharts of this application's embodiments, the order in which these steps are implemented is not limited to the order indicated by the arrows. Unless explicitly stated herein, in some implementation scenarios of this application's embodiments, the implementation steps in each flowchart can be executed in other orders as required. Furthermore, some or all steps in each flowchart, based on the actual implementation scenario, may include multiple sub-steps or multiple stages. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage can also be executed at different times. In scenarios where execution times differ, the execution order of these sub-steps or stages can be flexibly configured according to requirements, and this application's embodiments do not limit this.

[0253] The above description is only an optional implementation method for some implementation scenarios of this application. It should be noted that for those skilled in the art, other similar implementation methods based on the technical concept of this application without departing from the technical concept of this application also fall within the protection scope of the embodiments of this application.

Claims

1. A method for classifying multimedia data, characterized in that, include: Retrieve multimedia data containing at least two tags; Extract the first data vector corresponding to the multimedia data, and the first tag vector corresponding to each tag respectively; For the multimedia data, a classification search is performed in a preset sample database based on the first data vector to obtain first sample data; for each first sample data, a classification search is performed in the sample database to obtain second sample data corresponding to each first sample data; the information corresponding to the multimedia data, the first sample data, and the second sample data is aggregated to obtain the second data vector of the multimedia data. Based on the first tag vector, perform co-occurrence relationship analysis between tags to determine the second tag vector for each tag; The classification result of the multimedia data is determined based on the similarity between the second data vector and the second label vector.

2. The method according to claim 1, characterized in that, The aggregation of information corresponding to the multimedia data, the first sample data, and the second sample data to obtain the second data vector of the multimedia data includes: For each first sample data, a first directed edge is established with the multimedia data, and the first sample data is determined as a first-order neighbor of the multimedia data; For each second sample data, a second directed edge is established with respect to the corresponding first sample data, and the second sample data is determined as a second-order neighbor of the multimedia data; Based on the multimedia data, the first directed edge, the first-order neighbor, the second directed edge, and the second-order neighbor, a data directed graph is generated; Information is aggregated based on the directed graph of the data to obtain the second data vector of the multimedia data.

3. The method according to claim 2, characterized in that, The process of aggregating information based on the directed graph of the data to obtain the second data vector of the multimedia data includes: Based on the weight coefficients between the multimedia data and the first-order neighbors, the vector weighted sum of the first-order neighbors is calculated to obtain the intermediate data vector. Based on the weight coefficients between the nodes represented by the intermediate data vector and the second-order neighbors, the vector weighted sum of the second-order neighbors is calculated to obtain the second data vector.

4. The method according to claim 1, characterized in that, The step of performing co-occurrence relationship analysis between tags based on the first tag vector to determine the second tag vector for each tag includes: Perform the following co-occurrence analysis steps for each tag: Based on the first tag vector, a co-occurrence relationship analysis is performed in a preset sample tag library to obtain the first sample tag; Co-occurrence relationship analysis is performed on each first sample label in the sample label library to obtain the second sample label corresponding to each first sample label. By aggregating the information corresponding to the label, the first sample label, and the second sample label, a second label vector for the label is obtained.

5. The method according to claim 4, characterized in that, The co-occurrence relationship analysis includes: Determine the first frequency of occurrence of the tag to be analyzed in the sample tag library; In the sample label library, determine the second frequency of co-occurrence of each sample label with the label to be analyzed; The sample tags whose quotient of the second frequency and the first frequency is greater than a preset value are identified as tags that have a co-occurrence relationship with the tags to be analyzed.

6. The method according to claim 4, characterized in that, The process of aggregating the information corresponding to the label, the first sample label, and the second sample label to obtain the second label vector of the label includes: For each first sample label, establish a first association edge with that label, and determine the first sample label as a first-order neighbor of that label; For each second sample label, establish a second association edge with its corresponding first sample label, and determine the second sample label as a second-order neighbor of that label; Based on the label, the first associated edge, the first-order neighbor of the label, the second associated edge, and the second-order neighbor of the label, generate an undirected label graph; Information is aggregated based on the undirected graph of the label to obtain the second label vector of the label.

7. The method according to claim 6, characterized in that, The process of aggregating information based on the undirected graph of the label to obtain the second label vector includes: Based on the weight coefficients between the label and its first-order neighbors, the vector weighted sum of the first-order neighbors of the label is calculated to obtain the intermediate label vector; Based on the weight coefficients between the node represented by the intermediate label vector and the second-order neighbors of the label, the vector weighted sum of the second-order neighbors of the label is calculated to obtain the second label vector.

8. The method according to claim 1, characterized in that, The determination of the classification result of the multimedia data based on the similarity between the second data vector and the second label vector includes at least one of the following: Calculate the similarity score between the second data vector and each of the second label vectors, and determine at least one label with the highest similarity score and greater than a first preset threshold as the classification result of the multimedia data; Calculate the similarity score between the second data vector and each of the second tag vectors, and determine at least one tag with a similarity score lower than a second preset threshold as a tag with incorrect multimedia data labeling; Calculate the similarity score between the second data vector and each of the second tag vectors. If any similarity score is greater than a third preset threshold, it is determined that the multimedia data needs to be supplemented with additional tags.

9. A method for training a classification model, characterized in that, include: Acquire training data, which includes sample multimedia data, as well as sample labels and real labels corresponding to the sample multimedia data; The sample multimedia data and sample labels are input into the classification model to obtain a predicted data vector corresponding to the sample multimedia data and a predicted label vector corresponding to the sample label; a loss value is determined based on the similarity between the predicted data vector and the predicted label vector, and the true label; and the network parameters of the classification model are adjusted based on the loss value. The trained classification model is used to perform the classification method for multimedia data according to any one of claims 1-8.

10. A multimedia data classification device, characterized in that, include: The acquisition module is used to acquire multimedia data containing at least two tags; The extraction module is used to extract the first data vector corresponding to the multimedia data and the first tag vector corresponding to each tag. The retrieval module is used to perform classification retrieval in a preset sample database based on the first data vector for the multimedia data to obtain first sample data; and to perform classification retrieval in the sample database for each first sample data to obtain second sample data corresponding to each first sample data. By aggregating the information corresponding to the multimedia data, the first sample data, and the second sample data, a second data vector of the multimedia data is obtained. The analysis module is used to perform co-occurrence relationship analysis between tags based on the first tag vector, and determine the second tag vector of each tag; The determination module is used to determine the classification result of the multimedia data based on the similarity between the second data vector and the second label vector.

11. The apparatus according to claim 10, characterized in that, When the retrieval module aggregates the information corresponding to the multimedia data, the first sample data, and the second sample data to obtain the second data vector of the multimedia data, it is specifically used for: For each first sample data, a first directed edge is established with the multimedia data, and the first sample data is determined as a first-order neighbor of the multimedia data; For each second sample data, a second directed edge is established with respect to the corresponding first sample data, and the second sample data is determined as a second-order neighbor of the multimedia data; Based on the multimedia data, the first directed edge, the first-order neighbor, the second directed edge, and the second-order neighbor, a data directed graph is generated; Information is aggregated based on the directed graph of the data to obtain the second data vector of the multimedia data.

12. The apparatus according to claim 11, characterized in that, When the retrieval module performs information aggregation based on the directed graph of the data to obtain the second data vector of the multimedia data, it is specifically used for: Based on the weight coefficients between the multimedia data and the first-order neighbors, the vector weighted sum of the first-order neighbors is calculated to obtain the intermediate data vector. Based on the weight coefficients between the nodes represented by the intermediate data vector and the second-order neighbors, the vector weighted sum of the second-order neighbors is calculated to obtain the second data vector.

13. The apparatus according to claim 10, characterized in that, When the analysis module performs co-occurrence relationship analysis between tags based on the first tag vector to determine the second tag vector of each tag, it is specifically used for: Perform the following co-occurrence analysis steps for each tag: Based on the first tag vector, a co-occurrence relationship analysis is performed in a preset sample tag library to obtain the first sample tag; Co-occurrence relationship analysis is performed on each first sample label in the sample label library to obtain the second sample label corresponding to each first sample label. By aggregating the information corresponding to the label, the first sample label, and the second sample label, a second label vector for the label is obtained.

14. The apparatus according to claim 13, characterized in that, The co-occurrence relationship analysis includes: Determine the first frequency of occurrence of the tag to be analyzed in the sample tag library; In the sample label library, determine the second frequency of co-occurrence of each sample label with the label to be analyzed; The sample tags whose quotient of the second frequency and the first frequency is greater than a preset value are identified as tags that have a co-occurrence relationship with the tags to be analyzed.

15. The apparatus according to claim 13, characterized in that, When the analysis module aggregates the information corresponding to the label, the first sample label, and the second sample label to obtain the second label vector of the label, it is specifically used for: For each first sample label, establish a first association edge with that label, and determine the first sample label as a first-order neighbor of that label; For each second sample label, establish a second association edge with its corresponding first sample label, and determine the second sample label as a second-order neighbor of that label; Based on the label, the first associated edge, the first-order neighbor of the label, the second associated edge, and the second-order neighbor of the label, generate an undirected label graph; Information is aggregated based on the undirected graph of the label to obtain the second label vector of the label.

16. The apparatus according to claim 15, characterized in that, When the analysis module performs information aggregation based on the undirected graph of the label to obtain the second label vector, it is specifically used for: Based on the weight coefficients between the label and its first-order neighbors, the vector weighted sum of the first-order neighbors of the label is calculated to obtain the intermediate label vector; Based on the weight coefficients between the node represented by the intermediate label vector and the second-order neighbors of the label, the vector weighted sum of the second-order neighbors of the label is calculated to obtain the second label vector.

17. The apparatus according to claim 10, characterized in that, When determining the classification result of the multimedia data based on the similarity between the second data vector and the second label vector, the determining module specifically performs at least one of the following: Calculate the similarity score between the second data vector and each of the second label vectors, and determine at least one label with the highest similarity score and greater than a first preset threshold as the classification result of the multimedia data; Calculate the similarity score between the second data vector and each of the second tag vectors, and determine at least one tag with a similarity score lower than a second preset threshold as a tag with incorrect multimedia data labeling; Calculate the similarity score between the second data vector and each of the second tag vectors. If any similarity score is greater than a third preset threshold, it is determined that the multimedia data needs to be supplemented with additional tags.

18. A training device for a classification model, characterized in that, include: The training data acquisition module is used to acquire training data, which includes sample multimedia data, as well as sample labels and real labels corresponding to the sample multimedia data. The training module is used to input the sample multimedia data and sample labels into the classification model to obtain a predicted data vector corresponding to the sample multimedia data and a predicted label vector corresponding to the sample labels; determine a loss value based on the similarity between the predicted data vector and the predicted label vector and the real label; and adjust the network parameters of the classification model based on the loss value. The trained classification model is used to perform the classification method for multimedia data according to any one of claims 1-8.

19. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1-9.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-9.

21. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-9.

Images