Tag recommendation method and device, tag recommendation model training method and medium

Abstract

The application relates to the technical field of artificial intelligence, in particular to a label recommendation method and device, a label recommendation model training method and a medium, wherein the method comprises the following steps: obtaining a reference label and at least two candidate labels; determining semantic similarity between the reference label and the candidate labels; determining co-occurrence probability between the reference label and the candidate labels, the co-occurrence probability being used for describing the probability that the reference label and the candidate label belong to the same label of multimedia data; and selecting a target label from the at least two candidate labels according to the semantic similarity and the co-occurrence probability. Since the semantic similarity represents the semantic similarity of the labels and the co-occurrence probability represents the probability that the labels appear in the same multimedia data, the target label obtained based on the semantic similarity and the co-occurrence probability is more consistent with the distribution assumption of the multimedia data, so that the obtained target label is more accurate.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a label recommendation method, apparatus, training method and medium for a label recommendation model. Background Technology

[0002] With the advent of the video era, in addition to traditional self-media platforms like WeChat Official Accounts (which primarily use text and images), there are also video accounts where individual users can share short videos at any time. These include platforms like WeChat Video Accounts, Douyin (TikTok), and Kuaishou Video Accounts. Their main content expression is not text-based, but rather focuses on short videos combined with short text feeds. To improve the information environment and group information by topic, users can add hashtags to their feeds. Currently, hashtags on video accounts are primarily manually added by users when uploading their feeds. To improve the efficiency of hashtag addition, the platform offers an automatic hashtag recommendation capability; when a user enters a hashtag, the platform automatically recommends related hashtags.

[0003] When recommending topic tags, the relevant technology uses the open Google Word2Vec (a relevant model used to generate word vectors) based on news corpora to represent each topic tag as an N-dimensional word vector. Then, the cosine distance between the word vectors of different topic tags is calculated. Based on this distance, the relevance between different topic tags is determined, and potential topic tags are recommended based on the relevance.

[0004] However, since the word vectors for topic tags represented by native Google Word2vec are a byproduct of training a language model on unsupervised news corpora, they learn grammatical similarity. Therefore, words with high similarity may simply be words that frequently appear together in the news corpus, or words that are grammatically similar, or even antonyms, hyponyms, or appositives. For example, the word vectors for "Beijing" and "Shanghai" may be very similar in native Google Word2vec, but they often do not appear together frequently in the same video feed. In short, directly using word vectors obtained from native Google Word2vec results in low accuracy because, firstly, its distribution assumptions are inconsistent with the distribution assumptions of topic tags in the video feed, and secondly, because it only has weak constraints from the language model loss function, it cannot effectively control the distribution of the obtained word vectors to be consistent with the co-occurrence distribution required for the topic tags in the video feed. Summary of the Invention

[0005] Therefore, it is necessary to provide a tag recommendation method, device, training method and medium for tag recommendation models that can effectively improve the accuracy of tag recommendation, in order to address the above-mentioned technical problems.

[0006] A tag recommendation method for multimedia data includes:

[0007] Obtain the reference tag and at least two candidate tags;

[0008] Determine the semantic similarity between the reference tag and the candidate tag;

[0009] Determine the co-occurrence probability between reference tags and candidate tags. The co-occurrence probability describes the probability that the reference tags and candidate tags belong to the same multimedia data.

[0010] The target label is selected from at least two candidate labels based on semantic similarity and co-occurrence probability.

[0011] A method for training a tag recommendation model, comprising:

[0012] Obtain at least two sample words and at least two sample tags for the multimedia data;

[0013] Analyze the semantic similarity between each sample word and use it as the sample semantic similarity; analyze the semantic similarity between each sample label and use it as the sample semantic similarity.

[0014] Analyze the co-occurrence probability among the labels of each sample and use it as the sample co-occurrence probability;

[0015] A label recommendation model is trained based on the semantic similarity of samples and the co-occurrence probability of samples.

[0016] A tag recommendation device for multimedia data, comprising:

[0017] The first acquisition module is used to acquire a reference label and at least two candidate labels;

[0018] The first determining module is used to determine the semantic similarity between the reference tag and the candidate tag;

[0019] The second determining module is used to determine the co-occurrence probability between the reference tag and the candidate tag. The co-occurrence probability is used to describe the probability that the reference tag and the candidate tag belong to the same multimedia data.

[0020] The recommendation module is used to select the target label from at least two candidate labels based on semantic similarity and co-occurrence probability.

[0021] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described method for tag recommendation of multimedia data, or implements the steps of the above-described method for training a tag recommendation model.

[0022] The aforementioned tag recommendation method, apparatus, and training method and medium for the tag recommendation model acquire a reference tag and at least two candidate tags, determine the semantic similarity between the reference tag and the candidate tags, and determine the co-occurrence probability between the reference tag and the candidate tags. Based on the semantic similarity and co-occurrence probability, a target tag is selected from at least two candidate tags. Since semantic similarity characterizes the semantic similarity of tags, and co-occurrence probability characterizes the probability of tags appearing in the same multimedia data, the target tag obtained based on semantic similarity and co-occurrence probability better conforms to the distribution assumptions of multimedia data, thus making the obtained target tag more accurate. Attached Figure Description

[0023] Figure 1 This is an application environment diagram of a tag recommendation method for multimedia data in one embodiment;

[0024] Figure 2 This is a flowchart illustrating a tag recommendation method for multimedia data in one embodiment;

[0025] Figure 3 This is a schematic diagram illustrating the annotation of topic tags in the video account information stream in one embodiment;

[0026] Figure 4 This is a schematic diagram of a process for selecting a target label from at least two candidate labels in one embodiment;

[0027] Figure 5 This is a schematic diagram illustrating the training process of a tag recommendation model in one embodiment;

[0028] Figure 6 This is a schematic diagram of the tag recommendation model in one embodiment;

[0029] Figure 7 This is a flowchart illustrating the training method of a tag recommendation model in one embodiment;

[0030] Figure 8 This is a structural block diagram of a tag recommendation device for multimedia data in one embodiment;

[0031] Figure 9 This is a structural block diagram of a tag recommendation device for multimedia data in another embodiment;

[0032] Figure 10 This is a structural block diagram of a tag recommendation device for multimedia data in another embodiment. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0034] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. Machine learning (ML) is an important direction of AI, which mainly studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance.

[0035] Tag recommendation, as a branch of machine learning, uses the open Google Word2Vec (a relevant model used to generate word vectors) based on news corpora to represent each topic tag as an N-dimensional word vector. Then, the cosine distance between the word vectors of different topic tags is calculated, and the relevance between different topic tags is determined based on this distance. Potential topic tags are then recommended based on this relevance. However, since the topic tag word vectors represented by the native Google Word2Vec are a byproduct of training a language model on unsupervised news corpora, it learns the grammatical similarity of words. Therefore, words with high similarity may simply be words that frequently appear together in the news corpus, or words that are grammatically similar, or even antonyms, hyponyms, or appositives, resulting in low accuracy in topic tag recommendation. Based on this, this application provides a tag recommendation method for multimedia data, which can effectively improve the accuracy of topic tag recommendation. The tag recommendation method for multimedia data provided in this application can be applied to, for example... Figure 1In the application environment shown, terminal 102 and server 104 communicate via a network. Server 104 obtains reference tags input by the user on terminal 102, retrieves at least two candidate tags from a database, determines the semantic similarity between the reference tags and candidate tags, and determines the co-occurrence probability between the reference tags and candidate tags. The co-occurrence probability describes the probability that the reference tags and candidate tags belong to the same multimedia data. Based on the semantic similarity and co-occurrence probability, server 104 selects a target tag from the at least two candidate tags and sends the target tag to terminal 102 for display on terminal 102 for user selection. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, and tablets. Server 104 can be implemented using a standalone server or a server cluster consisting of at least two servers.

[0036] In one embodiment, such as Figure 2 As shown, a tag recommendation method for multimedia data is provided, which is then applied to... Figure 1 Taking the server in the example, the following steps are included:

[0037] Step 202: Obtain the reference label and at least two candidate labels.

[0038] Reference tags refer to the tags provided by the user for the current multimedia data; candidate tags refer to the tags of all multimedia data that have existed before this, and the specific tags can be based on the tags of the existing multimedia data that the server can provide, without any restrictions here.

[0039] Take, for example, a user publishing a video feed on WeChat Video Channel. (Reference) Figure 3 As shown, after a user uploads a short video about pocket cookies through the WeChat Video Channel in the WeChat app on terminal 102, the user can enter a short text about the video below it, such as "Pocket cookies are so delicious," and also enter a hashtag for the video, such as "pocket cookies." This hashtag can be used as a reference tag. Understandably, to distinguish between the short text and the hashtag, separators such as "#" can be added to both ends of the hashtag to separate the short text and different hashtags, so that the system and the user can quickly and accurately identify the short text and various hashtags corresponding to the short video.

[0040] After a user inputs a hashtag such as "pocket biscuit" for the aforementioned short video, terminal 102 sends the hashtag "pocket biscuit" as a reference tag to server 104. Upon receiving the reference tag such as the hashtag "pocket biscuit", server 104 retrieves all existing hashtags corresponding to multimedia data from the database, such as "home-style dishes", "breakfast biscuits", "food", "roujiamo", "lunch", and "dinner", and uses them as candidate tags.

[0041] Step 204: Determine the semantic similarity between the reference tag and the candidate tag.

[0042] Semantic similarity refers to the degree to which two words can be used interchangeably in different contexts without changing the grammatical and semantic structure of the text. Therefore, the greater the possibility that two words can be used interchangeably in different contexts without changing the grammatical and semantic structure of the text, the higher their similarity; otherwise, the lower the similarity.

[0043] In this application, the semantic similarity between two different labels can be determined by representing each of the two labels as an N-dimensional word vector, calculating the cosine distance between the word vectors of the two labels, and then using the cosine distance. Of course, other methods can also be used to obtain the semantic similarity between two different labels, and existing technologies can be used; no restrictions are imposed here.

[0044] After obtaining reference tags such as the topic tag "pocket biscuit" and at least two candidate tags such as the topic tags "home-style dishes", "breakfast biscuits", "food", "roujiamo", "lunch", and "dinner", server 104 can first represent the reference tags and each candidate tag as an N (e.g., 50) dimensional word vector, then calculate the cosine distance between the reference tags and each candidate tag, and determine the semantic similarity between the reference tags and each candidate tag based on the cosine distance.

[0045] Step 206: Determine the co-occurrence probability between the reference label and the candidate label.

[0046] Co-occurrence probability describes the probability that a reference tag and a candidate tag belong to the same multimedia data, that is, the probability of them appearing simultaneously in the same multimedia data. When the reference tag is "pocket bread" and the candidate tag is "food", since pocket bread is a type of food, the probability of "pocket bread" and "food" appearing simultaneously in the same multimedia data is high. When the reference tag is "pocket bread" and the candidate tag is "style", since there is no direct relationship between pocket bread and style, "pocket bread" and "style" will almost never appear simultaneously in the same multimedia data, that is, the probability of them appearing simultaneously in the same multimedia data is close to zero.

[0047] In one embodiment, determining the co-occurrence probability between a reference label and a candidate label includes: using the point mutual information between the reference label and the candidate label as the co-occurrence probability, wherein the point mutual information is used to describe the correlation between the reference label and the candidate label.

[0048] Pointwise Mutual Information (PMI) is derived from mutual information in information theory. Mutual information measures the correlation between two random variables, while pointwise mutual information measures the correlation between two things, such as two words. In this application, pointwise mutual information measures the correlation between a reference label and a candidate label. Since co-occurrence probability describes the probability that a reference label and a candidate label belong to the same multimedia data, and pointwise mutual information can quantitatively give the degree of correlation between a reference label and a candidate label, the pointwise mutual information between the reference label and the candidate label can be used as the co-occurrence probability.

[0049] The basic principle of point mutual information is relatively simple, and the corresponding formula is as follows:

[0050]

[0051] In the formula, p(x,y) represents the probability that things x and y occur together, p(x) represents the probability that thing x occurs alone, and p(y) represents the probability that thing y occurs alone. If things x and y are unrelated, then p(x,y) = p(x)p(y), and the point mutual information PMI(x;y) between things x and y is 0. The greater the correlation between things x and y, the larger p(x,y) is compared to p(x)p(y), and the larger the point mutual information PMI(x;y) between things x and y is. Furthermore, it can be seen from formula (1) that the point mutual information PMI(x;y) between thing x and thing y is essentially the conditional probability p(x|y) of thing x occurring when thing y occurs divided by the probability p(x) of thing x itself occurring, or the conditional probability p(y|x) of thing y occurring when thing x occurs divided by the probability p(y) of thing y itself occurring. Moreover, the point mutual information between thing x and thing y is symmetrical, i.e., PMI(x;y) = PMI(y;x).

[0052] In this application, point mutual information between two different labels can be obtained in the following ways:

[0053]

[0054] In the formula, PMI(X;Y) represents the point mutual information between tag X and tag Y, n represents the frequency of tag X and tag Y appearing together in all existing multimedia data, N represents the number of tags in all existing multimedia data, N1 represents the frequency of tag X appearing alone in all existing multimedia data, and N2 represents the frequency of tag Y appearing alone in all existing multimedia data.

[0055] For example, let's take the hashtags "roujiamo" (Chinese hamburger) and "food" as examples. Suppose that in all existing multimedia data, "roujiamo" and "food" appear simultaneously in 10 multimedia datasets, "roujiamo" appears in 15 multimedia datasets, and "food" appears in 20 multimedia datasets. And there are a total of N different hashtags in all existing multimedia data. Then the mutual information between "roujiamo" and "food" is... The greater the calculated point mutual information, the more often the two topic tags appear together, that is, the greater the co-occurrence probability. In other words, when many video account owners post videos in their feeds, they are very likely to use the topic tag "roujiamo" (Chinese hamburger) and also "food".

[0056] After obtaining reference tags such as the topic tag "pocket biscuit" and at least two candidate tags such as the topic tags "home-cooked dishes", "breakfast biscuits", "food", "roujiamo", "lunch", "dinner", etc., server 104 can first obtain from the database the frequency of the reference tag, the frequency of each candidate tag, and the frequency of the reference tag and each candidate tag appearing simultaneously in the same multimedia data. Then, based on the frequency of the reference tag, the frequency of each candidate tag, the frequency of the reference tag and each candidate tag appearing simultaneously in the same multimedia data, and the number of tags in all multimedia data, server 104 calculates the co-occurrence probability between the reference tag and each candidate tag using the above formula (2).

[0057] Step 208: Select the target label from at least two candidate labels based on semantic similarity and co-occurrence probability.

[0058] After obtaining the semantic similarity and co-occurrence probability between the reference tag, such as the topic tag "pocket bread", and each candidate tag, such as the topic tags "home-style dishes", "breakfast bread", "food", "roujiamo", "lunch", and "dinner", the server 104 can determine the candidate tags that are most semantically similar to the reference tag, such as the topic tag "pocket bread", and frequently co-occur in the same multimedia data as the target tags, such as the topic tags "home-style dishes", "breakfast bread", "food", and "roujiamo".

[0059] In the aforementioned tag recommendation method for multimedia data, a target tag is selected from at least two candidate tags based on the semantic similarity between the reference tag and the candidate tag and the co-occurrence probability between the reference tag and the candidate tag. Since semantic similarity characterizes the semantic similarity of tags and co-occurrence probability characterizes the probability of tags appearing in the same multimedia data, the target tag obtained based on semantic similarity and co-occurrence probability is more in line with the distribution assumption of multimedia data, thus making the obtained target tag more accurate.

[0060] In one embodiment, reference Figure 4 As shown, the target label is selected from at least two candidate labels based on semantic similarity and co-occurrence probability, including:

[0061] Step S402: Input the semantic similarity and co-occurrence probability into the pre-trained label recommendation model, and obtain the function value of the objective function output by the pre-trained label recommendation model. The pre-trained label recommendation model has learned the correspondence between semantic similarity, co-occurrence probability, and the function value of the objective function corresponding to semantic similarity and co-occurrence probability.

[0062] Step S404: Select the target label from at least two candidate labels based on the function value.

[0063] In other words, after obtaining the semantic similarity and co-occurrence probability between reference tags such as the topic tag "pocket pancake" and each candidate tag such as the topic tags "home-style dishes," "breakfast pancake," "food," "roujiamo," "lunch," and "dinner," server 104 can select the target tag from at least two candidate tags based on the semantic similarity and co-occurrence probability using a pre-trained tag recommendation model. Specifically, server 104 can input the obtained semantic similarity and co-occurrence probability between the reference tag and each candidate tag into the pre-trained tag recommendation model. After the objective function in the tag recommendation model is calculated, the corresponding function value is output, and then the target tag is selected from at least two candidate tags based on the function value.

[0064] In one embodiment, selecting a target label from at least two candidate labels based on a function value includes: sorting the function values ​​according to their size order; and selecting a predetermined number of candidate labels that are ranked first as the target label.

[0065] For example, after inputting the semantic similarity and co-occurrence probability between reference tags such as the topic tag "pocket pancake" and each candidate tag such as the topic tags "home-style dishes," "breakfast pancake," "gourmet food," "roujiamo," "lunch," and "dinner" into a pre-trained tag recommendation model, the server 104 calculates the objective function of the pre-trained tag recommendation model. The output function values ​​are 80%, 90%, 90%, 95%, 60%, and 60% respectively. Then, all function values ​​are sorted, and a predetermined number of candidate tags at the top of the sorted list are used as target tags such as the topic tags "roujiamo," "breakfast pancake," "gourmet food," and "home-style dishes." Then, the server 104 sends the obtained target tags to the terminal 102 and displays them on the terminal 102 so that the user can directly select from the target tags, thereby improving the tag annotation efficiency.

[0066] In one embodiment, reference Figure 5 As shown, before obtaining the reference label and at least two candidate labels, the process also includes:

[0067] Step 502: Obtain at least two sample words from the news corpus and at least two sample tags from the multimedia data.

[0068] Corpora are the fundamental resource for corpus-based linguistics research and a primary resource for empirical language research methods. They are applied in lexicography, language teaching, traditional language research, and statistical or instance-based research in natural language processing. Corpora store language materials that have actually occurred in real-world language use. They are the basic resource for carrying language knowledge on a computer platform, and these authentic corpora need to be processed (analyzed and manipulated) to become useful resources.

[0069] In this application, the corpus is a news corpus. The server 104 can obtain at least two sample words from the news corpus and use them as part of the training data for the label recommendation model. At the same time, the server 104 also obtains a large number of existing multimedia data labels as at least two sample labels and uses them as part of the training data for the label recommendation model.

[0070] In one embodiment, before obtaining at least two sample words from the news corpus, the method further includes: obtaining massive amounts of news data; obtaining corresponding news text based on the news data; performing word segmentation on the news text to obtain at least two word segments; and constructing a news corpus based on the at least two word segments.

[0071] For example, server 104 can obtain a large amount of news data from the Internet platform, including text data, voice data, or video data from various aspects such as history, geography, science and technology, culture, education, entertainment, society and law. Then, all news data are converted into formats, such as text conversion, to obtain a large amount of news text. Next, the news text is segmented to convert the original character stream into individual words, and finally at least two words are obtained. This can be achieved using existing technologies. Finally, a news corpus is constructed based on the obtained words, that is, the news corpus contains a large number of words related to the news data.

[0072] Step 504: Analyze the semantic similarity between each sample word and use it as the sample semantic similarity, and analyze the semantic similarity between each sample label and use it as the sample semantic similarity.

[0073] After obtaining at least two sample words and at least two sample tags, server 104 can acquire the semantic similarity between each sample word using the aforementioned method and use it as the sample semantic similarity. Simultaneously, it can acquire the semantic similarity between each sample tag using the aforementioned method and use it as the sample semantic similarity. In other words, sample semantic similarity includes two aspects: the semantic similarity between sample words and the semantic similarity between sample tags. Because the semantic similarity of both news corpora and tag corpora is used to train the tag recommendation model, the model can generalize to any word without limitations, thus improving the recognition accuracy of the tag recommendation model.

[0074] Step 506: Analyze the co-occurrence probability among the labels of each sample and use it as the sample co-occurrence probability.

[0075] After obtaining at least two sample labels, server 104 can also obtain the co-occurrence probability between each sample label in the aforementioned manner and use it as the sample co-occurrence probability.

[0076] Step 508: Train the initial label recommendation model based on the semantic similarity and co-occurrence probability of the samples until the loss value between the target value and the calibration value of the objective function output by the initial label recommendation model meets the set conditions, and the label recommendation model training is completed.

[0077] After obtaining the semantic similarity and co-occurrence probability of the samples, server 104 inputs the semantic similarity and co-occurrence probability of the samples into the initial label recommendation model to train the initial label recommendation model until the loss value between the target value and the calibration value of the objective function output by the initial label recommendation model meets the set conditions, and the label recommendation model training is completed.

[0078] In one embodiment, the initial tag recommendation model includes a first tag recommendation model and a second tag recommendation model. The objective function includes a first function corresponding to the first tag recommendation model and a second function corresponding to the second tag recommendation model. The first tag recommendation model and the second tag recommendation model share a word embedding layer.

[0079] In other words, the initial tag recommendation model can include two tag recommendation models: a first tag recommendation model and a second tag recommendation model. The first tag recommendation model serves as the main model and can be a word vector model. The second tag recommendation model serves as an auxiliary model, sharing the word embedding layer with the main model for joint optimization. That is, the function value output by the auxiliary model serves as the joint supervision signal for the main model, enabling joint training of the main model. This allows the trained tag recommendation model to learn the co-occurrence probability distribution of different tags in existing tags of multimedia data. The first tag recommendation model corresponds to a first function, which, when the first tag recommendation model is a word vector model, is the objective function of the word vector model. The second tag recommendation model corresponds to a second function, which can be a loss function that calculates the difference in co-occurrence probabilities, i.e., the difference in point mutual information. Specifically, it can be a hinge loss function or a mean squared error loss function, etc. The objective function includes both the first and second functions; for example, the objective function can be the first function plus the second function, or the absolute value of the first function plus the absolute value of the second function, or the multiplication of the first and second functions, etc.

[0080] In one embodiment, an initial label recommendation model is trained based on sample semantic similarity and sample co-occurrence probability until the loss value between the target value and the calibration value of the target function output by the initial label recommendation model meets a set condition, at which point the label recommendation model training is complete. This includes: training a first label recommendation model based on sample semantic similarity and obtaining the first function value of the first function output by the first label recommendation model; training a second label recommendation model based on sample co-occurrence probability and obtaining the second function value of the second function output by the second label recommendation model; and using the sum of the first function value and the second function value as the target value until the loss value between the target value and the calibration value meets a set condition, at which point the label recommendation model training is complete.

[0081] After obtaining the sample semantic similarity and sample co-occurrence probability, server 104 trains the first label recommendation model using the sample semantic similarity and obtains the function value of the first function corresponding to the first label recommendation model, denoted as the first function value. Simultaneously, it trains the second label recommendation model using the sample co-occurrence probability and obtains the function value of the second function corresponding to the second label recommendation model, denoted as the second function value. Then, the first function value and the second function value are summed, and the sum is used as the target value. This process continues until the loss value between the target value and the calibration value meets a set condition, at which point the label recommendation model training is complete. Specifically, the first function is used to find the maximum semantic similarity, the second function is used to find the maximum co-occurrence probability, and the target function is used to find the maximum value of both simultaneously. Furthermore, when the target function is the sum of the first and second functions, the target function is used to find the sum of their maximum values, i.e., to maximize the sum of the two values.

[0082] As a concrete example, see reference Figure 6 As shown, the initial tag recommendation model may include a first tag recommendation model (left side of the figure) and a second tag recommendation model (right side of the figure). The first tag recommendation model is the main model and can be a word vector model, specifically a skip-gram model or a glove model, etc. The second tag recommendation model is the auxiliary model and shares the word embedding layer with the main model for joint optimization.

[0083] When the first label recommendation model is the skip-gram model, the objective function of the model, i.e. the first function, can be expressed by the following formula (3):

[0084]

[0085] In the formula, Let p(w) be the objective function for the skip-gram model, specifically maximizing the semantic similarity likelihood function, which is used to find the maximum value of semantic similarity. T represents the total number of sample words input to the skip-gram model. t+j |w t ) indicates the central sample word w t When the sample word w appears t+j The conditional probability of occurrence, where c is the window size of the skip-gram model.

[0086] The objective function of the second-label recommendation model, i.e., the second function, can be expressed by the following formula (4):

[0087]

[0088] In the formula, The objective function for the second-label recommendation model is to maximize the co-occurrence probability likelihood function, which is used to find the maximum value of the co-occurrence probability. Specifically, this can be achieved by finding the difference between different co-occurrence probabilities using the hinge loss function or the mean squared error loss function, and then determining the maximum co-occurrence probability based on this difference. For example, when PMI(w) i ,w j > PMI(w i ,w k When ), the maximum co-occurrence probability is PMI(w). i ,w j PMI (w) i ,w j ) represents the sample label w i With sample label w j The point mutual information between them is the co-occurrence probability, PMI(w) i ,w k ) represents the sample label w i With sample label w k The mutual information between points is the co-occurrence probability.

[0089] The objective function of the initial label recommendation model can be expressed by the following formula (5):

[0090]

[0091] In the formula, The objective function of the initial label recommendation model is to maximize the sum of semantic similarity and co-occurrence probability.

[0092] It should be noted that in this example, the first function corresponding to the first label recommendation model includes the semantic similarity analysis process, while the second function corresponding to the second label recommendation model includes the co-occurrence probability analysis process. Therefore, when training the initial label recommendation model, sample words and sample labels can be directly input as training parameters into the initial label recommendation model and trained to obtain a pre-trained label recommendation model. Then, when the server 104 selects the target label from at least two candidate labels based on the pre-trained label recommendation model, it can directly input the reference label and at least two candidate labels into the pre-trained label recommendation model and learn to obtain the target label, thus eliminating the need to separately determine the semantic similarity and co-occurrence probability between the reference label and the candidate labels.

[0093] The target values ​​learned by the aforementioned tag recommendation model can not only ensure high similarity between words like "roujiamo" (meat sandwich) and "delicious food" in terms of cosine distance in their word vector representations, but also make the word vectors of words that are more likely to appear together frequently in other potential multimedia data closer together. For example, in the background technology, "Beijing" and "Shanghai" are unlikely to co-occur simultaneously in the video account information stream scenario. Therefore, the word vector distance between the two words learned by the aforementioned tag recommendation model will be significantly increased, i.e., their similarity will be low.

[0094] It should be understood that, although Figure 2 , 4 The steps in flowchart -5 are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order in which these steps are performed; they can be executed in other orders. Furthermore, Figure 2 , 4 At least some of the steps in -5 may include at least two sub-steps or at least two stages. These sub-steps or stages are not necessarily executed at the same time, but may be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0095] In summary, the aforementioned tag recommendation method for multimedia data obtains a reference tag and at least two candidate tags, determines the semantic similarity between the reference tag and the candidate tags, and determines the co-occurrence probability between the reference tag and the candidate tags. Based on the semantic similarity and co-occurrence probability, a target tag is selected from at least two candidate tags. Since semantic similarity characterizes the semantic similarity of tags, and co-occurrence probability characterizes the probability of tags appearing in the same multimedia data, the target tag obtained based on semantic similarity and co-occurrence probability better conforms to the distribution assumptions of multimedia data, thus making the obtained target tags more accurate.

[0096] In one embodiment, such as Figure 7 As shown, a training method for a tag recommendation model is provided, which is then applied to... Figure 1 Taking the server in the example, the following steps are included:

[0097] Step S702: Obtain at least two sample words and at least two sample tags of multimedia data.

[0098] Server 104 can obtain at least two sample words from the news corpus and use them as part of the training data for the label recommendation model. At the same time, server 104 also obtains a large number of existing multimedia data labels as at least two sample labels and uses them as part of the training data for the label recommendation model.

[0099] Step S704: Analyze the semantic similarity between each sample word and use it as the sample semantic similarity, and analyze the semantic similarity between each sample label and use it as the sample semantic similarity.

[0100] In this application, the semantic similarity between two different sample words can be determined by representing each of them as an N-dimensional word vector, calculating the cosine distance between the word vectors, and then using the cosine distance. Of course, other methods can also be used to obtain the semantic similarity between two different sample words; existing technologies can be employed, and no limitation is made here. For methods of obtaining the semantic similarity between two different sample tags, refer to the methods for obtaining the semantic similarity between two different sample words.

[0101] After obtaining at least two sample words and at least two sample tags, server 104 can acquire the semantic similarity between each sample word using the aforementioned method and use it as the sample semantic similarity. Simultaneously, it can acquire the semantic similarity between each sample tag using the aforementioned method and use it as the sample semantic similarity. In other words, sample semantic similarity includes two aspects: the semantic similarity between sample words and the semantic similarity between sample tags. Because the semantic similarity of both news corpora and tag corpora is used to train the tag recommendation model, the model can generalize to any word without limitations, thus improving the recognition accuracy of the tag recommendation model.

[0102] Step S706: Analyze the co-occurrence probability among the labels of each sample and use it as the sample co-occurrence probability.

[0103] In one embodiment, analyzing the co-occurrence probability among various sample labels and using it as the sample co-occurrence probability includes: using the point mutual information among various sample labels as the sample co-occurrence probability, wherein the point mutual information is used to describe the correlation between various sample labels.

[0104] After obtaining at least two sample tags, server 104 can first obtain the frequency of each sample tag in all existing multimedia data in the database, the frequency of each two different sample tags appearing in the same multimedia data, and then calculate the co-occurrence probability between each two sample tags using the above formula (2) based on the frequency of each sample tag, the frequency of each two different tags appearing in the same multimedia data, and the number of at least two sample tags.

[0105] Step S708: Train a label recommendation model based on sample semantic similarity and sample co-occurrence probability.

[0106] In one embodiment, training a label recommendation model based on sample semantic similarity and sample co-occurrence probability includes: training an initial label recommendation model based on sample semantic similarity and sample co-occurrence probability until the loss value between the target value and the calibration value of the objective function output by the initial label recommendation model meets the set conditions, at which point the label recommendation model training is complete.

[0107] After obtaining the semantic similarity and co-occurrence probability of the samples, server 104 inputs the semantic similarity and co-occurrence probability of the samples into the initial label recommendation model to train the initial label recommendation model until the loss value between the target value and the calibration value of the objective function output by the initial label recommendation model meets the set conditions, and the label recommendation model training is completed.

[0108] In one embodiment, the initial tag recommendation model includes a first tag recommendation model and a second tag recommendation model, and the objective function includes a first function corresponding to the first tag recommendation model and a second function corresponding to the second tag recommendation model. The first tag recommendation model and the second tag recommendation model share a word embedding layer.

[0109] In other words, the initial tag recommendation model can include two tag recommendation models: a first tag recommendation model and a second tag recommendation model. The first tag recommendation model serves as the main model and can be a word vector model. The second tag recommendation model serves as an auxiliary model, sharing the word embedding layer with the main model for joint optimization. That is, the function value output by the auxiliary model serves as the joint supervision signal for the main model, enabling joint training of the main model. This allows the trained tag recommendation model to learn the co-occurrence probability distribution of different tags in existing tags of multimedia data. The first tag recommendation model corresponds to a first function, which, when the first tag recommendation model is a word vector model, is the objective function of the word vector model. The second tag recommendation model corresponds to a second function, which can be a loss function that calculates the difference in co-occurrence probabilities, i.e., the difference in point mutual information. Specifically, it can be a hinge loss function or a mean squared error loss function, etc. The objective function includes both the first and second functions; for example, the objective function can be the first function plus the second function, or the absolute value of the first function plus the absolute value of the second function, or the multiplication of the first and second functions, etc.

[0110] In one embodiment, an initial label recommendation model is trained based on sample semantic similarity and sample co-occurrence probability until the loss value between the target value and the calibration value of the target function output by the initial label recommendation model meets a set condition, at which point the label recommendation model training is complete. This includes: training a first label recommendation model based on sample semantic similarity and obtaining the first function value of the first function output by the first label recommendation model; training a second label recommendation model based on sample co-occurrence probability and obtaining the second function value of the second function output by the second label recommendation model; and using the sum of the first function value and the second function value as the target value until the loss value between the target value and the calibration value meets a set condition, at which point the label recommendation model training is complete.

[0111] After obtaining the sample semantic similarity and sample co-occurrence probability, server 104 trains the first label recommendation model using the sample semantic similarity and obtains the function value of the first function corresponding to the first label recommendation model, denoted as the first function value. Simultaneously, it trains the second label recommendation model using the sample co-occurrence probability and obtains the function value of the second function corresponding to the second label recommendation model, denoted as the second function value. Then, the first function value and the second function value are summed, and the sum is used as the target value. This process continues until the loss value between the target value and the calibration value meets a set condition, at which point the label recommendation model training is complete. Specifically, the first function is used to find the maximum semantic similarity, the second function is used to find the maximum co-occurrence probability, and the target function is used to find the maximum value of both simultaneously. Furthermore, when the target function is the sum of the first and second functions, the target function is used to find the sum of their maximum values, i.e., to maximize the sum of the two values.

[0112] As a concrete example, see reference Figure 6As shown, the initial label recommendation model can include a first label recommendation model (left side of the figure) and a second label recommendation model (right side of the figure). The first label recommendation model is the main model and can be a word vector model, specifically a skip-gram model or a glove model, etc. The second label recommendation model is the auxiliary model and shares the word embedding layer with the main model for joint optimization. When the first label recommendation model is a skip-gram model, the objective function corresponding to this model, i.e., the first function, can be expressed by the above formula (3). The objective function corresponding to the second label recommendation model, i.e., the second function, can be expressed by the above formula (4). The objective function corresponding to the initial label recommendation model can be expressed by the above formula (5). It should be noted that in this example, the first function corresponding to the first label recommendation model includes the semantic similarity analysis process, and the second function corresponding to the second label recommendation model includes the co-occurrence probability analysis process. Therefore, when training the initial label recommendation model, the sample words and sample labels can be directly input as training parameters into the initial label recommendation model and trained to obtain the label recommendation model.

[0113] It should be noted that for details regarding the undisclosed training method of the tag recommendation model, please refer to the section on model training in the aforementioned tag recommendation method for multimedia data.

[0114] In one embodiment, such as Figure 8 As shown, a tag recommendation device for multimedia data is provided. The tag recommendation device 100 for multimedia data includes: a first acquisition module 101, a first determination module 102, a second determination module 103, and a recommendation module 104.

[0115] The first acquisition module 101 is used to acquire a reference tag and at least two candidate tags; the first determination module 102 is used to determine the semantic similarity between the reference tag and the candidate tags; the second determination module 103 is used to determine the co-occurrence probability between the reference tag and the candidate tags, where the co-occurrence probability describes the probability that the reference tag and the candidate tags belong to the same multimedia data; and the recommendation module 104 is used to select a target tag from at least two candidate tags based on the semantic similarity and the co-occurrence probability.

[0116] In one embodiment, the recommendation module 104 is specifically used to input semantic similarity and co-occurrence probability into a pre-trained label recommendation model, and obtain the function value of the objective function output by the pre-trained label recommendation model, wherein the pre-trained label recommendation model has learned the correspondence between semantic similarity, co-occurrence probability, and the function value of the objective function corresponding to semantic similarity and co-occurrence probability; and select a target label from at least two candidate labels based on the function value.

[0117] In one embodiment, the recommendation module 104 is specifically used to sort the function values ​​according to their size order and select the first preset number of candidate labels as the target labels.

[0118] In one embodiment, reference Figure 9 As shown, the tag recommendation device for multimedia data further includes: a second acquisition module 105, a first analysis module 106, a second analysis module 107, and a training module 108. Specifically, the second acquisition module 105 acquires at least two sample words from a news corpus and acquires at least two sample tags from the multimedia data; the first analysis module 106 analyzes the semantic similarity between each sample word and uses it as the sample semantic similarity, and analyzes the semantic similarity between each sample tag and uses it as the sample semantic similarity; the second analysis module 107 analyzes the co-occurrence probability between each sample tag and uses it as the sample co-occurrence probability; the training module 108 trains an initial tag recommendation model based on the sample semantic similarity and sample co-occurrence probability until the loss value between the target value and the calibration value of the objective function output by the initial tag recommendation model meets the set conditions, at which point the tag recommendation model training is complete.

[0119] In one embodiment, the initial tag recommendation model includes a first tag recommendation model and a second tag recommendation model, and the objective function includes a first function corresponding to the first tag recommendation model and a second function corresponding to the second tag recommendation model. The first tag recommendation model and the second tag recommendation model share a word embedding layer.

[0120] In one embodiment, the training module 108 is specifically used to: train a first label recommendation model using sample semantic similarity and obtain the first function value of the first function output by the first label recommendation model; train a second label recommendation model using sample co-occurrence probability and obtain the second function value of the second function output by the second label recommendation model; use the sum of the first function value and the second function value as the target value, until the loss value between the target value and the calibration value meets the set conditions, and the label recommendation model training is completed.

[0121] In one embodiment, reference Figure 10 As shown, the tag recommendation device for multimedia data further includes: a third acquisition module 109, a conversion module 110, a word segmentation module 111, and a construction module 112, wherein the third acquisition module 109 is used to acquire massive amounts of news data; the conversion module 110 is used to obtain corresponding news text based on the news data; the word segmentation module 111 performs word segmentation processing on the news text to obtain at least two words; and the construction module 112 constructs a news corpus based on the at least two words.

[0122] In one embodiment, the second determining module 103 is specifically used to obtain point mutual information between the reference label and the candidate label, and to use the point mutual information as the co-occurrence probability.

[0123] In one embodiment, the first label recommendation model is a word vector model.

[0124] In one embodiment, the second function is either a hinge loss function or a mean squared error loss function.

[0125] Specific limitations regarding the tag recommendation device for multimedia data can be found in the limitations of the tag recommendation method for multimedia data described above, and will not be repeated here. Each module in the aforementioned tag recommendation device for multimedia data can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0126] In some embodiments of this application, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in various optional implementations of the embodiments of the present invention, such as... Figure 2 , Figures 4-5 The method for recommending tags for multimedia data shown, or Figure 7The training method for the label recommendation model is shown. Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0127] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0128] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A training method for a tag recommendation model, characterized in that, include: At least two sample words are obtained from the news corpus, and sample tags for multiple multimedia data are obtained. The sample words in the news corpus are obtained by segmenting the news text, and there are sample words belonging to the same news text. Each multimedia data corresponds to at least two sample tags. The semantic similarity between each of the sample words is analyzed and used as the sample semantic similarity, and the semantic similarity between each of the sample tags is also analyzed and used as the sample semantic similarity. The co-occurrence probability among the various sample tags is analyzed and used as the sample co-occurrence probability, which describes the probability that each pair of sample tags belongs to the same multimedia data tag; The sample semantic similarity is used to train the first label recommendation model in the initial label recommendation model, and the first function value of the first function output by the first label recommendation model is obtained; The co-occurrence probabilities of the samples are used to train the second label recommendation model in the initial label recommendation model, and the second function value of the second function output by the second label recommendation model is obtained. The first label recommendation model and the second label recommendation model share the word embedding layer. The sum of the first function value and the second function value is used as the target value. The training of the label recommendation model is completed when the loss value between the target value and the calibration value meets the set conditions.

2. The training method for the tag recommendation model according to claim 1, characterized in that, The method further includes: Acquire massive amounts of news data; The corresponding news text is obtained based on the news data; The news text is segmented to obtain at least two words; The news corpus is constructed based on at least two word segments.

3. The training method for the tag recommendation model according to claim 2, characterized in that, The method further includes: Obtain the reference tag and at least two candidate tags; Determine the semantic similarity between the reference label and the candidate label; Determine the co-occurrence probability between the reference tag and the candidate tag, wherein the co-occurrence probability is used to describe the probability that the reference tag and the candidate tag belong to the same multimedia data; The target label is selected from the at least two candidate labels based on the semantic similarity and the co-occurrence probability.

4. The training method for the tag recommendation model according to claim 3, characterized in that, The step of selecting a target label from at least two candidate labels based on the semantic similarity and the co-occurrence probability includes: The semantic similarity and the co-occurrence probability are input into a pre-trained label recommendation model, and the function value of the objective function output by the pre-trained label recommendation model is obtained. The pre-trained label recommendation model has learned the correspondence between the semantic similarity, the co-occurrence probability, and the function value of the objective function corresponding to the semantic similarity and the co-occurrence probability. Based on the function value, a target label is selected from the at least two candidate labels.

5. The training method for the tag recommendation model according to claim 4, characterized in that, The step of selecting a target label from at least two candidate labels based on the function value includes: Sort the function values ​​according to their size order; The first preset number of candidate tags are selected as the target tags.

6. The training method for the tag recommendation model according to any one of claims 1-5, characterized in that, The analysis of the co-occurrence probability among the various sample labels and its use as the sample co-occurrence probability includes: The point mutual information between each of the sample labels is used as the sample co-occurrence probability, wherein the point mutual information is used to describe the correlation between each of the sample labels.

7. A training device for a tag recommendation model, characterized in that, The apparatus is used to perform the steps of the training method for the label recommendation model according to any one of claims 1-6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the training method for the label recommendation model according to any one of claims 1-6.

Images