Label construction method and device of multimedia resource, electronic equipment and storage medium

Abstract

The present disclosure relates to a label construction method and device of multimedia resources, an electronic device, a computer readable storage medium and a computer program product. The method comprises the following steps: obtaining a multimedia resource to be constructed; determining a target label set from a plurality of multimedia; obtaining statistical features of a label pair according to the multimedia corresponding to any label in the target label set; determining a relationship category of the label pair according to the statistical features of the label pair; and finally constructing a label system of the multimedia resource according to the relationship category of each label pair. The method can realize label mining and label system construction based on the multimedia resource to be constructed, thereby improving the accuracy and efficiency of defining labels.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of data processing, and particularly relates to a label construction method and device of multimedia resources, electronic equipment, computer readable storage medium and computer program product. BACKGROUND

[0002] With the development of Internet technology, short video technology has been widely applied. In the short video analysis scene, the label is also important structured data, which plays an important role in the arrangement, retrieval and distribution of short videos.

[0003] In the related art, the label system in the traditional information system is defined by experts. However, due to the characteristics of large increment and high timeliness of short videos, the traditional label definition method has low accuracy and efficiency. SUMMARY

[0004] The present disclosure provides a label construction method and device of multimedia resources, electronic equipment, computer readable storage medium and computer program product to at least solve the problem of low accuracy and efficiency of the label definition method in the related art. The technical solutions of the present disclosure are as follows:

[0005] According to a first aspect of the embodiments of the present disclosure, a label construction method of multimedia resources is provided, comprising:

[0006] Obtaining multimedia resources to be constructed labels, the multimedia resources comprising a plurality of multimedia;

[0007] Determining a target label set from the plurality of multimedia, the target label set comprising a plurality of target labels;

[0008] According to the statistical characteristics of the label pair, the relationship category of the label pair is determined.

[0009] According to the statistical characteristics of the label pair, the relationship category of the label pair is determined.

[0010] According to the relationship category of each label pair, a label system of the multimedia resources is constructed.

[0011] In one of the embodiments, the target label set is determined from the plurality of multimedia, comprising: obtaining text information of each multimedia, the text information comprising a splicing string composed of a title of the multimedia and a cover text of the multimedia; identifying keywords in the text information of each multimedia, and obtaining frequency of each keyword; obtaining candidate labels satisfying a set condition according to each keyword and the corresponding frequency; obtaining a feature representation of each candidate label, determining a target label according to the feature representation, and obtaining the target label set.

[0012] In one of the embodiments, the multimedia further comprises corresponding category information; the method for obtaining the feature representation of each candidate label, and determining the target label according to the feature representation comprises: obtaining the feature representation of each candidate label according to the multimedia corresponding to each candidate label and the category information corresponding to the multimedia, wherein the feature representation comprises the transition feature of the candidate label to any category in the category information and the corresponding feature entropy; inputting the feature representation of the candidate label into the trained prediction model to obtain the feature parameter of the candidate label output by the prediction model; and obtaining the candidate label whose feature parameter reaches the parameter threshold as the target label.

[0013] In one of the embodiments, the method for obtaining the transition feature comprises: obtaining the first occurrence frequency of the candidate label in the multimedia corresponding to any category information in the category information corresponding to the multimedia resource, and obtaining the second occurrence frequency of the candidate label in all multimedia in the multimedia resource; and determining the transition feature of the candidate label to any category according to the first occurrence frequency and the second occurrence frequency.

[0014] In one of the embodiments, the method for obtaining the feature entropy comprises: calculating the feature entropy corresponding to the transition feature of the candidate label to any category according to the category information corresponding to the multimedia in the multimedia resource and the transition feature of the candidate label to any category.

[0015] In one of the embodiments, the base model of the prediction model adopts a random forest classifier; and the method for obtaining the prediction model comprises: obtaining a training set, wherein the training set comprises the same number of label sample positives and label sample negatives, the label sample positives are label samples selected from a preset confidence label set, and the label sample negatives are label samples selected from the candidate labels; obtaining the sample feature representation of each label sample in the training set, wherein the sample feature representation comprises the sample transition feature of the label sample to any category and the corresponding sample feature entropy; and training the random forest classifier by using the sample feature representation of each label sample to obtain the trained prediction model.

[0016] In one of the embodiments, the method for obtaining the keyword satisfying the set condition as the candidate label according to each keyword and the corresponding frequency comprises: when there is a keyword with a character length satisfying the set length, and the frequency of the keyword reaches the set frequency threshold, determining that the keyword satisfies the set condition; and determining the keyword satisfying the set condition as the candidate label.

[0017] In one of the embodiments, the statistical feature comprises a transition probability of the label pair; the label pair comprises a first target label and a second target label; and the acquiring the statistical feature of the label pair according to the multimedia corresponding to any of the target labels comprises: acquiring a first quantity of the multimedia corresponding to both the first target label and the second target label, and acquiring a second quantity of the multimedia corresponding to the first target label; and determining the transition probability of the first target label and the second target label according to the first quantity and the second quantity.

[0018] In one of the embodiments, the statistical feature comprises mutual information of the label pair; the acquiring the statistical feature of the label pair according to the multimedia corresponding to any of the target labels comprises: acquiring a first quantity of the multimedia corresponding to both the first target label and the second target label, and acquiring a second quantity of the multimedia corresponding to the first target label and a third quantity of the multimedia corresponding to the second target label; and calculating the mutual information between the first target label and the second target label according to the first quantity, the second quantity and the third quantity.

[0019] In one of the embodiments, the statistical feature comprises information entropy corresponding to the label pair; the acquiring the statistical feature of the label pair according to the multimedia corresponding to any of the target labels comprises: acquiring a first label set matched with the second target label, the first label set comprising a plurality of first target labels, and the first target label and the second target label having at least one common corresponding multimedia; acquiring a transition probability of each first target label and the second target label; and calculating the information entropy corresponding to the first label set matched with the second target label according to the first label set and the transition probability of each first target label and the second target label in the first label set.

[0020] In one of the embodiments, the statistical feature comprises a multimodal transition probability corresponding to the label pair; the acquiring the statistical feature of the label pair according to the multimedia corresponding to any of the target labels comprises: acquiring visual information and text information of each multimedia in the multimedia resource, and performing multimodal coding on the visual information and the text information of the multimedia to obtain a multimodal feature corresponding to the multimedia; acquiring a distance between the first target label and the second target label according to the multimedia corresponding to the first target label and the second target label respectively and the multimodal feature of the multimedia; performing normalized average processing on the distance to obtain a probability mass score of the first target label and the second target label; and calculating the multimodal transition probability corresponding to the label pair according to the probability mass score.

[0021] In one of the embodiments, the relationship category of the label pair includes any one of the second target label being a parent label of the first target label, the second target label being a child label of the first target label, and the second target label being irrelevant to the first target label; and the constructing the label system of the multimedia resource according to the relationship category of each of the label pairs includes: when the relationship category of the label pair is the second target label being a parent label of the first target label or the second target label being a child label of the first target label, generating a pointing relationship between the second target label and the first target label in the label pair according to the relationship category; and obtaining the label system of the multimedia resource according to the target label set and the pointing relationship between the second target label and the first target label in each of the label pairs.

[0022] In one of the embodiments, the obtaining the multimedia resource to be constructed with labels includes: obtaining full-candidate multimedia to be constructed with labels, filtering the full-candidate multimedia based on a preset condition to obtain a plurality of filtered multimedia, and obtaining the multimedia resource to be constructed with labels based on the plurality of filtered multimedia.

[0023] According to a second aspect of the embodiments of the present disclosure, a label construction device of a multimedia resource is provided, including:

[0024] The multimedia resource obtaining module is configured to perform the obtaining of the multimedia resource to be constructed with labels, and the multimedia resource includes a plurality of multimedia.

[0025] The target label set determining module is configured to perform the determining of the target label set from the plurality of multimedia, and the target label set includes a plurality of target labels.

[0026] The statistical feature obtaining module is configured to perform the obtaining of the statistical feature of any label pair in the target label set based on the multimedia corresponding to the label pair, and the label pair is any two target labels appearing in the same multimedia.

[0027] The relationship category identifying module is configured to perform the determining of the relationship category of the label pair based on the statistical feature of the label pair.

[0028] The label system constructing module is configured to perform the constructing of the label system of the multimedia resource based on the relationship category of each of the label pairs.

[0029] In one of the embodiments, the target tag set determining module comprises: a text information obtaining unit configured to obtain text information of each multimedia, the text information comprising a spliced string of a title of the multimedia and cover text of the multimedia; a keyword information identifying unit configured to identify keywords in the text information of each multimedia and obtain frequency of each keyword; a candidate tag determining unit configured to obtain keywords satisfying a set condition as candidate tags according to each keyword and corresponding frequency; and a target tag set obtaining unit configured to obtain feature representation of each candidate tag and determine target tags according to the feature representation to obtain the target tag set.

[0030] In one of the embodiments, the multimedia further comprises corresponding category information; and the target tag set obtaining unit further comprises: a feature representation obtaining subunit configured to obtain feature representation of the candidate tag according to the multimedia corresponding to each candidate tag and the category information corresponding to the multimedia, the feature representation comprising transition feature of the candidate tag to any category in the category information and corresponding feature entropy; a feature parameter obtaining subunit configured to input the feature representation of the candidate tag into a trained prediction model to obtain feature parameter of the candidate tag output by the prediction model; and a target tag determining subunit configured to obtain the candidate tag whose feature parameter reaches a parameter threshold as a target tag.

[0031] In one of the embodiments, the feature representation obtaining subunit is further configured to: obtain first occurrence frequency of the candidate tag in the multimedia corresponding to any category information in the category information corresponding to the multimedia resource, and obtain second occurrence frequency of the candidate tag in all multimedia in the multimedia resource; and determine the transition feature of the candidate tag to any category according to the first occurrence frequency and the second occurrence frequency.

[0032] In one of the embodiments, the feature representation obtaining subunit is further configured to: calculate feature entropy corresponding to the transition feature of the candidate tag to any category according to the category information corresponding to the multimedia in the multimedia resource and the transition feature of the candidate tag to any category.

[0033] In one of the embodiments, the target label set obtaining unit further comprises a prediction model obtaining subunit, configured to perform: obtaining a training set, the training set comprising the same number of label sample positives and label sample negatives, the label sample positives being label samples selected from the preset confidence label set, and the label sample negatives being label samples selected from the candidate labels; obtaining a sample feature representation of each label sample in the training set, the sample feature representation comprising a sample transition feature of the label sample to any category and a corresponding sample feature entropy; and training the random forest classifier using the sample feature representation of each label sample to obtain a trained prediction model.

[0034] In one of the embodiments, the candidate label determining unit is configured to perform: when there is a keyword with a character length satisfying a set length, and the frequency of the keyword reaches a set frequency threshold, determining that the keyword satisfies a set condition; and determining the keyword satisfying the set condition as a candidate label.

[0035] In one of the embodiments, the statistical features comprise a transition probability of the label pair; the label pair comprises a first target label and a second target label; and the statistical feature obtaining module is configured to perform: obtaining a first quantity of multimedia corresponding to both the first target label and the second target label, and obtaining a second quantity of multimedia corresponding to the first target label; and determining the transition probability of the first target label and the second target label according to the first quantity and the second quantity.

[0036] In one of the embodiments, the statistical features further comprise mutual information of the label pair; and the statistical feature obtaining module is further configured to perform: obtaining a first quantity of multimedia corresponding to both the first target label and the second target label, and obtaining a second quantity of multimedia corresponding to the first target label and a third quantity of multimedia corresponding to the second target label; and calculating the mutual information between the first target label and the second target label according to the first quantity, the second quantity, and the third quantity.

[0037] In one of the embodiments, the statistical features further comprise information entropy corresponding to the label pair; and the statistical feature obtaining module is further configured to perform: obtaining a first label set matching the second target label, the first label set comprising a plurality of first target labels, and the first target labels and the second target label at least having one common corresponding multimedia; obtaining a transition probability of each first target label and the second target label; and calculating the information entropy corresponding to the first label set matching the second target label according to the first label set and the transition probability of each first target label and the second target label in the first label set.

[0038] In one of the embodiments, the statistical features further include a multi-modal transition probability corresponding to the label pair; the statistical feature acquisition module is further configured to perform: acquiring visual information and text information of each multimedia in the multimedia resource, performing multi-modal encoding on the visual information and the text information of the multimedia to obtain a multi-modal feature corresponding to the multimedia; acquiring a distance between the first target label and the second target label according to the multimedia corresponding to the first target label and the second target label respectively and the multi-modal feature of the multimedia; performing normalized average processing on the distance to obtain a probability mass score of the first target label and the second target label; and calculating the multi-modal transition probability corresponding to the label pair according to the probability mass score.

[0039] In one of the embodiments, the relationship category of the label pair includes any one of the second target label being a parent label of the first target label, the second target label being a child label of the first target label, and the second target label being irrelevant to the first target label; the label system construction module is further configured to perform: when the relationship category of the label pair is the second target label being a parent label of the first target label or the second target label being a child label of the first target label, generating a pointing relationship between the second target label and the first target label in the label pair according to the relationship category; and obtaining a label system of the multimedia resource according to the target label set and the pointing relationship between the second target label and the first target label in each of the label pairs.

[0040] In one of the embodiments, the multimedia resource acquisition module is further configured to perform: acquiring full-candidate multimedia of a label to be constructed, filtering the full-candidate multimedia based on a preset condition to obtain a plurality of filtered multimedia; and obtaining the multimedia resource of the label to be constructed according to the plurality of filtered multimedia.

[0041] According to a third aspect of the embodiments of the present disclosure, an electronic device is provided, including: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the label construction method of the multimedia resource as described in the first aspect above.

[0042] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute the label construction method of the multimedia resource as described in the first aspect above.

[0043] According to a fifth aspect of the embodiments of the present disclosure, a computer program product is provided, which comprises instructions executed by a processor of an electronic device to enable the electronic device to perform the method for constructing a label system of a multimedia resource according to the first aspect.

[0044] The embodiments of the present disclosure provide at least the following beneficial effects: by obtaining a multimedia resource to be constructed, determining a target label set from a plurality of multimedia, and obtaining statistical features of a label pair corresponding to the multimedia according to any label in the target label set, further determining a relationship category of the label pair according to the statistical features of the label pair, and finally constructing a label system of the multimedia resource according to the relationship category of each label pair, the label mining based on the multimedia resource to be constructed and the construction of the label system are realized, thereby improving the accuracy and efficiency of defining labels.

[0045] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0046] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the specification, serve to explain the principles of the present disclosure, and do not constitute an undue limitation on the present disclosure.

[0047] Figure 1 is a flowchart of a method for constructing a label system of a multimedia resource according to an exemplary embodiment.

[0048] Figure 2 is a schematic diagram of a label system according to an exemplary embodiment.

[0049] Figure 3 is a flowchart of a step of obtaining a multimedia resource according to an exemplary embodiment.

[0050] Figure 4 is a flowchart of a step of determining a target label set according to an exemplary embodiment.

[0051] Figure 5 is a flowchart of a step of determining a target label according to an exemplary embodiment.

[0052] Figure 6 is a flowchart of a step of obtaining a prediction model according to an exemplary embodiment.

[0053] Figure 7 is a flowchart of a step of obtaining statistical features according to an exemplary embodiment.

[0054] Figure 8is a schematic diagram of multi-modal encoding according to an example embodiment.

[0055] Figure 9 is a block diagram of a tag construction apparatus of a multimedia resource according to an example embodiment.

[0056] Figure 10 is a block diagram of an electronic device according to an example embodiment. DETAILED DESCRIPTION

[0057] In order for those skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below in conjunction with the accompanying drawings.

[0058] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation described in the following exemplary embodiments does not represent all implementations consistent with the present disclosure. Rather, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0059] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, analyzed data, etc.) involved in the present disclosure are all information and data authorized by the user or authorized by all parties.

[0060] Currently, traditional label mining models require a given text corpus and corresponding language confidence term name, so as to output candidate N-grams and their quality scores, and high-score N-grams can be used as labels. This method first obtains high-frequency N-grams from the corpus, and then counts text features from two aspects of continuity and information quantity. Then, the scoring of N-grams is defined as a 0-1 classification task: the positive samples of the training set come from the repeated part of N-grams and confidence term names, and the negative samples come from random sampling of N-grams; the model uses a random forest constructed by an unpruned decision tree. Finally, the N-grams and their classifier scores are output. From the functional point of view, this scheme can complete the label discovery, but it cannot complete the label structuring. From the methodological point of view, this scheme only designs features from two aspects of continuity and information quantity, but cannot reflect user interest, often resulting in candidate words unrelated to user interest, such as “like and collect”; in addition, this scheme uses an unpruned decision tree to construct a random forest to reduce the noise of the training set, but requires that the error N-grams in the negative samples are independent and identically distributed with the positive samples. However, in fact, since the positive samples come from the confidence term names, there is a difference in the field between the short video website, and this requirement cannot be met.

[0061] Traditional label system construction models require user-uploaded images in image websites and labels marked by users when uploading, and output a label system, but the output label system is a directed acyclic graph. From the functional point of view, this scheme can only construct a label system, and its label library depends on user supplementation, and cannot automatically mine a label library from a corpus. From the methodological point of view, this scheme uses artificially set rules to discover the inclusion relationship between labels combined with manual features, which is highly subjective and has low accuracy.

[0062] Traditional label system construction method one requires user search history records, and the model outputs labels through data mining, and the labels are organized into a tree structure, with parent nodes containing child nodes. This method first mines search terms from user search logs, and extracts search term labels through text. The hierarchical relationship between labels is obtained by text clustering of search terms. From the functional point of view, the label system constructed by this scheme is a tree structure, and the representation ability of semantic information between labels is limited. From the methodological point of view, this scheme cannot utilize visual information on short video platforms, and can only be applied to the construction of label systems for text platforms, thereby affecting the accuracy.

[0063] The second traditional label system construction method requires a given user historical search word set. The model outputs labels through text clustering. The labels are organized into a tree structure, and the parent node contains the child node. This method performs multi-level text clustering on the full user search words, uses search word vectors and clustering center vectors for text matching to find the corresponding labels of each node, thereby forming a label system. From the functional point of view, the label system constructed by this scheme is a tree structure, and the semantic information between labels has limited representation ability. Moreover, the source of the text corpus is all user search words, which is biased from natural Internet data. Simply constructing a label system based on search words will limit the coverage. From the method point of view, this scheme cannot use visual information on the short video platform, thereby affecting the accuracy.

[0064] Based on this, in one embodiment, as shown in Figure 1 a label construction method of a multimedia resource is provided. In this embodiment, the method is applied to a server for illustration. It can be understood that the method can also be applied to a terminal and can also be applied to a system including a terminal and a server and can be realized through the interaction of the terminal and the server. In this embodiment, the method includes the following steps:

[0065] In step S110, a multimedia resource to be constructed is obtained.

[0066] The multimedia resource includes a plurality of multimedia, and the multimedia can be a video, a short video, etc. The multimedia resource to be constructed can be a multimedia that needs to be classified and arranged, for example, a short video that needs to be classified and arranged in a live broadcast platform. Since the label can also reflect the interests and hobbies of the user, the multimedia resource to be constructed can also be a multimedia related to a user, for example, a multimedia published, browsed, collected, and liked by a user on the network. In this embodiment, based on the purpose of constructing the label, the multimedia resource in the corresponding purpose range is obtained, and the label system is constructed through the subsequent steps.

[0067] In step S120, a target label set is determined from the plurality of multimedia.

[0068] The target label set includes a plurality of target labels, and the target label is a label finally constructed in the label system from the plurality of multimedia of the multimedia resource. Specifically, the target label can be obtained by using a natural language processing method, for example, N-Gram (language model), etc. In this embodiment, a plurality of target labels are obtained by performing label mining on the plurality of multimedia, and the set of the plurality of target labels is the target label set.

[0069] In step S130, the statistical features of the label pair are obtained according to the corresponding multimedia of any label in the target label set.

[0070] wherein the tag pair is any two target tags appearing in the same multimedia. The statistical feature is a feature reflecting some characteristics associated with the tag pair, which is obtained by statistical calculation based on statistical methods. For example, but not limited to, the transfer probability of the tag pair, the multi-modal transfer probability, the information entropy, and the mutual information.

[0071] Since the target tags are mined from multiple multimedia, in this embodiment, any two target tags appearing in the same multimedia are determined as a tag pair, and then the statistical features of the tag pair are obtained by statistical analysis according to the multimedia corresponding to the tag pair.

[0072] In step S140, the relationship category of the tag pair is determined according to the statistical features of the tag pair.

[0073] wherein the relationship category of the tag pair includes any one of the parent tag of one target tag being another target tag, the child tag of one target tag being another target tag, and the two target tags being irrelevant to each other.

[0074] In this embodiment, the relationship category of the tag pair can be determined according to the statistical features of the tag pair. Specifically, the statistical features of the tag pair obtained above can be input into a pre-established label relationship model for recognition, so as to obtain the relationship category of the tag pair output by the model. The label relationship model is a machine learning model pre-established for identifying the relationship category of the tag pair. Specifically, the label relationship model is a random forest classifier established by labeling a small amount of tag pairs.

[0075] In step S150, the label system of the multimedia resource is constructed according to the relationship category of each tag pair.

[0076] Specifically, based on the above steps S130 and S140, the relationship category of each tag pair in the target tag set is obtained, and then the label system of the multimedia resource is constructed according to the relationship category of each tag pair.

[0077] In the above label construction method of the multimedia resource, the target tag set is determined from multiple multimedia by obtaining the multimedia resource to be constructed, the statistical features of the tag pair are obtained according to the multimedia corresponding to any tag pair in the target tag set, the relationship category of the tag pair is determined according to the statistical features of the tag pair, and finally the label system of the multimedia resource is constructed according to the relationship category of each tag pair. The label mining and label system construction based on the multimedia resource to be constructed are realized, so as to improve the accuracy and efficiency of defining the label.

[0078] In an example embodiment, the tag pair includes a first target tag and a second target tag. The relationship category of the tag pair includes any one of the second target tag being a parent tag of the first target tag, the second target tag being a child tag of the first target tag, and the second target tag being irrelevant to the first target tag. In step S150, a tag system of the multimedia resource is constructed according to the relationship category of each tag pair, specifically including: when the relationship category of the tag pair is the second target tag being a parent tag of the first target tag, or the second target tag being a child tag of the first target tag, a pointing relationship between one target tag and another target tag in the tag pair is generated according to the relationship category, and then a tag system of the multimedia resource is obtained according to the target tag set and the pointing relationship between the second target tag and the first target tag in each tag pair.

[0079] Specifically, as shown in Figure 2 In the tag system, all target tags in the target tag set constitute a directed graph, each node in the graph is a target tag (the target tag set is determined based on the above step S120, that is, each node in the graph is obtained), and the relationship category of the tag pair is mapped to a directed edge in the graph, that is, the relationship category of the tag pair obtained based on the above step S140 constructs the directed edge between the nodes. For example, taking the first target tag in the tag pair as "food" and the second target tag as "fruit" as an example, through relationship category identification, if the corresponding relationship category is "food" being a parent tag of "fruit", or the corresponding relationship category is "fruit" being a child tag of "food". Then a pointing relationship is generated between the nodes "food" and "fruit" based on the relationship category, that is, the child tag points to the parent tag, so that the node "fruit" in the graph points to the node "food" through a directed edge. Then all directed edges of the graph are obtained based on the relationship category of all nodes (that is, each tag pair) in the graph, so that the tag system as shown in Figure 2 is obtained.

[0080] The embodiment generates a corresponding pointing relationship based on the relationship category between the target tags in the tag pair, and then obtains a tag system of the multimedia resource according to the pointing relationship between the target tags of each tag pair in the target tag set, thereby improving the accuracy of the tag system construction.

[0081] In an example embodiment, as shown in Figure 3 In step S110, a multimedia resource to be constructed with a tag is obtained, which can be achieved through the following steps:

[0082] In step S112, a full-quantity candidate multimedia to be constructed with a tag is obtained, and the full-quantity candidate multimedia is filtered based on a preset condition to obtain a plurality of filtered multimedia.

[0083] The full quantity candidate multimedia is all multimedia in a corresponding range determined based on a purpose of constructing the tag. For example, if the purpose of constructing the tag is to determine the interest of the target user, the full quantity candidate multimedia to be constructed into the tag refers to all multimedia related to the target user, such as multimedia published, browsed, collected, and liked by the target user in the network. For another example, if the purpose of constructing the tag is to effectively classify and organize multimedia in a network platform, the full quantity candidate multimedia to be constructed into the tag includes all multimedia in the network platform.

[0084] The preset condition is a preset filtering condition, and in this embodiment, the full quantity candidate multimedia is filtered through the preset condition, so as to obtain the plurality of multimedia after removing noise. Specifically, the preset condition can be a condition for measuring the legality, integrity, and meaning of the candidate multimedia. For example, based on the preset condition, the illegal candidate multimedia, the incomplete candidate multimedia, and the meaningless candidate multimedia (such as multimedia of a cheating user, non-public multimedia, non-compliant multimedia, multimedia without a title, and multimedia with a meaningless title) are filtered from the full quantity candidate multimedia, so as to obtain the plurality of filtered multimedia.

[0085] In step S114, the multimedia resource to be constructed into the tag is obtained according to the plurality of filtered multimedia.

[0086] Specifically, the multimedia resource to be constructed into the tag is obtained according to the plurality of filtered multimedia.

[0087] In the above embodiment, the full quantity candidate multimedia to be constructed into the tag is filtered through the preset condition, so as to obtain the plurality of multimedia after removing noise, and then the tag mining is performed based on the plurality of filtered multimedia, so as to facilitate improving the accuracy of the tag.

[0088] In an exemplary embodiment, as shown in FIG. 12, in step S120, a target tag set is determined from the plurality of multimedia, which can be achieved through the following steps: Figure 4

[0089] In step S410, the text information of each multimedia is obtained.

[0090] ​The text information includes a spliced string composed of a title of the multimedia and cover text of the multimedia. Specifically, taking a short video in a live broadcast platform as an example, usually, when a user publishes a short video in the live broadcast platform, the user is required to draft a title of the short video and select or design a frame of video image as a video cover. Therefore, each short video in the live broadcast platform has a corresponding title and a video cover. The cover text refers to an identification result obtained by performing OCR (Optical Character Recognition) on the video cover.

[0091] In this embodiment, the cover text corresponding to each multimedia is obtained by performing OCR identification on the cover of each multimedia, and then the title of the multimedia and the corresponding cover text are spliced to obtain a spliced string, that is, the text information of the corresponding multimedia.

[0092] In step S420, a keyword in the text information of each multimedia is identified, and the frequency of each keyword is obtained.

[0093] The keyword refers to a byte segment obtained by identifying the text information of the multimedia. The identification of the keyword can be implemented based on N-Grams (n-gram). Specifically, the keyword, that is, the byte segment, in the text information of each multimedia is identified based on N-Grams, and the frequency of each keyword is counted.

[0094] In this embodiment, the frequency of each keyword is counted based on a plurality of multimedia in the multimedia resource. For example, for a keyword A, if the keyword A appears in 100 multimedia in the multimedia resource, the frequency of the keyword A is 100.

[0095] In step S430, a keyword satisfying a set condition is obtained as a candidate label according to each keyword and the corresponding frequency.

[0096] The candidate label is a keyword preliminarily selected from the keywords and possibly becoming a label. The set condition is a condition for screening the candidate label, which is set in advance. It can include a length condition of the keyword and a frequency condition. In this embodiment, when a keyword satisfies the corresponding length condition and the frequency condition, the keyword can be determined as a candidate label.

[0097] Specifically, for example, if the length condition is a set length and the frequency condition is a set frequency threshold, when there is a keyword whose character length meets the set length and the frequency of the keyword reaches the set frequency threshold, it can be determined that the keyword meets the set condition, and thus the keyword can be determined as a candidate label. Generally, the length condition is the character length of the keyword, for example, the character length should be less than or equal to 7, the frequency should be greater than 100, and the like. The embodiment filters high-frequency keywords as candidate labels based on the length condition and the frequency condition, thereby improving the accuracy and efficiency of label mining.

[0098] In step S440, a feature representation of each candidate label is obtained, and a target label is determined according to the feature representation, thereby obtaining a target label set.

[0099] The feature representation is a feature vector used to filter target labels from candidate labels. In the embodiment, the feature representation of each candidate label is calculated, and then the target label is determined according to the feature representation, thereby obtaining the target label set.

[0100] In the above embodiment, the text information of each multimedia is obtained, and the keywords in the text information of each multimedia are identified, the frequency of each keyword is obtained, the keywords meeting the set condition are obtained as candidate labels according to each keyword and the corresponding frequency, the feature representation of each candidate label is obtained, and the target label is determined according to the feature representation, thereby obtaining the target label set. Since the target label is mined from the text information of the multimedia through multiple filtering, the accuracy of the finally determined target label is higher.

[0101] In an exemplary embodiment, the multimedia further includes corresponding category information, and as shown in FIG. 4B, in step S440, the feature representation of each candidate label is obtained, and the target label is determined according to the feature representation, specifically including: Figure 5

[0102] In step S442, the feature representation of the candidate label is obtained according to the multimedia corresponding to each candidate label and the category information corresponding to the multimedia.

[0103] The category information refers to the category to which the multimedia belongs, for example, music, game, food, and the like. The feature representation includes vertical features, for example, the transfer feature of the candidate label to any category and the corresponding feature entropy. In the embodiment, the feature representation of each candidate label is obtained by statistical calculation according to the multimedia corresponding to each candidate label and the category information corresponding to the multimedia.

[0104] ​Specifically, in an example embodiment, the acquisition method of the transfer feature can specifically include: acquiring a first occurrence frequency of a candidate label in multimedia corresponding to any category information in the multimedia resource, and acquiring a second occurrence frequency of the candidate label in all multimedia in the multimedia resource, and determining the transfer feature of the candidate label to any category according to the first occurrence frequency and the second occurrence frequency. For example, for a candidate label Ngram, the transfer probability of the candidate label Ngram to any category K can be expressed as: Wherein, the first occurrence frequency is the occurrence frequency of the candidate label Ngram in multimedia with category K in the multimedia resource, and the second occurrence frequency is the occurrence frequency of the candidate label Ngram in all multimedia of the multimedia resource.

[0105] In an example embodiment, the acquisition method of the feature entropy can specifically include: calculating the feature entropy corresponding to the transfer feature of the candidate label to any category according to the category information corresponding to the multimedia in the multimedia resource and the transfer feature of the candidate label to any category.

[0106] Specifically, the feature entropy can be expressed as:

[0107]

[0108] Wherein, n is the category set of all multimedia in the multimedia resource, and p(category n|Ngram) is the transfer probability of the candidate label Ngram to any category in the category set n, and the summation range is all elements in the category set n.

[0109] The embodiment calculates the transfer feature of the candidate label to any category and the corresponding feature entropy, and then performs screening of the target label, thereby facilitating to improve the accuracy of the target label.

[0110] In step S444, the feature representation of the candidate label is input into the trained prediction model to obtain the feature parameters of the candidate label output by the prediction model.

[0111] Wherein, the prediction model is a model for predicting feature parameters based on the feature representation of the candidate label. Specifically, the feature representation of the candidate label is input into the trained prediction model, thereby obtaining the feature parameters of the candidate label output by the prediction model. Based on this, the feature parameters of each candidate label can be obtained.

[0112] In step S446, the candidate label whose feature parameters reach the parameter threshold is acquired as the target label.

[0113] The parameter threshold is a threshold condition for screening the candidate labels to obtain the target labels. Specifically, only the candidate labels whose feature parameters reach the parameter threshold can be the target labels. For example, if the set parameter threshold is X, when the feature parameter of a candidate label is greater than X, the candidate label can be determined as a target label.

[0114] In the above embodiment, according to the multimedia corresponding to each candidate label and the category information corresponding to the multimedia, the feature representation of the candidate label is obtained, the feature representation of the candidate label is input into the trained prediction model, the feature parameter of the candidate label output by the prediction model is obtained, and then the candidate label whose feature parameter reaches the parameter threshold is obtained as the target label, thereby improving the accuracy of screening the target labels.

[0115] In an exemplary embodiment, the base model of the prediction model adopts a random forest classifier, and as shown in FIG. 6, the method for obtaining the prediction model includes the following steps: Figure 6

[0116] In step S610, the training set is obtained.

[0117] The training set is training data for training the random forest classifier to obtain the prediction model. The training set includes the same number of label sample positives and label sample negatives, and the label sample positives are label samples selected from a preset confidence label set, and the label sample negatives are label samples selected from candidate labels. Specifically, the confidence label set is a set of labeled label sample positives.

[0118] In step S620, the sample feature representation of each label sample in the training set is obtained.

[0119] The sample feature representation includes the sample transfer feature of the label sample to any category and the corresponding sample feature entropy. Specifically, the method for obtaining the sample transfer feature and the sample feature entropy is similar to the method for obtaining the transfer feature and the corresponding feature entropy in the above embodiment, and this embodiment will not be described again.

[0120] In step S630, the random forest classifier is trained using the sample feature representation of each label sample to obtain the trained prediction model.

[0121] In this embodiment, the random forest classifier is trained using the sample feature representation of each label sample to obtain the trained prediction model.

[0122] ​Specifically, since there are some label samples that are not in the confidence label set but are actually positive examples in the negative examples of the label samples in the training set construction stage, noise suppression measures can be taken when training the model. For example, based on the method used by AutoPhrase (a new word discovery method), a random forest classifier can be trained using non-pruned decision trees, so that the errors of each decision tree can randomly reduce noise. Strategies such as MixUp (data augmentation), noisy label (noisy label), and the like can also be used to construct a training set by mixing features, thereby effectively suppressing noise. Finally, by sampling multiple training sets, the results of each are integrated by voting to form a scoring model.

[0123] In the above embodiment, by obtaining the training set, and then obtaining the sample feature representation of each label sample in the training set, i.e., the vertical class feature, and training a random forest classifier using the sample feature representation of each label sample, a trained prediction model is obtained. Since only a small number of labeled samples, i.e., label sample positive examples, are used in the model training process, and by combining the noise reduction strategy, the precision problem of model training with only a small number of samples is overcome. The prediction model can effectively improve the mining accuracy of the target label by predicting the feature parameters of the candidate label and mining the target label, and the mined target label can better express the user interest.

[0124] In an example embodiment, the statistical feature includes a transition probability of a label pair, and the label pair includes a first target label and a second target label. Then, in step S130, the statistical feature of the label pair is obtained according to the multimedia corresponding to any label pair in the target label set, specifically including: obtaining a first quantity of multimedia corresponding to the first target label and the second target label together, and obtaining a second quantity of multimedia corresponding to the first target label; and determining the transition probability of the first target label and the second target label according to the first quantity and the second quantity.

[0125] Specifically, the transition probability of the label pair can be expressed as:

[0126] Where v is the first target label and u is the second target label. The first quantity refers to the number of all multimedia in the multimedia resource that simultaneously appear the first target label and the second target label. The second quantity refers to the number of all multimedia in the multimedia resource that only appear the first target label. This embodiment identifies the relationship category of the label pair based on the transition probability of the label pair, which can effectively improve the accuracy of identification.

[0127] In an example embodiment, the statistical features further include mutual information of the label pair. Then in step S130, the statistical features of the label pair are obtained according to the multimedia corresponding to any label pair in the target label set, specifically including: obtaining a first quantity of the multimedia corresponding to the first target label and the second target label together, and obtaining a second quantity of the multimedia corresponding to the first target label and a third quantity of the multimedia corresponding to the second target label, and then calculating the mutual information between the first target label and the second target label according to the first quantity, the second quantity and the third quantity.

[0128] Specifically, the mutual information of the label pair can be expressed as:

[0129] Wherein, the first quantity refers to the quantity of all multimedia in the multimedia resource in which the first target label and the second target label appear together. The second quantity refers to the quantity of all multimedia in the multimedia resource in which only the first target label appears. Similarly, the third quantity refers to the quantity of all multimedia in the multimedia resource in which only the second target label appears. The embodiment based on the mutual information of the label pair identifies the relationship category of the label pair, which can effectively improve the accuracy of identification.

[0130] In an example embodiment, the statistical features further include the information entropy corresponding to the label pair. Then in step S130, the statistical features of the label pair are obtained according to the multimedia corresponding to any label pair in the target label set, specifically including: obtaining a first label set matched with the second target label, wherein the first label set includes a plurality of first target labels, and the first target label and the second target label have at least one common corresponding multimedia. Then the transition probability of each first target label and the second target label is obtained, and the information entropy corresponding to the first label set matched with the second target label is calculated according to the first label set and the transition probability of each first target label and the second target label in the first label set.

[0131] Specifically, the information entropy corresponding to the label pair can be expressed as:

[0132] H(u) = -∑ wp(u|w)log(p(u|w)), where u is the second target label, and w is the first label set, i.e., the set of first target labels that co-occur with the second target label u in the same multimedia. For example, if the multimedia resource includes multimedia A, B and C, where multimedia A includes target labels a1, a2 and u, multimedia B includes target labels B1 and B2, and multimedia C includes target labels c1 and u, the first target labels that co-occur with the second target label in the same multimedia include a1, a2 and c1, and the first target labels a1, a2 and c1 form the first label set. p(u|w) is the transition probability corresponding to any element in the set w. The sum is taken over all elements in the set w. This embodiment identifies the relationship category of the label pair based on the information entropy of the label pair, which can effectively improve the accuracy of identification.

[0133] In an example embodiment, the statistical features further include a multi-modal transition probability corresponding to the label pair. Then, as shown in step S130, the statistical features of the label pair are obtained according to the multimedia corresponding to any label pair in the target label set, specifically including: Figure 7

[0134] In step S132, the visual information and the text information of each multimedia in the multimedia resource are obtained, and the visual information and the text information of the multimedia are multi-modal encoded to obtain the multi-modal features corresponding to the multimedia.

[0135] The multi-modal features are a vectorized representation of the visual information and the text information of the multimedia. Specifically, the text information includes a spliced string composed of the title of the multimedia and the cover text of the multimedia. The visual information is composed of the extracted frames from the multimedia. For example, taking a short video as the multimedia, the visual information can be the frame in the short video, which can be a plurality of frames uniformly extracted from the short video.

[0136] In this embodiment, as shown in Figure 8 The text encoder can be used to encode the text information of the multimedia to obtain the corresponding first encoding, the visual encoder can be used to encode the visual information of the multimedia to obtain the corresponding second encoding, and the multi-modal joint encoder can be used to fuse the first encoding and the second encoding to obtain the multi-modal features corresponding to the multimedia.

[0137] Specifically, a multi-modal pre-training model (such as ActionBERT, ClipBERT, etc.) can be used to obtain the multi-modal features corresponding to the multimedia. The visual information and the text information of the multimedia are input into the multi-modal pre-training model to obtain the multi-modal features of the multimedia output by the model.

[0138] ​In step S134, a distance between the first target label and the second target label is obtained according to the multimedia and the multi-modal features of the multimedia corresponding to the first target label and the second target label respectively.

[0139] Specifically, the distance calculation can adopt the following formula:

[0140]

[0141] wherein, Specifically, σ is a constant, v is the first target label, and u is the second target label. denotes a set of all multimedia in the multimedia resource in which only the second target label u appears, denotes a set of all multimedia in the multimedia resource in which only the first target label v appears, and f(q) denotes a multi-modal feature corresponding to an element (i.e. a multimedia in which only the first target label v appears) in the set f(q), and f(p) denotes a multi-modal feature corresponding to an element (i.e. a multimedia in which only the second target label u appears) in the set f(p). The summation range is all elements in the set f(q) and the set f(p).

[0142] In step S136, the distance is normalized and averaged to obtain a probability mass score of the first target label and the second target label.

[0143] wherein the normalized and averaged processing can adopt the following formula: m(u, v) = MeanPool(Sigmoid(d(u, v))). Specifically, d(u, v) represents the distance between the first target label v and the second target label u. Sigmoid is a normalization function, and MeanPool is an average function.

[0144] In step S138, a multi-modal transition probability corresponding to a label pair is calculated according to the probability mass score.

[0145] Specifically, the multi-modal transition probability corresponding to a label pair can be calculated by the following formula:

[0146]

[0147] wherein, if v is the first target label and u is the second target label, then z is a second label set, i.e. a set of all second target labels that appear together with the first target label v in the same multimedia, which has a similar meaning to the first label set w described above, and will not be described again here.

[0148] ​The embodiment is based on the multi-modal transition probability corresponding to the label pair to identify the relationship category of the label pair. Since the visual information is considered when determining the multi-modal transition probability, the characteristics of the multimedia can be comprehensively reflected, and the accuracy of identifying the target label can be effectively improved.

[0149] It should be understood that, although Figures 1-8 The steps in the flowchart of the method can be displayed in sequence according to the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, Figures 1-8 At least part of the steps in the method can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be alternately executed with other steps or steps or stages in other steps.

[0150] It can be understood that the same / similar parts of each embodiment of the method in the specification can be mutually referred to, and each embodiment focuses on the differences from other embodiments, and the related parts can be referred to the description of other method embodiments.

[0151] Figure 9 is a block diagram of a label construction device of a multimedia resource according to an exemplary embodiment. Referring to Figure 9 The device includes a multimedia resource acquisition module 902, a target label set determination module 904, a statistical feature acquisition module 906, a relationship category identification module 908, and a label system construction module 910.

[0152] The multimedia resource acquisition module 902 is configured to acquire a multimedia resource to be constructed, and the multimedia resource includes a plurality of multimedia;

[0153] The target label set determination module 904 is configured to determine a target label set from the plurality of multimedia, and the target label set includes a plurality of target labels;

[0154] The statistical feature acquisition module 906 is configured to acquire statistical features of a label pair from the multimedia corresponding to any label pair in the target label set, and the label pair is any two target labels appearing in the same multimedia;

[0155] The relationship category identification module 908 is configured to determine the relationship category of the label pair according to the statistical features of the label pair;

[0156] The tag system construction module 910 is configured to construct a tag system of the multimedia resource according to a relationship category of each tag pair.

[0157] In an example embodiment, the target tag set determination module specifically comprises: a text information acquisition unit configured to acquire text information of each multimedia, the text information comprising a spliced string of a title of the multimedia and cover text of the multimedia; a keyword information identification unit configured to identify keywords in the text information of each multimedia and acquire a frequency of each keyword; a candidate tag determination unit configured to acquire a keyword satisfying a set condition as a candidate tag according to each keyword and the corresponding frequency; and a target tag set acquisition unit configured to acquire a feature representation of each candidate tag, determine a target tag according to the feature representation, and obtain the target tag set.

[0158] In an example embodiment, the multimedia further comprises corresponding category information; and the target tag set acquisition unit further comprises: a feature representation acquisition subunit configured to acquire the feature representation of the candidate tag according to the multimedia corresponding to each candidate tag and the category information corresponding to the multimedia, the feature representation comprising a transition feature of the candidate tag to any category and a corresponding feature entropy; a feature parameter acquisition subunit configured to input the feature representation of the candidate tag into a trained prediction model to obtain a feature parameter of the candidate tag output by the prediction model; and a target tag determination subunit configured to acquire a candidate tag whose feature parameter reaches a parameter threshold as a target tag.

[0159] In an example embodiment, the feature representation acquisition subunit is further configured to: acquire a first occurrence frequency of the candidate tag in the multimedia corresponding to any category information in the multimedia resource, and acquire a second occurrence frequency of the candidate tag in all multimedia in the multimedia resource; and determine the transition feature of the candidate tag to any category according to the first occurrence frequency and the second occurrence frequency.

[0160] In an example embodiment, the feature representation acquisition subunit is further configured to: calculate the feature entropy corresponding to the transition feature of the candidate tag to any category according to the category information corresponding to the multimedia in the multimedia resource and the transition feature of the candidate tag to any category.

[0161] In an example embodiment, the target label set obtaining unit further comprises a prediction model obtaining subunit configured to perform: obtaining a training set, the training set comprising a same number of label sample positives and label sample negatives, the label sample positives being label samples selected from a preset confidence label set, and the label sample negatives being label samples selected from the candidate labels; obtaining a sample feature representation of each label sample in the training set, the sample feature representation comprising a sample transition feature of the label sample to any category and a corresponding sample feature entropy; and training the random forest classifier using the sample feature representation of each label sample to obtain a trained prediction model.

[0162] In an example embodiment, the candidate label determining unit is configured to perform: when there is a keyword with a character length satisfying a set length, and a frequency of the keyword reaches a set frequency threshold, determining that the keyword satisfies a set condition; and determining the keyword satisfying the set condition as a candidate label.

[0163] In an example embodiment, the statistical feature comprises a transition probability of the label pair; the label pair comprises a first target label and a second target label; and the statistical feature obtaining module is configured to perform: obtaining a first number of multimedia corresponding to both the first target label and the second target label, and obtaining a second number of multimedia corresponding to the first target label; and determining the transition probability of the first target label and the second target label according to the first number and the second number.

[0164] In an example embodiment, the statistical feature further comprises mutual information of the label pair; and the statistical feature obtaining module is further configured to perform: obtaining a first number of multimedia corresponding to both the first target label and the second target label, and obtaining a second number of multimedia corresponding to the first target label and a third number of multimedia corresponding to the second target label; and calculating mutual information between the first target label and the second target label according to the first number, the second number, and the third number.

[0165] In an example embodiment, the statistical feature further comprises information entropy corresponding to the label pair; and the statistical feature obtaining module is further configured to perform: obtaining a first label set matching the second target label, the first label set comprising a plurality of first target labels, and there being at least one multimedia corresponding to both the first target label and the second target label; obtaining a transition probability of each first target label to the second target label; and calculating information entropy corresponding to the first label set matching the second target label according to the first label set and the transition probability of each first target label to the second target label.

[0166] In an example embodiment, the statistical features further include a multi-modal transition probability corresponding to the label pair; the statistical feature acquisition module is further configured to perform: acquiring visual information and text information of each multimedia in the multimedia resource, performing multi-modal encoding on the visual information and the text information of the multimedia to obtain a multi-modal feature corresponding to the multimedia; acquiring a distance between the first target label and the second target label according to the multimedia corresponding to the first target label and the second target label respectively and the multi-modal feature of the multimedia; performing normalized average processing on the distance to obtain a probability mass score of the first target label and the second target label; and calculating the multi-modal transition probability corresponding to the label pair according to the probability mass score.

[0167] In an example embodiment, the relationship category of the label pair includes any one of the second target label being a parent label of the first target label, the second target label being a child label of the first target label, and the second target label being irrelevant to the first target label; and the label system construction module is further configured to perform: when the relationship category of the label pair is the second target label being a parent label of the first target label or the second target label being a child label of the first target label, generating a pointing relationship between the second target label and the first target label in the label pair according to the relationship category; and obtaining a label system of the multimedia resource according to the target label set and the pointing relationship between the second target label and the first target label in each of the label pairs.

[0168] In an example embodiment, the multimedia resource acquisition module is further configured to perform: acquiring full-candidate multimedia of a label to be constructed, filtering the full-candidate multimedia based on a preset condition to obtain a plurality of filtered multimedia; and obtaining a multimedia resource of the label to be constructed according to the plurality of filtered multimedia.

[0169] As to the apparatus in the above embodiments, the specific manners in which various modules perform operations have been described in detail in the embodiments of the method, and will not be described here in detail.

[0170] Figure 10 FIG. 8 is a block diagram illustrating an electronic device S00 for label construction of a multimedia resource according to an example embodiment. For example, the electronic device S00 can be a server. Refer to FIG. 1 for the components of the electronic device S00. The electronic device S00 can include a multimedia resource acquisition module S01, a label pair generation module S02, a statistical feature acquisition module S03, and a label system construction module S04. Figure 10The electronic device S00 includes a processing component S20, which further includes one or more processors, and a memory resource represented by a memory S22 for storing instructions, such as application programs, executable by the processing component S20. The application programs stored in the memory S22 can include one or more than one module each corresponding to a set of instructions. In addition, the processing component S20 is configured to execute the instructions to perform the above method.

[0171] The electronic device S00 can further include a power component S24 configured to perform power management of the electronic device S00, a wired or wireless network interface S26 configured to connect the electronic device S00 to a network, and an input and output (I / O) interface S28. The electronic device S00 can operate based on an operating system stored in the memory S22, such as Windows Server, Mac OSX, Unix, Linux, FreeBSD, or the like.

[0172] In an exemplary embodiment, a computer readable storage medium including instructions, such as the memory S22 including instructions, is also provided, which can be executed by the processor of the electronic device S00 to complete the above method. The storage medium can be a computer readable storage medium, such as a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0173] In an exemplary embodiment, a computer program product including instructions, which can be executed by the processor of the electronic device S00 to complete the above method, is also provided.

[0174] It should be noted that the above apparatus, electronic device, computer readable storage medium, computer program product, etc. according to the description of the method embodiment can also include other implementations, and the specific implementation can refer to the description of the related method embodiment, which will not be repeated here.

[0175] Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the aspects of the present disclosure disclosed herein. It is intended that the present disclosure cover any and all variations of the present disclosure including those variations that can be incorporated into other aspects of the present disclosure. It is intended that the present disclosure include modifications and variations of the present disclosure that are obvious to those skilled in the art from the foregoing description. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0176] It should be understood that the present disclosure is not limited to the precise structures herein described and illustrated in the drawings and that various modifications and changes can be made therein without departing from the scope thereof. The scope of the present disclosure is indicated by the appended claims rather than by the description and examples.

Claims

1. A method for constructing a label of a multimedia resource, characterized in that, The method comprises: acquiring a multimedia resource to be constructed with a label, the multimedia resource comprising a plurality of multimedia to be classified and arranged; determining a target label set from the plurality of multimedia, the target label set comprising a plurality of target labels; acquiring a statistical feature of a label pair corresponding to the plurality of multimedia according to any label in the target label set, the label pair being any two target labels appearing in the same multimedia; the statistical feature comprising at least one of a transition probability, a multi-modal transition probability, an information entropy and a mutual information of the label pair; the multi-modal transition probability comprises: acquiring visual information and text information of each multimedia in the multimedia resource and performing multi-modal encoding to obtain a multi-modal feature, calculating a distance and a probability quality score between the label pair based on the multi-modal feature to obtain the multi-modal transition probability; determining a relationship category of the label pair according to the statistical feature of the label pair, the relationship category of the label pair comprising any one of the label pair in which one target label is a parent label of another target label, one target label is a child label of another target label, and two target labels are irrelevant to each other; constructing a label system of the multimedia resource according to the relationship category of each label pair, the label system being used to represent a directed graph of a plurality of target labels in the target label set, each node in the graph representing a target label, and a directed edge in the graph representing a pointing relationship corresponding to the label pair; the method of determining the target label set from the plurality of multimedia comprises: acquiring text information of each multimedia, the text information comprising a spliced string composed of a title of the multimedia and cover text of the multimedia; identifying keywords in the text information of each multimedia and acquiring a frequency of each keyword; acquiring a keyword satisfying a set condition as a candidate label according to each keyword and the corresponding frequency; acquiring a feature representation of each candidate label, determining a target label according to the feature representation, and obtaining the target label set, the feature representation comprising a transition feature of the candidate label to any category in the category information corresponding to the multimedia and a corresponding feature entropy.

2. The method of claim 1, wherein, The multimedia further comprises corresponding category information; the method of acquiring a feature representation of each candidate label and determining a target label according to the feature representation comprises: acquiring a feature representation of the candidate label according to the multimedia corresponding to each candidate label and the category information corresponding to the multimedia, the feature representation comprising a transition feature of the candidate label to any category in the category information and a corresponding feature entropy; inputting the feature representation of the candidate label into a trained prediction model to obtain a feature parameter of the candidate label output by the prediction model; acquiring a candidate label whose feature parameter reaches a parameter threshold as a target label.

3. The method of claim 2, wherein, The method of acquiring the transition feature comprises: acquiring a first occurrence frequency of the candidate label in the multimedia corresponding to any category information in the category information corresponding to the multimedia resource, and acquiring a second occurrence frequency of the candidate label in all multimedia in the multimedia resource; The transition feature of the candidate label to any category is determined according to the first occurrence frequency and the second occurrence frequency.

4. The method of claim 3, wherein, The feature entropy acquisition method comprises: According to the category information corresponding to the multimedia in the multimedia resource and the transition feature of the candidate label to any category, the feature entropy corresponding to the transition feature of the candidate label to any category is calculated.

5. The method of claim 2, wherein, The base model of the prediction model adopts a random forest classifier; and the acquisition method of the prediction model comprises: A training set is acquired, the training set comprising the same number of label sample positives and label sample negatives, the label sample positives being label samples selected from a preset confidence label set, and the label sample negatives being label samples selected from the candidate labels; Sample feature representations of each label sample in the training set are acquired, the sample feature representations comprising sample transition features of the label samples to any category and corresponding sample feature entropies; The random forest classifier is trained by using the sample feature representations of each label sample, to obtain a trained prediction model.

6. The method of claim 1, wherein, The acquisition of the keyword satisfying the set condition as the candidate label according to each keyword and the corresponding frequency comprises: When there is a keyword with a character length satisfying a set length, and the frequency of the keyword reaches a set frequency threshold, it is determined that the keyword satisfies the set condition; The keyword satisfying the set condition is determined as the candidate label.

7. The method of claim 1, wherein, The statistical feature comprises a transition probability of the label pair; the label pair comprises a first target label and a second target label; and the acquisition of the statistical feature of the label pair according to the multimedia corresponding to any label in the target label comprises: A first number of multimedia corresponding to the first target label and the second target label is acquired, and a second number of multimedia corresponding to the first target label is acquired; The transition probability of the first target label and the second target label is determined according to the first number and the second number.

8. The method of claim 7, wherein, The statistical feature comprises mutual information of the label pair; and the acquisition of the statistical feature of the label pair according to the multimedia corresponding to any label in the target label comprises: A first number of multimedia corresponding to the first target label and the second target label is acquired, and a second number of multimedia corresponding to the first target label and a third number of multimedia corresponding to the second target label are acquired; The mutual information between the first target label and the second target label is calculated according to the first number, the second number and the third number.

9. The method of claim 7, wherein, The statistical feature comprises information entropy corresponding to the label pair; and the acquisition of the statistical feature of the label pair according to the multimedia corresponding to any label in the target label comprises: A first label set matching the second target label is acquired, the first label set comprising a plurality of first target labels, and the first target label and the second target label having at least one common corresponding multimedia; The transition probability of each first target label and the second target label is acquired; and The information entropy of the first target label and the second target label is calculated according to the transition probability of each first target label and the second target label. According to the first label set and a transition probability of each first target label in the first label set and the second target label, information entropy corresponding to the first label set matched with the second target label is calculated.

10. The method of claim 1, wherein, The label pair includes a first target label and a second target label; the distance and the probability mass score between the label pair are calculated based on the multi-modal feature to obtain a multi-modal transition probability, including: According to the multi-modal feature of the multimedia corresponding to the first target label and the second target label respectively, a distance between the first target label and the second target label is obtained; The distance is normalized and averaged to obtain a probability mass score of the first target label and the second target label; According to the probability mass score, a multi-modal transition probability corresponding to the label pair is calculated.

11. The method of claim 7, wherein, The relationship category of the label pair includes any one of the second target label being a parent label of the first target label, the second target label being a child label of the first target label, and the second target label being irrelevant to the first target label; The label system of the multimedia resource is constructed according to the relationship category of each label pair, including: When the relationship category of the label pair is that the second target label is a parent label of the first target label or the second target label is a child label of the first target label, a pointing relationship between the second target label and the first target label in the label pair is generated according to the relationship category; According to the target label set and the pointing relationship between the second target label and the first target label in each label pair, a label system of the multimedia resource is obtained.

12. The method according to any one of claims 1 to 11, characterized in that, The multimedia resource of the label to be constructed includes: Full-candidate multimedia of the label to be constructed is obtained, and the full-candidate multimedia is filtered based on a preset condition to obtain a plurality of filtered multimedia; According to the plurality of filtered multimedia, the multimedia resource of the label to be constructed is obtained.

13. A tag-building device for multimedia resources, characterized in that, It includes: A multimedia resource acquisition module is configured to execute the acquisition of the multimedia resource of the label to be constructed, and the multimedia resource includes a plurality of multimedia to be classified and arranged; A target label set determination module is configured to execute the determination of a target label set from a plurality of multimedia, and the target label set includes a plurality of target labels; A statistical feature acquisition module is configured to execute the acquisition of a statistical feature of a label pair corresponding to any label in the target label set, and the label pair is any two target labels appearing in the same multimedia; the statistical feature includes at least one of a transition probability, a multi-modal transition probability, an information entropy, and a mutual information of the label pair; the statistical feature acquisition module is further configured to execute: visual information and text information of each multimedia in the multimedia resource are obtained and multi-modal encoded to obtain a multi-modal feature, a distance and a probability mass score between the label pair are calculated based on the multi-modal feature to obtain a multi-modal transition probability; The relationship category recognition module is configured to determine a relationship category of the label pair according to statistical features of the label pair, the relationship category of the label pair including any one of a parent label of one target label being another target label, a child label of one target label being another target label, and two target labels being irrelevant to each other; The label system construction module is configured to construct a label system of the multimedia resource according to the relationship category of each label pair, the label system being used to represent a directed graph of multiple target labels in the target label set, each node in the graph representing a target label, and a directed edge in the graph representing a pointing relationship corresponding to the label pair; The target label set determination module specifically includes: The text information acquisition unit is configured to acquire text information of each multimedia, the text information including a spliced string composed of a title of the multimedia and cover text of the multimedia; The keyword information recognition unit is configured to recognize a keyword in the text information of each multimedia and acquire a frequency of each keyword; The candidate label determination unit is configured to acquire a keyword satisfying a set condition as a candidate label according to each keyword and the corresponding frequency; The target label set acquisition unit is configured to acquire a feature representation of each candidate label, determine a target label according to the feature representation, and obtain the target label set, the feature representation including a transition feature of the candidate label to any category in category information corresponding to the multimedia and a corresponding feature entropy.

14. The apparatus of claim 13, wherein, The multimedia further includes corresponding category information; The target label set acquisition unit further includes: The feature representation acquisition subunit is configured to acquire the feature representation of the candidate label according to the multimedia corresponding to each candidate label and the category information corresponding to the multimedia, the feature representation including the transition feature of the candidate label to any category in the category information and the corresponding feature entropy; The feature parameter acquisition subunit is configured to input the feature representation of the candidate label into a trained prediction model to obtain a feature parameter of the candidate label output by the prediction model; The target label determination subunit is configured to acquire the candidate label whose feature parameter reaches a parameter threshold as a target label.

15. The apparatus of claim 14, wherein, The feature representation acquisition subunit is further configured to: acquire a first occurrence frequency of the candidate label in the multimedia corresponding to any category information in the category information of the multimedia resource, and acquire a second occurrence frequency of the candidate label in all multimedia in the multimedia resource; determine the transition feature of the candidate label to any category according to the first occurrence frequency and the second occurrence frequency.

16. The apparatus of claim 15, wherein, The feature representation acquisition subunit is further configured to: calculate the feature entropy corresponding to the transition feature of the candidate label to any category according to the category information corresponding to the multimedia in the multimedia resource and the transition feature of the candidate label to any category.

17. The apparatus of claim 14, wherein, The base model of the prediction model adopts a random forest classifier; the target label set acquisition unit further comprises a prediction model acquisition subunit configured to perform: acquiring a training set, the training set comprising the same number of label sample positives and label sample negatives, the label sample positives being label samples selected from a preset confidence label set, and the label sample negatives being label samples selected from the candidate labels; acquiring a sample feature representation of each label sample in the training set, the sample feature representation comprising a sample transition feature of the label sample to any category and a corresponding sample feature entropy; training the random forest classifier using the sample feature representation of each label sample to obtain a trained prediction model.

18. The apparatus of claim 13, wherein, The candidate label determination unit is configured to perform: when there is a keyword with a character length satisfying a set length, and the frequency of the keyword reaches a set frequency threshold, determining that the keyword satisfies a set condition; determining the keyword satisfying the set condition as a candidate label.

19. The apparatus of claim 13, wherein, The statistical feature comprises a transition probability of the label pair; the label pair comprises a first target label and a second target label; and the statistical feature acquisition module is configured to perform: acquiring a first quantity of multimedia corresponding to both the first target label and the second target label, and acquiring a second quantity of multimedia corresponding to the first target label; determining the transition probability of the first target label and the second target label according to the first quantity and the second quantity.

20. The apparatus of claim 19, wherein, The statistical feature further comprises mutual information of the label pair; and the statistical feature acquisition module is further configured to perform: acquiring a first quantity of multimedia corresponding to both the first target label and the second target label, and acquiring a second quantity of multimedia corresponding to the first target label and a third quantity of multimedia corresponding to the second target label; calculating mutual information between the first target label and the second target label according to the first quantity, the second quantity, and the third quantity.

21. The apparatus of claim 19, wherein, The statistical feature further comprises information entropy corresponding to the label pair; and the statistical feature acquisition module is further configured to perform: acquiring a first label set matching the second target label, the first label set comprising a plurality of first target labels, and the first target label and the second target label having at least one common corresponding multimedia; acquiring a transition probability of each first target label and the second target label; calculating information entropy corresponding to the first label set matching the second target label according to the first label set and the transition probability of each first target label and the second target label in the first label set.

22. The apparatus of claim 13, wherein, The label pair comprises a first target label and a second target label; and the statistical feature acquisition module is further configured to perform: acquiring a distance between the first target label and the second target label according to multimedia respectively corresponding to the first target label and the second target label and multi-modal features of the multimedia; performing normalized average processing on the distance to obtain a probability mass score of the first target label and the second target label; and According to the probability mass score, a multi-modal transition probability corresponding to the label pair is calculated.

23. The apparatus of claim 19, wherein, The relationship category of the label pair includes any one of the second target label being a parent label of the first target label, the second target label being a child label of the first target label, and the second target label being irrelevant to the first target label; and the label system construction module is further configured to perform: When the relationship category of the label pair is the second target label being a parent label of the first target label or the second target label being a child label of the first target label, a pointing relationship between the second target label and the first target label in the label pair is generated according to the relationship category. According to the target label set and the pointing relationship between the second target label and the first target label in each label pair, a label system of the multimedia resource is obtained.

24. The apparatus of any one of claims 13 to 23, wherein, The multimedia resource acquisition module is further configured to perform: Obtaining full-candidate multimedia of a label to be constructed, filtering the full-candidate multimedia based on a preset condition to obtain a plurality of filtered multimedia; According to the plurality of filtered multimedia, obtaining a multimedia resource of the label to be constructed.

25. An electronic device, comprising: Comprise: A processor; A memory for storing instructions executable by the processor; Wherein the processor is configured to execute the instructions to implement the label construction method of the multimedia resource according to any one of claims 1 to 12.

26. A computer-readable storage medium, characterized in that, When the instructions in the computer readable storage medium are executed by the processor of the electronic device, the electronic device can execute the label construction method of the multimedia resource according to any one of claims 1 to 12.

27. A computer program product, comprising instructions therein, wherein When the instructions are executed by the processor of the electronic device, the electronic device can execute the label construction method of the multimedia resource according to any one of claims 1 to 12.

Images