Label classification method and device for multimedia file, equipment and medium

Abstract

The invention discloses a label classification method and device for multimedia files, equipment, a medium and a program product, and belongs to the field of multimedia classification. The method comprises the following steps: acquiring description information of a multimedia file, and acquiring a contradictory label pair of the multimedia file; calling a first large language model based on the description information, the contradiction tag pair and the first prompt word, and predicting to obtain a correct multimedia tag of the multimedia file; and outputting a multimedia output tag corresponding to the multimedia file based on the correct multimedia tag. According to the method, the correct multimedia labels in the contradictory label pairs of the multimedia files are obtained based on the description information by calling the first large language model, and finally the multimedia output labels corresponding to the multimedia files are output, so that the problem that semantic conflicts exist between the multimedia labels corresponding to the same multimedia file is reduced, and the efficiency of the multimedia files is improved. And the accuracy of the finally output multimedia output label is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of multimedia classification, and in particular to a multimedia tag classification method, device, equipment, medium and program product. Background Art

[0002] With the development of terminal devices, more and more users listen to music through music applications on terminal devices. In order to recommend music that users may be interested in, it is necessary to add music tags to the music and recommend corresponding music based on the music tags.

[0003] In the related art, by collecting music-related comments, the comment music tags corresponding to the music are obtained from the comments. Compared with the reference music tags added by the music provider, the comment music tags are usually closer to the user's real opinion.

[0004] However, due to different opinions among different users, a piece of music may include incorrect comment music tags. For example, comments corresponding to music expressing sad emotions may also include comments expressing happy emotions, thereby adding comment music tags corresponding to happy emotions to the music. However, incorrect comment music tags will affect the accuracy of music recommendations. Summary of the invention

[0005] The present application provides a method, device, equipment, medium and program product for label classification of multimedia files, and the technical solution includes at least one of the following aspects.

[0006] According to one aspect of an embodiment of the present application, a method for label classification of multimedia files is provided, the method comprising:

[0007] Obtaining description information of a multimedia file and obtaining a contradictory tag pair of the multimedia file, wherein the description information is used to describe the content of the multimedia file, and the contradictory tag pair includes two multimedia tags with opposite semantics;

[0008] Based on the description information, the contradictory label pair and the first prompt word, a first language model is called to predict a correct multimedia label of the multimedia file, wherein the first prompt word is used to indicate the correct multimedia label in the contradictory label pair determined based on the description information;

[0009] Based on the correct multimedia tag, the multimedia output tag corresponding to the multimedia file is output.

[0010] According to another aspect of an embodiment of the present application, a device for label classification of multimedia files is provided, the device comprising:

[0011] An acquisition module, used to acquire description information of a multimedia file and to acquire a contradictory tag pair of the multimedia file, wherein the description information is used to describe the content of the multimedia file, and the contradictory tag pair includes two multimedia tags with opposite semantics;

[0012] A prediction module, configured to call a first language model based on the description information, the contradictory label pair and the first prompt word to predict a correct multimedia label of the multimedia file, wherein the first prompt word is used to indicate a correct multimedia label in the contradictory label pair determined based on the description information;

[0013] The output module is used to output the multimedia output tag corresponding to the multimedia file based on the correct multimedia tag.

[0014] According to another aspect of an embodiment of the present application, a computer device is provided, comprising: a processor and a memory, wherein at least one program is stored in the memory; the processor is used to execute at least one program in the memory to implement the above-mentioned tag classification method for multimedia files.

[0015] According to another aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which at least one program is stored. The at least one program is loaded and executed by a processor to implement the above-mentioned tag classification method for multimedia files.

[0016] According to another aspect of an embodiment of the present application, a computer program product or a computer program is provided, which includes computer instructions, the computer instructions are stored in a computer-readable storage medium, a processor obtains the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to implement the above-mentioned multimedia file tag classification method.

[0017] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0018] The method obtains the correct multimedia tag in the contradictory tag pair of the multimedia file based on the description information by calling the first language model, and finally outputs the multimedia output tag corresponding to the multimedia file, thereby reducing the problem of semantic conflict between multimedia tags corresponding to the same multimedia file and improving the accuracy of the finally output multimedia output tag. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 A schematic diagram of a computer system provided by an exemplary embodiment of the present application is shown;

[0021] Figure 2 A schematic diagram showing a method for label classification of multimedia files provided by an exemplary embodiment of the present application is shown;

[0022] Figure 3 A flowchart of a method for label classification of multimedia files provided by an exemplary embodiment of the present application is shown;

[0023] Figure 4 A flowchart of a method for label classification of multimedia files provided by an exemplary embodiment of the present application is shown;

[0024] Figure 5 A schematic diagram showing an audio processing process provided by an exemplary embodiment of the present application is shown;

[0025] Figure 6 A schematic diagram of a feature mapping network provided by an exemplary embodiment of the present application is shown;

[0026] Figure 7 A flowchart of a method for label classification of multimedia files provided by an exemplary embodiment of the present application is shown;

[0027] Figure 8 A flowchart of a method for label classification of multimedia files provided by an exemplary embodiment of the present application is shown;

[0028] Fig. 9 A flowchart of a method for label classification of multimedia files provided by an exemplary embodiment of the present application is shown;

[0029] Fig.10 A flowchart of a method for label classification of multimedia files provided by an exemplary embodiment of the present application is shown;

[0030] Fig.11 A flowchart of obtaining high-quality candidate reviews provided by an exemplary embodiment of the present application is shown;

[0031] Fig.12 A block diagram of a multimedia file tag classification device provided by an exemplary embodiment of the present application is shown;

[0032] Fig.13 A structural block diagram of a computer device provided by an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0033] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0034] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0035] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0036] It should be noted that the object information (including but not limited to object device information, object personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the object or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0037] It should be understood that, although the terms first, second, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first parameter may also be referred to as the second parameter, and similarly, the second parameter may also be referred to as the first parameter. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0038] First, some of the terms used in this application are explained:

[0039] Pre-Training Model (PTM): also known as cornerstone model or big model, refers to a deep neural network (DNN) with large parameters that is trained on massive unlabeled data. The function approximation capability of large-parameter DNN is used to enable PTM to extract common features from the data. After fine-tuning and efficient parameter fine-tuning, it is suitable for downstream tasks. Therefore, the pre-training model can achieve ideal results in small sample or zero sample scenarios. PTM can be divided into language model, visual model, speech model, multimodal model, etc. according to the data modality processed. The multimodal model refers to a model that establishes feature representations of two or more data modalities. The pre-training model is an important tool for outputting artificial intelligence generated content, and can also be used as a general interface to connect multiple specific task models.

[0040] Natural Language Processing (NLP): It is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can enable effective communication between people and computers using natural language. Natural language processing involves natural language, that is, the language people use in daily life. It is closely related to linguistics research and also involves computer science and mathematics. The pre-training model, an important technology for model training in the field of artificial intelligence, is developed from the large language model (LLM) in the field of NLP. After fine-tuning, the large language model can be widely used in downstream tasks. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0041] Multimodal data: refers to data of multiple different media types or modalities, each modality represents a type of data, such as text modal data, image modal data, audio modal data, video modal data, etc. Modal data refers to data of a certain modality.

[0042] Figure 1 A schematic diagram of a computer system 10 provided by an exemplary embodiment of the present application is shown. The computer system 10 may include: a terminal 100 and a server 200 .

[0043] The terminal 100 may be an electronic device such as a mobile phone, a tablet computer, a vehicle-mounted terminal (vehicle computer), a wearable device, a personal computer (PC), etc. The terminal 100 may be installed with a client that runs a target application, and the target application may be an application that supports tag classification of multimedia files, which is not limited in this application. In addition, this application does not limit the form of the target application, including but not limited to an application (Application, App), a mini-program, a client, etc. installed in the terminal 100, and may also be in the form of a web page.

[0044] The server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services, cloud database, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and cloud servers providing basic cloud computing services such as big data and artificial palm image recognition platforms. The server 200 may be a backend server of the above-mentioned target application, used to provide backend services for the client of the target application.

[0045] Among them, cloud technology refers to a hosting technology that unifies hardware, software, network and other resources in a wide area network or local area network to realize data calculation, storage, processing and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool, which is used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of technical network systems require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the high development and application of the Internet industry, in the future, each item may have its own identification mark, which needs to be transmitted to the background system for logical processing. Data of different levels will be processed separately. All kinds of industry data require strong system backing support, which can only be achieved through cloud computing.

[0046] In some embodiments, the above-mentioned server can also be implemented as a node in a blockchain system. Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block.

[0047] The terminal 100 and the server 200 may communicate with each other via a network, such as a wired network or a wireless network.

[0048] In the method for labeling and classifying multimedia files provided in the embodiment of the present application, the execution subject of each step may be a computer device, which refers to an electronic device with data calculation, processing and storage capabilities. Figure 1 Taking the computer system 10 shown as an example, the multimedia file tag classification method can be executed by the terminal 100, the multimedia file tag classification method can also be executed by the server 200, or the terminal 100 and the server 200 can interact and cooperate to execute it, which is not limited in the present application.

[0049] With the development of terminal devices, more and more users listen to music through music applications on terminal devices. In order to recommend music that users may be interested in, it is necessary to add music tags to the music and recommend the corresponding music based on the music tags. In the related art, by collecting music-related comments, the comment music tags corresponding to the music are obtained from the comments. Compared with the reference music tags added by the music provider, the comment music tags are usually closer to the user's real opinion. However, due to different opinions of different users, a piece of music may include an incorrect comment music tag. For example, in the comments corresponding to the music expressing sad emotions, there may also be comments expressing happy emotions, thereby adding the comment music tags corresponding to happy emotions to the music, but the incorrect comment music tags will affect the accuracy of music recommendations.

[0050] In order to solve the above problems, the present application proposes a label classification method for multimedia files. Figure 2 A schematic diagram of a method for label classification of multimedia files provided by an exemplary embodiment of the present application is shown.

[0051] A multimedia file refers to a file containing data of multiple media types or modalities. A multimedia file includes at least one of the following modalities: text modality, image modality, audio modality, and video modality. The embodiments of the present application are generally described by taking the multimedia file as an audio file as an example.

[0052] Taking the multimedia file as an audio file as an example, the label classification method of the audio file is divided into two parts: stage one and stage two.

[0053] In the first stage, the comments 210 related to the audio file are processed to obtain an initial candidate tag pool 230 , wherein the initial candidate tag pool 230 includes an unscreened first candidate audio tag.

[0054] In the second stage, the first candidate audio tag is processed by calling the first language model 250 and the second language model 220 to obtain an audio output tag 290. The audio output tag 290 is an audio tag used to describe the audio file from one or more dimensions.

[0055] Phase 1:

[0056] In some embodiments, comments 210 associated with the audio file are obtained.

[0057] Exemplarily, in the music playing program, each piece of music (audio file) has a corresponding comment area, and the comment area includes comments 210 related to the audio file, such as: "This song is really nice!", "The old songs are still good", etc.

[0058] In some embodiments, the comments 210 are filtered.

[0059] Since the label classification method of the audio file needs to call the second largest language model 220, and the number of comments 210 is sometimes too large, it will bring a large calculation pressure to the second largest language model 220. Therefore, it is necessary to filter the comments 210 and input the selected high-quality candidate comments into the second largest language model 220.

[0060] In some embodiments, the comment 210 is input into a comment classification model to obtain a comment type corresponding to the comment 210 and a corresponding confidence score.

[0061] A review classification model is a model that can classify input reviews into different review types and output confidence scores corresponding to different review types.

[0062] Exemplarily, comment 210 is “This song is so nice!”, and it is assumed that the comment types include the following four types: multimedia evaluation type, question feedback type, story impression type, and low-quality comment type.

[0063] After inputting the comment 210 into the comment classification model, the confidence score of the output multimedia review type is 0.9, the confidence score of the problem feedback type is 0.02, the confidence score of the story impression type is 0.05, and the confidence score of the low-quality comment type is 0.03. Since the confidence score of the multimedia review type is the highest, the comment 210 is classified as the multimedia review type.

[0064] In some embodiments, multimedia evaluation type comments are filtered out from the comments related to the audio file;

[0065] Based on a preset comment database, the comments of multimedia evaluation type are filtered to obtain actual candidate comments; wherein the actual candidate comments do not include meaningless comments, the comment database includes multiple meaningless comments, and the semantics of the meaningless comments are irrelevant to the audio file.

[0066] Generally, it is more likely to extract suitable audio tags from multimedia evaluation type comments, the second most likely to extract suitable audio tags from story impression type comments, and the likelihood of extracting suitable audio tags from other types of comments is smaller. The embodiment of the present application does not limit the specific types of the filtered comments, and only takes multimedia evaluation type comments as an example for explanation.

[0067] Nonsense comments are also called spam comments. They are comments that are semantically irrelevant to the audio file and have the following two characteristics:

[0068] (1) They are not closely related to the current audio file and may appear in the comment area of ​​any audio file. Therefore, the audio tags extracted from such comments are unlikely to be used as audio tags for the audio file.

[0069] (2) Since some of the meaningless comments are very popular, that is, they have a high number of likes and replies, they cannot be filtered using indicators such as likes and replies.

[0070] For example, in the comments of a song expressing melancholy, a comment such as "I am on the first floor" that is irrelevant to the audio file is a meaningless comment, but it may have a high number of likes because it is the first comment.

[0071] Through the preset comment database, the comments of multimedia evaluation type are compared with the comment database to remove meaningless comments. The comment database includes a large number of meaningless comments collected historically. Through the comparison function of the comment database, the similarity between the comments of multimedia evaluation type and the meaningless comments in the comment database can be compared, and the comments with similarity higher than a certain threshold are determined as meaningless comments and removed, so as to obtain the actual candidate comments.

[0072] In some embodiments, high-quality candidate reviews may also be screened out based on review scores of actual candidate reviews, where high-quality candidate reviews are actual candidate reviews whose review scores are greater than or equal to a score threshold.

[0073] Optionally, the comment score of the actual candidate comment is related to the confidence score of the multimedia comment type corresponding to the actual candidate comment. For example, if the confidence score of the multimedia comment type corresponding to comment A is 0.9, then the comment score of the actual candidate comment is 0.9. Assuming the score threshold is 0.8, comment A is a high-quality candidate comment.

[0074] The review score of the actual candidate review may also be related to other information, which is not limited in this embodiment of the present application.

[0075] In some embodiments, the high-quality candidate comments and the second prompt word are input into the second language model 220 to obtain the first candidate audio tag.

[0076] By inputting the high-quality candidate comments and the second prompt word into the second largest language model 220, the first candidate audio tags corresponding to the high-quality candidate comments are mined. The second prompt word is used to indicate the semantic-based extraction of the first candidate audio tags and the tag dimension type corresponding to the first candidate audio tags.

[0077] Exemplarily, the content of the high-quality candidate comment is: "Singer A's voice sings "Song A", and adds a section of "Song B" in the middle. He is so sweet as soon as he opens his mouth, and the guitar is matched with the cello, and the modern and classical are intertwined, which is fascinating to listen to. Singer A and singer B are people who have accompanied many people through their youth. Listening to singer A singing the song created by singer B, it seems that those beautiful memories are overlapping, bringing us infinite reverie", and the second prompt word is: extract the place suitable for listening to the song, the instruments involved in the song, the genre of the song, the emotions expressed in the song, and the people associated with the song from the high-quality candidate comment (it can also include other content, and the above prompt words are only used as examples here).

[0078] By calling the second largest language model 220, the first candidate audio tag obtained includes: "none" is a place suitable for listening to the song; "guitar, cello" is instruments involved in the song; "modern, classical" is the genre of the song; "nostalgia" is the emotion expressed by the song; and "singer A, singer B" is people associated with the song.

[0079] In some embodiments, an initial candidate tag pool is constructed, and the initial candidate tag pool includes a first candidate audio tag.

[0080] Phase 2:

[0081] After entering the second stage, since the first candidate audio tags have been obtained, the first candidate audio tags need to be screened and incorrect audio tags in the first candidate audio tags need to be removed, so as to improve the accuracy of the audio tags finally outputted.

[0082] In some embodiments, based on the statistical index, the first candidate audio tags are classified to obtain a high-frequency tag 241 and a low-frequency tag 242, wherein the high-frequency tag 241 is a first candidate audio tag whose statistical index is higher than a statistical threshold, and the low-frequency tag 242 is a first candidate audio tag whose statistical index is lower than the statistical threshold;

[0083] Based on the statistical index analysis of the first candidate audio tag, a first candidate audio tag whose any statistical index is greater than or equal to the corresponding threshold is a high-frequency tag 241, and a first candidate audio tag whose all statistical indexes are less than the corresponding threshold is a low-frequency tag 242.

[0084] By way of example and not limitation, the statistical indicator includes at least one of the following: the number of likes for comments corresponding to the first candidate audio tag; the frequency with which the first candidate audio tag appears in comments related to the audio file.

[0085] Exemplarily, assuming that the like threshold is 100 likes and the frequency threshold is 50 times, the first candidate audio tags include: tag 1 "happiness", tag 2 "youth", tag 3 "nostalgia", tag 4 "progress", and tag 5 "pain".

[0086] Among them, the number of likes for the comments corresponding to label 1 "happiness" is 200, and the frequency of occurrence in the comments is 90 times; the number of likes for the comments corresponding to label 2 "youth" is 150, and the frequency of occurrence in the comments is 20 times; the number of likes for the comments corresponding to label 3 "nostalgia" is 50, and the frequency of occurrence in the comments is 100 times; the number of likes for the comments corresponding to label 4 "progress" is 50, and the frequency of occurrence in the comments is 20 times; the number of likes for the comments corresponding to label 5 "pain" is 20, and the frequency of occurrence in the comments is 10 times. Therefore, label 1 "happiness", label 2 "youth" and label 3 "nostalgia" are classified as high-frequency labels 241, and label 4 "progress" and label 5 "pain" are classified as low-frequency labels 242.

[0087] In some embodiments, the low-frequency tags 242 and the judgment prompt words are input into the first large language model 250 to obtain the credible low-frequency tags 251 , where the judgment prompt words are used to indicate the credible low-frequency tags 251 in the low-frequency tags 242 determined based on the description information 240 .

[0088] The description information 240 is used to describe the content of the audio file, including: the name of the audio file; the lyrics of the audio file; the audio of the audio file; the composer of the audio file; the lyricist of the audio file; the singer of the audio file, etc.

[0089] For example, label 4 "progress" and label 5 "pain" and a judgment prompt word are input into the first language model 250, and the judgment prompt word is "judge whether label 4 and label 5 are consistent with the lyrics, and output labels that are consistent with the lyrics." Since "progress" exists in the lyrics, label 4 "progress" is output as a credible low-frequency label 251.

[0090] In some embodiments, a secondary candidate tag pool 260 is constructed based on the high-frequency tags 241 and the trusted low-frequency tags 251 .

[0091] In some embodiments, the credible tag 270 and the third prompt word in the first candidate audio tag are input into the second large language model 220 to obtain a second candidate audio tag 272 and a contradictory tag pair 271 .

[0092] Optionally, the trusted tag includes a high frequency tag 241 and a trusted low frequency tag 251 .

[0093] In some embodiments, the third prompt word includes a classification prompt word and a de-duplication prompt word, the classification prompt word is used to indicate the extraction of semantically opposite contradictory label pairs 271 and semantically independent second candidate audio labels 272, and the de-duplication prompt word is used to indicate the merging of semantically repeated credible labels;

[0094] Input the trusted label 270 and the duplicate removal prompt word in the first candidate audio label into the second language model 220 to obtain the trusted label after duplicate removal;

[0095] The deduplicated credible labels and classification prompt words are input into the second largest language model 220 to obtain a second candidate audio label 272 and a contradictory label pair 271 .

[0096] For example, in addition to the above tags 1 to 5, there is also a tag 6 "joy". The deduplication prompt is "merge semantically repeated tags". Since "joy" and "happiness" have semantic repetition, tag 6 "joy" and tag 1 "happiness" are merged into "happiness".

[0097] The classification prompt word is "extract semantically opposite contradictory label pairs, and semantically independent labels". Since "pain" and "happiness" have opposite semantics, the contradictory label pair 1 "happiness-pain" is obtained. "Youth", "nostalgia", and "progress" are independent of "pain" and "happiness", so the second candidate audio label 272 is obtained: label 2 "youth", label 3 "nostalgia", and label 4 "progress".

[0098] In some embodiments, the first language model 250 is called based on the description information 240 , the contradictory tag pair 271 and the first prompt word to predict the correct audio tag 280 of the audio file. The first prompt word is used to indicate the correct audio tag 280 in the contradictory tag pair 271 determined based on the description information 240 .

[0099] For example, the description information 240 includes the lyrics of the audio file, the contradictory label pair 271 is the contradictory label pair 1 "happy-pain", and the first prompt word is "determine the correct audio label in the contradictory label pair based on the lyrics". Since there are many words expressing the meaning of "happy" and "joy" in the lyrics, the correct audio label 280 is determined to be label 1 "happy".

[0100] In some embodiments, based on the correct audio tag 280 and the second candidate audio tag 272 , an audio output tag 290 corresponding to the audio file is output.

[0101] Exemplarily, the correct audio tag 280 includes tag 1 "happy", the second candidate audio tag 272 includes tag 2 "youth", tag 3 "nostalgia", and tag 4 "progress", so the final audio output tag 290 includes: "happy", "youth", "nostalgia", and "progress".

[0102] To sum up, the method provided in this embodiment, by calling the first language model, obtains the correct multimedia tag in the contradictory tag pair of the audio file based on the description information, and finally outputs the audio output tag corresponding to the audio file, thereby reducing the problem of semantic conflicts between audio tags corresponding to the same audio file and improving the accuracy of the final output audio output tag.

[0103] The method provided in this embodiment also screens out high-quality candidate comments by using comment classification models, comment databases, comment scores, etc., thereby reducing the computational pressure of the second largest language model called and improving the efficiency of outputting audio output labels.

[0104] Figure 3 A flowchart of a method for label classification of multimedia files provided by an exemplary embodiment of the present application is shown. The method includes at least one of the following steps.

[0105] Step 310: Obtain description information of the multimedia file, and obtain conflicting tag pairs of the multimedia file.

[0106] The description information is used to describe the content of the multimedia file, and the contradictory tag pair includes two multimedia tags with opposite semantics.

[0107] A multimedia file refers to a file containing data of multiple media types or modalities. A multimedia file includes at least one of the following modalities: text modality, image modality, audio modality, and video modality. The embodiments of the present application are generally described by taking the multimedia file as an audio file as an example.

[0108] The description information is information used to describe the content of a multimedia file. Taking the multimedia file as an audio file as an example, the description information is information used to describe the content of the audio file, including: the name of the audio file; the lyrics of the audio file; the audio of the audio file; the composer of the audio file; the lyricist of the audio file; the singer of the audio file, etc.

[0109] Multimedia tags are used to annotate multimedia files, and each multimedia tag has corresponding semantics. For example, if the multimedia file is an audio file, the multimedia tag corresponds to the audio tag. For example, if audio tag 1 is "happy", then the semantics corresponding to audio tag 1 is "expressing the emotion of happiness".

[0110] The contradictory tag pair includes two multimedia tags with opposite semantics. For example, if multimedia tag 1 is "happiness" and multimedia tag 2 is "pain", then multimedia tag 1 and multimedia tag 2 are a contradictory tag pair.

[0111] Step 320: Based on the description information, the contradictory tag pair and the first prompt word, the first language model is called to predict a correct multimedia tag for the multimedia file.

[0112] The first prompt word is used to indicate the correct multimedia tag in the contradictory tag pair determined based on the description information.

[0113] The first large language model is a natural language processing model that can understand input text and generate output text. The large language model uses a large amount of text data for pre-training to learn the patterns and rules in natural language, so that it can understand the input text and generate relevant output text.

[0114] The first prompt word is a prompt word, which is an input text used to guide the large language model to perform a specified task. It contains instructions and / or examples corresponding to the specified task, helping the large language model understand the type and format of the text to be output.

[0115] Exemplarily, taking the multimedia file as an audio file, the description information includes the lyrics of the audio file, and the contradictory label pair "happiness-pain" and the first prompt word are input into the first language model, and the first prompt word is "determine the correct audio label in the contradictory label pair based on the lyrics."

[0116] Since the semantics of the two multimedia tags in the contradictory tag pair are opposite, and usually, a multimedia file does not correspond to opposite multimedia tags, there is a correct multimedia tag and an incorrect multimedia tag in the contradictory tag pair, and the correct multimedia tag needs to be determined as the output tag.

[0117] Step 330: Based on the correct multimedia tag, output the multimedia output tag corresponding to the multimedia file.

[0118] Multimedia output tags are multimedia tags that have been screened and optimized. The number of multimedia output tags is usually less than that of input multimedia tags, but multimedia files can be labeled more accurately.

[0119] Exemplarily, after the contradictory label pair "happy-pain" and the first prompt word are input into the first language model, since there are many words in the lyrics that express the semantics of "happy" and "joy", the correct audio label is determined to be "happy", so the audio output label (multimedia output label) includes "happy".

[0120] To sum up, the method provided in this embodiment, by calling the first language model, obtains the correct multimedia tag in the contradictory tag pair of the multimedia file based on the description information, and finally outputs the multimedia output tag corresponding to the multimedia file, thereby reducing the problem of semantic conflicts between multimedia tags corresponding to the same multimedia file and improving the accuracy of the multimedia output tag finally output.

[0121] Figure 4 A flowchart of a method for label classification of multimedia files provided by an exemplary embodiment of the present application is shown. The method includes at least one of the following steps.

[0122] Step 312: Obtain description information of the multimedia file in at least two modes, and obtain conflicting tag pairs of the multimedia file.

[0123] Among them, the at least two modalities include at least two of the following modalities: text modality, image modality, audio modality, and video modality.

[0124] Text modality refers to multimedia files that include text information, such as titles, descriptions, subtitles, etc.; image modality refers to multimedia files that include static visual information, such as photos, graphics, etc.; audio modality refers to multimedia files that include sound information, such as music, voice, sound effects, etc.; video modality refers to multimedia files that include dynamic visual information, such as movies, animations, etc.

[0125] The at least two modes may also include other types of modes. This application only uses the above modes as examples for illustration and does not limit this.

[0126] Step 322: Map the description information of other modalities except the text modality to the text feature space corresponding to the text modality to obtain feature representations of the description information of other modalities.

[0127] There is a lack of direct correspondence between the description information of different modalities. For example, the image modality corresponds to visual information, and the audio modality corresponds to auditory information. Therefore, it is necessary to map the description information of modalities other than the text modality to the text feature space corresponding to the text modality so that the feature representation of the description information of other modalities can be aligned with the feature representation of the description information of the text modality, and thus can be aligned with the feature representation of the input text.

[0128] In some embodiments, the description information of other modalities except the text modality is mapped to the text feature space corresponding to the text modality to obtain the feature representation of the description information of other modalities, including:

[0129] Encode the description information of other modalities through encoders corresponding to other modalities to obtain original feature representations of the description information of other modalities in feature spaces corresponding to other modalities;

[0130] The original feature representation is mapped from the feature space corresponding to other modalities to the text feature space corresponding to the text modality through the feature mapping network to obtain the feature representation of the descriptive information of other modalities.

[0131] Figure 5 A schematic diagram of an audio processing process provided by an exemplary embodiment of the present application is shown.

[0132] Taking the multimedia file as an audio file as an example, the audio 510 corresponding to the audio file is input into the audio encoder 520, and the audio 510 is encoded by the audio encoder 520 to obtain the original feature representation 521 of the audio 510 in the audio feature space. The original feature representation 521 is mapped from the audio feature space to the text feature space corresponding to the text modality through the feature mapping network 530 to obtain the audio feature representation 531 of the audio 510. The audio feature representation 531 and the input text 540 are input into the first large language model 550 to obtain the output text 560.

[0133] Exemplarily, the length of the text feature in the text feature space is 4, and after the original feature representation 521 is mapped to the text feature space, the length of the audio feature representation 531 is also 4 (indicated by 4 black squares).

[0134] Figure 6 A schematic diagram of a feature mapping network provided by an exemplary embodiment of the present application is shown.

[0135] The feature mapping network is a multilayer perceptron (MLP) structure. A multilayer perceptron is a feedforward artificial neural network consisting of an input layer, at least one hidden layer, and an output layer. Each layer contains a number of neurons (or nodes), each of which is connected to each neuron in the next layer, forming a fully connected network structure. MLP is trained using the back-propagation algorithm, which minimizes the difference between the output layer and the target by adjusting the weights between the layers, thereby improving the performance of the model.

[0136] Taking the multimedia file as an audio file as an example, assuming that the text feature length is d1, the input layer includes the original feature representation of the audio in the audio feature space, and the feature length of the original feature representation of the audio in the audio feature space is d2. The multiple MLP units in the hidden layer convert the original feature representation of d2 dimensions into an audio feature representation of d1 dimensions that is consistent with the text feature length (output layer). After the input text is converted into a text feature of d1 dimensions, it is spliced ​​with the audio feature representation and input into the first language model to perform specific tasks.

[0137] Step 3242: When at least two modalities include a text modality, the feature representation of the description information of the other modalities, the description information of the text modality, the contradictory label pair and the first prompt word are input into the first language model to predict the correct multimedia label of the multimedia file.

[0138] In some embodiments, feature representations of input text and descriptive information of other modalities are input into a first large language model; wherein the input text includes: descriptive information of the text modality, contradictory tag pairs, and a first prompt word.

[0139] like Figure 5 As shown, in the case where at least two modalities include a text modality, the input text 540 includes: description information 541 of the text modality, a contradictory tag pair 542 and a first prompt word 543 .

[0140] For example, if the multimedia file is an audio file, assuming that the description information 541 of the text modality is the lyrics of the audio file, the contradictory label pair 542 is "happy-painful", and the first prompt word is "extract the correct audio label based on the audio and lyrics". Since there are many words in the lyrics that express the meaning of "happy" and "joy", and the rhythmic features of the audio, such as speech speed, volume, and pitch, represent the emotion of happiness, the correct audio label is determined to be "happy".

[0141] Step 3244: When at least two modalities do not include a text modality, the feature representation of the description information of other modalities, the contradictory label pair and the first prompt word are input into the first language model to predict the correct multimedia label of the multimedia file.

[0142] In some embodiments, feature representations of input text and descriptive information of other modalities are input into a first large language model; wherein the input text includes: a contradictory tag pair and a first prompt word.

[0143] like Figure 5 As shown, when at least two modalities do not include a text modality, the input text 540 includes: a contradictory tag pair 542 and a first prompt word 543 .

[0144] For example, if the multimedia file is an audio file, the description information 541 of the text modality is the lyrics of the audio file, the contradictory label pair 542 is "happy-painful", and the first prompt word is "based on the audio, extract the correct audio label". Since the rhythmic features of the audio, such as speech speed, volume and pitch, represent the emotion of happiness, the correct audio label is determined to be "happy".

[0145] Step 330: Based on the correct multimedia tag, output the multimedia output tag corresponding to the multimedia file.

[0146] In summary, the method provided in this embodiment obtains description information of a multimedia file in at least two modes, predicts correct multimedia tags from the dimensions of multiple modes, and improves the accuracy of prediction.

[0147] The method provided in this embodiment also obtains feature representations of the description information of other modalities by mapping the description information of other modalities except the text modality into the text feature space corresponding to the text modality, and executes corresponding steps respectively in the case where at least two modalities include the text modality or do not include the text modality to predict the correct multimedia tags, thereby expanding the applicable scenarios for predicting the correct multimedia tags and ensuring that the correct multimedia tags can be predicted regardless of whether the text modality is included.

[0148] The method provided in this embodiment also obtains feature representations of descriptive information of other modalities by using encoders corresponding to other modalities and feature mapping networks, so that the feature representations of descriptive information of other modalities are aligned with the feature representations of descriptive information of text modality, and can also be aligned with the feature representations of input text, thereby enabling the first language model to understand information from different dimensions and improve its ability to understand descriptive information of other modalities.

[0149] Figure 7 A flowchart of a method for label classification of multimedia files provided by an exemplary embodiment of the present application is shown. The method includes at least one of the following steps.

[0150] Step 305: Input the comments related to the multimedia file and the second prompt word into the second largest language model to obtain a first candidate multimedia tag.

[0151] The second prompt word is used to instruct to extract the first candidate multimedia tag based on semantics.

[0152] For example, if the multimedia file is an audio file and the first candidate multimedia tag corresponds to the first candidate audio tag, in the music player program, each piece of music (audio file) has a corresponding comment area, and the comment area includes comments related to the audio file, such as "this song is really nice!", "old songs are still the best", etc.

[0153] The comments related to the multimedia file correspond to one or more first candidate multimedia tags, and the multiple first candidate multimedia tags are semantically independent, overlapping, or opposite to each other.

[0154] Exemplarily, the comment related to the audio file is "This song is very happy, but I feel very painful when I think of the friends who left me because of this song", and the second prompt word is "Extract audio tags in comments based on semantics", and the first candidate audio tags extracted include: "happy" and "painful", which are opposite in semantics.

[0155] Step 314: Obtain description information of the multimedia file, and input the credible tag and the third prompt word in the first candidate multimedia tag into the second largest language model to obtain a second candidate multimedia tag and a contradictory tag pair.

[0156] The credible tags include multimedia tags obtained by the first language model based on judgment data, the judgment data include first candidate multimedia tags and description information, and the third prompt word is used to indicate the semantic extraction of second candidate multimedia tags and contradictory tag pairs.

[0157] Trusted tags are the first candidate multimedia tags that are selected by the first language model and the first candidate multimedia tags that cannot accurately describe the multimedia file are removed. For example, among the multimedia tags corresponding to 1,000 comments, a first candidate multimedia tag that only appears once usually cannot accurately describe the multimedia file.

[0158] The second candidate multimedia tags are semantically independent multimedia tags, and the contradictory tag pair is two multimedia tags with opposite semantics.

[0159] Step 320: Based on the description information, the contradictory tag pair and the first prompt word, the first language model is called to predict a correct multimedia tag for the multimedia file.

[0160] Step 332: Based on the correct multimedia tag and the second candidate multimedia tag, output a multimedia output tag corresponding to the multimedia file.

[0161] In addition to the correct multimedia tag in the contradictory tag pair, the semantically independent second candidate multimedia tag is also a multimedia tag for annotating multimedia files. The multimedia output tag output based on the correct multimedia tag and the second candidate multimedia tag is more comprehensive and accurate.

[0162] Exemplarily, the correct multimedia label for the contradictory label pair "happiness-pain" is "happiness", and the second candidate multimedia labels include: "youth", "nostalgia", and "progress", so the final multimedia output labels include: "happiness", "youth", "nostalgia", and "progress".

[0163] In summary, the method provided in this embodiment obtains a first candidate multimedia tag by inputting comments and a second prompt word into a second language model, and obtains a second candidate multimedia tag and a contradictory tag pair by inputting a trusted tag and a third prompt word in the first candidate multimedia tag into the second language model, and finally outputs a multimedia output tag corresponding to the multimedia file based on the correct multimedia tag and the second candidate multimedia tag, thereby mining multimedia tags (first candidate multimedia tags) from comments, and distinguishing the second candidate multimedia tags and the correct multimedia tags from the first candidate multimedia tags. The multimedia output tags finally outputted are more accurate and cover a wider range, and can recommend the multimedia file to more users who may be interested in a certain multimedia output tag.

[0164] Figure 8 A flowchart of a method for label classification of multimedia files provided by an exemplary embodiment of the present application is shown. The method includes at least one of the following steps.

[0165] Step 305: Input the comments related to the multimedia file and the second prompt word into the second largest language model to obtain a first candidate multimedia tag.

[0166] Step 308: Based on the statistical indicators, classify the first candidate multimedia tags to obtain high-frequency tags and low-frequency tags.

[0167] The high-frequency tag is a first candidate multimedia tag whose statistical index is higher than the statistical threshold, and the low-frequency tag is a first candidate multimedia tag whose statistical index is lower than the statistical threshold.

[0168] Since the first candidate multimedia tag is a multimedia tag extracted based on the comment, it may only reflect the personal opinion of the comment and may not be accurate. Therefore, it is necessary to combine other information, such as statistical indicators, to measure the accuracy of the first candidate multimedia tag.

[0169] The first candidate multimedia tags are analyzed based on statistical indicators. A first candidate multimedia tag whose any statistical indicator is greater than or equal to a corresponding threshold is a high-frequency tag, and a first candidate multimedia tag whose all statistical indicators are less than the corresponding threshold is a low-frequency tag.

[0170] By way of example and not limitation, the statistical indicator includes at least one of the following: the number of likes of the comments corresponding to the first candidate multimedia tag; the frequency of the first candidate multimedia tag appearing in the comments related to the multimedia file.

[0171] Exemplarily, assuming that the like threshold is 100 likes and the frequency threshold is 50 times, the first candidate multimedia tags include: tag 1 "happiness", tag 2 "youth", tag 3 "nostalgia", tag 4 "progress", and tag 5 "pain".

[0172] Among them, the number of likes for the comments corresponding to label 1 "happiness" is 200, and the frequency of occurrence in the comments is 90 times; the number of likes for the comments corresponding to label 2 "youth" is 150, and the frequency of occurrence in the comments is 20 times; the number of likes for the comments corresponding to label 3 "nostalgia" is 50, and the frequency of occurrence in the comments is 100 times; the number of likes for the comments corresponding to label 4 "progress" is 50, and the frequency of occurrence in the comments is 20 times; the number of likes for the comments corresponding to label 5 "pain" is 20, and the frequency of occurrence in the comments is 10 times. Therefore, label 1 "happiness", label 2 "youth" and label 3 "nostalgia" are classified as high-frequency labels, and label 4 "progress" and label 5 "pain" are classified as low-frequency labels.

[0173] Step 309: Input the low-frequency label and the judgment prompt word into the first language model to obtain a reliable low-frequency label.

[0174] The judgment prompt word is used to indicate the determination of a reliable low-frequency tag among the low-frequency tags based on the description information.

[0175] High-frequency tags are usually multimedia tags that can accurately label multimedia files, but low-frequency tags are not necessarily multimedia tags that incorrectly label multimedia files. They may be classified as low-frequency tags because the number of comments involving the low-frequency tags is small.

[0176] For example, if the multimedia file is an audio file, the description information includes the lyrics of the audio file, and label 4 "progress" and label 5 "pain" and the judgment prompt word are input into the first language model, and the judgment prompt word is "judge whether label 4 and label 5 are consistent with the lyrics, and output the label that is consistent with the lyrics." Since "progress" exists in the lyrics, label 4 "progress" is output as a credible low-frequency label.

[0177] Step 316: Obtain description information of the multimedia file, and input the high-frequency tag, the credible low-frequency tag and the third prompt word into the second language model to obtain a second candidate multimedia tag and a contradictory tag pair.

[0178] In some embodiments, the trusted tags include high frequency tags and trusted low frequency tags.

[0179] By inputting high-frequency tags and credible low-frequency tags into the second largest language model for the next step of screening, the comprehensiveness of the multimedia tag source is ensured, avoiding or reducing the omission of multimedia tags that have the possibility of accurately annotating multimedia files.

[0180] Step 320: Based on the description information, the contradictory tag pair and the first prompt word, the first language model is called to predict a correct multimedia tag for the multimedia file.

[0181] Step 332: Based on the correct multimedia tag and the second candidate multimedia tag, output a multimedia output tag corresponding to the multimedia file.

[0182] In summary, the method provided in this embodiment classifies the first candidate multimedia tags into high-frequency tags and low-frequency tags based on statistical indicators, extracts credible low-frequency tags from the low-frequency tags through the first largest language model, and inputs the high-frequency tags and credible low-frequency tags as credible tags into the second largest language model to obtain the second candidate multimedia tags and the contradictory tag pairs, thereby ensuring the comprehensiveness of the sources of the second candidate multimedia tags and the contradictory tag pairs, and avoiding or reducing the omission of multimedia tags that have the possibility of accurately marking multimedia files.

[0183] Fig. 9 A flowchart of a method for label classification of multimedia files provided by an exemplary embodiment of the present application is shown. The method includes at least one of the following steps.

[0184] Step 305: Input the comments related to the multimedia file and the second prompt word into the second largest language model to obtain a first candidate multimedia tag.

[0185] Step 318: Obtain description information of the multimedia file, and input the credible tag and the classification prompt word in the first candidate multimedia tag into the second largest language model to obtain a second candidate multimedia tag and a contradictory tag pair.

[0186] The third prompt words include classification prompt words, and the classification prompt words are used to indicate the extraction of semantically opposite contradictory tag pairs and semantically independent second candidate multimedia tags.

[0187] In some embodiments, the third prompt word further includes a de-duplication prompt word, which is used to indicate the credible tag with repeated semantics to be merged; the credible tag and the classification prompt word in the first candidate multimedia tag are input into the second largest language model to obtain the second candidate multimedia tag and the contradictory tag pair, including:

[0188] Inputting the credible tags and duplicate removal prompt words in the first candidate multimedia tags into the second largest language model to obtain the credible tags after duplicate removal;

[0189] The deduplicated credible tags and classification prompt words are input into the second largest language model to obtain the second candidate multimedia tags and contradictory tag pairs.

[0190] Exemplarily, the first candidate multimedia tags include: tag 1 "happy", tag 2 "youth", tag 3 "nostalgia", tag 4 "progress", tag 5 "pain", tag 6 "pleasure". The deduplication prompt is "merge semantically repeated tags". Since "pleasure" and "happy" have semantic repetition, tag 6 "pleasure" is merged with tag 1 "happy" into "happy".

[0191] The classification prompt is "extract semantically opposite contradictory label pairs and semantically independent labels". Since "pain" and "happiness" have opposite semantics, we get contradictory label pair 1 "happiness-pain". "Youth", "nostalgia", "progress" and "pain" are independent of each other, so we get the second candidate multimedia labels: label 2 "youth", label 3 "nostalgia", label 4 "progress".

[0192] Step 320: Based on the description information, the contradictory tag pair and the first prompt word, the first language model is called to predict a correct multimedia tag for the multimedia file.

[0193] Step 332: Based on the correct multimedia tag and the second candidate multimedia tag, output a multimedia output tag corresponding to the multimedia file.

[0194] In summary, the method provided in this embodiment improves the accuracy of distinguishing credible tags by dividing credible tags into second candidate multimedia tags and contradictory tag pairs based on classification prompt words.

[0195] Alternatively, based on deduplication prompt words and classification prompt words, the trusted tags are deduplicated and then divided into second candidate multimedia tags and contradictory tag pairs, thereby reducing the number of second candidate multimedia tags and contradictory tag pairs with the same semantics and reducing the computational pressure of the second largest language model.

[0196] Fig.10 A flowchart of a method for label classification of multimedia files provided by an exemplary embodiment of the present application is shown. The method includes at least one of the following steps.

[0197] Step 301: Input comments related to the multimedia file into a comment classification model to obtain comment types and corresponding confidence scores corresponding to the comments related to the multimedia file.

[0198] A review classification model is a model that can classify input reviews into different review types and output confidence scores corresponding to different review types.

[0199] Exemplarily, the comment classification model includes a Bidirectional Encoder Representations from Transformers (BERT) model. The comment classification model is obtained by fine-tuning and optimizing the BERT model. For example, a classification layer is added to the end of the BERT model, and full parameter fine-tuning, Adam optimizer, and the maximum iteration round is set to 15, and the learning rate is 1*10. -5 .

[0200] The Adam optimizer calculates the mean and variance of the gradient and uses the mean and variance of the gradient to adjust the learning rate of each parameter. The maximum number of iterations refers to the number of times the BERT model fully propagates forward and backward on the entire training set. In each iteration, the BERT model observes each sample in the training set once and updates the parameters based on the sample. The learning rate is a hyperparameter used to control the step size of the model parameter update. In each iteration, the model parameters are updated according to the gradient of the loss function and the learning rate.

[0201] By way of example and not limitation, the comment type includes at least one of the following: a multimedia evaluation type, a question feedback type, a story impression type, and a low-quality comment type.

[0202] Among them, multimedia evaluation type comments are comments made by users on multimedia content. This type of comments may include comments on the quality of the video, the clarity of the audio, the aesthetics of the image, etc. For example: "This song is so nice!";

[0203] Problem feedback type comments are feedback provided by users when they encounter problems with products or services. This type of comment may include a description of the problem encountered, the context in which the problem occurred, etc. For example: "The video keeps freezing when I watch it";

[0204] Story impression type comments are user comments on the stories related to multimedia files. This type of comment may include the user's feelings about the story, their views on the characters, the rationality of the plot, and overall satisfaction. For example: "The characters in this story...";

[0205] Low-quality comments are comments that lack substantive content, are illogical, or are malicious. These comments may contain spam, irrelevant content, malicious attacks, or simply repetitive words. For example: "XXX (name not related to the multimedia file) is really not a good person."

[0206] For example, the comment is "This song is so nice!", and the comment types include the following four types: multimedia evaluation type, question feedback type, story impression type, and low-quality comment type.

[0207] After the comment is input into the comment classification model, the confidence score of the output multimedia comment type is 0.9, the confidence score of the problem feedback type is 0.02, the confidence score of the story feeling type is 0.05, and the confidence score of the low-quality comment type is 0.03. Since the confidence score of the multimedia comment type is the highest, the comment is classified as the multimedia comment type.

[0208] Step 302: Filter out multimedia evaluation type comments from the comments related to the multimedia files.

[0209] Generally, it is more likely to extract appropriate multimedia tags from multimedia evaluation type comments, the second most likely to extract appropriate multimedia tags from story impression type comments, and the likelihood of extracting appropriate multimedia tags from other types of comments is smaller. The embodiment of the present application does not limit the specific types of comments screened, and only takes multimedia evaluation type comments as an example for explanation.

[0210] Step 303: Based on a preset comment database, filter the comments of the multimedia evaluation type to obtain actual candidate comments.

[0211] Among them, the actual candidate comments do not include meaningless comments, the comment database includes multiple meaningless comments, and the semantics of the meaningless comments are irrelevant to the multimedia files.

[0212] Meaningless comments, also known as spam comments, refer to comments that are semantically irrelevant to multimedia files and have the following two characteristics:

[0213] (1) They are not closely related to the current multimedia file and may appear in the comment area of ​​any multimedia file. Therefore, the multimedia tags extracted from such comments are unlikely to be used as multimedia tags for the multimedia file.

[0214] (2) Since some of the meaningless comments are very popular, that is, they have a high number of likes and replies, they cannot be filtered using indicators such as likes and replies.

[0215] For example, in the comments of a song expressing melancholy, a comment such as "I am on the first floor" that is irrelevant to the audio file is a meaningless comment, but it may have a high number of likes because it is the first comment.

[0216] Through the preset comment database, the comments of multimedia evaluation type are compared with the comment database to remove meaningless comments. The comment database includes a large number of meaningless comments collected historically. Through the comparison function of the comment database, the similarity between the comments of multimedia evaluation type and the meaningless comments in the comment database can be compared, and the comments with similarity higher than a certain threshold are determined as meaningless comments and removed, so as to obtain the actual candidate comments.

[0217] Step 304: Filter out high-quality candidate reviews based on the review scores of the actual candidate reviews.

[0218] In some embodiments, a high-quality candidate review is an actual candidate review with a review score greater than or equal to a score threshold.

[0219] Optionally, the comment score of the actual candidate comment is related to the confidence score of the multimedia comment type corresponding to the actual candidate comment. For example, if the confidence score of the multimedia comment type corresponding to comment A is 0.9, then the comment score of the actual candidate comment is 0.9. Assuming the score threshold is 0.8, comment A is a high-quality candidate comment.

[0220] By way of example and not limitation, the review score is related to at least one of the following information:

[0221] Confidence score of the multimedia review type corresponding to the actual candidate review;

[0222] The number of likes corresponding to the actual candidate comments;

[0223] The number of reference multimedia tags that have the same semantics as the actual candidate review, and the description information includes the reference multimedia tags.

[0224] The reference multimedia tag is a multimedia tag added by the provider of the multimedia file for reference. For example, for an audio file, the reference multimedia tag added by the provider of the audio file is "nostalgia".

[0225] Fig.11 A flowchart for obtaining high-quality candidate reviews provided by an exemplary embodiment of the present application is shown.

[0226] After obtaining the actual candidate comments 1110, based on the comment scores 1120, the comments with the top K comment scores (i.e., the comments ranked in the top K in comment scores) in the actual candidate comments 1110 are screened, the text features of the top K comments are extracted, clustering and deduplication are performed, and finally the comments with the top N comment scores (i.e., the comments ranked in the top N in comment scores) are retained as high-quality candidate comments. K and N are positive integers, and N is less than K.

[0227] For example, K is 200 and N is 10, that is, the top 10 comments in terms of comment scores are finally retained as high-quality candidate comments.

[0228] Optionally, the calculation formula of the comment score 1120 is as follows:

[0229] S=Z+3*min(1, float(D) / 100)+6*min(1, float(C) / 3);

[0230] Among them, S represents the comment score 1120, Z represents the confidence score 1121 of the multimedia evaluation type corresponding to the actual candidate comment 1110, the min function represents selecting the minimum value between 1 and the float function value, the float function is used to convert a value into a floating point number, D represents the number of likes 1122 corresponding to the actual candidate comment 1110, and C represents the number of reference multimedia tags 1123 with the same semantics as the actual candidate comment 1110.

[0231] Exemplarily, the actual candidate comment 1110 is "Listening to this song makes me miss the past. I was so happy back then. People cannot have both youth and the understanding of youth at the same time." The number of likes corresponding to the actual candidate comment 1110 is 500, and the confidence score 1121 of the multimedia evaluation type corresponding to the actual candidate comment 1110 is 0.9. Reference multimedia tags include: "nostalgia", "happiness", "youth", "friendship", etc.

[0232] Since the semantics of the actual candidate comment 1110 include: "nostalgia", "happiness", and "youth", the number 1123 of reference multimedia tags having the same semantics as the actual candidate comment 1110 is 3. Then the comment score S = 0.9 + 3*min(1, float(500) / 100) + 6*min(1, float(3) / 3) = 9.9.

[0233] The calculation formula for the comment score 1120 may also be other calculation formulas, which is not limited in the embodiment of the present application.

[0234] Step 3051: Input the high-quality candidate comments and the second prompt word into the second largest language model to obtain the first candidate multimedia tag.

[0235] Since the label classification method of multimedia files requires the use of the second largest language model, and the number of comments is sometimes too large, it will bring great computational pressure to the second largest language model. Therefore, it is necessary to filter the comments and input the selected high-quality candidate comments into the second largest language model.

[0236] In some embodiments, the second prompt word is used to instruct to extract the first candidate multimedia tag based on semantics, and the tag dimension type corresponding to the first candidate multimedia tag.

[0237] By way of example and not limitation, the label dimension type includes at least one of the following:

[0238] Places suitable for enjoying multimedia files; the region where the multimedia files originate; the musical instruments involved in the multimedia files; the genres of the multimedia files; the emotions expressed in the multimedia files; the language used in the multimedia files; the groups of people the multimedia files are aimed at; the themes of the multimedia files; the characters associated with the multimedia files; and other multimedia associated with the multimedia files.

[0239] Exemplarily, taking the multimedia file as an audio file, the content of the high-quality candidate comment is: "The voice of singer A sings "Song A", and adds a section of "Song B" in the middle. He is so sweet as soon as he opens his mouth, the guitar is matched with the cello, and the modern and classical are intertwined. It is intoxicating to listen to. Singer A and singer B are people who have accompanied many people through their youth. Listening to singer A singing the song created by singer B is like those beautiful memories overlapping, bringing us infinite reverie", and the second prompt word is: extract the place suitable for listening to the song, the instruments involved in the song, the genre of the song, the emotions expressed in the song, and the characters associated with the song from the high-quality candidate comment (it may also include other content, and the above prompt words are only used as examples here).

[0240] By calling the second largest language model 220, the first candidate audio tag obtained includes: "none" is a place suitable for listening to the song; "guitar, cello" is instruments involved in the song; "modern, classical" is the genre of the song; "nostalgia" is the emotion expressed by the song; and "singer A, singer B" is people associated with the song.

[0241] Step 314: Obtain description information of the multimedia file, and input the credible tag and the third prompt word in the first candidate multimedia tag into the second largest language model to obtain a second candidate multimedia tag and a contradictory tag pair.

[0242] Step 320: Based on the description information, the contradictory tag pair and the first prompt word, the first language model is called to predict a correct multimedia tag for the multimedia file.

[0243] Step 332: Based on the correct multimedia tag and the second candidate multimedia tag, output a multimedia output tag corresponding to the multimedia file.

[0244] In summary, the method provided in this embodiment obtains the comment types and confidence scores corresponding to the comments related to the multimedia file by using the comment classification model, and preliminarily classifies the comments to facilitate subsequent screening of the comments.

[0245] The method provided in this embodiment further screens out multimedia evaluation type comments from the comments, and filters multimedia evaluation type comments based on the comment database to obtain actual candidate comments. It is more likely that suitable multimedia output tags will be extracted from multimedia evaluation type comments, and removing meaningless comments from multimedia evaluation type comments can leave actual candidate comments with substantial meaning, thereby reducing the computational pressure of the second largest language model called.

[0246] The method provided in this embodiment also screens out high-quality candidate comments based on the comment scores of the actual candidate comments, further reducing the number of actual candidate comments, extracting the first candidate multimedia tags from the selected high-quality candidate comments, reducing the computational pressure of the second largest language model, and improving the efficiency of outputting multimedia output tags.

[0247] In the method provided in this embodiment, the second prompt word is used to indicate the semantic extraction of the first candidate multimedia tag and the tag dimension type corresponding to the first candidate multimedia tag. The first candidate multimedia tags are extracted from different dimension types and displayed, and various multimedia tags can be filtered, screened and purified more flexibly and finely in the subsequent processing and analysis.

[0248] In the above embodiment, steps with the same sequence number can be considered as the same step. Figure 3 The corresponding embodiments, Figure 4 The corresponding embodiments, Figure 7 The corresponding embodiments, Figure 8 The corresponding embodiments, Fig. 9 The corresponding embodiments and Fig.10 The corresponding embodiments may be implemented individually or in combination, and this application does not limit this.

[0249] Taking the application of the label classification method of multimedia files in the audio file scenario as an example, after obtaining comments related to the audio file, the comments related to the audio file and the second prompt word are input into the second largest language model to obtain the first candidate audio label.

[0250] Exemplarily, the comment related to the audio file is: "Listening to this song makes me miss the past. I was so happy back then. People cannot have both youth and the understanding of youth, but I am very painful now", and the second prompt word is: "Extract the emotions expressed by the song and the theme of the song in the comment", and the first candidate audio tags are obtained: Tag 1 "Emotions expressed by the song: happiness", Tag 2 "Emotions expressed by the song: pain", Tag 3 "Theme of the song: youth", Tag 4 "Theme of the song: nostalgia".

[0251] In some embodiments, description information of the audio file is obtained, and the credible tag and the third prompt word in the first candidate audio tag are input into the second largest language model to obtain the second candidate audio tag and the contradictory tag pair.

[0252] Exemplarily, the description information of the audio file includes the lyrics and audio of the audio file. The credible tags in the first candidate audio tags include tags 1 to 4, and the third prompt word is: "Extract semantically opposite contradictory tag pairs and semantically independent tags", and obtain the contradictory tag pair "emotions expressed by the song: happiness-emotions expressed by the song: pain", and the second candidate audio tags: tags 3 and 4.

[0253] In some embodiments, a first language model is called based on the description information, the contradictory tag pair, and the first prompt word to predict a correct audio tag for the audio file.

[0254] For example, the first prompt word is "extract the correct audio tag based on the audio and lyrics". Since there are many words in the lyrics that express the meaning of "happy" and "joy", and the rhythmic features of the audio, such as speech speed, volume and pitch, represent happy emotions, the correct audio tag is determined to be "happy".

[0255] In some embodiments, based on the correct audio tag and the second candidate audio tag, an audio output tag corresponding to the audio file is output.

[0256] Exemplarily, the correct audio tag is "happy", the second candidate audio tag includes tag 3 and tag 4, so the final multimedia output tags include: "Emotion expressed by the song: happiness", "Theme of the song: youth", "Theme of the song: nostalgia".

[0257] The above embodiments are all described by taking the application of the multimedia file tag classification method in the audio file scenario as an example. In some embodiments, the multimedia file tag classification method can also be applied to video file scenarios or other multimedia file scenarios. This application is not limited to this and only describes it by taking the video file scenario as an example.

[0258] In some embodiments, the tag classification method of the video file includes at least one of the following steps.

[0259] Obtaining description information of a video file and obtaining a contradictory tag pair of the video file, wherein the description information is used to describe the content of the video file, and the contradictory tag pair includes two video tags with opposite semantics;

[0260] Based on the description information, the contradictory label pair and the first prompt word, a first language model is called to predict a correct video label of the video file, wherein the first prompt word is used to indicate the correct video label in the contradictory label pair determined based on the description information;

[0261] Based on the correct video tag, output the video output tag corresponding to the video file.

[0262] In some embodiments, obtaining description information of a video file includes:

[0263] Descriptive information of a video file in at least two modes is obtained, where the at least two modes include at least two of the following modes: text mode, image mode, audio mode, and video mode.

[0264] In some embodiments, calling the first language model based on the description information, the contradictory tag pair, and the first prompt word to predict a correct video tag for the video file includes:

[0265] Mapping the description information of other modalities except the text modality to the text feature space corresponding to the text modality to obtain the feature representation of the description information of other modalities;

[0266] In the case where at least two modalities include a text modality, feature representations of description information of other modalities, description information of the text modality, contradictory label pairs and a first prompt word are input into a first language model to predict a correct video label for the video file.

[0267] In some embodiments, calling the first language model based on the description information, the contradictory tag pair, and the first prompt word to predict a correct video tag for the video file includes:

[0268] Mapping the description information of other modalities except the text modality to the text feature space corresponding to the text modality to obtain the feature representation of the description information of other modalities;

[0269] In the case that at least two modalities do not include a text modality, feature representations of description information of other modalities, contradictory label pairs, and a first prompt word are input into a first language model to predict a correct video label of the video file.

[0270] In some embodiments, the description information of other modalities except the text modality is mapped to the text feature space corresponding to the text modality to obtain the feature representation of the description information of other modalities, including:

[0271] Encode the description information of other modalities through encoders corresponding to other modalities to obtain original feature representations of the description information of other modalities in feature spaces corresponding to other modalities;

[0272] The original feature representation is mapped from the feature space corresponding to other modalities to the text feature space corresponding to the text modality through the feature mapping network to obtain the feature representation of the descriptive information of other modalities.

[0273] In some embodiments, the method further comprises:

[0274] Inputting the comments and the second prompt word related to the video file into the second largest language model to obtain a first candidate video tag, wherein the second prompt word is used to indicate the extraction of the first candidate video tag based on semantics;

[0275] Get contradictory label pairs for video files, including:

[0276] Inputting the credible label and the third prompt word in the first candidate video label into the second largest language model to obtain the second candidate video label and the contradictory label pair, wherein the credible label includes the video label obtained by the first largest language model based on the judgment data, the judgment data includes the first candidate video label and the description information, and the third prompt word is used to indicate the extraction of the second candidate video label and the contradictory label pair based on semantics;

[0277] Based on the correct video tag, the video output tag corresponding to the output video file includes:

[0278] Based on the correct video tag and the second candidate video tag, a video output tag corresponding to the output video file is output.

[0279] In some embodiments, the method further comprises:

[0280] Based on the statistical indicators, the first candidate video labels are classified to obtain high-frequency labels and low-frequency labels, wherein the high-frequency labels are first candidate video labels whose statistical indicators are higher than the statistical threshold, and the low-frequency labels are first candidate video labels whose statistical indicators are lower than the statistical threshold;

[0281] Inputting the low-frequency label and the judgment prompt word into the first language model to obtain a reliable low-frequency label, wherein the judgment prompt word is used to indicate a reliable low-frequency label among the low-frequency labels determined based on the description information;

[0282] The credible label and the third prompt word in the first candidate video label are input into the second language model to obtain the second candidate video label and the contradictory label pair, including:

[0283] The high-frequency labels, the credible low-frequency labels and the third prompt word are input into the second largest language model to obtain the second candidate video label and the contradictory label pair.

[0284] In some embodiments, the statistical indicator includes at least one of the following: the number of likes of the comments corresponding to the first candidate video tag; the frequency of the first candidate video tag appearing in the comments related to the video file.

[0285] In some embodiments, the third prompt word includes a classification prompt word, and the classification prompt word is used to indicate the extraction of semantically opposite contradictory label pairs and semantically independent second candidate video labels;

[0286] The credible label and the third prompt word in the first candidate video label are input into the second language model to obtain the second candidate video label and the contradictory label pair, including:

[0287] The credible label and the classification prompt word in the first candidate video label are input into the second largest language model to obtain the second candidate video label and the contradictory label pair.

[0288] In some embodiments, the third prompt word further includes a de-duplication prompt word, and the de-duplication prompt word is used to indicate the credible label of merging semantic repetitions;

[0289] The credible label and classification prompt word in the first candidate video label are input into the second largest language model to obtain the second candidate video label and the contradictory label pair, including:

[0290] Inputting the credible labels and duplicate removal prompt words in the first candidate video labels into the second largest language model to obtain the credible labels after duplicate removal;

[0291] The deduplicated credible labels and classification prompt words are input into the second largest language model to obtain the second candidate video label and the contradictory label pair.

[0292] In some embodiments, the method further comprises:

[0293] The comments related to the video file are input into the comment classification model to obtain the comment types and corresponding confidence scores of the comments related to the video file.

[0294] In some embodiments, the comment type includes at least one of the following: a video evaluation type, a question feedback type, a story impression type, and a low-quality comment type.

[0295] In some embodiments, the method further comprises:

[0296] Filter out comments of the video evaluation type from comments related to the video file;

[0297] Based on the preset comment database, filter the comments of the video evaluation type to obtain the actual candidate comments;

[0298] Among them, the actual candidate comments do not include meaningless comments, the comment database includes multiple meaningless comments, and the semantics of the meaningless comments are irrelevant to the video file.

[0299] In some embodiments, the method further comprises:

[0300] Filter out high-quality candidate reviews based on the review scores of actual candidate reviews;

[0301] The comments related to the video file and the second prompt word are input into the second language model to obtain the first candidate video tag, including:

[0302] The high-quality candidate comments and the second prompt word are input into the second largest language model to obtain the first candidate video label.

[0303] In some embodiments, the review score is related to at least one of the following information:

[0304] The confidence score of the video review type corresponding to the actual candidate comment;

[0305] The number of likes corresponding to the actual candidate comments;

[0306] The number of reference video tags with the same semantics as the actual candidate review. The description information includes the reference video tags.

[0307] In some embodiments, the second prompt word is used to indicate the semantic-based extraction of the first candidate video tag and the tag dimension type corresponding to the first candidate video tag.

[0308] In some embodiments, the tag dimension type includes at least one of the following:

[0309] Places suitable for viewing video files; the region where the video file originates; the instruments involved in the video file; the genre of the video file; the emotions expressed in the video file; the language used in the video file; the group of people the video file is aimed at; the theme of the video file; the characters associated with the video file; and other multimedia files associated with the video file.

[0310] For specific implementation details, please refer to the embodiment in the above audio file scenario, which will not be repeated here.

[0311] Fig.12 A block diagram of a multimedia file tag classification device provided by an exemplary embodiment of the present application is shown. The device can be implemented as a computer device or as a part of a computer device through software or hardware or a combination of both. The device includes:

[0312] An acquisition module 1210 is used to acquire description information of a multimedia file and to acquire a contradictory tag pair of the multimedia file, wherein the description information is used to describe the content of the multimedia file, and the contradictory tag pair includes two multimedia tags with opposite semantics;

[0313] Prediction module 1220, used to call the first language model based on the description information, the contradictory label pair and the first prompt word to predict a correct multimedia label of the multimedia file, wherein the first prompt word is used to indicate the correct multimedia label in the contradictory label pair determined based on the description information;

[0314] The output module 1230 is used to output the multimedia output tag corresponding to the multimedia file based on the correct multimedia tag.

[0315] In a possible design of this embodiment, the acquisition module 1210 is used to obtain description information of a multimedia file in at least two modalities, and the at least two modalities include at least two of the following modalities: text modality, image modality, audio modality, and video modality.

[0316] In a possible design of this embodiment, the prediction module 1220 is used to map the description information of other modalities except the text modality to the text feature space corresponding to the text modality to obtain the feature representation of the description information of other modalities;

[0317] In the case where at least two modalities include a text modality, feature representations of description information of other modalities, description information of the text modality, contradictory label pairs and a first prompt word are input into a first language model to predict a correct multimedia label for the multimedia file.

[0318] In a possible design of this embodiment, the prediction module 1220 is used to map the description information of other modalities except the text modality to the text feature space corresponding to the text modality to obtain the feature representation of the description information of other modalities;

[0319] In the case that at least two modalities do not include a textual modality, feature representations of description information of other modalities, contradictory label pairs, and a first prompt word are input into a first language model to predict a correct multimedia label for the multimedia file.

[0320] In a possible design of this embodiment, the prediction module 1220 is used to encode the description information of other modalities through encoders corresponding to other modalities to obtain original feature representations of the description information of other modalities in feature spaces corresponding to other modalities;

[0321] The original feature representation is mapped from the feature space corresponding to other modalities to the text feature space corresponding to the text modality through the feature mapping network to obtain the feature representation of the descriptive information of other modalities.

[0322] In a possible design of this embodiment, the acquisition module 1210 is further used to input the comments related to the multimedia file and the second prompt word into the second language model to obtain the first candidate multimedia tag, and the second prompt word is used to indicate the extraction of the first candidate multimedia tag based on semantics;

[0323] The acquisition module 1210 is used to input the credible tag and the third prompt word in the first candidate multimedia tag into the second largest language model to obtain the second candidate multimedia tag and the contradictory tag pair, wherein the credible tag includes the multimedia tag obtained by the first largest language model based on the judgment data, the judgment data includes the first candidate multimedia tag and the description information, and the third prompt word is used to indicate the extraction of the second candidate multimedia tag and the contradictory tag pair based on semantics;

[0324] The output module 1230 is used to output a multimedia output tag corresponding to the multimedia file based on the correct multimedia tag and the second candidate multimedia tag.

[0325] In a possible design of this embodiment, the first classification module 1209 is used to classify the first candidate multimedia tags based on the statistical indicators to obtain high-frequency tags and low-frequency tags, where the high-frequency tags are the first candidate multimedia tags whose statistical indicators are higher than the statistical threshold, and the low-frequency tags are the first candidate multimedia tags whose statistical indicators are lower than the statistical threshold;

[0326] An acquisition module 1210 is used to input the low-frequency tags and the judgment prompt words into the first language model to obtain credible low-frequency tags, wherein the judgment prompt words are used to indicate the credible low-frequency tags among the low-frequency tags determined based on the description information;

[0327] The acquisition module 1210 is used to input the high-frequency tag, the credible low-frequency tag and the third prompt word into the second language model to obtain a second candidate multimedia tag and a contradictory tag pair.

[0328] In a possible design of this embodiment, the statistical indicator includes at least one of the following: the number of likes of the comments corresponding to the first candidate multimedia tag; the frequency of the first candidate multimedia tag appearing in the comments related to the multimedia file.

[0329] In a possible design of this embodiment, the third prompt word includes a classification prompt word, and the classification prompt word is used to indicate the extraction of semantically opposite contradictory tag pairs and semantically independent second candidate multimedia tags;

[0330] The acquisition module 1210 is used to input the credible tag and the classification prompt word in the first candidate multimedia tag into the second largest language model to obtain the second candidate multimedia tag and the contradictory tag pair.

[0331] In a possible design of this embodiment, the third prompt word also includes a de-duplication prompt word, which is used to indicate the merged semantically repeated trusted tags; the acquisition module 1210 is used to input the trusted tags and de-duplication prompt words in the first candidate multimedia tags into the second largest language model to obtain the de-duplication trusted tags; the de-duplication trusted tags and classification prompt words are input into the second largest language model to obtain the second candidate multimedia tags and the contradictory tag pair.

[0332] In a possible design of this embodiment, the second classification module 1201 is used to input the comments related to the multimedia file into the comment classification model to obtain the comment type and the corresponding confidence score corresponding to the comments related to the multimedia file.

[0333] In a possible design of this embodiment, the comment type includes at least one of the following: a multimedia evaluation type, a problem feedback type, a story impression type, and a low-quality comment type.

[0334] In a possible design of this embodiment, the screening module 1202 is used to screen out multimedia evaluation type comments from the comments related to the multimedia files;

[0335] A filtering module 1203 is used to filter the comments of the multimedia evaluation type based on a preset comment database to obtain actual candidate comments;

[0336] Among them, the actual candidate comments do not include meaningless comments, the comment database includes multiple meaningless comments, and the semantics of the meaningless comments are irrelevant to the multimedia files.

[0337] In a possible design of this embodiment, the screening module 1202 is used to screen out high-quality candidate reviews based on the review scores of the actual candidate reviews;

[0338] The acquisition module 1210 is used to input the high-quality candidate comments and the second prompt word into the second largest language model to obtain the first candidate multimedia tag.

[0339] In a possible design of this embodiment, the review score is related to at least one of the following information:

[0340] The confidence score of the multimedia review type corresponding to the actual candidate review;

[0341] The number of likes corresponding to the actual candidate comments;

[0342] The number of reference multimedia tags with the same semantics as the actual candidate review. The description information includes the reference multimedia tags.

[0343] In a possible design of this embodiment, the second prompt word is used to indicate the semantic-based extraction of the first candidate multimedia tag and the tag dimension type corresponding to the first candidate multimedia tag.

[0344] In a possible design of this embodiment, the label dimension type includes at least one of the following:

[0345] Places suitable for enjoying multimedia files; the region where the multimedia files originate; the musical instruments involved in the multimedia files; the genres of the multimedia files; the emotions expressed in the multimedia files; the language used in the multimedia files; the groups of people the multimedia files are aimed at; the themes of the multimedia files; the characters associated with the multimedia files; and other multimedia associated with the multimedia files.

[0346] This embodiment is illustrated by taking a second classification module 1201, a screening module 1202, a filtering module 1203, a first classification module 1209, an acquisition module 1210, a prediction module 1220 and an output module 1230 as an example, and the number of the second classification module 1201, the screening module 1202, the filtering module 1203, the first classification module 1209, the acquisition module 1210, the prediction module 1220 and the output module 1230 is not limited.

[0347] For the functional introduction of the second classification module 1201, please refer to Fig.10 The content of step 301 in the embodiment.

[0348] For an introduction to the functions of the screening module 1202, please refer to Fig.10 The contents of step 302 and step 304 in the embodiment.

[0349] For an introduction to the functions of the filtering module 1203, please refer to Fig.10 The content of step 303 in the embodiment.

[0350] For the functional introduction of the first classification module 1209, please refer to Figure 8 The content of step 308 in the embodiment.

[0351] For an introduction to the functions of the acquisition module 1210, please refer to Figure 3 The content of step 310 in the embodiment, Figure 4 The content of step 312 in the embodiment, Figure 7 The contents of step 305 and step 314 in the embodiment are: Figure 8 The contents of step 309 and step 316 in the embodiment are: Fig. 9 The content of step 318 in the embodiment, and Fig.10 The content of step 3051 in the embodiment.

[0352] For an introduction to the functions of the prediction module 1220, please refer to Figure 3 The content of step 320 in the embodiment, Figure 4 The contents of step 322, step 3242 and step 3244 in the embodiment.

[0353] For an introduction to the functions of the output module 1230, please refer to Figure 3 The content of step 330 in the embodiment, Figure 7 The content of step 332 in the embodiment.

[0354] Fig.13The structural block diagram of a computer device 1300 shown in an exemplary embodiment of the present application is shown. The computer device can be used to implement the tag classification method of multimedia files provided in the above embodiments. The computer device 1300 includes a central processing unit (CPU) 1301, a system memory 1304 including a random access memory (RAM) 1302 and a read-only memory (ROM) 1303, and a system bus 1305 connecting the system memory 1304 and the central processing unit 1301. The computer device 1300 also includes an input / output (I / O) system 1306 that helps transmit information between various devices in the computer device, and a large-capacity storage device 1307 for storing an operating system 1313, an application program 1314, and other program modules 1315.

[0355] The input / output system 1306 includes a display 1308 for displaying information and an input device 1309 such as a mouse and a keyboard for user inputting information. The display 1308 and the input device 1309 are connected to the central processing unit 1301 through an input / output controller 1310 connected to the system bus 1305. The input / output system 1306 may also include an input / output controller 1310 for receiving and processing inputs from a plurality of other devices such as a keyboard, a mouse, or an electronic stylus. Similarly, the input / output controller 1310 also provides output to a display screen, a printer, or other types of output devices.

[0356] The mass storage device 1307 is connected to the central processing unit 1301 through a mass storage controller (not shown) connected to the system bus 1305. The mass storage device 1307 and its associated computer-readable storage medium provide non-volatile storage for the terminal device 1300. That is, the mass storage device 1307 may include a computer-readable storage medium (not shown) such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0357] Without loss of generality, computer-readable storage media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable storage instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, Erasable Programmable Read Only Memory (EPROM), Electronically-Erasable Programmable Read-Only Memory (EEPROM), flash memory or other solid-state storage technology, CD-ROM, Digital Versatile Disc (DVD) or other optical storage, cassettes, tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that computer storage media are not limited to the above. The above-mentioned system memory 1304 and mass storage device 1307 can be collectively referred to as memory.

[0358] The memory stores one or more programs, and the one or more programs are configured to be executed by one or more central processing units 1301. The one or more programs contain instructions for implementing the above-mentioned method embodiments. The central processing unit 1301 executes the one or more programs to implement the label classification method of multimedia files provided by the above-mentioned method embodiments.

[0359] According to various embodiments of the present application, the computer device 1300 can also be connected to a remote terminal device on a network through a network such as the Internet. That is, the computer device 1300 can be connected to the network 1312 through a network interface unit 1311 connected to the system bus 1305, or the network interface unit 1311 can be used to connect to other types of networks or remote terminal device systems (not shown).

[0360] The memory also includes one or more programs, one or more programs are stored in the memory, and one or more programs include steps executed by the terminal device in the method provided in the embodiment of the present application.

[0361] An embodiment of the present application also provides a computer device, which includes: a processor and a memory, wherein at least one program is stored in the memory; the processor is used to execute at least one program in the memory to implement the label classification method for multimedia files provided by the above-mentioned method embodiments.

[0362] An embodiment of the present application also provides a computer-readable storage medium, in which at least one program is stored. The at least one program is loaded and executed by a processor to implement the label classification method for multimedia files provided by the above-mentioned method embodiments.

[0363] An embodiment of the present application also provides a computer program product, which includes computer instructions, the computer instructions are stored in a computer-readable storage medium, a processor obtains the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to implement the label classification method of multimedia files provided in the above-mentioned method embodiments.

[0364] It should be understood that the "plurality" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0365] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented with hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable storage medium or transmitted as one or more instructions or codes on a computer-readable storage medium. Computer-readable storage media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access.

[0366] The above are only optional embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for label classification of multimedia files, characterized in that: The method comprises: Acquire description information of the multimedia file, and acquire a contradictory tag pair of the multimedia file, wherein the description information is used to describe the content of the multimedia file, and the contradictory tag pair includes two multimedia tags with opposite semantics; Based on the description information, the contradictory label pair and a first prompt word, a first language model is called to predict a correct multimedia label for the multimedia file, wherein the first prompt word is used to indicate the correct multimedia label in the contradictory label pair determined based on the description information; Based on the correct multimedia tag, a multimedia output tag corresponding to the multimedia file is output.

2. The method according to claim 1, characterized in that The obtaining the description information of the multimedia file includes: Descriptive information of the multimedia file in at least two modes is obtained, where the at least two modes include at least two of the following modes: text mode, image mode, audio mode, and video mode.

3. The method according to claim 2, characterized in that The calling of the first language model based on the description information, the contradictory tag pair and the first prompt word to predict a correct multimedia tag of the multimedia file includes: Mapping description information of other modalities except the text modality into a text feature space corresponding to the text modality to obtain a feature representation of the description information of the other modalities; In a case where the at least two modalities include the text modality, feature representations of the description information of the other modalities, the description information of the text modality, the contradictory label pairs and the first prompt word are input into the first large language model to predict a correct multimedia label for the multimedia file.

4. The method according to claim 2, characterized in that: The calling of the first language model based on the description information, the contradictory tag pair and the first prompt word to predict a correct multimedia tag of the multimedia file includes: Mapping description information of other modalities except the text modality into a text feature space corresponding to the text modality to obtain a feature representation of the description information of the other modalities; In a case where the at least two modalities do not include the text modality, the feature representation of the description information of the other modalities, the contradictory label pair and the first prompt word are input into the first large language model to predict a correct multimedia label for the multimedia file.

5. The method according to claim 3 or 4, characterized in that: The mapping of the description information of other modalities except the text modality to the text feature space corresponding to the text modality to obtain the feature representation of the description information of the other modalities includes: Encoding the description information of the other modality through an encoder corresponding to the other modality to obtain an original feature representation of the description information of the other modality in a feature space corresponding to the other modality; The original feature representation is mapped from the feature space corresponding to the other modalities to the text feature space corresponding to the text modality through a feature mapping network to obtain a feature representation of the description information of the other modalities.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: Inputting the comments and second prompt words related to the multimedia file into the second largest language model to obtain a first candidate multimedia tag, wherein the second prompt word is used to indicate that the first candidate multimedia tag is extracted based on semantics; The obtaining of the contradictory tag pairs of the multimedia file includes: Inputting the credible tag and the third prompt word in the first candidate multimedia tag into the second large language model to obtain the second candidate multimedia tag and the contradictory tag pair, wherein the credible tag includes the multimedia tag obtained by the first large language model based on judgment data, the judgment data includes the first candidate multimedia tag and the description information, and the third prompt word is used to indicate that the second candidate multimedia tag and the contradictory tag pair are extracted based on semantics; The step of outputting a multimedia output tag corresponding to the multimedia file based on the correct multimedia tag includes: Based on the correct multimedia tag and the second candidate multimedia tag, a multimedia output tag corresponding to the multimedia file is output.

7. The method according to claim 6, characterized in that The method further comprises: Based on the statistical index, classify the first candidate multimedia tags to obtain high-frequency tags and low-frequency tags, wherein the high-frequency tags are first candidate multimedia tags whose statistical index is higher than a statistical threshold, and the low-frequency tags are first candidate multimedia tags whose statistical index is lower than the statistical threshold; Inputting the low-frequency label and the judgment prompt word into the first large language model to obtain a credible low-frequency label, wherein the judgment prompt word is used to indicate the credible low-frequency label among the low-frequency labels determined based on the description information; The step of inputting the credible tag and the third prompt word in the first candidate multimedia tag into the second language model to obtain the second candidate multimedia tag and the contradictory tag pair includes: The high-frequency tag, the credible low-frequency tag, and the third prompt word are input into the second large language model to obtain the second candidate multimedia tag and the contradictory tag pair.

8. The method according to claim 7, characterized in that The statistical indicator includes at least one of the following: the number of likes of the comments corresponding to the first candidate multimedia tag; the frequency of the first candidate multimedia tag appearing in the comments related to the multimedia file.

9. The method according to claim 6, characterized in that The third prompt word includes a classification prompt word, and the classification prompt word is used to indicate the extraction of the contradictory tag pair with opposite semantics and the second candidate multimedia tag that is independent in semantics; The step of inputting the credible tag and the third prompt word in the first candidate multimedia tag into the second language model to obtain the second candidate multimedia tag and the contradictory tag pair includes: The credible tag in the first candidate multimedia tag and the classification prompt word are input into the second large language model to obtain the second candidate multimedia tag and the contradictory tag pair.

10. The method according to claim 9, characterized in that The third prompt word also includes a de-duplication prompt word, and the de-duplication prompt word is used to indicate the credible label of merging semantic repetitions; The step of inputting the credible tag in the first candidate multimedia tag and the classification prompt word into the second language model to obtain the second candidate multimedia tag and the contradictory tag pair includes: Inputting the credible tag and the duplicate removal prompt word in the first candidate multimedia tag into the second largest language model to obtain a credible tag after duplicate removal; The deduplicated credible tags and the classification prompt words are input into the second language model to obtain the second candidate multimedia tags and the contradictory tag pair.

11. The method according to any one of claims 6 to 10, characterized in that: The method further comprises: The comments related to the multimedia file are input into a comment classification model to obtain comment types and corresponding confidence scores corresponding to the comments related to the multimedia file.

12. The method according to claim 11, characterized in that The comment type includes at least one of the following: a multimedia evaluation type, a question feedback type, a story impression type, and a low-quality comment type.

13. The method according to claim 12, characterized in that The method further comprises: Filtering comments of the multimedia evaluation type from comments related to the multimedia file; Based on a preset comment database, filtering comments of the multimedia evaluation type to obtain actual candidate comments; The actual candidate comments do not include meaningless comments, the comment database includes a plurality of the meaningless comments, and the semantics of the meaningless comments are irrelevant to the multimedia file.

14. The method according to claim 13, characterized in that The method further comprises: Filter out high-quality candidate reviews based on the review scores of the actual candidate reviews; The step of inputting the comments related to the multimedia file and the second prompt word into the second language model to obtain the first candidate multimedia tag includes: The high-quality candidate comment and the second prompt word are input into the second language model to obtain the first candidate multimedia tag.

15. The method according to claim 14, characterized in that The review score is related to at least one of the following information: A confidence score of the multimedia review type corresponding to the actual candidate comment; The number of likes corresponding to the actual candidate comment; The number of reference multimedia tags having the same semantics as the actual candidate comment, the description information including the reference multimedia tags.

16. The method according to any one of claims 6 to 15, characterized in that: The second prompt word is used to instruct the semantic-based extraction of the first candidate multimedia tag and the tag dimension type corresponding to the first candidate multimedia tag.

17. The method according to claim 16, characterized in that The label dimension type includes at least one of the following: A place suitable for enjoying the multimedia file; the region where the multimedia file originates; the musical instruments involved in the multimedia file; the genre of the multimedia file; the emotions expressed by the multimedia file; the language used in the multimedia file; the group of people the multimedia file is aimed at; the theme of the multimedia file; the characters associated with the multimedia file; and other multimedia associated with the multimedia file.

18. A label classification device for multimedia files, characterized in that: The device comprises: an acquisition module, used to acquire description information of the multimedia file and to acquire a contradictory tag pair of the multimedia file, wherein the description information is used to describe the content of the multimedia file, and the contradictory tag pair includes two multimedia tags with opposite semantics; A prediction module, configured to call a first language model based on the description information, the contradictory label pair and a first prompt word to predict a correct multimedia label for the multimedia file, wherein the first prompt word is used to indicate the correct multimedia label in the contradictory label pair determined based on the description information; The output module is used to output the multimedia output tag corresponding to the multimedia file based on the correct multimedia tag.

19. A computer device, characterized in that: The computer device includes: a processor and a memory, wherein the memory stores at least one program; the processor is used to execute the at least one program in the memory to implement the tag classification method for multimedia files as described in any one of claims 1 to 17.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one program, and the at least one program is loaded and executed by the processor to implement the tag classification method for multimedia files as described in any one of claims 1 to 17.

21. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium. The processor obtains the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to implement the tag classification method for multimedia files as described in any one of claims 1 to 17.

Images