Multimedia information topic analysis method, device, equipment and storage medium

Abstract

The present disclosure relates to a method, device, equipment and computer-readable storage medium for analyzing topics of multimedia information. The present disclosure obtains speech recognition text through audio data and obtains subtitle text through key frames of video data; extracts topic information from three aspects of entity, keyword and semantic tag for the speech recognition text and subtitle text, thereby realizing comprehensive and multi-granular text topic extraction based on text data; extracts topic information from two aspects of face tag and picture tag for the key frames of video data, thereby realizing visual topic extraction based on image data; the embodiments of the present disclosure fully consider the multimodal characteristics of video data, comprehensively analyze video topics, and improve the accuracy of topic analysis; further, through accurate topic analysis, the audience can quickly and effectively obtain the main information of the video, thereby improving work efficiency, and can be widely used in scenarios such as personalized video recommendation and video content retrieval.

Description

Technical Field

[0001] The present disclosure relates to the fields of deep learning, computer vision and natural language processing, and in particular to a method, device, equipment and computer-readable storage medium for analyzing multimedia information topics. Background Art

[0002] In the Internet environment, multimedia information is becoming more and more important in daily life. Accurate analysis of the topics of multimedia information is crucial for multimedia information screening, reading, multimedia content retrieval, and multimedia personalized recommendation. As video is an important form of multimedia information, it is particularly important to accurately analyze the topic of the video.

[0003] Video is a typical multimodal combination type, which includes different modal features such as text, image, audio, etc. However, in the prior art, the topic analysis technology for videos usually directly uses the video title and similar key frames of the video to analyze the video topic; due to the multimodal characteristics of the video, the topic analysis is only performed based on the video title and similar key frames of the video, resulting in incomplete topic analysis and low accuracy of topic analysis results, which further leads to low accuracy in video screening, reading, video content retrieval, and personalized video recommendation. Summary of the invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a multimedia information topic analysis method, device, equipment and computer-readable storage medium, which fully consider the multimodal characteristics of video data, comprehensively analyze video topics, and improve the accuracy of topic analysis; further, through accurate topic analysis, the audience can quickly and effectively obtain the main information of the video, improve work efficiency, and can be widely used in scenarios such as personalized video recommendation and video content retrieval.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for analyzing a topic of multimedia information, including:

[0006] Extracting audio data and video data from multimedia information;

[0007] Converting the audio data into first text information, and obtaining speech recognition text according to the first text information;

[0008] Extracting voice information from the audio data to obtain voice recognition text;

[0009] Extracting subtitle information from key frames of the video data to obtain subtitle text;

[0010] Extracting entities and keywords from the speech recognition text and the subtitle text, and determining semantic tags of the speech recognition text and the subtitle text;

[0011] Extracting face tags in the key frame and determining image tags of the key frame;

[0012] A target topic of the multimedia information is determined according to the entity, the keyword, the semantic tag, the face tag, and the picture tag.

[0013] In a second aspect, an embodiment of the present disclosure provides a multimedia information topic analysis device, including:

[0014] A first extraction module, used for extracting audio data and video data from multimedia information;

[0015] A second extraction module, used to extract voice information from the audio data to obtain a voice recognition text;

[0016] A third extraction module, used to extract subtitle information from the key frames of the video data to obtain subtitle text;

[0017] A fourth extraction module, used to extract entities and keywords from the speech recognition text and the subtitle text;

[0018] A first determination module, used to determine semantic tags of the speech recognition text and the subtitle text;

[0019] A fifth extraction module, used to extract face tags from the key frame;

[0020] A second determination module, used to determine the image label of the key frame;

[0021] The third determination module is used to determine the target topic of the multimedia information according to the entity, the keyword, the semantic tag, the face tag and the picture tag.

[0022] In a third aspect, an embodiment of the present disclosure provides an electronic device, including:

[0023] Memory;

[0024] Processor; and

[0025] Computer programs;

[0026] The computer program is stored in the memory and is configured to be executed by the processor to implement the method as described in the first aspect.

[0027] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the method described in the first aspect.

[0028] In a fifth aspect, an embodiment of the present disclosure further provides a computer program product, which includes a computer program or instructions, and when the computer program or instructions are executed by a processor, the multimedia information topic analysis method as described above is implemented.

[0029] The multimedia information topic analysis method, device, equipment and computer-readable storage medium provided by the embodiments of the present disclosure obtain speech recognition text through audio data and obtain subtitle text through key frames of video data; extract topic information from three aspects of entity, keyword and semantic label for the above-mentioned speech recognition text and subtitle text, thereby realizing comprehensive and multi-granular text topic extraction based on text data; extract topic information from two aspects of face label and picture label for the key frames of video data, thereby realizing visual topic extraction based on image data; the embodiments of the present disclosure fully consider the multimodal information of video data, comprehensively analyze video topics, and improve the accuracy of topic analysis; further, through accurate topic analysis, the audience can quickly and effectively obtain the main information of the video, thereby improving work efficiency, and can be widely used in scenarios such as personalized video recommendation and video content retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0032] Figure 1 A flowchart of a multimedia information topic analysis method provided by an embodiment of the present disclosure;

[0033] Figure 2 A flowchart of a multimedia information topic analysis method provided by an embodiment of the present disclosure;

[0034] Figure 3 A flowchart for extracting topic information based on text data provided by an embodiment of the present disclosure;

[0035] Figure 4 A flowchart for extracting topic information based on image data provided by an embodiment of the present disclosure;

[0036] Figure 5 A flowchart of another multimedia information topic analysis method provided by an embodiment of the present disclosure;

[0037] Figure 6 A schematic diagram of the structure of a multimedia information topic analysis device provided by an embodiment of the present disclosure;

[0038] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0039] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0040] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0041] In the Internet environment, multimedia information is becoming more and more important in daily life. Accurate analysis of the topics of multimedia information is crucial for multimedia information screening, reading, multimedia content retrieval, and multimedia personalized recommendation. As video is an important form of multimedia information, it is particularly important to accurately analyze the topic of the video.

[0042] Normally, most of the topic analysis technologies for videos directly use video titles or similar key frames of videos to analyze video topics. However, due to the multimodal characteristics of videos, topic analysis is only performed based on video titles and similar key frames of videos, resulting in incomplete topic analysis and low accuracy of topic analysis results, which further leads to low accuracy in video screening, reading, video content retrieval, and personalized video recommendations.

[0043] To address this problem, an embodiment of the present disclosure provides a method for analyzing topics in multimedia information, which will be introduced below in conjunction with a specific embodiment.

[0044] Figure 1 A flowchart of a multimedia information topic analysis method provided by an embodiment of the present disclosure; Figure 2A flowchart for implementing a multimedia information topic analysis method provided in an embodiment of the present disclosure; the multimedia information topic analysis method can be executed by a multimedia information topic analysis device, which can be implemented using software and / or hardware and can be integrated on any electronic device with computing capabilities, such as a terminal, specifically including a smart phone, a PDA, a tablet computer, a wearable device with a display screen, a laptop computer, etc.

[0045] At present, video has become a very important form of multimedia information. The multimedia information topic analysis method provided in this embodiment is suitable for multimedia information displayed in the form of video. Therefore, the multimedia information topic analysis method is introduced from the perspective of video below.

[0046] like Figure 1 As shown, the multimedia information topic analysis method provided by the embodiment of the present disclosure includes the following steps:

[0047] S101. Extracting audio data and video data from multimedia information.

[0048] Specifically, in this embodiment, extracting audio data and video data from multimedia information refers to separating the multimedia information into two parts: audio data and video data. The audio data includes the voice content in the multimedia information, and the video data includes all video data except the audio data.

[0049] It should be noted that any available method, tool or device may be used to separate multimedia data, and the embodiments of the present disclosure are not limited thereto.

[0050] S102: Extract voice information from the audio data to obtain voice recognition text.

[0051] Specifically, the main purpose of this step is to obtain speech recognition text. Speech information is extracted from audio data through automatic speech recognition technology (Automatic Speech Recognition, ASR) to obtain speech recognition information.

[0052] Exemplarily, in some embodiments, extracting voice information from separated audio data to obtain speech recognition text mainly includes the following processes: feature extraction, acoustic model, language model, dictionary and decoding. Specifically, the audio data is essentially a set of sound signals, and the features of the above sound signals are extracted, and the sound signals are converted from the time domain to the frequency domain to provide suitable feature vectors for the acoustic model; the acoustic model then calculates the score of each feature vector on the acoustic feature based on the acoustic feature; then, the language model calculates the probability of the possible phrase sequence corresponding to the sound signal based on the relevant linguistics theory; finally, the phrase sequence is decoded according to the existing dictionary to obtain the speech recognition text.

[0053] It should be noted that the above-mentioned method of extracting voice information from audio data to obtain voice recognition text is only a feasible technical solution, and this application does not limit the method of extracting voice information from audio data to obtain voice recognition text.

[0054] S103: Extract subtitle information from the key frames of the video data to obtain subtitle text.

[0055] Specifically, in this embodiment, key frames in the video data are extracted to obtain the minimum unit of video processing, and optical character recognition (OCR) is used to recognize text information in the key frames.

[0056] Optionally, a text recognition model may be used to recognize text information in a key frame, and a Paddle OCR model may be used as an example.

[0057] Because the video title in the video can best reflect the topic of the video. For example, in the video of the news broadcast, the video title will be displayed on the video page to summarize the core content of the news. Therefore, the subtitle text in the video title plays an important role in video topic analysis.

[0058] Therefore, in this embodiment, when using the text recognition model, what needs to be obtained is the subtitle text of the video title in the video data; therefore, in this embodiment, when using the Paddle OCR text recognition model, the input is the key frame, and the output is the subtitle text.

[0059] In order to avoid interference from areas unrelated to the subtitle text of the video title, it is necessary to add a mask to the key frame extracted from the video data before using the Paddle OCR text recognition model for text recognition. In this embodiment, the role of the mask is to shield the areas on the key frame that are unrelated to the subtitle text of the video title so that they do not participate in text recognition; then the Paddle OCR text recognition model is used to perform text recognition on the unshielded areas to obtain the subtitle text.

[0060] S104: extracting entities and keywords from the speech recognition text and the subtitle text, and determining semantic tags of the speech recognition text and the subtitle text.

[0061] Specifically, in this embodiment, topic information is obtained by extracting entities from speech recognition text and subtitle text, extracting keywords from speech recognition text and subtitle text, and determining semantic tags of speech recognition text and subtitle text. The flowchart of this step is as follows: Figure 3 As shown below, combined Figure 3 For a detailed introduction:

[0062] First, in the field of natural language, entities are structured attributes in unstructured text data. Named Entity Recognition (NER) is used to obtain entities such as names of people, places, institutions, and time. From a semantic point of view, entities such as specific time, specific place, designated person or institution have a limiting effect on the text content. Introducing this feature in topic detection can improve the accuracy of topic analysis.

[0063] Specifically, in this embodiment, obtaining the entity can be achieved in the following ways:

[0064] First, the speech recognition text and subtitle text are preprocessed by sentence segmentation, word segmentation, stop word removal and part-of-speech tagging. Figure 3 Text preprocessing in .

[0065] Among them, sentence segmentation and word segmentation refer to dividing the above-mentioned speech recognition text and subtitle text into separate sentences, separate words, etc.;

[0066] Stop word removal refers to removing the characters or words that need to be automatically filtered out from the speech recognition text and subtitle text. Removing stop words can not only improve the efficiency of text data processing, but also save storage space, and at the same time improve the accuracy of topic analysis.

[0067] Part-of-speech tagging refers to tagging the parts of speech of characters or words in speech recognition text and subtitle text, where the parts of speech include but are not limited to nouns, verbs, adjectives, etc.

[0068] Through this step, a word segmentation set with stop words removed and parts of speech marked is obtained.

[0069] Exemplarily, the sentence segmentation, word segmentation, and part-of-speech tagging of the speech recognition text and the subtitle text can be implemented using the Chinese word segmentation open source library jieba. The removal of stop words from the speech recognition text and the subtitle text can be matched with the speech recognition text and the subtitle text using an existing stop word list. If a character or word in the speech recognition text and the subtitle text is in the stop word list, the character or word is removed. Among them, the stop word list is, for example, a Baidu stop word list, or a custom stop word list, which is not limited in the embodiments of the present disclosure.

[0070] Secondly, based on the speech recognition text and subtitle text processed as above, words with the part of speech being noun are screened to obtain a noun segmentation set.

[0071] Because most entities are nouns, in this embodiment, characters or words with a noun part of speech are selected to obtain a word segment set with a noun part of speech. In some other embodiments, words with other parts of speech can also be selected, which is not limited in the present embodiment.

[0072] Then, identify entities such as names of people, places, organizations, and time, that is, Figure 3 Entity recognition in .

[0073] Optionally, a deep learning model may be used to extract entities in the speech recognition text and the subtitle text.

[0074] For example, the long short-term memory artificial neural network model (LSTM) and conditional random field (CRF) can be used to identify entities such as names of people, places, organizations, time, etc. in the noun segmentation set. In this embodiment, the above two models are used, the input is the segmentation sequence, and the output is the entity recognition sequence.

[0075] Finally, the entity recognition sequence output above is subjected to entity filtering and entity deduplication to filter out entities that are not of interest and duplicate entities, that is, Figure 3 Filter the entities shown to obtain the required entities.

[0076] Secondly, keywords, as important words representing a text, express the main idea of ​​the entire text to a certain extent. Therefore, extracting keywords from the text is equally important for topic analysis.

[0077] Specifically, in this embodiment, extracting keywords from the speech recognition text and the subtitle text can be achieved in the following ways:

[0078] First, the speech recognition text and subtitle text are processed by sentence segmentation, word segmentation, and stop word removal. Figure 3 The text preprocessing shown in the figure is the same as the sentence segmentation, word segmentation and stop word removal in the above entity, which will not be repeated here.

[0079] Through this step, a word segmentation set with stop words removed is obtained.

[0080] Secondly, extract the words that appear more frequently and are uncommon in the text set from the above word segmentation set without stop words as keywords, that is, Figure 3 The keyword extraction shown.

[0081] Optionally, a keyword extraction model may be used to extract keywords from the speech recognition text and the subtitle text.

[0082] Exemplarily, a keyword extraction model (Term Frequency–Inverse Document Frequency, TF-IDF) can be used. The main logic of the TF-IDF model is: if a certain word appears frequently in a text and rarely appears in other articles, it is considered that the word has a good ability to distinguish this document; if a certain word has a high word frequency in a text and a low file frequency in the entire file collection, the word can generate a high weight. The above model is applied to this embodiment, and the input of the model is a word set with stop words removed, and the output is keywords and the weights of each keyword.

[0083] Optionally, the keywords output above can be sorted according to weights, i.e. Figure 3 The keyword ranking shown in the figure shows that the higher the ranking is, the more important it is. The top K keywords can be selected as the final keywords as needed.

[0084] Thirdly, since the above-mentioned entities and keywords are partial contents extracted from the text and lack global information, fusing the speech recognition text and subtitle text and determining the semantic labels of the fused speech recognition text and subtitle text is to analyze the video topic from a global perspective.

[0085] In this embodiment, determining the semantic labels of the speech recognition text and the subtitle text may specifically be determining the primary label and the secondary label corresponding to the speech recognition text and the subtitle text. Specifically, determining the semantic labels of the fused speech recognition text and the subtitle text may be achieved in the following manner:

[0086] First, the primary tags of the speech recognition text and the subtitle text are determined. For example, the primary tags include: finance, military, technology, sports, entertainment, etc.

[0087] Optionally, a multi-classification model may be used to determine primary labels for the speech recognition text and the subtitle text.

[0088] Exemplarily, the multi-classification model can use a pre-trained model (Bidirectional Encoder Representations from Transformers, BERT). Before using the BERT model, the model needs to be trained. The classification data in the training phase can use the currently available Chinese text classification dataset or a custom Chinese text classification dataset.

[0089] After the model training is completed, it is applied in this embodiment. The input of the model is the speech recognition text and the subtitle text, and the output is the primary label of the speech recognition text and the subtitle text.

[0090] Secondly, determine the secondary tags of the speech recognition text and the subtitle text. Before determining the secondary tags, it is necessary to pre-set the secondary tags under each primary tag. For example, for the sports in the above primary tag, the corresponding secondary tags include: basketball, football, sports stars, etc.

[0091] Optionally, for the secondary labels, a multi-label classification model may be used to determine the secondary labels of the speech recognition text and the subtitle text.

[0092] For example, the multi-label classification model can use the BERT model. Before using the model, the model also needs to be trained, and the classification data used in the training phase uses a publicly available or customized Chinese multi-label classification dataset.

[0093] After the model training is completed, it is applied in this embodiment. The input of the model is the speech recognition text and the subtitle text, and the output is multiple secondary labels of the speech recognition text and the subtitle text.

[0094] Based on the above implementation scheme, a first-level label and multiple second-level labels for speech recognition text and subtitle text can be determined, namely Figure 3 The semantic tags shown.

[0095] It should be noted that when the video contains only speech recognition text but no subtitle text, it is only necessary to determine the primary and secondary tags corresponding to the speech recognition text. When the video contains only subtitle text but no speech recognition text, it is only necessary to determine the primary and secondary tags corresponding to the subtitle text.

[0096] It can be understood that since both the speech recognition text and the subtitle text belong to text data, this step realizes the extraction of topic information based on text data, that is, Figure 3 The text topic information based on the text data is shown.

[0097] S105: extracting face tags from key frames and determining image tags of key frames.

[0098] Specifically, for a video, in addition to speech recognition text and subtitle text, it also includes key frame images. The key frame images have a greater information capacity than this article to a certain extent and are more representative of real topics.

[0099] For the key frame images included in the video, in this embodiment, topic information is obtained through two aspects: face tags and picture tags. The flowchart of this step is as follows: Figure 4 As shown below, combined Figure 4 Give a detailed introduction.

[0100] In this embodiment, the face label can specifically be the face name, and for example, the face label is Zhang San, Li Si, etc.; the picture label can specifically be the classification label obtained after classifying the picture, and for example, the picture label is people, animals, plants, etc.

[0101] First, for face labeling, it can be achieved in the following ways.

[0102] Specifically, face label extraction is divided into several steps: face detection, face feature extraction, and feature comparison.

[0103] First, facial key points are detected from the key frame data, that is, Figure 4 Face detection shown;

[0104] Exemplarily, a face detection model (Multi Task Cascaded Convolutional Networks, MTCNN) can be used for facial key point detection. The input of the model is key frame data, and the output is the position information of facial key points.

[0105] Secondly, based on the facial key point detection results, the facial features contained in the corresponding positions are extracted, that is, Figure 4 Feature extraction shown.

[0106] Exemplarily, the facial features contained in the corresponding positions can be extracted using the facial feature extraction model FaceNet, the input of the model is the position information of the above-mentioned facial key points, and the output is the facial feature vector corresponding to the above-mentioned facial key points.

[0107] Finally, the face feature vector extracted from the key frame is compared with the face feature vector in the face feature library one by one to obtain the face label extraction result, that is, Figure 4 Feature comparison shown.

[0108] It is understandable that before comparing the facial feature vectors extracted from the key frame with the facial feature vectors in the facial feature library one by one, it is necessary to pre-build the facial feature library. The facial feature library is derived from facial images with different postures, lighting, and expressions included in the facial library, and the facial library can be built according to the business scenario.

[0109] Specifically, after establishing the face library, the facial images in the face library are used with the face feature extraction model FaceNet to obtain the face feature vectors in the face feature library, the face feature vectors extracted from the key frame are compared with the face feature vectors in the face feature library one by one, and the face labels are extracted. For example, according to the model, the name of the person in the key frame is Zhang San.

[0110] Secondly, the image labeling of key frames can be achieved in the following way.

[0111] Optionally, the key frames may be classified using a picture classification model to determine the picture labels. Exemplarily, the picture classification model is a ResNet model, i.e. Figure 4 The picture categories shown.

[0112] Optionally, before using the image classification model ResNet, the image classification model is pre-trained on the large-scale image dataset ImageNet, that is, Figure 4 The model is trained as shown. After the training is completed, it is applied in this example. The input of the model is the key frame and the output is the image label. Exemplarily, the image label of the key frame obtained according to the model is an animal, etc.

[0113] It can be understood that since the keyframes belong to image data, this step realizes the visual topic extraction based on image data, that is, Figure 4 The visual topic information based on the image data is shown.

[0114] S106: Determine the target topic of the multimedia information based on the entities, keywords, semantic tags, face tags, and image tags.

[0115] Specifically, for the speech recognition text and subtitle text in the video, topic information is extracted from three aspects: entities, keywords, and semantic tags to obtain multiple topics; for the key frames contained in the video, topic information is extracted from two aspects: face tags and image tags to obtain multiple topics; the topics obtained above are used as target topics of multimedia information.

[0116] The disclosed embodiment obtains speech recognition text through audio data and obtains subtitle text through key frames of video data; extracts topic information from three aspects, namely entities, keywords and semantic tags, for the speech recognition text and subtitle text, thereby realizing comprehensive and multi-granular text topic extraction based on text data; extracts topic information from two aspects, namely face tags and picture tags, for key frames in video data, thereby realizing visual topic extraction based on image data; the disclosed embodiment fully considers the multimodal characteristics of video data, comprehensively analyzes video topics, and improves the accuracy of topic analysis; further, through accurate topic analysis, the audience can quickly and effectively obtain the main information of the video, thereby improving work efficiency, and can be widely used in scenarios such as personalized video recommendation and video content retrieval.

[0117] Figure 5 Another multimedia new topic analysis method provided by the embodiment of the present disclosure includes the following steps:

[0118] S501: Extract audio data and video data from multimedia information.

[0119] In this embodiment, this step is the same as S101 and will not be described again.

[0120] S502: extract first text information from the voice information from the audio data, perform error correction on the first text information, and obtain a voice recognition text based on the first text information after the error correction.

[0121] Specifically, in this embodiment, first, voice information is extracted from the separated audio data to obtain the first text information. The extraction of voice information from the separated audio data to obtain the first text information can be implemented in the implementation method described in S101.

[0122] Secondly, error correction is performed on the first text information, and the first text information after the error correction is used as the speech recognition text.

[0123] Specifically, error correction processing refers to correcting errors in similar words, similar pronunciations, idiom usage, collocation of quantifiers, grammar, etc. in the first text information. Exemplarily, the first text information can be matched with a pre-collected vocabulary, and the characters, words or sentences that do not match the vocabulary can be corrected, so that the corrected first text information is used as the speech recognition text.

[0124] S503: extracting second text information from the key frame of the video data, performing error correction processing on the second text information; and obtaining subtitle text based on the second text information after the error correction processing.

[0125] Specifically, in this embodiment, first, the second text information is obtained from the key frame of the separated video data. The second text information is obtained from the key frame of the separated video data by using the implementation method described in S103.

[0126] Secondly, error correction processing is performed on the second text information, and the second text information after error correction processing is used as the subtitle text. The error correction processing of the second text information can be performed in the same manner as described in S202, which will not be described in detail here.

[0127] S504: extracting entities and keywords from the speech recognition text and the subtitle text, and determining semantic tags of the speech recognition text and the subtitle text.

[0128] In this embodiment, this step is the same as S104 and will not be described again.

[0129] S505: extracting face tags from key frames and determining image tags of key frames.

[0130] In this embodiment, this step is the same as S105 and will not be described in detail here.

[0131] S506, merge the entity, keyword and semantic tag to obtain merged text topic information; merge the face tag and the picture tag to obtain merged visual topic information; determine multiple topics appearing in the multimedia information based on the merged text topic information and the merged visual topic information. Optionally, use the picture classification model to determine the picture tag of the key frame; remove the duplicates of the face tag and the picture tag of the key frame and merge them to obtain the merged visual topic information.

[0132] Specifically, in this embodiment, because the video key frames contain many similar key frames, the face label extraction results obtained in S205 contain repeated content. Similarly, the key frame image label extraction results will also contain repeated content. Therefore, it is necessary to deduplicate the face label extraction results and the key frame image label extraction results respectively, and then merge the deduplicated face labels and image labels. For speech recognition text and subtitle text, text topic information is extracted from three aspects: entities, keywords, and semantic tags. By merging the above entities, keywords, and semantic tags, multiple text topic information can be obtained; for the key frames contained in the video, visual topic information is extracted from two aspects: face labels and image labels. By merging the above face labels and image labels, multiple visual topic information can be obtained. By merging the above text topic information and visual topic information, multiple topics can be determined.

[0133] For example, from a video, entities including roses, festivals, etc. are extracted; keywords include plants, roses, red, etc.; semantic tags include first-level tags such as plants and second-level tags such as roses, etc. are extracted, and entities, keywords, and semantic tags are merged to obtain multiple text topic information; face tags include Zhang San; picture tags include roses, lilies, etc., and face tags and picture tags are merged to obtain multiple visual topic information. Based on the above multiple text topic information and multiple visual topic information, it can be determined that the topics of the video include plants, roses, etc.

[0134] S507, determining a weight value of each topic according to the source of each topic in the multiple topics; determining a target topic of the multimedia information according to the weight value of each topic, the target topic being a topic whose weight value meets a preset condition.

[0135] Specifically, in this embodiment, the origin of each topic refers to whether the source of each topic is text data or image data, or is derived from both text data and image data.

[0136] In this embodiment, the weight value of each topic is determined according to the source of each topic.

[0137] Optionally, if the topic appears in text data, the weight value of the topic is determined to be a first weight value, and the text data includes speech recognition text and / or subtitle text; exemplarily, the first weight value is 0.4.

[0138] If the topic appears in the image data, the weight value of the topic is determined to be a second weight value, and the image data includes a key frame; illustratively, the second weight value is 0.6

[0139] If the topic appears in both the text data and the image data, the weight value of the topic is determined to be a third weight value, where the third weight value is the sum of the first weight value and the second weight value.

[0140] Then, based on the weight of each topic, when the weight value of a topic meets a preset condition, the topic corresponding to the weight value is the target topic.

[0141] Exemplarily, for the weight value of each topic, the preset condition may be to use a topic with a topic weight value greater than or equal to 0.6 as a target topic.

[0142] For example, the topics may be arranged in descending order according to the weight values, and the preset condition may be that after the topic weights are arranged in descending order, the first K topics are taken as target topics. This embodiment does not limit the preset condition.

[0143] The disclosed embodiment obtains the first text information by extracting voice information from audio data and the second text information from key frames of video data, and performs error correction processing on the first text information and the second text information, thereby avoiding interference of incorrect grammar, typos, etc. on topic analysis, and further improving the accuracy of topic analysis; at the same time, different weight values ​​are determined according to the different sources of the topic, which can not only analyze the topic in an all-round and multi-granular manner, but also treat topics from different sources differently, thereby improving the accuracy of topic analysis.

[0144] Figure 6 The structure diagram of the multimedia information topic analysis device provided in the embodiment of the present disclosure. The multimedia information topic analysis device may be the terminal device described in the above embodiment, or the multimedia information topic analysis device may be a terminal device component or assembly as described above. The multimedia information topic analysis device provided in the embodiment of the present disclosure may execute the processing flow provided in the multimedia information topic analysis method embodiment, such as Figure 6 As shown, the multimedia information topic analysis device 60 includes:

[0145] The first extraction module 61 is used to extract audio data and video data from multimedia information.

[0146] The second extraction module 62 is used to extract voice information from the audio data to obtain voice recognition text.

[0147] The third extraction module 63 is used to extract subtitle information from the key frames of the video data to obtain subtitle text.

[0148] The fourth extraction module 64 is used to extract entities and keywords from the speech recognition text and the subtitle text.

[0149] The first determination module 65 is used to determine the semantic tags of the speech recognition text and the subtitle text.

[0150] The fifth extraction module 66 is used to extract the face label in the key frame.

[0151] The second determining module 67 is used to determine the image label of the key frame.

[0152] The third determination module 68 is used to determine the target topic of the multimedia information according to the entity, the keyword, the text semantic tag, the face tag and the picture tag.

[0153] Optionally, the third determination module includes a merging unit 681, a first determination unit 682, a second determination unit 683, and a third determination unit 684. The merging unit 681 is used to merge the face tag and the picture tag to obtain merged visual topic information; the first determination unit 682 is used to determine multiple topics appearing in the multimedia information according to the entity, the keyword, the semantic tag, and the merged visual topic information; the second determination unit 683 is used to determine the weight value of each topic according to the source of each topic in the multiple topics; the third determination unit 684 is used to determine the target topic of the multimedia information according to the weight value of each topic, and the target topic is a topic whose weight value meets a preset condition.

[0154] Optionally, the second determination unit 683 is used to determine the weight value of each topic according to the source of each topic in the multiple topics, specifically for: if the topic appears in text data, then the weight value of the topic is determined to be a first weight value, and the text data includes the speech recognition text and / or the subtitle text; if the topic appears in image data, then the weight value of the topic is determined to be a second weight value, and the image data includes the key frame; if the topic appears in both the text data and the image data, then the weight value of the topic is determined to be a third weight value, and the third weight value is the sum of the first weight value and the second weight value.

[0155] Optionally, the merging unit 681 is used to merge the face tag and the picture tag to obtain the merged visual topic information, specifically for: deduplicating the face tag and the picture tag of the key frame respectively and then merging them to obtain the merged visual topic information.

[0156] Optionally, the second extraction module 62 is used to extract voice information from the audio data. When obtaining the voice recognition text, it is specifically used to: extract first text information in the voice information from the audio data, and perform error correction processing on the first text information; and obtain the voice recognition text based on the first text information after error correction processing.

[0157] Optionally, the third extraction module 63 is used to extract subtitle information from the key frames of the video data. When obtaining the subtitle text, it is specifically used to: extract second text information from the key frames of the video data, perform error correction on the second text information; and obtain the subtitle text based on the second text information after error correction.

[0158] Optionally, when the fourth extraction module 64 is used to extract entities and keywords from the speech recognition text and the subtitle text, it is specifically used to: extract entities from the speech recognition text and the subtitle text using a deep learning model; and extract keywords from the speech recognition text and the subtitle text using a keyword extraction model.

[0159] Optionally, when the first determination module 65 is used to determine the semantic labels of the speech recognition text and the subtitle text, it is specifically used to: determine the semantic labels in the speech recognition text and the subtitle text using a multi-classification model and a multi-label classification model.

[0160] Figure 6 The multimedia information topic analysis device of the illustrated embodiment can be used to execute the technical solution of the above method embodiment, and its implementation principle and technical effect are similar and will not be described in detail here.

[0161] Figure 7 The electronic device provided in the embodiment of the present disclosure can execute the processing flow provided in the embodiment of the multimedia information topic analysis method, such as Figure 7 As shown, the device 70 includes: a memory 71, a processor 72, a computer program and a communication interface 73; wherein the computer program is stored in the memory 71 and is configured so that the processor 72 executes the method as described above.

[0162] In addition, an embodiment of the present disclosure further provides a computer-readable storage medium on which a computer program is stored. The computer program is executed by a processor to implement the multimedia information topic analysis method described in the above embodiment.

[0163] In addition, an embodiment of the present disclosure further provides a computer program product, which includes a computer program or instructions, and when the computer program or instructions are executed by a processor, the multimedia information topic analysis method described above is implemented.

[0164] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0165] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for analyzing multimedia information topics, characterized in that: The method comprises: Extracting audio data and video data from multimedia information; Extracting voice information from the audio data to obtain voice recognition text; Extract subtitle information from the key frames of the video data to obtain subtitle text; Extracting entities and keywords from the speech recognition text and the subtitle text, and determining semantic tags of the speech recognition text and the subtitle text; Extracting face tags in the key frame and determining image tags of the key frame; Determining a target topic of the multimedia information according to the entity, the keyword, the semantic tag, the face tag, and the picture tag; Determining a target topic of the multimedia information according to the entity, the keyword, the semantic tag, the face tag, and the picture tag includes: Merging the entity, the keyword and the semantic tag to obtain merged text topic information; Merging the face tag and the picture tag to obtain merged visual topic information; determining multiple topics appearing in the multimedia information according to the merged text topic information and the merged visual topic information; Determining a weight value of each topic according to the source of each topic in the plurality of topics; Determining a target topic of the multimedia information according to the weight value of each topic, wherein the target topic is a topic whose weight value satisfies a preset condition; Determining a weight value of each topic according to a source of each topic in the plurality of topics includes: If the topic appears in text data, determining the weight value of the topic to be a first weight value, the text data including the speech recognition text and / or the subtitle text; If the topic appears in the image data, determining the weight value of the topic to be a second weight value, the image data including the key frame; If the topic appears in both the text data and the image data, the weight value of the topic is determined to be a third weight value, where the third weight value is the sum of the first weight value and the second weight value.

2. The method according to claim 1, characterized in that The face label and the picture label are merged to obtain merged visual topic information, including: The face labels and the image labels of the key frames are respectively deduplicated and then merged to obtain merged visual topic information.

3. The method according to claim 1, characterized in that The step of extracting voice information from the audio data to obtain voice recognition text includes: Extracting first text information from the voice information from the audio data, and performing error correction processing on the first text information; The speech recognition text is obtained based on the first text information after error correction processing.

4. The method according to claim 1, characterized in that: The step of extracting subtitle information from the key frames of the video data to obtain subtitle text includes: extracting second text information from the key frame of the video data, and performing error correction processing on the second text information; The subtitle text is obtained based on the second text information after error correction processing.

5. The method according to claim 1, characterized in that The extracting entities and keywords from the speech recognition text and the subtitle text, and determining semantic tags of the speech recognition text and the subtitle text, includes: Extracting entities from the speech recognition text and the subtitle text using a deep learning model; Extracting keywords from the speech recognition text and the subtitle text using a keyword extraction model; The semantic labels in the speech recognition text and the subtitle text are determined using a multi-classification model and a multi-label classification model.

6. A multimedia information topic analysis device, characterized in that: The device comprises: A first extraction module, used for extracting audio data and video data from multimedia information; A second extraction module, used to extract voice information from the audio data to obtain a voice recognition text; A third extraction module, used to extract subtitle information from the key frames of the video data to obtain subtitle text; A fourth extraction module, used to extract entities and keywords from the speech recognition text and the subtitle text; A first determination module, used to determine semantic tags of the speech recognition text and the subtitle text; A fifth extraction module, used to extract face tags from the key frame; A second determination module, used to determine the image label of the key frame; A third determination module, configured to determine a target topic of the multimedia information according to the entity, the keyword, the semantic tag, the face tag, and the picture tag; The third determination module includes: A merging unit, used to merge the face label and the picture label to obtain merged visual topic information; A first determining unit, configured to determine a plurality of topics appearing in the multimedia information according to the entity, the keyword, the semantic tag, and the merged visual topic information; A second determining unit, configured to determine a weight value of each topic according to a source of each topic among the plurality of topics; A third determining unit, configured to determine a target topic of the multimedia information according to a weight value of each topic, wherein the target topic is a topic whose weight value satisfies a preset condition; The second determination unit is used to determine the weight value of each topic according to the source of each topic in the multiple topics, specifically for: if the topic appears in text data, then the weight value of the topic is determined to be a first weight value, and the text data includes the speech recognition text and / or the subtitle text; if the topic appears in image data, then the weight value of the topic is determined to be a second weight value, and the image data includes the key frame; if the topic appears in both the text data and the image data, then the weight value of the topic is determined to be a third weight value, and the third weight value is the sum of the first weight value and the second weight value.

7. An electronic device, characterized in that: include Memory; processor; as well as Computer programs; The computer program is stored in the memory and is configured to be executed by the processor to implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images