A method, device, storage medium and equipment for mining multimedia data

Abstract

The present application discloses a method, device, storage medium and equipment for mining multimedia data, and relates to the field of data mining technology. The method includes obtaining the transcription result and translation result of multimedia data of minority languages, and obtaining the first keyword and the second keyword, wherein the first keyword is the keyword of the official language, and the second keyword is the keyword of the minority language obtained by translation; according to the transcription result and the translation result, the confidence of the keyword fragments existing in the majority of media data is obtained by combining the first keyword and the second keyword, and then according to the confidence of the keyword fragments existing in the majority of media data and the keyword recognition threshold, the recognition result is obtained, and the recognition result includes the keywords recognized in the multimedia data. It can be seen that the method can realize the mining of multimedia data of minority languages.

Description

Technical Field

[0001] The present application relates to the technical field of data mining, and in particular to a method, device, storage medium and equipment for mining multimedia data. Background Art

[0002] With the continuous development of the Internet, especially the mobile Internet, massive data has been generated. By mining massive data, different needs of users can be met. For example, in the application scenario of document retrieval, text data can be retrieved by keywords, and text data related to the keyword can be returned to the user.

[0003] In some scenarios, in addition to text data, a large amount of multimedia data is also included. The multimedia data includes one or more of voice data, image data, video data, etc. Mining the above multimedia data through artificial intelligence (AI) technologies such as voice recognition, image recognition, and video analysis has become one of the mainstream implementation methods.

[0004] AI technology is usually data-driven. For the six official languages ​​of the United Nations, such as English, French, Russian, Chinese, Spanish, and Arabic, they are widely used and generate large amounts of data. By learning from large amounts of data, we can obtain stronger AI processing capabilities for multimedia data of official languages. However, for minority languages, such as Uyghur, Tibetan, Mongolian, Kazakh, and Yi, they are used by a small population and generate small amounts of data. Based on small amounts of data, it is difficult to mine multimedia data of minority languages.

[0005] How to provide a method for mining multimedia data in minority languages ​​has become a key issue in the industry. Summary of the invention

[0006] The main purpose of the present application is to provide a multimedia data mining method, device, storage medium and equipment, which can realize the mining of multimedia data in minority languages.

[0007] In a first aspect, the present application provides a method for mining multimedia data, comprising:

[0008] Obtaining transcription results and translation results of multimedia data in a minority language, and obtaining a first keyword and a second keyword, wherein the first keyword is a keyword in the official language, and the second keyword is a keyword in the minority language obtained by translation;

[0009] According to the transcription result and the translation result, combined with the first keyword and the second keyword, obtaining the confidence level of the keyword segment in the multimedia data;

[0010] According to the confidence of the keyword segment existing in the multimedia data and the keyword recognition threshold, a recognition result is obtained, and the recognition result includes the keywords recognized in the multimedia data.

[0011] In a second aspect, the present application provides a multimedia data mining device, comprising:

[0012] An acquisition module, used to acquire the transcription result and translation result of the multimedia data in the minority language, and to acquire a first keyword and a second keyword, wherein the first keyword is a keyword in the official language, and the second keyword is a keyword in the minority language obtained by translation;

[0013] A calculation module, for obtaining the confidence of the keyword segment in the multimedia data according to the transcription result and the translation result, combined with the first keyword and the second keyword

[0014] The recognition module is used to obtain a recognition result according to the confidence of the keyword segment existing in the multimedia data and the keyword recognition threshold, and the recognition result includes the keywords recognized in the multimedia data.

[0015] In a third aspect, the present application also provides a multimedia data mining device, including: a processor, a memory, and a system bus;

[0016] The processor and the memory are connected via the system bus;

[0017] The memory is used to store one or more programs, and the one or more programs include instructions. When the instructions are executed by the processor, the processor executes any one of the implementations of the multimedia data mining method mentioned above.

[0018] In a fourth aspect, the present application also provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes any one of the implementation methods of the above-mentioned multimedia data mining method.

[0019] In a fifth aspect, the present application also provides a computer program product, which, when running on a terminal device, enables the terminal device to execute any one of the implementations of the above-mentioned multimedia data mining method.

[0020] It can be seen from the above technical solution that the present application has at least the following beneficial effects:

[0021] The present application provides a method for mining multimedia data in minority languages. Specifically, when the keywords of the official language are known, the keywords of the official language and the keywords of the minority language corresponding to the keywords of the official language are used, and the transcription results and translation results of the multimedia data in the minority language are combined to obtain the confidence of the keyword fragments in the multimedia data, and then obtain the recognition results according to the confidence of the keyword fragments in the multimedia data and the keyword recognition threshold.

[0022] The above method integrates the corpus of the official language and the corpus of the minority language, thereby increasing the scale of the corpus of the minority language, avoiding the situation where the corpus of the minority language is too small and leading to poor AI processing capabilities, and improving the recognition rate of keywords in the multimedia data of the minority language, thereby improving the mining effect of the multimedia data of the minority language.

[0023] Furthermore, the method provides sentence-level keyword recognition capabilities, realizes keyword recognition of multiple sentences in a multimedia data, such as a voice, and establishes a correspondence between keywords and sentences in multimedia data. By presenting the above correspondence, it can also support manual verification to further improve accuracy. In this way, more sample data can be accumulated and used to update the model and improve model performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 A flowchart of a multimedia data mining method provided in an embodiment of the present application;

[0026] Figure 2A A schematic diagram of a human-computer interaction interface provided in an embodiment of the present application;

[0027] Figure 2B A schematic diagram of a human-computer interaction interface provided in an embodiment of the present application;

[0028] Figure 2C A schematic diagram of a human-computer interaction interface provided in an embodiment of the present application;

[0029] Figure 3 A schematic diagram of processing multimedia data provided by an embodiment of the present application;

[0030] Figure 4A schematic diagram of a multimedia data mining device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The terms "first" and "second" in the embodiments of the present application are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0032] First, some technical terms involved in the embodiments of the present application are introduced.

[0033] Data mining refers to the process of automatically searching for useful information hidden in a large amount of data. At present, it is possible to mine a large amount of data by building an artificial intelligence model to obtain useful information hidden in it. Taking multimedia data as an example, multimedia data can be divided into normal information and sensitive information. Based on the artificial intelligence model, a large amount of multimedia data can be mined to obtain multimedia data containing sensitive information from the large amount of multimedia data, so as to issue an early warning or further process the keyword fragments of the sensitive information.

[0034] Training an AI model requires a large amount of sample data. For multimedia data in official languages, the data generated is large because official languages ​​are widely used. Learning based on this large-scale data can produce an AI model with good recognition effect. For multimedia data in unofficial languages, such as multimedia data in minority languages, the data generated is small because minority languages ​​are less used. Small-scale data makes it difficult for AI models to learn good recognition effects, and thus it is difficult to mine multimedia data in minority languages.

[0035] In view of this, an embodiment of the present application provides a method for mining multimedia data, which can be performed by a processing device. The processing device can be a terminal or a server. Terminals include but are not limited to smart phones, tablet computers, laptops, personal digital assistants (PDAs) or smart wearable devices. The server can be a cloud server, such as a central server in a central cloud computing cluster, or an edge server in an edge cloud computing cluster. Of course, the server can also be a server in a local data center. A local data center refers to a data center directly controlled by a user.

[0036] The processing device obtains the transcription result and translation result of the multimedia data in the minority language; the transcription result includes the transcription text and the confidence of at least one sentence in the transcription text, the confidence and word vector of at least one word in at least one sentence of the transcription text, the transcription text is the text in the minority language, and the transcription text can also be called the first text; the translation result includes the translation text obtained by translating the first text and the confidence of at least one sentence in the translation text, the confidence and word vector of at least one word in at least one sentence of the translation text, the translation text is the text of the official language obtained by translating the first text, and the translation text can also be called the second text. The processing device also obtains the first keyword, the weight of the first keyword, the word vector of the first keyword, the second keyword, and the word vector of the second keyword; wherein the first keyword is the keyword of the official language, and the second keyword is the keyword of the minority language obtained by translation. Then, the processing device obtains the confidence of the keyword segment corresponding to the keyword (for example, the first keyword and the second keyword) in the multimedia data according to the transcription result and the translation result, combined with the first keyword, the weight of the first keyword, the word vector of the first keyword, the second keyword, and the word vector of the second keyword. The processing device obtains a recognition result according to the confidence of the keyword segment corresponding to the keyword included in the multimedia data and the keyword recognition threshold, and the recognition result includes the keyword recognized in the multimedia data.

[0037] In this method, the processing device integrates the corpus of the official language and the corpus of the minority language, thereby increasing the size of the corpus of the minority language, avoiding the situation where too little corpus of the minority language leads to poor AI processing capabilities, and increasing the recognition rate of keywords in multimedia data of the minority language, thereby improving the mining effect of multimedia data of the minority language.

[0038] The multimedia data mining method provided in the embodiment of the present application can be applied to a variety of scenarios. For example, in a communication scenario, the method can be used to identify the multimedia data transmitted by the communicating parties, so as to determine whether the identified multimedia data contains sensitive information, where sensitive information can refer to illegal information or preset information. For another example, in a content publishing scenario, the method can be used to review the multimedia data to be published, so as to assist in determining whether the multimedia data to be published contains sensitive information.

[0039] It should be noted that the multimedia data mining method provided in the embodiments of the present application is not only applicable to the above two scenarios, and those skilled in the art can select other application scenarios according to actual needs.

[0040] In order to make the technical solution of the present application clearer and easier to understand, the multimedia data mining method provided by the embodiment of the present application is introduced below in conjunction with the accompanying drawings. Figure 1 As shown, the figure is a flowchart of a multimedia data mining method provided by an embodiment of the present application, the method comprising:

[0041] S101: A processing device obtains transcription results and translation results of multimedia data in a minority language.

[0042] Minority languages ​​include but are not limited to Uyghur, Tibetan, Mongolian, Kazakh, and Yi. Multimedia data includes one or more of speech, image, and video. For example, the multimedia data of minority languages ​​may be speech in Uyghur, video in Tibetan, image of Tibetan text, and the like.

[0043] Taking the multimedia data in a minority language as a speech in a minority language as an example, the processing device can recognize the speech in the minority language based on a minority speech recognition model, and then obtain a transcription result of the speech in the minority language. The transcription result includes a first text, a confidence of at least one sentence in the first text, and a confidence of at least one word in the sentence, and the first text is a text in a minority language.

[0044] In some examples, the first text may include N sentences, and the first text may be represented as “S1, S2, …, S N ". The confidence of the N sentences in the first text can be expressed as "S_weight1, S_weight2, ..., S_weight N ". The i-th sentence in the N sentences in the first text may include M i words, the i-th sentence can be expressed as “W i1 ,W i2 ,…, ”, in the i-th sentence, M i The confidence of a word can be expressed as "W_weight i1 ,W_weight i2 ,…, The processing device can process the N sentences of the first text through the minority language word vector model to obtain the word vector of each word in the N sentences. i The word vector of a word can be expressed as “Vec i1 ,Vec i2 ,…, ". The embodiment of the present application does not specifically limit the dimension of the word vector, and the dimension of the word vector can be 512 dimensions.

[0045] Continuing with the above example, after the processing device performs speech recognition on the speech of the minority language to obtain the transcription result, it processes the transcription result based on the language translation model to obtain the translation result of the speech of the minority language. The translation result includes a second text translated from the first text, the confidence of at least one sentence in the second text, the confidence and word vector of at least one word in the sentence, and the second text can be a text in an official language, such as a text in an official language obtained by translating the first text, and the official languages ​​include but are not limited to English, French, Russian, Chinese, Spanish, and Arabic.

[0046] In some examples, the second text may also include N sentences. Based on this, the second text may be represented as “ST1, ST2, …, ST N ". The confidence of the N sentences in the second text can be expressed as "ST_weight1, ST_weight2, ..., ST_weight N ". The i-th sentence in the N sentences in the second text may include P i words, the i-th sentence can be expressed as “WT i1 ,WT i2 ,…, ”, P in the i-th sentence i The confidence of a word can be expressed as "WT_weight i1 ,WT_weight i2 ,…, ". The processing device can process the N sentences of the second text through the official language word vector model to obtain the word vector of each word in the N sentences. i The word vector of a word can be expressed as "WTVec i1 ,WTVec i2 ,…, ". The embodiment of the present application does not specifically limit the dimension of the word vector, and the dimension of the word vector can be 512 dimensions.

[0047] It should be noted that the above is only an exemplary introduction using the example of the multimedia data in the minority language being the speech in the minority language. In other embodiments, the multimedia data in the minority language may also be an image of a text in the minority language, and the processing device may recognize the image of the text in the minority language based on the image recognition model, and then obtain the transcription result of the image of the text in the minority language; the multimedia data in the minority language may also be a video including the speech in the minority language, and the way in which the processing device obtains the transcription result of the video including the speech in the minority language is similar to the above embodiment, and will not be repeated here.

[0048] S102: The processing device obtains a first keyword and a second keyword.

[0049] The first keyword refers to a keyword in the official language, and the second keyword refers to a keyword in a minority language obtained by translation, such as a keyword in a minority language obtained by translating a keyword in the official language. In some examples, the first keyword can be set according to specific business needs. For example, in the above content publishing scenario, the word corresponding to the uncivilized official language speech (such as swear words) can be set as the first keyword.

[0050] In some embodiments, the processing device may obtain the weights of the first keywords, and the Q first keywords may be represented as “cn_key1, cn_key2, …, cn_key Q ", the weights of the Q first keywords can be obtained based on the statistical information of the historical data corresponding to the official language, and the statistical information can be information such as word frequency.

[0051] Historical data can be divided into full historical data and partial historical data. Full historical data refers to all historical data related to preset services generated on the Internet, partial historical data refers to historical data related to the first keyword in the full historical data, and partial historical data is a part of the full historical data. Specifically, the processing device can calculate the word frequency of Q first keywords in the full historical data based on the full historical data. The word frequency of Q first keywords in the full historical data can be expressed as “cn_key_all_freq1,cn_key_all_freq2,…,cn_key_all_freq Q ”, wherein the word frequencies of the Q first keywords in the full amount of historical data may be normalized values. The processing device may also calculate the word frequencies of the Q first keywords in the partial historical data based on the partial historical data, and the word frequencies of the Q first keywords in the partial historical data may be expressed as “cn_key_part_freq1,cn_key_part_freq2,…,cn_key_part_freq Q ", wherein the word frequencies of the Q first keywords in part of the historical data may be normalized values.

[0052] Taking the i-th first keyword among the Q first keywords as an example, the processing device obtains the weight of the i-th first keyword based on the word frequency of the i-th first keyword in the partial historical data and the word frequency of the i-th first keyword in the full historical data. For example, the processing device can obtain the weight of the i-th first keyword among the Q first keywords based on the ratio between the word frequency of the i-th first keyword among the Q first keywords in the full historical data and the word frequency of the i-th first keyword in the partial historical data. Specifically, the processing device can calculate the weight of the i-th first keyword by the following formula:

[0053]

[0054] Among them, cn_key_freq i is the weight of the i-th first keyword among the Q first keywords, cn_key_part_freq i is the frequency of the i-th first keyword among the Q first keywords in some historical data, cn_key_all_freq i It is the word frequency of the i-th first keyword among the Q first keywords in the entire historical data.

[0055] The processing device can obtain the weight of each of the Q first keywords according to the above formula (1). The weights of the Q first keywords can be expressed as “cn_key_freq1, cn_key_freq2, …, cn_key_freq Q ”.

[0056] It should be noted that the above formula (1) is only an optional way to obtain the weight of the first keyword. In other embodiments, the processing device can also calculate the ratio between the word frequency of the i-th first keyword in the full amount of historical data and the word frequency of the i-th first keyword in the partial historical data, and then multiply the ratio with the position factor and / or part-of-speech factor of the i-th first keyword, and use the product as the final weight of the i-th first keyword. For example, the processing device can multiply the ratio with the position factor of the i-th first keyword, and use the product as the final weight of the i-th first keyword; for another example, the processing device can multiply the ratio with the part-of-speech factor of the i-th first keyword, and use the product as the final weight of the i-th first keyword; for another example, the processing device can also add the ratio and the product of the above-mentioned position factor and part-of-speech factor to obtain the average value, and use the average value as the final weight of the i-th first keyword.

[0057] The processing device can also process the Q first keywords based on the official language word vector model to obtain word vectors of the Q first keywords. The word vectors of the Q first keywords can be expressed as “cn_key_vec1, cn_key_vec2, …, cn_key_vec Q ", the embodiment of the present application does not specifically limit the dimension of the word vector, and the dimension of the word vector can be 512 dimensions.

[0058] In some embodiments, the processing device may translate the first keyword through a language translation model to obtain a second keyword. The embodiment of the present application does not specifically limit the manner in which the processing device obtains the second keyword corresponding to the first keyword. In other embodiments, the second keyword obtained by manually translating the first keyword may also be provided to the processing device so that the processing device obtains the second keyword. Among them, a first keyword can be translated to obtain at least one second keyword. For ease of understanding, the following is an introduction to translating a first keyword to obtain a second keyword as an example.

[0059] Continuing with the above example, the Q second keywords corresponding to the Q first keywords can be expressed as “T_key1, T_key2, …, T_key Q Similarly, the processing device can process the Q second keywords based on the minority language word vector model to obtain the word vectors of the Q second keywords. The word vectors of the Q second keywords can be expressed as “T_key_vec1, T_key_vec2, …, T_key_vec Q ", the embodiment of the present application does not specifically limit the dimension of the word vector, and the dimension of the word vector can be 512 dimensions.

[0060] It should be noted that the embodiments of the present application do not specifically limit the execution order of S101 and S102. In some examples, the processing device may execute S101 and S102 simultaneously, or may execute S102 first and then S101.

[0061] S103: The processing device obtains the confidence level of the keyword segment in the multimedia data based on the transcription result and the translation result in combination with the first keyword and the second keyword.

[0062] Taking the multimedia data in the minority language as the speech in the minority language as an example, the keyword segment corresponding to the keyword may refer to the speech segment in the minority language corresponding to the first keyword or the second keyword. The transcription result of the keyword segment corresponding to the keyword includes the first keyword or the translation result includes the second keyword.

[0063] In some embodiments, the processing device calculates the confidence of the keyword segment corresponding to the first keyword included in the multimedia data and the confidence of the keyword segment corresponding to the second keyword included in the multimedia data, respectively, and then obtains the confidence of the keyword segment corresponding to the keyword included in the multimedia data based on the confidence of the keyword segment corresponding to the first keyword included in the multimedia data and the confidence of the keyword segment corresponding to the second keyword included in the multimedia data.

[0064] Specifically, the processing device determines the confidence level of the second keyword in the transcription result according to the transcription result and the second keyword.

[0065] In some examples, the confidence that at least one sentence in the transcription result includes the second keyword may include the first confidence and the second confidence that the i-th sentence in the first text includes the j-th second keyword. The processing device may use the first confidence alone as the confidence that at least one sentence in the transcription result includes the second keyword, or use the second confidence alone as the confidence that at least one sentence in the transcription result includes the second keyword, or perform a weighted sum of the first confidence and the second confidence, and use the sum result as the confidence that at least one sentence in the transcription result includes the second keyword. Among them, the first confidence is the confidence determined based on the result of character matching, and the second confidence is the confidence determined based on the similarity between word vectors, which are introduced below.

[0066] The processing device may perform character matching on the second keyword and the transcription result, and obtain a first confidence that the second keyword exists in the transcription result based on the result of the character matching. For example, the processing device may determine the first confidence that the i-th sentence in the first text includes the j-th second keyword based on the result of the character matching between the i-th sentence in the first text and the j-th second keyword, and the weight of the j-th first keyword. In some examples, when the second keyword matches the sentence, the first confidence that the second keyword exists in the sentence is the product of the weight of the first keyword corresponding to the second keyword and a preset coefficient, otherwise the first confidence is zero. The preset coefficient may be 1 or other values, such as 0.8. Taking the preset coefficient of 1 as an example, the processing device may calculate the first confidence by the following formula:

[0067]

[0068] Among them, key_fun1 ij is the first confidence that the i-th sentence in the first text includes the j-th second keyword, cn_key_freq j is the weight of the jth first keyword. When the character of the jth second keyword is matched in the i-th sentence in the first text, the first confidence is cn_key_freqj 1; when the jth character of the second keyword is not matched in the i-th sentence in the first text, the first confidence is cn_key_freq j ·0.

[0069] The processing device calculates the first confidence based on the result of character matching between the i-th sentence and the j-th second keyword in the first text, which can reduce the negative impact of the recognition error of the minority speech recognition model (for example, word segmentation error, word segmentation granularity error) on the calculation of the first confidence and improve the accuracy of the calculated first confidence.

[0070] In other embodiments, the processing device may determine the result of the match based on multiple matching degrees when matching the i-th sentence in the first text with the characters of the j-th second keyword. The matching degrees may be divided into three, such as 0-%40, 40%-80% and 80%-100%. When the matching degrees are different, the preset coefficients may be different. For example, when the result of the matching process between the i-th sentence in the first text and the characters of the j-th second keyword is within the matching degree of 40%-80%, the preset coefficient may be 0.5.

[0071] The embodiment of the present application does not specifically limit the manner in which the processing device obtains the result of the character matching between the i-th sentence and the j-th second keyword in the first text. After obtaining the i-th sentence and the j-th second keyword, the processing device may match the characters of the j-th second keyword in the i-th sentence to obtain the matching result; the processing device may also obtain the matching result obtained by other devices after matching the characters of the i-th sentence and the j-th second keyword in the first text.

[0072] It should be noted that the character can be a single character or a string consisting of multiple characters.

[0073] The processing device can determine the similarity between the word vector of the second keyword and the word vector of at least one word in the transcription result, and obtain a second confidence that the second keyword exists in the transcription result based on the similarity. Wherein, the second confidence that the second keyword exists in the transcription result can be represented by the second confidence that the second keyword exists in the sentence of the transcription result. The similarity between the word vector of the second keyword and the word vector of the word in the transcription result can be represented by the distance between the word vector of the second keyword and the word vector of the word in the sentence of the transcription result, and the second confidence that the second keyword exists in the sentence of the transcription result is the weighted sum of the target distance, the target distance is the distance between the word vector of the target word in the sentence and the word vector of the second keyword, the target word is a word whose distance is not less than a preset distance, and the weight of the target distance is the product of the weight of the first keyword and the weight of the target word in the sentence.

[0074] In some examples, the processing device may determine the second confidence that the i-th sentence in the first text includes the j-th second keyword based on the similarity between the word vectors of each word in the i-th sentence in the first text and the word vector of the j-th second keyword, as well as the confidence of each word in the i-th sentence and the weight of the j-th first keyword. For example, the processing device may first calculate the product of the word vector of the k-th word in the i-th sentence in the first text and the word vector of the j-th second keyword, and use the product as the target distance, then calculate the product of the weight of the j-th first keyword and the confidence of the k-th word in the i-th sentence in the first text, and use the product as the weight of the target distance, and then perform weighted summation on multiple target distances based on the weights of the target distances corresponding to multiple words, thereby obtaining the second confidence that the i-th sentence in the first text includes the j-th second keyword. Specifically, the processing device may calculate the second confidence by the following formula:

[0075]

[0076] Among them, key_fun2 ij The second confidence level of the jth second keyword included in the i-th sentence in the first text; cn_key_freq j is the weight of the jth first keyword; W_weight ik is the confidence of the kth word in the i-th sentence in the first text; Vec ik is the word vector of the kth word in the i-th sentence in the first text, T_key_vec j is the word vector of the jth second keyword, Vec ik T_key_vec j Represents the similarity between the word vector of the kth word in the ith sentence in the first text and the word vector of the jth second keyword; W_threshold is the first similarity threshold value, for example, W_threshold = 0.9, and those skilled in the art may also set the first similarity to other values ​​according to actual needs, such as 0.8, 0.7, etc.

[0077] It should be noted that the above formula (3) is only an optional way to obtain the second confidence that the second keyword exists in the sentence of the transcription result. In other embodiments, the processing device can also introduce a distance factor. After calculating the target distance, the target distance is multiplied by the distance factor, and then a weighted sum is performed based on the weight of the target distance to obtain the final second confidence.

[0078] In the above formula (3), for the i-th sentence in the first text, when Vec ik T_key_vec j≥W_threshold, it indicates that the kth word in the i-th sentence has a high similarity with the j-th second keyword, that is, the i-th sentence in the first text includes the j-th second keyword, and the processing device can record the j-th second keyword. Furthermore, the processing device can also establish a corresponding relationship between the j-th second keyword and the keyword segment in the i-th sentence for subsequent verification.

[0079] Since the second keyword is obtained by translating the first keyword, when calculating the confidence that at least one sentence in the transcription result includes the second keyword, the processing device not only combines the information of the second keyword (such as the second keyword and its confidence), but also combines the information of the first keyword (such as the first keyword and its weight and confidence). In this way, the calculated confidence that at least one sentence in the transcription result includes the second keyword can be made more accurate, thereby further reducing errors.

[0080] Similarly, the processing device may also determine the confidence level of the first keyword in the translation result based on the translation result and the first keyword.

[0081] In some examples, the confidence that at least one sentence in the translation result includes the first keyword may include the third confidence and the fourth confidence that the i-th sentence in the second text includes the j-th first keyword. The processing device may use the third confidence alone as the confidence that at least one sentence in the translation result includes the first keyword, or use the fourth confidence alone as the confidence that at least one sentence in the translation result includes the first keyword, or perform a weighted sum of the third confidence and the fourth confidence, and use the sum result as the confidence that at least one sentence in the translation result includes the first keyword. Among them, the third confidence is the confidence determined based on the result of character matching, and the fourth confidence is the confidence determined based on the similarity between word vectors, which are introduced below.

[0082] The processing device may perform character matching on the first keyword and the translation result, and obtain a third confidence that the first keyword exists in the translation result based on the result of the character matching. For example, the processing device may determine the third confidence that the i-th sentence in the second text includes the j-th first keyword based on the result of the string matching between the i-th sentence in the second text and the j-th first keyword and the weight of the j-th first keyword. In some examples, when the first keyword matches the sentence, the third confidence that the first keyword exists in the sentence is the product of the weight of the first keyword and a preset coefficient, otherwise the third confidence is zero. The preset coefficient may be 1 or other values, such as 0.7. Taking the preset coefficient of 1 as an example, the processing device may calculate the third confidence by the following formula:

[0083]

[0084] Among them, key_fun3 ij The third confidence level of the jth first keyword included in the i-th sentence in the second text, cn_key_freq j is the weight of the jth first keyword. When the character of the jth first keyword is matched in the i-th sentence in the second text, the third confidence is cn_key_freq j 1; when the jth character of the first keyword is not matched in the i-th sentence in the second text, the third confidence is cn_key_freq j ·0.

[0085] The processing device calculates the third confidence based on the result of character matching between the i-th sentence and the j-th first keyword in the second text, which can reduce the negative impact of translation errors (for example, word segmentation errors, word segmentation granularity errors) of the language translation model on the calculation of the third confidence and improve the accuracy of the calculated third confidence.

[0086] In other embodiments, the processing device may determine the result of the match based on multiple matching degrees when matching the i-th sentence in the second text with the characters of the j-th first keyword. The matching degrees may be divided into three, such as 0-%40, 40%-80% and 80%-100%. When the matching degrees are different, the preset coefficients may be different. For example, when the result of the matching process between the i-th sentence in the second text and the characters of the j-th first keyword is within the matching degree of 80%-100%, the preset coefficient may be 1.

[0087] The way in which the processing device obtains the result of the string matching between the i-th sentence in the second text and the j-th first keyword is similar to the way in which the processing device obtains the result of the string matching between the i-th sentence in the first text and the j-th second keyword, which will not be repeated here.

[0088] The processing device can determine the similarity between the word vector of the first keyword and the word vector of at least one word in the translation result, and obtain the fourth confidence that the first keyword exists in the translation result based on the similarity. Among them, the fourth confidence that the first keyword exists in the translation result can be represented by the fourth confidence that the first keyword exists in the sentence of the translation result, the similarity between the word vector of the first keyword and the word vector of the word in the translation result is represented by the distance between the word vector of the first keyword and the word vector of the word in the sentence of the translation result, the fourth confidence that the first keyword exists in the sentence of the translation result is the weighted sum of the target distance, the target distance is the distance between the word vector of the target word in the sentence and the word vector of the first keyword, the target word is a word whose distance is not less than a preset distance, and the weight of the target distance is the product of the weight of the first keyword and the weight of the target word in the sentence.

[0089] In some examples, the processing device determines the fourth confidence that the i-th sentence in the second text includes the j-th first keyword based on the similarity between the word vectors of each word in the i-th sentence in the second text and the word vector of the j-th first keyword, as well as the confidence of each word in the i-th sentence and the weight of the j-th first keyword. For example, the processing device may first calculate the product of the word vector of the k-th word in the i-th sentence in the second text and the word vector of the j-th second keyword, and use the product as the target distance, and then calculate the product of the weight of the j-th first keyword and the confidence of the k-th word in the i-th sentence in the second text, and use the product as the weight of the target distance, and then perform weighted summation on multiple target distances based on the weights of the target distances corresponding to multiple words, thereby obtaining the fourth confidence that the i-th sentence in the second text includes the j-th first keyword. Specifically, the processing device may calculate the fourth confidence by the following formula:

[0090]

[0091] Among them, key_fun4 ij The fourth confidence level of the i-th sentence in the second text including the j-th first keyword; cn_key_freq j is the weight of the jth first keyword; WT_weight ik is the confidence of the kth word in the i-th sentence in the second text; WTVec ik is the word vector of the kth word in the i-th sentence in the second text, cn_key_vec j is the word vector of the jth second keyword, WTVec ik cn_key_vec j Represents the similarity between the word vector of the kth word in the ith sentence in the second text and the word vector of the jth second keyword; Wcn_threshold is the second similarity threshold value, for example, Wcn_threshold = 0.9, and those skilled in the art may also set the second similarity to other values ​​according to actual needs, such as 0.8, 0.7, etc.

[0092] It should be noted that the above formula (5) is only an optional way to obtain the fourth confidence that the first keyword exists in the sentence of the translation result. In other embodiments, the processing device can also introduce a distance factor. After calculating the target distance, the target distance is multiplied by the distance factor, and then a weighted sum is performed based on the weight of the target distance to obtain the final fourth confidence.

[0093] In the above formula (5), for the i-th sentence in the second text, when WTVec ikcn_key_vec j ≥Wcn_threshold, it indicates that the kth word in the i-th sentence in the second text has a high similarity with the j-th first keyword, that is, the i-th sentence in the second text includes the j-th first keyword, and the processing device can record the j-th first keyword. Further, the processing device can also establish a corresponding relationship between the j-th first keyword and the keyword segment in the i-th sentence for subsequent verification.

[0094] Then, the processing device obtains the confidence that the multimedia data includes the keyword segment corresponding to the keyword based on the confidence that at least one sentence in the transcription result includes the second keyword and the confidence that at least one sentence in the translation result includes the first keyword.

[0095] Continuing with the above example, the confidence that at least one sentence in the transcription result includes the second keyword may include the first confidence and the second confidence, and the confidence that at least one sentence in the translation result includes the first keyword includes the third confidence and the fourth confidence. For example, the processing device may calculate the sum of the third confidence and the fourth confidence, and then multiply the sum by the confidence of the i-th sentence in the second text, and add the product with the product of the confidence of the i-th sentence in the first text and the first confidence and the second confidence to obtain the confidence that the keyword segment exists in the multimedia data. Specifically, the processing device may calculate the confidence that the keyword segment exists in the multimedia data by the following formula:

[0096]

[0097] Among them, key_sent ij The multimedia data includes confidences of keyword segments corresponding to the keywords; is the confidence of the i-th sentence in the first text, is the confidence of the i-th sentence in the second text; key_fun1 ij is the first confidence that the i-th sentence in the first text includes the j-th second keyword, key_fun2 ij The second confidence that the i-th sentence in the first text includes the j-th second keyword, key_fun3 ij The third confidence level of the jth first keyword included in the i-th sentence in the second text, key_fun4 ij The fourth confidence level that the i-th sentence in the second text includes the j-th first keyword.

[0098] It should be noted that the above formula (6) is only an optional way to obtain the confidence of the existence of keyword fragments in multimedia data. In other embodiments, the processing device also introduces a confidence factor. After calculating the confidence of the existence of keyword fragments in the multimedia data, the confidence is multiplied by the confidence factor to obtain the final confidence of the existence of keyword fragments in the multimedia data.

[0099] S104: The processing device obtains a recognition result according to the confidence level of the keyword segment existing in the multimedia data and the keyword recognition threshold.

[0100] The processing device may compare the confidence level of the keyword segment in the multimedia data with the keyword recognition threshold to obtain the recognition result. Specifically, the processing device may determine the recognition result by the following formula:

[0101]

[0102] Among them, sent_have_key i For the recognition result, key_sent ij is the confidence of the keyword segment in the multimedia data; key_threshold is the keyword recognition threshold. When the confidence of the keyword segment in the multimedia data is greater than or equal to the keyword recognition threshold, the recognition result is "1", indicating that the keyword segment exists in the multimedia data; when the confidence of the keyword segment in the multimedia data is less than the keyword recognition threshold, the recognition result is "0", indicating that the keyword segment does not exist in the multimedia data. The recognition result also includes keywords obtained after the processing device recognizes the multimedia data, such as the first keyword, the second keyword, or the first keyword and the second keyword.

[0103] It can be seen from formula (7) that the larger the keyword recognition threshold is, the greater the confidence that the keyword segment exists in the multimedia data is required to obtain the recognition result of the keyword segment in the multimedia data. In some examples, the keyword recognition threshold can be 50; in other examples, the keyword recognition threshold can support the user to set it within a preset range, for example, the preset range is 0-100, and the user can set the keyword recognition threshold to any value between 0-100, for example, to 55.

[0104] In this embodiment, the processing device can receive the keyword recognition threshold configured by the user, and obtain the recognition result based on the keyword recognition threshold configured by the user. This can support the user's personalized configuration of the keyword recognition threshold, meet the user's needs, and expand the application scope of the method.

[0105] In some embodiments, the recognition result may also include a keyword segment corresponding to the keyword recognized in the multimedia data, such as Figure 2A Schematic diagram of a human-computer interaction interface shown in FIG. 1 , a processing device can present a keyword 210 and a corresponding keyword segment 221 identified in multimedia data to a user. In some examples, the multimedia data can be a voice. When the processing device recognizes that there is a keyword in the voice, it intercepts and processes the voice. For example, the voice of 5 seconds before and after the keyword appears in the voice can be intercepted to obtain a keyword segment, and a corresponding relationship between the keyword and the keyword segment can be established. In this way, the keyword segment corresponding to the keyword can be determined based on the established corresponding relationship between the keyword and the keyword segment, and the keyword segment can be presented to the user. The processing device can play the keyword segment 221 based on the user's click operation on the keyword segment 221, such as playing the intercepted voice.

[0106] In other examples, the multimedia data may also be a video or an image. When the multimedia data is a video, when the processing device recognizes that there is a keyword in the video, it intercepts the video, for example, it intercepts the video 5 seconds before and after the keyword appears in the video, and then obtains the keyword segment. Figure 2B A schematic diagram of another human-computer interaction interface is shown, where the processing device can play the keyword segment 222 based on a user's click operation on the keyword segment 222, such as playing a captured video.

[0107] When the multimedia data is an image, the processing device can intercept and process the image when it recognizes that there are keywords in the image. For example, it can intercept the image including the keyword with the keyword position as the center to obtain the keyword fragment. The processing device can also use the entire image as the keyword fragment. Figure 2C A schematic diagram of another human-computer interaction interface is shown, and the processing device can view the keyword segment 222 based on the user's click operation on the keyword segment 222, such as zooming in or out on the image.

[0108] In some embodiments, the above-mentioned human-computer interaction interface also includes a verification component 230, and the verification component 230 includes a confirmation control 231 and a modification control 232. The processing device can receive verification information of the user on the keywords and corresponding keyword fragments recognized in the multimedia data. For example, the user can verify the recognition result based on the verification component 230, and then the processing device can obtain the verified recognition result. For example, the user can confirm the recognition result by clicking the confirmation control 231, and the processing device can use the recognition result confirmed by the user as the verified recognition result; the user can also modify the recognition result through the modification control 232, such as supplementing the missed keywords or modifying the misrecognized keywords, and the processing device can use the user's modified recognition result as the verified recognition result.

[0109] In some embodiments, the processing device may also update the sample data set according to the recognition result, and use the updated sample data set to update the model, such as the above-mentioned minority speech recognition model, minority language word vector model, language translation model, and official language word vector model. The above-mentioned recognition result can be obtained according to business needs, for example, according to the keyword recognition threshold configured by the user. Based on this, after updating the sample data set with the recognition result, the model is updated, so that the updated model can better meet business needs and improve recognition accuracy.

[0110] Based on the above description, the embodiment of the present application provides a method for mining multimedia data. In the method, the processing device merges the corpus of the official language and the corpus of the minority language, thereby increasing the scale of the corpus of the minority language, avoiding the situation where the corpus of the minority language is too small and the AI ​​processing capability is poor, and increasing the recognition rate of keywords in the multimedia data of the minority language, thereby improving the mining effect of the multimedia data of the minority language.

[0111] Furthermore, the method provides sentence-level keyword recognition capabilities, realizes keyword recognition of multiple sentences in a multimedia data, such as a voice, and establishes a correspondence between keywords and sentences in multimedia data. By presenting the above correspondence, it can also support manual verification to further improve accuracy. In this way, more sample data can be accumulated and used to update the model and improve model performance.

[0112] To facilitate understanding, the following describes the process of a processing device mining speech of a minority language by taking the processing device mining speech of a minority language as an example.

[0113] like Figure 3As shown, in S310, the processing device processes the speech of the minority language, and the processing device can obtain the first text included in the speech of the minority language based on the minority speech recognition model (the first text includes S1, S2, ..., S N ), S1 represents the first sentence in the first text, and the confidence of each sentence is obtained; then, combined with the minority language word vector model, the word vector and its confidence of each word in each sentence in the first text are obtained.

[0114] The processing device processes the acquired first text, and the processing device can obtain a second text based on the language translation model (the second text includes ST1, ST2, ..., ST N ), ST1 represents the first sentence in the second text, and the confidence of each sentence is obtained; then the word vector and its confidence of each word in each sentence in the second text are obtained by combining the official language word vector model.

[0115] In S320, the processing device obtains the weight and word vector of the first keyword. For example, the processing device may obtain the weight of the first keyword based on the statistical information of the historical data, and obtain the word vector of the first keyword based on the official language word vector model. The first keyword may be a preset keyword or a user-configured keyword. The processing device may translate the first keyword to obtain a second keyword. For example, the processing device may translate the first keyword based on the language translation model to obtain the second keyword, and then obtain the word vector of the second keyword based on the minority language word vector model.

[0116] In S330, taking the example of a processing device processing a sentence in a first text and the corresponding sentence of the sentence in a second text, the processing device obtains the confidence that the sentence in the first text includes the second keyword based on the confidence of the sentence in the first text, the word vectors of each word in the sentence in the first text and their confidence, the weight and word vector of the first keyword, and the word vector of the second keyword.

[0117] The processing device obtains the confidence that the sentence in the second text includes the first keyword based on the confidence of a sentence in the second text, the word vectors of each word in the sentence in the second text and their confidences, and the weight and word vector of the first keyword.

[0118] The processing device may calculate the confidence level that each sentence in the first text includes the second keyword and the confidence level that each sentence in the second text includes the first keyword in a similar manner.

[0119] The processing device obtains the confidence that the speech of the minority language includes the keyword based on the confidence that the sentence in the first text includes the second keyword and the confidence that the sentence in the second text includes the first keyword. Then the processing device compares the confidence that the speech of the minority language includes the keyword with the keyword recognition threshold to obtain a recognition result. When the confidence that the speech of the minority language includes the keyword is greater than or equal to the keyword recognition threshold, the recognition result is that the speech of the minority language includes the keyword; otherwise, the recognition result is that the speech of the minority language does not include the keyword.

[0120] In S340, the processing device may receive the keyword recognition threshold configured by the user, so as to support personalized configuration of multimedia data mining to meet business needs. The processing device may also present the recognition result, and receive verification information from the user on the keyword included in the recognition result and the keyword segment corresponding to the keyword, to obtain the verified recognition result.

[0121] In S350, the processing device can update the sample data set based on the verified recognition results, and use the updated sample data set to regularly update the above-mentioned minority speech recognition model, minority language word vector model, official language word vector model and language translation model, so as to further improve the mining effect of multimedia data of minority languages ​​and meet business needs.

[0122] In an embodiment of the present application, the processing device performs data mining on the speech of the minority language based on the information of the first text and the information of the second text mentioned above, thereby improving the accuracy of identifying keywords included in the speech of the minority language; the processing device can also receive a keyword recognition threshold configured by the user to meet various business needs and expand the scope of application of the method; the processing device can also regularly update the model based on the recognition results, thereby further improving the accuracy of identifying keywords included in the speech of the minority language.

[0123] The present application also provides a multimedia data mining device, such as Figure 4 As shown, the multimedia data mining device 400 includes:

[0124] The acquisition module 401 is used to acquire the transcription result and translation result of the multimedia data in the minority language, and acquire the first keyword and the second keyword, wherein the first keyword is the keyword of the official language, and the second keyword is the keyword of the minority language obtained by translation;

[0125] The calculation module 402 is used to obtain the confidence of the keyword segment in the multimedia data according to the transcription result and the translation result, combined with the first keyword and the second keyword.

[0126] The recognition module 403 is used to obtain a recognition result according to the confidence of the keyword segment existing in the multimedia data and the keyword recognition threshold, and the recognition result includes the keywords recognized in the multimedia data.

[0127] In a possible implementation, the calculation module 402 is specifically used to determine the confidence of the presence of the second keyword in the transcription result based on the transcription result and the second keyword, and to determine the confidence of the presence of the first keyword in the translation result based on the translation result and the first keyword; and to obtain the confidence of the presence of the keyword fragment in the multimedia data based on the confidence of the presence of the second keyword in the transcription result and the confidence of the presence of the first keyword in the translation result.

[0128] In one possible implementation, the calculation module 402 is specifically used to perform character matching on the second keyword and the transcription result, and obtain a first confidence that the second keyword exists in the transcription result based on the result of the character matching; and / or determine the similarity between the word vector of the second keyword and the word vector of at least one word in the transcription result, and obtain a second confidence that the second keyword exists in the transcription result based on the similarity.

[0129] In one possible implementation, the first confidence that the second keyword exists in the transcription result is represented by the first confidence that the second keyword exists in the sentence of the transcription result; when the second keyword matches the sentence, the first confidence that the second keyword exists in the sentence is the product of the weight of the first keyword corresponding to the second keyword and a preset coefficient, otherwise the first confidence is zero.

[0130] In a possible implementation, the second confidence that the second keyword exists in the transcription result is represented by the second confidence that the second keyword exists in the sentence of the transcription result; the similarity between the word vector of the second keyword and the word vector of the word in the transcription result is represented by the distance between the word vector of the second keyword and the word vector of the word in the sentence of the transcription result; the second confidence that the second keyword exists in the sentence of the transcription result is the weighted sum of the target distances, the target distance being the distance between the word vector of the target word in the sentence and the word vector of the second keyword, the target word being a word whose distance is not less than a preset distance, and the weight of the target distance being the product of the weight of the first keyword and the weight of the target word in the sentence.

[0131] In one possible implementation, the calculation module 402 is specifically used to perform character matching on the first keyword and the translation result, and obtain a third confidence that the first keyword exists in the translation result based on the result of the character matching; and / or determine the similarity between the word vector of the first keyword and the word vector of at least one word in the translation result, and obtain a fourth confidence that the first keyword exists in the translation result based on the similarity.

[0132] In a possible implementation, the third confidence of the first keyword in the translation result is represented by the third confidence of the first keyword in the sentence of the translation result; when the first keyword matches the sentence, the third confidence of the first keyword in the sentence is the product of the weight of the first keyword and a preset coefficient, otherwise the third confidence is zero.

[0133] In a possible implementation, the fourth confidence that the first keyword exists in the translation result is represented by the fourth confidence that the first keyword exists in the sentence of the translation result; the similarity between the word vector of the first keyword and the word vector of the word in the translation result is represented by the distance between the word vector of the first keyword and the word vector of the word in the sentence of the translation result; the fourth confidence that the first keyword exists in the sentence of the translation result is a weighted sum of target distances, where the target distance is the distance between the word vector of the target word in the sentence and the word vector of the first keyword, the target word is a word whose distance is not less than a preset distance, and the weight of the target distance is the product of the weight of the first keyword and the weight of the target word in the sentence.

[0134] In a possible implementation manner, the weight of the first keyword is obtained according to statistical information in historical data.

[0135] In one possible implementation, the recognition result also includes the keyword fragment; the device also includes a display module for presenting the keywords and corresponding keyword fragments recognized in the multimedia data to the user; receiving the user's verification information on the keywords and corresponding keyword fragments recognized in the multimedia data, and obtaining the verified recognition result.

[0136] In a possible implementation, the device further includes an updating module, configured to update the sample data set according to the recognition result; and update the model using the updated sample data set.

[0137] The embodiment of the present application also provides a multimedia data mining device, including: a processor, a memory, and a system bus;

[0138] The processor and the memory are connected via the system bus;

[0139] The memory is used to store one or more programs, and the one or more programs include instructions. When the instructions are executed by the processor, the processor executes any one of the implementation methods of the above-mentioned multimedia data mining method.

[0140] Furthermore, an embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes any one of the implementation methods of the above-mentioned multimedia data mining method.

[0141] Furthermore, an embodiment of the present application also provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute any one of the implementation methods of the above-mentioned multimedia data mining method.

[0142] It can be known from the description of the above implementation mode that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment method can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., including several instructions for enabling a computer device (which can be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0143] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0144] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0145] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for mining multimedia data, characterized in that: include: Obtaining transcription results and translation results of multimedia data in a minority language, and obtaining a first keyword and a second keyword, wherein the transcription result is obtained by identifying the multimedia data in the minority language, the transcription result includes a first text, a confidence of at least one sentence in the first text, and a confidence of at least one word in the sentence, the first keyword is a keyword in the official language, and the second keyword is a keyword in the minority language obtained by translation; According to the transcription result and the translation result, combined with the first keyword and the second keyword, obtaining the confidence level of the keyword segment in the multimedia data; Obtaining a recognition result according to a confidence level of the keyword segment existing in the multimedia data and a keyword recognition threshold, wherein the recognition result includes the keywords recognized in the multimedia data; The obtaining, according to the transcription result and the translation result, the confidence of the keyword segment in the multimedia data in combination with the first keyword and the second keyword, includes: Determining the confidence level of the second keyword in the transcription result according to the transcription result and the second keyword, and determining the confidence level of the first keyword in the translation result according to the translation result and the first keyword; The confidence level of the keyword segment in the multimedia data is obtained according to the confidence level of the second keyword in the transcription result and the confidence level of the first keyword in the translation result.

2. The method according to claim 1, characterized in that The determining, according to the transcription result and the second keyword, the confidence level of the second keyword in the transcription result includes: Performing character matching on the second keyword and the transcription result, and obtaining a first confidence level that the second keyword exists in the transcription result according to the character matching result; and / or, Determine the similarity between the word vector of the second keyword and the word vector of at least one word in the transcription result, and obtain a second confidence that the second keyword exists in the transcription result based on the similarity.

3. The method according to claim 2, characterized in that The first confidence level of the second keyword in the transcription result is represented by the first confidence level of the second keyword in the sentence of the transcription result; When the second keyword matches the sentence, the first confidence that the second keyword exists in the sentence is the product of the weight of the first keyword corresponding to the second keyword and a preset coefficient, otherwise the first confidence is zero.

4. The method according to claim 2, characterized in that: The second confidence level of the second keyword in the transcription result is represented by the second confidence level of the second keyword in the sentence of the transcription result; The similarity between the word vector of the second keyword and the word vector of the word in the transcription result is represented by the distance between the word vector of the second keyword and the word vector of the word in the sentence of the transcription result; The second confidence that the second keyword exists in the sentence of the transcription result is the weighted sum of the target distances, where the target distance is the distance between the word vector of the target word in the sentence and the word vector of the second keyword, the target word is a word whose distance is not less than a preset distance, and the weight of the target distance is the product of the weight of the first keyword and the weight of the target word in the sentence.

5. The method according to claim 1, characterized in that Determining the confidence level of the first keyword in the translation result according to the translation result and the first keyword includes: Performing character matching on the first keyword and the translation result, and obtaining a third confidence level that the first keyword exists in the translation result according to the character matching result; and / or, Determine the similarity between the word vector of the first keyword and the word vector of at least one word in the translation result, and obtain a fourth confidence that the first keyword exists in the translation result according to the similarity.

6. The method according to claim 5, characterized in that The third confidence level that the first keyword exists in the translation result is represented by the third confidence level that the first keyword exists in the sentence of the translation result; When the first keyword matches the sentence, the third confidence that the first keyword exists in the sentence is the product of the weight of the first keyword and a preset coefficient, otherwise the third confidence is zero.

7. The method according to claim 5, characterized in that The fourth confidence level that the first keyword exists in the translation result is represented by the fourth confidence level that the first keyword exists in the sentence of the translation result; The similarity between the word vector of the first keyword and the word vector of the word in the translation result is represented by the distance between the word vector of the first keyword and the word vector in the sentence of the translation result; The fourth confidence that the first keyword exists in the sentence of the translation result is the weighted sum of the target distances, where the target distance is the distance between the word vector of the target word in the sentence and the word vector of the first keyword, the target word is a word whose distance is not less than a preset distance, and the weight of the target distance is the product of the weight of the first keyword and the weight of the target word in the sentence.

8. The method according to claim 3, 4, 6 or 7, characterized in that: The weight of the first keyword is obtained according to statistical information in historical data.

9. The method according to any one of claims 1 to 7, characterized in that: The recognition result also includes the keyword fragment; The method further comprises: Presenting keywords and corresponding keyword segments identified in the multimedia data to a user; Receive verification information from a user on the keywords and corresponding keyword segments recognized in the multimedia data, and obtain the verified recognition results.

10. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: Update the sample data set according to the recognition result; The model is updated using the updated sample data set.

11. A multimedia data mining device, characterized in that: include: an acquisition module, used to acquire transcription results and translation results of multimedia data in a minority language, and to acquire a first keyword and a second keyword, wherein the transcription result is obtained by identifying the multimedia data in the minority language, the transcription result includes a first text, a confidence of at least one sentence in the first text, and a confidence of at least one word in the sentence, the first keyword is a keyword in the official language, and the second keyword is a keyword in the minority language obtained by translation; a calculation module, configured to obtain a confidence level of a keyword segment in the multimedia data based on the transcription result and the translation result in combination with the first keyword and the second keyword; an identification module, configured to obtain an identification result according to a confidence level of a keyword segment existing in the multimedia data and a keyword identification threshold, wherein the identification result includes the keywords identified in the multimedia data; The calculation module is specifically used for: Determining the confidence level of the second keyword in the transcription result according to the transcription result and the second keyword, and determining the confidence level of the first keyword in the translation result according to the translation result and the first keyword; The confidence level of the keyword segment in the multimedia data is obtained according to the confidence level of the second keyword in the transcription result and the confidence level of the first keyword in the translation result.

12. A multimedia data mining device, characterized in that: include: Processor, memory, system bus; The processor and the memory are connected via the system bus; The memory is used to store one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by the processor, the processor executes the method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device executes the method according to any one of claims 1 to 10.

Images