Audio essence segment determination method, electronic device, and computer-readable storage medium

Abstract

Embodiments of the present application disclose a method for determining an audio essence segment, an electronic device and a computer readable storage medium, wherein the method comprises: obtaining audio data and text content corresponding to the audio data; inputting the text content into a first supervised model to determine a first set of wonderful sentences in the text content, and inputting the text content into a first unsupervised model to determine a second set of wonderful sentences in the text content; determining an essence paragraph in the text content based on the first set of wonderful sentences and the second set of wonderful sentences; and determining audio data corresponding to the essence paragraph in the audio data as an audio essence segment. The present application can be applied to the field of audio processing technology, and combines supervised learning and unsupervised learning to determine the essence segment in the audio. Compared with the manner of using supervised learning in the prior art, the present application can reduce the manual labeling of the essence segment and reduce the labor cost.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of audio processing, and in particular to an audio essence segment determination method, an electronic device, and a computer readable storage medium. BACKGROUND

[0002] With the rapid development of multimedia technology, audio as a carrier that bears a large amount of rich semantic information is listened to and loved by more and more users. With the substantial increase in the amount of audio data, when a user searches for an audio file (such as a song, a radio program, etc.) that the user likes, the user may need to quickly browse through an essence segment (or a highlight segment) in the audio file.

[0003] Currently, a supervised learning-based machine learning model is mainly used to determine a highlight segment from an audio file. When the model is trained, a large amount of manual annotation (that is, manual finding of all possible essence segments) needs to be performed on essence segments contained in full-text content of a long audio in advance, and the labor cost is high. SUMMARY

[0004] The present application provides an audio essence segment determination method, an electronic device, and a computer readable storage medium, which can reduce manual annotation of essence segments and reduce labor costs.

[0005] In a first aspect, an audio essence segment determination method is provided, and the method includes:

[0006] obtaining audio data and text content corresponding to the audio data;

[0007] inputting the text content into a first supervised model to determine a first highlight sentence set in the text content, and inputting the text content into a first unsupervised model to determine a second highlight sentence set in the text content;

[0008] determining an essence paragraph in the text content based on the first highlight sentence set and the second highlight sentence set;

[0009] determining audio data corresponding to the essence paragraph in the audio data as an audio essence segment.

[0010] The method described in the first aspect can determine an essence segment in audio based on supervised learning and unsupervised learning, which can reduce manual annotation of essence segments and reduce labor costs compared with the way of using supervised learning in the prior art.

[0011] In a possible implementation, the inputting of the text content into the first supervised model to determine the first highlight sentence set in the text content includes:

[0012] inputting the text content into the first supervised model to obtain a first index value of each sentence in the text content, and determining a first highlight sentence set based on the first index value;

[0013] The inputting the text content into the first unsupervised model to determine a second highlight sentence set in the text content comprises:

[0014] inputting the text content into the first unsupervised model to obtain a second index value of each sentence in the text content, and determining a second highlight sentence set based on the second index value;

[0015] The first index value or the second index value is any one or more of the following: a highlight score of each sentence in the text content, a semantic similarity of each sentence in the text content to the text content, and a probability of each sentence in the text content being a summary of the text content.

[0016] Based on this mode, the index parameter can be used to accurately determine whether a sentence is a highlight sentence.

[0017] In a possible implementation, the inputting the text content into the first supervised model to determine a first highlight sentence set in the text content comprises:

[0018] inputting the text content and an audio signal of the audio data into the first supervised model to determine a first highlight sentence set in the text content;

[0019] The inputting the text content into the first unsupervised model to determine a second highlight sentence set in the text content comprises:

[0020] inputting the text content and an audio signal of the audio data into the first unsupervised model to determine a second highlight sentence set in the text content.

[0021] Based on this mode, the text content and the audio signal can be combined to determine the highlight sentence set, and the accuracy of determining the highlight sentence set is improved.

[0022] In a possible implementation, the inputting the text content into the first supervised model to determine a first highlight sentence set in the text content comprises:

[0023] inputting the text content into the first supervised model and inputting an audio signal of the audio data into a second supervised model to determine a first highlight sentence set in the text content;

[0024] The inputting the text content into the first unsupervised model to determine a second highlight sentence set in the text content comprises:

[0025] input the text content into a first unsupervised model and input the audio signal of the audio data into a second unsupervised model to determine a second highlight sentence set in the text content.

[0026] Based on this manner, inputting the text content and the audio signal into different supervised models or unsupervised models to determine the highlight sentence set can decouple the dependence between the text content and the audio signal and improve the number of determined highlight sentences.

[0027] In a possible implementation, the inputting the text content into the first supervised model and the inputting the audio signal of the audio data into the second supervised model to determine the first highlight sentence set in the text content includes:

[0028] inputting the text content into the first supervised model to obtain a first index value of each sentence in the text content, inputting the audio signal of the audio data into the second supervised model to obtain a third index value of each sentence in the text content, and determining the first highlight sentence set in the text content based on the first index value and the third index value.

[0029] The inputting the text content into the first unsupervised model and the inputting the audio signal of the audio data into the second unsupervised model to determine the second highlight sentence set in the text content includes:

[0030] inputting the text content into the first unsupervised model to obtain a second index value of each sentence in the text content, inputting the audio signal of the audio data into the second unsupervised model to obtain a fourth index value of each sentence in the text content, and determining the second highlight sentence set in the text content based on the second index value and the fourth index value.

[0031] The first index value or the second index value or the third index value or the fourth index value is any one or more of the following: a highlight score of each sentence in the text content, a semantic similarity of each sentence in the text content to the text content, and a probability that each sentence in the text content is a summary of the text content.

[0032] Based on this manner, whether a sentence is a highlight sentence can be accurately determined by using multiple index parameters.

[0033] In a possible implementation, the determining the highlight paragraph in the text content based on the first highlight sentence set and the second highlight sentence set includes:

[0034] determining a highlight sentence union set of the first highlight sentence set and the second highlight sentence set.

[0035] determine an essence paragraph in the text content based on the union of the highlight sentences.

[0036] Based on this manner, the highlight sentence sets determined by two models can be first synthesized, and then the highlight paragraph is determined based on the synthesized highlight sentences, thereby improving the accuracy of determining the highlight paragraph.

[0037] In a possible implementation, the determining the essence paragraph in the text content based on the first highlight sentence set and the second highlight sentence set comprises:

[0038] determining a first highlight paragraph based on the first highlight sentence set and a second highlight paragraph based on the second highlight sentence set;

[0039] determining a union of the first highlight paragraph and the second highlight paragraph as the essence paragraph in the text content.

[0040] Based on this manner, two highlight paragraphs can be respectively determined based on the highlight sentence sets determined by two models, and the final highlight paragraph is determined by synthesizing the two highlight paragraphs, thereby improving the accuracy of determining the highlight paragraph.

[0041] In a possible implementation, after determining the essence paragraph in the text content, and before determining the audio data corresponding to the essence paragraph in the audio data as the audio essence clip, the method further comprises:

[0042] inputting the essence paragraph and context adjacent sentences of the essence paragraph in the text content into a first deep learning model to obtain a probability that the context adjacent sentences and the essence paragraph belong to a same paragraph;

[0043] if the probability is greater than a probability threshold, adding the context adjacent sentences to the essence paragraph.

[0044] Based on this manner, the context information of the audio essence clip can be perfected, so that the processed audio essence clip is more complete in semantics and more fluent in syntax.

[0045] In a possible implementation, if the audio data is multiple, the audio essence clip comprises audio essence clips corresponding to the multiple audio data respectively, and the method further comprises:

[0046] determining a highlight degree score of the audio essence clip of each of the audio data respectively;

[0047] sorting the multiple audio essence clips based on the highlight degree scores to obtain a sorting result;

[0048] recommending an audio essence clip to a user based on the sorting result.

[0049] Based on this manner, the audio essence segment can be scored and sorted, and the sorting result will help subsequent recommendation distribution business and help accurately recommend the audio segment that the user likes.

[0050] In a second aspect, an embodiment of the present application provides an audio essence segment determination apparatus, which comprises:

[0051] An acquisition module, configured to acquire audio data and text content corresponding to the audio data;

[0052] A processing module, configured to input the text content into a first supervised model to determine a first wonderful sentence set in the text content;

[0053] input the text content into a first unsupervised model to determine a second wonderful sentence set in the text content;

[0054] determine an essence paragraph in the text content based on the first wonderful sentence set and the second wonderful sentence set.

[0055] determine audio data corresponding to the essence paragraph in the audio data as an audio essence segment.

[0056] In a possible implementation, the processing module inputs the text content into the first supervised model to determine the first wonderful sentence set in the text content in the following manner:

[0057] input the text content into the first supervised model to obtain a first index value of each sentence in the text content and determine the first wonderful sentence set based on the first index value;

[0058] The processing module inputs the text content into the first unsupervised model to determine the second wonderful sentence set in the text content in the following manner:

[0059] input the text content into the first unsupervised model to obtain a second index value of each sentence in the text content and determine the second wonderful sentence set based on the second index value;

[0060] The first index value or the second index value is any one or more of the following: a wonderful degree score of each sentence in the text content, a semantic similarity of each sentence in the text content to the text content, and a probability of each sentence in the text content being a summary of the text content.

[0061] In a possible implementation, the processing module inputs the text content into the first supervised model to determine the first wonderful sentence set in the text content in the following manner:

[0062] inputting the text content and an audio signal of the audio data into a first supervised model to determine a first highlight sentence set in the text content;

[0063] The processing module inputs the text content into a first unsupervised model to determine a second highlight sentence set in the text content in the following manner:

[0064] The processing module inputs the text content into a first unsupervised model to determine a second highlight sentence set in the text content in the following manner:

[0065] In a possible implementation, the processing module inputs the text content into a first supervised model to determine a first highlight sentence set in the text content in the following manner:

[0066] The processing module inputs the text content into a first supervised model and inputs an audio signal of the audio data into a second supervised model to determine a first highlight sentence set in the text content.

[0067] The processing module inputs the text content into a first unsupervised model to determine a second highlight sentence set in the text content in the following manner:

[0068] The processing module inputs the text content into a first unsupervised model and inputs an audio signal of the audio data into a second unsupervised model to determine a second highlight sentence set in the text content.

[0069] In a possible implementation, the processing module inputs the text content into a first supervised model and inputs an audio signal of the audio data into a second supervised model to determine a first highlight sentence set in the text content in the following manner:

[0070] The processing module inputs the text content into a first supervised model to obtain a first index value of each sentence in the text content, inputs an audio signal of the audio data into a second supervised model to obtain a third index value of each sentence in the text content, and determines a first highlight sentence set in the text content based on the first index value and the third index value.

[0071] The processing module inputs the text content into a first unsupervised model and inputs an audio signal of the audio data into a second unsupervised model to determine a second highlight sentence set in the text content in the following manner: the processing module inputs the text content into the first unsupervised model to obtain a second index value of each sentence in the text content, inputs the audio signal of the audio data into the second unsupervised model to obtain a fourth index value of each sentence in the text content, and determines a second highlight sentence set in the text content based on the second index value and the fourth index value.

[0072] The first index value or the second index value or the third index value or the fourth index value is any one or more of the following: a highlight score of each sentence in the text content, a semantic similarity between each sentence in the text content and the text content, and a probability that each sentence in the text content is a summary of the text content.

[0073] In a possible implementation, the processing module determines the essence paragraph in the text content based on the first highlight sentence set and the second highlight sentence set in the following manner:

[0074] determining a highlight sentence union of the first highlight sentence set and the second highlight sentence set;

[0075] determining the essence paragraph in the text content based on the highlight sentence union.

[0076] In a possible implementation, the processing module determines the essence paragraph in the text content based on the first highlight sentence set and the second highlight sentence set in the following manner:

[0077] determining a first highlight paragraph based on the first highlight sentence set and a second highlight paragraph based on the second highlight sentence set;

[0078] determining a highlight sentence union of the first highlight paragraph and the second highlight paragraph as the essence paragraph in the text content.

[0079] In a possible implementation, the processing module is further configured to input the essence paragraph and context adjacent sentences of the essence paragraph in the text content into a first deep learning model to obtain a probability that the context adjacent sentences and the essence paragraph belong to a same paragraph.

[0080] If the probability is greater than a probability threshold, the context adjacent sentences are added to the essence paragraph.

[0081] In a possible implementation, if the audio data is multiple, the audio essence segment includes multiple audio essence segments corresponding to the audio data respectively, and the processing module is further configured to determine a highlight score of each audio essence segment.

[0082] The multiple audio essence segments are sorted based on the highlight scores of the multiple audio essence segments to obtain a sorting result.

[0083] An audio essence segment is recommended to a user based on the sorting result.

[0084] In a third aspect, an electronic device is provided, which includes:

[0085] a memory for storing a computer program;

[0086] a processor for invoking the computer program from the memory to perform the method according to any one of the first aspect.

[0087] In a fourth aspect, the embodiments of the present application provide a computer readable storage medium, the computer storage medium storing a computer program, the computer program including program instructions, when the program instructions are executed by a processor, causing the processor to perform the method according to any one of the first aspect.

[0088] The advantages of the possible implementation manners in the second aspect to the fourth aspect can be referred to the corresponding description in the first aspect, and will not be described here. BRIEF DESCRIPTION OF DRAWINGS

[0089] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows.

[0090] Figure 1 is a schematic diagram of a communication system provided by the embodiments of the present application;

[0091] Figure 2 is a flowchart of a method for determining an audio essence segment provided by the embodiments of the present application;

[0092] Figure 3 is a flowchart of another method for determining an audio essence segment provided by the embodiments of the present application;

[0093] Figure 4 is a structural diagram of an apparatus for determining an audio essence segment provided by the embodiments of the present application;

[0094] Figure 5 is a structural diagram of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0095] The technical solutions of the embodiments of the present application will be described clearly and completely in the embodiments of the present application in combination with the drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0096] The terms "first" and "second" and the like in the description, claims and drawings of the present application are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. Furthermore, the terms "comprises", "comprising", "includes", "including" and the like are used herein to mean either "consists" or "consisting of" and include the terms "consists" and "consisting of". It is specifically intended that any total singular reference include the element only and does not exclude additional elements. It is also specifically intended that the use of "one" or "another" in the description of a process, method, product or device includes the use of more than one.

[0097] In order to better understand the scheme of the present application, the technical field and professional terms involved in the present application are introduced as follows:

[0098] I. Automatic Speech Recognition (ASR)

[0099] Automatic speech recognition is closely related to acoustics, phonetics, linguistics, digital signal processing theory, information theory, computer science and many other disciplines, and is a multi-disciplinary technical field. It converts input audio data into corresponding text data through acoustic models and language models. However, due to the diversity and complexity of audio signals, different acoustic models and language models result in different accuracy of the recognized text results.

[0100] II. Machine Learning (ML)

[0101] Machine learning is a multi-disciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and many other disciplines. It is a subject that studies how computers simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge structure to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent, and its applications are widespread in various fields of artificial intelligence. Machine learning / deep learning usually includes artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and inductive learning.

[0102] In one classification, machine learning includes supervised learning and unsupervised learning. Supervised learning learns from training samples with labels to classify or predict data outside the training sample set as much as possible. The training samples of unsupervised learning do not have labels. In learning, the feature relationship inside the training sample is analyzed, and finally the data outside the training sample set can be classified or predicted. As can be seen, in supervised learning, a large number of labels of training data need to be collected, and the setting of the labels is greatly influenced by the subjectivity of the marker.

[0103] The communication system of the embodiments of the present application is introduced as follows:

[0104] Referring to Figure 1 , Figure 1 is a schematic diagram of a communication system provided by the embodiments of the present application. As shown in Figure 1 , the communication system includes a terminal device 101 and an audio essence segment determination apparatus 102. The terminal device 101 is a device where a client of a playing platform is located, and is a device with an audio playing function, including but not limited to: a smart phone, a tablet computer, a notebook computer, and the like. The audio essence segment determination apparatus 102 is a background device of a playing platform or a chip in a background device, and can determine an essence segment in audio data. Exemplarily, the audio essence segment determination apparatus 102 can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform.

[0105] When one or more audio files (or audio data) are uploaded to the audio essence segment determination apparatus 102, the apparatus can extract and store the essence segments in the audio files. When the terminal device 101 browses and listens to the audio files, the audio essence segment determination apparatus 102 sends the stored essence segments in the audio files to the terminal device 101, so that the terminal device 101 outputs the essence segments through a loudspeaker or the like output device. The terminal device 101 and the audio essence segment determination apparatus 102 can be directly or indirectly connected through wired communication or wireless communication, which is not limited in the present application.

[0106] It should be noted that Figure 1The number of terminal devices 101 and audio highlight segment determining apparatuses 102 in the illustrated communication system can be one or more, and the present application does not limit this. For the convenience of description, the audio highlight segment determining apparatus 102 will be taken as an example of a server to further describe the audio highlight segment determining method provided by the embodiments of the present application.

[0107] Referring to Figure 2 A flowchart of an audio highlight segment determining method provided by an embodiment of the present application is shown. The audio highlight segment determining method comprises steps 201-204.

[0108] 201. The server acquires audio data.

[0109] In the embodiment of the present application, the audio data can be long audio podcast products, radio audio data, songs, etc.

[0110] The server can acquire one audio data or a batch of audio data. For example, the server can acquire an audio data when detecting that the audio data is online, so as to determine the highlight segment in the audio data. Alternatively, the server can acquire a batch of newly online audio data in a preset time period, so as to determine the highlight segment in the batch of audio data. Alternatively, the server can extract the highlight segment from the stored audio data, and the data source of the audio data is not limited in the embodiment of the present application.

[0111] The audio data can be a complete audio file. Alternatively, the audio data can be part of the audio data extracted from an audio file. For example, an audio file has a duration of 20 minutes. The audio data can be the audio data from the 10th minute to the 20th minute of the audio file.

[0112] 202. The server determines the text content corresponding to the audio data.

[0113] In a possible implementation, the server can use ASR technology to convert the audio data into text content aligned with the timestamp of the audio data. The text content can be sentence-level text content. The ASR technology can determine the pause point of each sentence in the audio data according to the audio features such as the tone pause in the audio data, and convert the audio data into sentence-level text content. The server can also determine the text content corresponding to the audio data by other speech recognition technology, and the embodiments of the present application do not limit this.

[0114] In a possible implementation, the server can also not acquire the text information corresponding to the audio data in a manner of voice recognition. For example, a song has corresponding lyrics information, a person recording a network audio program such as a podcast product is equipped with subtitle information, and the like. The lyrics information, the subtitle information, and the like can be uploaded to the server as the text content corresponding to the audio data at the same time when the audio data is acquired.

[0115] 203. The server determines the highlight passage in the text content based on the first supervised model and the first unsupervised model.

[0116] In the embodiments of the present application, the server can determine one or more highlight passages in the text content. The following introduces three specific implementation manners in which the server determines the highlight passage in the text content based on the first supervised model and the first unsupervised model.

[0117] Manner one:

[0118] The server inputs the text content into the first supervised model to determine a first highlight sentence set in the text content; the server inputs the text content into the first unsupervised model to determine a second highlight sentence set in the text content; and the server determines the highlight passage in the text content based on the first highlight sentence set and the second highlight sentence set.

[0119] The first highlight sentence set includes N1 highlight sentences, and the second highlight sentence set includes N2 highlight sentences. N1 and N2 are positive integers greater than or equal to 1. The highlight sentences in the first highlight sentence set can be the same as or different from the highlight sentences in the second highlight sentence set. Each highlight passage in the text content can be composed of a plurality of continuous highlight sentences in the N1 highlight sentences and the N2 highlight sentences.

[0120] Optionally, the server can first determine the union set of the highlight sentences in the first highlight sentence set and the second highlight sentence set, and then determine the highlight passage according to the union set of the highlight sentences.

[0121] Exemplarily, if the text content includes 50 sentences, represented as X={x1, x2, x3, x4,..., x50}, xi represents the i-th sentence in the text content, the first highlight sentence set obtained through the first supervised model is X1={x1, x2, x3, x4, x5, x6, x7, x8, x9, x10, x11, x12, x13, x14, x15, x16, x17, x18, x19, x20, x21, x22, x23, x24, x25, x26, x27, x28, x29, x30, x31, x32, x33, x34, x35, x36, x37, x38, x39, x40, x41, x42, x43, x44, x45, x46, x47, x48, x49, x50}, and the second highlight sentence set obtained through the first unsupervised model is X2={x1, x2, x3, x4, x5, x6, x7, x8, x9, x10, x11, x12, x13, x14, x15, x16, x17, x18, x19, x20, x21, x22, x23, x24, x25, x26, x27, x28, x29, x30, x31, x32, x33, x34, x35, x36, x37, x38, x39, x40, x41, x42, x43, x44, x45, x46, x47, x48, x49, x50}. 50 i 16 17 30 31 35 36 40 41}​​​​​​​​​18 x 25 x 28 x 31 x 32 x 33 x 34 x 35 The union of X1 and X2 is determined to be {x1, x2, x3, x4, x5, x...}. 16 x 17 x 18 x 25 x 28 x 30 x 31 x 32 x 33 x 34 x 35 x 36 x 40 x 41 Based on the union of X1 and X2, four essential segments can be identified: {x1, x2, x3, x4, x5}, {x...}. 16 x 17 x 18}、{x 30 x 31 x 32 x 33 x 34 x 35 x 36}、{x 40 x 41}

[0122] Optionally, the minimum number of key sentences contained in the essential paragraph can be set, for example, to 2, 3, 4, 5, etc. This application does not impose any restrictions on this.

[0123] Optionally, the server can first determine the first essential paragraph based on the first set of excellent sentences, and then determine the second essential paragraph based on the second set of excellent sentences. The union of the essential paragraphs from the first and second sets is then used to determine the essential paragraphs within the text content.

[0124] For example, regarding the text content X in the aforementioned example, if the first set of excellent sentences X1 = {x1, x2, x3, x4, x...} 16 x 17 x 18 x 19 x 35 x 36 x 40 x 41}, then the exciting paragraphs determined by X1 are {x1, x2, x3, x4}, {x 16, x 17 , x 18 , x 19}, {x 35 , x 36}, {x 40 , x 41}. If the second highlight sentence set X2 = {x1, x2, x3, x4, x 20 , x 21 , x 22}, the highlight passages determined by X2 are {x1, x2, x3, x4}, {x 20 , x 21 , x 22}. The final highlight passages are the union of all the above highlight passages, which are {x1, x2, x3, x4}, {x 16 , x 17 , x 18 , x 19}, {x 20 , x 21 , x 22}, {x 35 , x 36}, {x 40 , x 41}.

[0125] In one possible implementation, the specific implementation of the server inputting the text content into the first supervised model to determine the first highlight sentence set in the text content and the server inputting the text content into the first unsupervised model to determine the second highlight sentence set in the text content is: the server inputting the text content into the first supervised model to determine the first index value of each sentence in the text content; the server determining the first highlight sentence set based on the first index value; the server inputting the text content into the first unsupervised model to obtain the second index value of each sentence in the text content; and the server determining the second highlight sentence set based on the second index value. The first index value or the second index value is one or more of the following types: the highlight degree score of each sentence in the text content, the semantic similarity of each sentence in the text content to the text content, and the probability that each sentence in the text content is a summary of the text content.

[0126] wherein the summary is another description form of the highlight sentence, and the probability that each sentence in the text content is a summary of the text content refers to the probability that each sentence in the text content is a highlight sentence. The first supervised model and the first unsupervised model are pre-trained models that can determine the highlight degree of each sentence in the text content. Moreover, the training data used in the training of the first supervised model contains the highlight degree annotation of each sentence, and the first unsupervised model does not need to annotate the highlight degree of each sentence in the training data during the training.

[0127] When the server determines the set of excellent sentences based on indicator values, if the first indicator value of a sentence is not less than the first indicator threshold, the sentence is determined to be an excellent sentence and added to the first set of excellent sentences; similarly, if the second indicator value of a sentence is not less than the second indicator threshold, the sentence is determined to be an excellent sentence and added to the second set of excellent sentences. The first indicator threshold and the second indicator threshold can be the same or different, and this application does not impose any restrictions on this.

[0128] Taking the first indicator parameter as the eloquence score of each sentence in the text content as an example, the specific process of determining the first set of eloquent sentences is explained. For example, for the aforementioned text content X, for the first 10 sentences in X {x1, x2, x3, x4, x5, x6, x7, x8, x9, x...} 10 The brilliance scores for each sentence are {70, 70, 80, 85, 60, 67, 56, 67, 48, 50}. If the first indicator threshold is 70, then the brilliance scores of the first to fourth sentences in X are not less than the first indicator threshold, and they can be added to the first set of brilliant sentences, {x1, x2, x3, x4}. The specific process for determining the second set of brilliant sentences can be found in the process for determining the first set of brilliant sentences, and will not be elaborated here.

[0129] It should be noted that the supervised models described above can employ traditional machine learning algorithms such as K-nearest neighbors, decision trees, and Naive Bayes, or deep learning algorithms such as convolutional neural networks (CNNs). The unsupervised learning models described above can employ algorithms such as k-means, autoencoders, principal component analysis, generative adversarial networks (GANs), or other deep learning algorithms; this application makes no restrictions on these methods.

[0130] Method 2:

[0131] The server inputs the audio signals of the text content and audio data into a first supervised model to determine the first set of key sentences in the text content; the server inputs the audio signals of the text content and audio data into a first unsupervised model to determine the second set of key sentences in the text content; based on the first set of key sentences and the second set of key sentences, the server determines the essential paragraphs in the text content.

[0132] The audio signal of the audio data includes timbre, tone and other characteristics. The text content and the audio signal are input into the first supervised model and the first unsupervised model, so that the first supervised model and the first unsupervised model can extract multi-modal features as a basis for determining the first highlight sentence set and the second highlight sentence set.

[0133] Optionally, the text content and the audio signal of the audio data are input into the first supervised model to determine a first index value of each sentence in the text content; the first highlight sentence set is determined based on the first index value; the text content and the audio signal of the audio data are input into the first unsupervised model to obtain a second index value of each sentence in the text content; and the second highlight sentence set is determined based on the second index value.

[0134] The possible forms of the first index value or the second index value and the specific process of determining the two highlight sentence sets and the essence paragraph in the text content according to the two highlight sentence sets can be referred to the corresponding description in Mode 1, which will not be described here in detail.

[0135] It should be noted that when training the first supervised model, the text content of the training audio data, the audio signal of the training audio data, and the corresponding highlight degree label are also input into the model for training. When training the first unsupervised model, the text content of the training audio data and the audio signal of the training audio data are input into the model for training. Thus, the trained first supervised model or first unsupervised model can extract the text features and audio signal features in the newly input audio data and determine the highlight degree of the audio data to determine the first index value or the second index value.

[0136] Mode 3:

[0137] The server inputs the text content into the first supervised model and inputs the audio signal of the audio data into the second supervised model to determine a first highlight sentence set in the text content; the server inputs the text content into the first unsupervised model and inputs the audio signal of the audio data into the second unsupervised model to determine a second highlight sentence set in the text content; and the server determines an essence paragraph in the text content based on the first highlight sentence set and the second highlight sentence set.

[0138] This mode can use different models to decouple and extract different features, avoid the mutual dependence between features, and improve the number of determined highlight sentences.

[0139] Optionally, the server can also obtain the corresponding index values after inputting the text content and the audio signal into different supervised models or unsupervised models, and determine the highlight sentence set according to the index values. For example, the server determines the first index values of each sentence in the text content after inputting the text content into the first supervised model, determines the third index values of each sentence corresponding to the audio signal after inputting the audio signal into the second supervised model, and determines the first highlight sentence set based on the first index values and the third index values. In addition, the server determines the second index values of each sentence in the text content after inputting the text content into the first unsupervised model, determines the fourth index values of each sentence corresponding to the audio signal after inputting the audio signal into the second unsupervised model, and determines the second highlight sentence set based on the second index values and the fourth index values.

[0140] The possible forms of the third index values and the fourth index values in this mode can refer to the possible forms of the first index values or the second index values described above, and how to determine the highlight paragraph in the text content based on the two highlight sentence sets can refer to the corresponding description in Mode 1, which will not be described here in detail.

[0141] In a possible implementation, the specific implementation of the server determining the first highlight sentence set based on the first index values and the third index values, or determining the second highlight sentence set based on the second index values and the fourth index values is that the server adds the union of the highlight sentences determined based on the first index values and the highlight sentences determined based on the third index values to the first highlight sentence set, and adds the union of the highlight sentences determined based on the second index values and the highlight sentences determined based on the fourth index values to the second highlight sentence set. For example, the highlight scores of 10 sentences obtained after inputting the 10 sentences into the first supervised model are {50, 60, 65, 70, 72, 73, 75, 80, 65, 40} in turn, and the 4th sentence to the 8th sentence are determined as highlight sentences according to the highlight scores. The highlight scores of 10 sentences obtained after inputting the audio signals corresponding to the 10 sentences into the second supervised model are {46, 52, 70, 73, 70, 75, 62, 60, 60, 50} in turn, and the 3rd sentence to the 6th sentence are determined as highlight sentences according to the highlight scores. Therefore, the first highlight sentence set can be the union of the two highlight sentence results, including the 3rd sentence to the 8th sentence.

[0142] In another possible implementation, the specific implementation in which the server determines the first highlight sentence set based on the first index value and the third index value, or determines the second highlight sentence set based on the second index value and the fourth index value is as follows: When determining the first highlight sentence set or the second highlight sentence set, the server can set weights for the two obtained index parameters to obtain new index parameters, and determine the highlight sentence set based on the new index parameters. For example, after obtaining the first index parameter and the third index parameter output by the two supervised models, the server multiplies the first index parameter by a first weight and multiplies the third index parameter by a second weight to obtain a fifth index parameter, and determines the first highlight sentence set based on the fifth index parameter. Exemplarily, after inputting 10 sentences into the first supervised model, the highlight scores of the 10 sentences are {50, 60, 65, 70, 72, 73, 75, 80, 65, 40} in sequence, and after inputting the audio signals corresponding to the 10 sentences into the second supervised model, the highlight scores of the 10 sentences are {46, 52, 70, 73, 70, 75, 62, 60, 60, 50} in sequence. If the first weight is 0.7 and the second weight is 0.3, the final highlight score of the first sentence of the 10 sentences is 50*0.7+46*0.3=48.8, and the final highlight scores of the other sentences are obtained in the same manner. The final highlight scores of the 10 sentences are {48.8, 57.6, 66.5, 70.9, 71.4, 73.6, 71.1, 74, 63.5, 43}, and the sentences greater than the first index threshold are the 4th to 8th sentences. The 4th to 8th sentences are taken as the highlight sentences and added to the first highlight sentence set.

[0143] 204、The server determines the audio data corresponding to the essence paragraph in the audio data as the audio essence segment.

[0144] Specifically, the essence paragraph is composed of highlight sentences in the text content, and the audio essence segment can be determined by searching for a segment in the entire audio data that is aligned with the time stamp of the essence paragraph. For example, the highlight sentences in the essence paragraph are the 20th to 220th sentences in the text content, and the audio data corresponding to the time stamp of 17:30 to 17:31 is determined as the audio essence segment.

[0145] Based on Figure 2 In the illustrated embodiment, supervised learning and unsupervised learning can be combined to determine the essence segment in the audio data by inputting the text information of the audio data. Compared with the prior art using supervised learning, the manual labeling of the essence segment can be reduced, and the labor cost can be reduced.

[0146] Referring to Figure 3This is a flowchart illustrating another method for determining audio essence segments provided in this application. The method includes steps 301 to 306. Wherein:

[0147] 301. The server retrieves audio data.

[0148] 302. The server determines the text content corresponding to the audio data.

[0149] 303. The server identifies the essential paragraphs in the text content based on the first supervised model and the first unsupervised model.

[0150] For specific implementation methods of 301 to 303, please refer to the corresponding descriptions in 201 to 203 above, which will not be repeated here.

[0151] 304. The server inputs the featured paragraph and its contextual adjacent sentences into the first deep learning model to determine whether the contextual adjacent sentences need to be added to the featured paragraph.

[0152] In one possible implementation, 304 is implemented as follows: the server inputs the essential paragraph and its contextual adjacent sentences into the first deep learning model to obtain the probability that the contextual adjacent sentences belong to the same segment as the essential paragraph; if the probability is greater than the probability threshold, the contextual adjacent sentences are added to the essential paragraph.

[0153] Specifically, the first step is to identify the contextual adjacent sentences of the essential paragraph. These adjacent sentences can be one or more sentences preceding the starting sentence of the essential paragraph, or one or more sentences following the ending sentence of the essential paragraph. The contextual adjacent sentences and the essential paragraph are then input into a first deep learning model. By analyzing word similarity, semantic similarity, and grammatical logic relationships between the contextual adjacent sentences and sentences in the essential paragraph, the probability that the contextual adjacent sentences and the essential paragraph belong to the same segment is determined. If this probability is greater than a probability threshold (e.g., 0.8), the contextual adjacent sentences and the essential paragraph are considered to belong to the same segment and are added to the essential paragraph. If the probability is less than the probability threshold (e.g., 0.8), the contextual adjacent sentences and the essential paragraph are considered not to belong to the same segment and are not added to the essential paragraph. The sentence preceding the starting sentence and the sentence following the ending sentence of the essential paragraph are most likely to belong to the same segment as the essential paragraph. Therefore, the closest sentences are judged first. If the closest sentences do not meet the above conditions, the judgment of the remaining sentences can be stopped. If the above conditions are met, the next closest sentence is judged, and so on. This possible implementation method can improve the contextual information of the essential paragraphs, making the processed essential paragraphs more semantically complete and grammatically fluent.

[0154] 305、The server determines the audio data corresponding to the essence paragraph in the audio data as the audio essence segment.

[0155] The specific implementation of step 305 can refer to the corresponding description in 204 described above, and details are not described herein.

[0156] 306、The server determines the highlight score of the multiple audio essence segments, and recommends the audio essence segment to the user based on the highlight score.

[0157] In a possible implementation, if the audio data is multiple, the audio essence segment includes the audio essence segment corresponding to each of the multiple audio data, and the specific implementation of step 306 is: the server respectively determines the highlight score of the audio essence segment of each audio data; sorts the multiple audio essence segments based on the highlight score to obtain a sorting result; and recommends the audio essence segment to the user based on the sorting result.

[0158] The server can input the corresponding essence paragraph into the second deep learning model, the second deep learning model can extract feature vectors for representing text themes, plots, characters, etc. from the essence paragraph according to the semantics of the essence paragraph, and evaluate the highlight score of the audio essence segment according to the feature vectors. The multiple audio essence segments are sorted according to the highlight score obtained by the evaluation, and the sorting result is obtained, and the audio essence segment is recommended to the user based on the sorting result.

[0159] Optionally, the server can store the correspondence between the audio essence segment and the highlight score of the audio essence segment. When it is necessary to push the audio to the user, one or more audio essence segments with higher highlight scores are determined according to the correspondence and pushed to the user. The audio essence segment is sent to the terminal device of the user by the server, and is rendered by the terminal device of the user, and is played through the loudspeaker.

[0160] In this way, the highlight of the audio essence segment is further scored, and the audio essence segment is sorted, and the sorting result will help the subsequent recommendation and distribution business, and help to accurately recommend the favorite audio segment to the user.

[0161] Based on Figure 3 As shown in the embodiment, after the essence paragraph is determined, context information can be further added to the essence paragraph to improve the completeness of the audio essence segment, and the highlight of the audio essence segment is scored and sorted to obtain an audio segment that is more representative and more liked by the user.

[0162] Referring to Figure 4 is a structural schematic diagram of an audio essence segment determination device provided by an embodiment of the present application, the device 40 includes an acquisition module 401 and a processing module 402. Wherein:

[0163] The acquisition module 401 is configured to acquire audio data and text content corresponding to the audio data.

[0164] The processing module 402 is configured to input the text content into a first supervised model to determine a first set of wonderful sentences in the text content, and input the text content into a first unsupervised model to determine a second set of wonderful sentences in the text content.

[0165] The processing module 402 is further configured to determine an essence paragraph in the text content based on the first set of wonderful sentences and the second set of wonderful sentences.

[0166] The processing module 402 is further configured to determine audio data corresponding to the essence paragraph in the audio data as an audio essence segment.

[0167] In a possible implementation, the processing module 402 inputs the text content into the first supervised model to determine the first set of wonderful sentences in the text content in the following manner: the text content is input into the first supervised model to obtain a first index value of each sentence in the text content; and the first set of wonderful sentences is determined based on the first index value.

[0168] The processing module 402 inputs the text content into the first unsupervised model to determine the second set of wonderful sentences in the text content in the following manner: the text content is input into the first unsupervised model to obtain a second index value of each sentence in the text content; and the second set of wonderful sentences is determined based on the second index value.

[0169] The first index value or the second index value is any one or more of the following: a wonderful degree score of each sentence in the text content, a semantic similarity of each sentence in the text content to the text content, and a probability of each sentence in the text content being an abstract of the text content.

[0170] In a possible implementation, the processing module 402 inputs the text content into the first supervised model to determine the first set of wonderful sentences in the text content in the following manner: the text content and an audio signal of the audio data are input into the first supervised model to determine the first set of wonderful sentences in the text content.

[0171] The processing module 402 inputs the text content into the first unsupervised model to determine the second set of wonderful sentences in the text content in the following manner: the text content and the audio signal of the audio data are input into the first unsupervised model to determine the second set of wonderful sentences in the text content.

[0172] In a possible implementation, the processing module 402 inputs the text content into the first supervised model to determine the first set of wonderful sentences in the text content in the following manner:

[0173] The processing module 402 inputs the text content into the first supervised model and inputs the audio signal of the audio data into the second supervised model to determine the first highlight sentence set in the text content.

[0174] The processing module 402 inputs the text content into the first unsupervised model and inputs the audio signal of the audio data into the second unsupervised model to determine the second highlight sentence set in the text content.

[0175] In a possible implementation, the processing module 402 inputs the text content into the first supervised model and inputs the audio signal of the audio data into the second supervised model to determine the first highlight sentence set in the text content in the following manner:

[0176] The processing module 402 inputs the text content into the first supervised model to obtain a first index value of each sentence in the text content, inputs the audio signal of the audio data into the second supervised model to obtain a third index value of each sentence in the text content, and determines the first highlight sentence set in the text content based on the first index value and the third index value.

[0177] The processing module 402 inputs the text content into the first unsupervised model and inputs the audio signal of the audio data into the second unsupervised model to determine the second highlight sentence set in the text content in the following manner:

[0178] The processing module 402 inputs the text content into the first unsupervised model to obtain a second index value of each sentence in the text content, inputs the audio signal of the audio data into the second unsupervised model to obtain a fourth index value of each sentence in the text content, and determines the second highlight sentence set in the text content based on the second index value and the fourth index value.

[0179] The first index value or the second index value or the third index value or the fourth index value is any one or more of the following: a highlight score of each sentence in the text content, a semantic similarity of each sentence in the text content to the text content, and a probability that each sentence in the text content is an abstract of the text content.

[0180] In a possible implementation, the processing module 402 determines the essence paragraph in the text content based on the first highlight sentence set and the second highlight sentence set in the following manner:

[0181] Determines a highlight sentence union set of the first highlight sentence set and the second highlight sentence set.

[0182] Determines the essence paragraph in the text content based on the highlight sentence union set.

[0183] In a possible implementation, the processing module 402 determines the specific manner of the essence paragraph in the text content based on the first highlight sentence set and the second highlight sentence set as follows:

[0184] determining the first highlight paragraph based on the first highlight sentence set and determining the second highlight paragraph based on the second highlight sentence set;

[0185] determining the union of the first highlight paragraph and the second highlight paragraph as the essence paragraph in the text content.

[0186] In a possible implementation, the processing module 402 is further configured to input the essence paragraph and context-adjacent sentences of the essence paragraph in the text content into the first deep learning model to obtain a probability that the context-adjacent sentences belong to the same paragraph as the essence paragraph, and add the context-adjacent sentences to the essence paragraph if the probability is greater than a probability threshold.

[0187] In a possible implementation, if the audio data is multiple, the audio essence segment includes audio essence segments corresponding to the multiple audio data respectively, and the processing module 402 is further configured to determine the highlight scores of the audio essence segments respectively.

[0188] sort the multiple audio essence segments based on the highlight scores of the multiple audio essence segments to obtain a sorting result.

[0189] recommend the audio essence segment to the user based on the sorting result.

[0190] It should be noted that the functions of the modules of the audio essence segment determination apparatus according to the embodiments of the present application can be implemented according to the methods in the above method embodiments, and the specific implementation process and beneficial effects can be referred to the related descriptions of the above method embodiments, which will not be repeated here.

[0191] Referring to Figure 5 FIG. 1 shows a structural schematic diagram of an electronic device according to an embodiment of the present application. The electronic device 50 can include one or more processors 501, a memory 502, and a transceiver 503. The processor 501, the memory 502, and the transceiver 503 are connected through a bus 504. The memory 502 is configured to store a computer program including program instructions, and the processor 501 and the transceiver 503 are configured to execute the program instructions stored in the memory 502 to perform the following operations:

[0192] obtain audio data and text content corresponding to the audio data;

[0193] input the text content into a first supervised model to determine a first highlight sentence set in the text content, and input the text content into a first unsupervised model to determine a second highlight sentence set in the text content;

[0194] Based on the first and second sets of excellent sentences, the essential paragraphs in the text content are identified.

[0195] The audio data corresponding to the most important segments in the audio data is identified as the audio essence segments.

[0196] It should be understood that in some feasible implementations, the processor 501 described above may be a central processing unit (CPU), which may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor. The memory 502 may include read-only memory and random access memory, and provides instructions and data to the processor 501. A portion of the memory 502 may also include non-volatile random access memory. For example, the memory 502 may also store device type information.

[0197] In specific implementation, the aforementioned terminal device can perform the above-described actions through its built-in functional modules. Figure 2 , Figure 3 The implementation methods, specific implementation processes, and beneficial effects provided for each step can be found in the implementation methods provided for each step above, and will not be repeated here.

[0198] This application embodiment also provides a computer-readable storage medium storing computer-readable instructions executed by the aforementioned audio synthesis apparatus, wherein the computer-readable instructions include program instructions, and when the processor executes the program instructions, it can perform the aforementioned... Figure 2 , Figure 3 The methods described in the corresponding embodiments will not be repeated here. Furthermore, the beneficial effects of using the same methods will also not be repeated. For technical details not disclosed in the computer-readable storage medium embodiments related to this application, please refer to the description of the method embodiments of this application. As an example, program instructions can be deployed on a computer device, or executed on multiple computer devices located in one location, or executed on multiple computer devices distributed across multiple locations and interconnected via a communication network. These multiple computer devices distributed across multiple locations and interconnected via a communication network can constitute a blockchain system.

[0199] According to an aspect of the present application, a computer program product or computer program is provided, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device can perform the method in the above-mentioned Figure 2 、 Figure 3 、 Figure 4 、 Figure 5 embodiments, and thus, will not be described again here. A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The above-mentioned program can be stored in a computer readable storage medium, and when the program is executed, the program can include the processes of the above-mentioned embodiments. The above-mentioned storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM), a random access memory (RAM), or the like.

[0200] The above describes only specific embodiments of the present application, but the protection scope of the present application is not limited to this. Any person of ordinary skill in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An audio essence segment determination method, characterized in that, The method comprises: acquiring audio data and text content corresponding to the audio data; inputting the text content into a first supervised model to obtain a first index value of each sentence in the text content, inputting an audio signal of the audio data into a second supervised model to obtain a third index value of each sentence in the text content, determining a first highlight sentence set in the text content based on the first index value and the third index value, inputting the text content into a first unsupervised model to obtain a second index value of each sentence in the text content, inputting the audio signal of the audio data into a second unsupervised model to obtain a fourth index value of each sentence in the text content, and determining a second highlight sentence set in the text content based on the second index value and the fourth index value; determining an essence paragraph in the text content based on the first highlight sentence set and the second highlight sentence set; inputting the essence paragraph and context adjacent sentences of the essence paragraph in the text content into a first deep learning model to obtain a probability that the context adjacent sentences and the essence paragraph belong to a same paragraph; the first deep learning model is used to analyze a preset relationship between the context adjacent sentences and sentences in the essence paragraph in the text content, and the preset relationship comprises word similarity, semantic similarity and syntax logical relationship; if the probability is greater than a probability threshold, the context adjacent sentences are added to the essence paragraph; determining audio data corresponding to the essence paragraph in the audio data as an audio essence clip.

2. The method of claim 1, wherein, The first index value or the second index value or the third index value or the fourth index value is any one or more of the following: a highlight score of each sentence in the text content, a semantic similarity between each sentence in the text content and the text content, and a probability that each sentence in the text content is a summary of the text content.

3. The method of claim 1, wherein, The determination of the essence paragraph in the text content based on the first highlight sentence set and the second highlight sentence set comprises: determining a highlight sentence union set of the first highlight sentence set and the second highlight sentence set; determining the essence paragraph in the text content based on the highlight sentence union set.

4. The method of claim 1, wherein, The determination of the essence paragraph in the text content based on the first highlight sentence set and the second highlight sentence set comprises: determining a first highlight paragraph based on the first highlight sentence set and a second highlight paragraph based on the second highlight sentence set; determining a union set of the first highlight paragraph and the second highlight paragraph as the essence paragraph in the text content.

5. The method according to any one of claims 1 to 4, characterized in that, If the audio data is multiple, the audio essence clip comprises audio essence clips corresponding to the multiple audio data respectively. The method further comprises: determining a highlight score of the audio essence clip of each audio data respectively; sorting the multiple audio essence clips based on the highlight scores to obtain a sorting result; recommending an audio essence clip to a user based on the sorting result.

6. An electronic device, comprising: The electronic device comprises a memory and a processor; the memory is used to store a computer program. The processor is configured to call the computer program from the memory to execute the method of any one of claims 1-5.

7. A computer readable storage medium characterized in that, The computer storage medium stores a computer program, and the computer program comprises program instructions. When the program instructions are executed by a processor, the processor is caused to execute the method of any one of claims 1-5.

8. A computer program product, comprising computer instructions stored in a computer readable storage medium, wherein a processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to cause the computer device to execute the method of any one of claims 1-5.

Images