Audio Summary Generation Method, Apparatus, Electronic Device, and Readable Storage Medium

Abstract

Embodiments of the present invention provide an audio abstract generation method, apparatus, electronic device, and readable storage medium, which relate to the field of computer technology. By obtaining multiple target key sentences from the speech recognition text corresponding to the target audio, the target key sentence represents a written complete sentence whose content matches the content of the target audio. Then, at least one is selected from the multiple target key sentences as a candidate summary sentence, and the candidate summary sentence represents a smoothly expressed summary content corresponding to the target audio. In this way, a corresponding candidate summary sentence can be automatically generated according to the speech recognition text corresponding to the target audio, eliminating the labor cost and time cost of manually writing an abstract for the target audio in the prior art.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and more particularly, to a method, apparatus, electronic device, and readable storage medium for generating an audio abstract. Background Art

[0002] Today, the consumption of Internet information content is booming, and a huge amount of new content is generated every day. In the audio scenario, recommending audio to users is based on the matching degree between the content of the audio title and introduction and the user tags, and the search results obtained by users using keywords to search for audio are also based on the matching degree between the keywords and the content of the audio title and introduction. Neither of these two methods involves audio abstracts. However, an audio abstract can reflect the key and wonderful parts of the audio content, and can show the essence of the audio to users or locate the key content of the abstract part according to user needs.

[0003] However, there is no related technology for automatically generating audio abstracts in the prior art, and the labor cost and time cost of manually listening to audio and writing abstracts are very high. Summary of the Invention

[0004] The purpose of the present invention is to provide a method, apparatus, electronic device, and readable storage medium for generating an audio abstract to improve the problems existing in the prior art.

[0005] Embodiments of the present invention may be implemented as follows:

[0006] In a first aspect, the present invention provides a method for generating an audio abstract, including:

[0007] Obtaining a plurality of target key sentences from the speech recognition text corresponding to the target audio; the target key sentence represents a complete written sentence whose content matches the content of the target audio;

[0008] Selecting at least one of the plurality of target key sentences as a candidate abstract sentence, where the candidate abstract sentence represents a smooth and summarizing content corresponding to the target audio.

[0009] In an optional implementation, the step of obtaining a plurality of target key sentences from the speech recognition text corresponding to the target audio includes:

[0010] Extracting a plurality of first key sentences from the speech recognition text; the first key sentence represents a complete sentence whose content matches the content of the target audio;

[0011] Using a sequence labeling model to respectively identify the punctuation error information existing in each first key sentence and correct each punctuation error information to obtain all the corrected first key sentences;

[0012] For each of the corrected first key sentence end-to-end rewriting models, obtain all the target key sentences; the end-to-end rewriting model is used to convert the colloquial expression part in the input text sequence into a written expression to obtain the target key sentence.

[0013] In an alternative embodiment, the step of extracting a plurality of first key sentences from the speech recognition text includes:

[0014] Use a classification model to screen at least one candidate sentence from the speech recognition text; the candidate sentence represents a sentence with complete expression and correct grammar;

[0015] Based on a plurality of evaluation dimensions, obtain the comprehensive score of each candidate sentence respectively; the plurality of evaluation dimensions include at least one or a combination of the importance degree of the candidate sentence, the matching degree with the field to which the target audio belongs, the relevance with the target audio, and the richness of vocabulary;

[0016] Determine the first key sentence from at least one of the candidate sentences according to all the comprehensive scores.

[0017] In an alternative embodiment, the step of selecting at least one of the plurality of target key sentences as a summary candidate sentence includes:

[0018] If it is recognized that any of the target key sentences contains a proper noun error, correct each proper noun error in combination with the proper noun information; the proper noun information is included in the title information and / or the introduction information of the target audio;

[0019] Use the GPT-2 model to calculate the fluency score of each corrected target key sentence;

[0020] Obtain the similarity information between each corrected target key sentence and the title information;

[0021] Determine at least one of the summary candidate sentences from the plurality of corrected target key sentences according to the similarity information and the fluency score.

[0022] In an alternative embodiment, before the step of obtaining a plurality of target key sentences from the speech recognition text corresponding to the target audio, the method further includes:

[0023] Obtain the speech recognition text corresponding to the target audio;

[0024] Clean up the redundant content in the speech recognition text.

[0025] In an alternative embodiment, the end-to-end rewriting model is trained in the following manner:

[0026] Obtain a training sample set and a corresponding set of reference statements; the training sample set includes multiple training samples, and the set of reference statements includes multiple reference statements;

[0027] Train a pre-established end-to-end rewriting model according to the training sample set and the set of reference statements to obtain a trained end-to-end rewriting model.

[0028] In an optional implementation manner, the step of obtaining the training sample set and the corresponding set of reference statements includes:

[0029] Obtain training audio and the corresponding title of the training audio from an audio database;

[0030] Obtain a target statement in the training audio that matches the title, use the target statement as a training sample, and use the title as the reference statement corresponding to the training sample; and / or,

[0031] Obtain training samples from an audio database and clean the training samples using a template to obtain reference statements; and / or,

[0032] Obtain an abstract from an open-source abstract task corpus as a reference statement;

[0033] Perform self-supervised data augmentation on the reference statements to add noise to obtain training samples.

[0034] In a second aspect, the present invention provides an audio abstract generation device, including:

[0035] An acquisition module, configured to obtain multiple target key sentences from the speech recognition text corresponding to the target audio; the target key sentence represents a written complete sentence whose content matches the content of the target audio;

[0036] A screening module, configured to select at least one from the multiple target key sentences as a summary candidate sentence, and the summary candidate sentence represents a smooth and summarizing content corresponding to the target audio.

[0037] In a third aspect, the present invention provides an electronic device, including: a memory and a processor, where the memory stores machine-readable instructions executable by the processor, and when the electronic device runs, the processor executes the machine-readable instructions to implement the method according to any one of the foregoing implementation manners.

[0038] In a fourth aspect, the present invention provides a readable storage medium, where the readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method according to any one of the foregoing implementation manners.

[0039] In summary, the embodiments of the present invention provide an audio abstract generation method, apparatus, electronic device, and readable storage medium, which relate to the field of computer technology. By obtaining multiple target key sentences from the speech recognition text corresponding to the target audio, the target key sentence represents a written complete sentence whose content matches the content of the target audio. Then, at least one is selected from the multiple target key sentences as the abstract candidate sentence, and the abstract candidate sentence represents a smooth summary content corresponding to the target audio. The beneficial effect is that the corresponding abstract candidate sentence can be automatically generated according to the speech recognition text corresponding to the target audio, eliminating the labor cost and time cost of manually writing an abstract for the target audio in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required in the embodiments. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0041] Figure 1 One of the flow diagrams of an audio abstract generation method provided by an embodiment of the present invention.

[0042] Figure 2 Another flow diagram of an audio abstract generation method provided by an embodiment of the present invention.

[0043] Figure 3 Another flow diagram of an audio abstract generation method provided by an embodiment of the present invention.

[0044] Figure 4 Another flow diagram of an audio abstract generation method provided by an embodiment of the present invention.

[0045] Figure 5 Another flow diagram of an audio abstract generation method provided by an embodiment of the present invention.

[0046] Figure 6 Module diagram of an audio abstract generation apparatus provided by an embodiment of the present invention.

[0047] Figure 7 Structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0048] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. Components of the embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a variety of different configurations.

[0049] Therefore, the detailed description of the embodiments of the present invention provided in the drawings below is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the scope of protection of the present invention.

[0050] It should be noted that like reference numerals and letters denote like items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0051] In the description of the present invention, it should be noted that if terms such as "upper", "lower", "inner", "outer", etc. are used to indicate the orientation or positional relationship, it is based on the orientation or positional relationship shown in the drawings or the orientation or positional relationship in which the product of the invention is usually placed during use. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus should not be construed as a limitation to the present invention.

[0052] In addition, terms such as "first" and "second" are only used for descriptive distinction and should not be construed as indicating or implying relative importance.

[0053] It should be noted that, without conflict, the features in the embodiments of the present invention may be combined with each other.

[0054] Nowadays, the consumption of Internet information content is booming, and a huge amount of new content is generated every day. In the audio scenario, how to quickly display the content essence of audio to users, or locate the key content of audio according to user needs, and generate an audio summary based on the audio content are exactly the technical difficulties that need to be overcome urgently at present.

[0055] In the prior art, audio recommendation to users is mostly based on the matching degree between the content of the audio title and the audio introduction and the user tags. When a user searches for an audio using keywords, the search results are also obtained by matching the input keywords with the keywords included in the audio title and the audio introduction. That is, in the audio scenario, neither of the two algorithms of user recommendation and user search recall involves the audio abstract. However, the audio abstract can reflect the wonderful key parts of the audio content, and can show the essence of the audio to the user or locate the key content of the abstract part according to the user's needs.

[0056] However, there is no related technology for automatically generating audio abstracts in the prior art, and the labor cost and time cost of manually listening to the audio and writing the abstract are very high.

[0057] In view of this, the embodiments of the present invention provide an audio abstract generation method, which can automatically generate corresponding abstract candidate sentences according to the speech recognition text corresponding to the target audio, eliminating the labor cost and time cost of manually writing the abstract for the target audio in the prior art. The following will be described in detail through embodiments and in conjunction with the accompanying drawings.

[0058] Please refer to Figure 1 , Figure 1 which is a schematic flowchart of an audio abstract generation method provided by an embodiment of the present invention. It should be noted that the audio abstract generation method of the present invention is not limited to the specific order of the steps described below. It should be understood that in other embodiments, the order of some steps of the audio abstract generation method of the present invention can be interchanged according to actual needs, or some of the steps can also be omitted or deleted. The steps of the method include:

[0059] S104. Obtain multiple target key sentences from the speech recognition text corresponding to the target audio.

[0060] In this embodiment, the target key sentence can be a written complete sentence whose content matches the content of the target key sentence. The content included in the target key sentence can match the content of the target audio, and the target key sentence can be a complete sentence in written expression form.

[0061] S105. Select at least one from the multiple target key sentences as the abstract candidate sentence.

[0062] In this embodiment, the summary candidate sentence can represent a well-expressed summary content corresponding to the target audio. The audio is uploaded after being recorded by the user. There may be situations where the user's expressions are mostly colloquial, and there may also be situations where the user's pronunciation is unclear. Correspondingly, there may be many colloquial expressions and unsmooth expressions in the corresponding speech recognition text. However, the summary candidate sentence finally obtained according to the speech recognition text in this embodiment can be a well-expressed summary content.

[0063] An audio summary generation method provided by an embodiment of the present invention obtains multiple target key sentences from a speech recognition text corresponding to a target audio, where the target key sentence represents a written complete sentence whose content matches the content of the target audio. Then, at least one is selected from the multiple target key sentences as the summary candidate sentence, and the summary candidate sentence represents a well-expressed summary content corresponding to the target audio. Its beneficial effect is that it can automatically generate a corresponding summary candidate sentence according to the speech recognition text corresponding to the target audio, eliminating the labor cost and time cost of manually writing a summary for the target audio in the prior art.

[0064] In an optional implementation manner, before processing the speech recognition text to obtain the summary candidate sentence, redundant content included in the speech recognition text can also be cleaned. Please refer to Figure 2 , correspondingly, before step S104, the method may further include the step of:

[0065] S102. Obtain the speech recognition text corresponding to the target audio.

[0066] In this embodiment, the speech recognition text can be obtained by performing speech recognition on the target audio, or can be pre-stored in an audio database.

[0067] S103. Clean the redundant content in the speech recognition text.

[0068] In this embodiment, there may be redundant content such as grammar errors and semantic redundancy noises in the speech recognition text, such as semantic repetitions, reduplicated words, modal particles, and filler words. The speech recognition text can be first cleaned using a rule template, and then the redundant expressions can be filtered by calculating the lexical richness of the speech recognition text.

[0069] In an optional example, the rule template can use regular expressions; in linguistics, the lexical richness can be measured by calculating TTR (Type token radio) and MTLD (Measure of Textual Lexical Diversity) metrics.

[0070] In an alternative embodiment, the target key sentence may be obtained by correcting the first key sentence extracted from the speech recognition text. Please refer to Figure 3 , correspondingly, step S104 may include sub-steps:

[0071] S104-1. Extract multiple first key sentences from the speech recognition text.

[0072] In this embodiment, the first key sentence may represent a complete sentence whose content matches the content of the target audio. That is, the content included in the first key sentence may match the content of the target audio, and the first key sentence may be a complete sentence.

[0073] S104-2. Use a sequence labeling model to identify the punctuation error information existing in each first key sentence respectively, and correct each punctuation error information to obtain all the corrected first key sentences.

[0074] In this embodiment, the sequence labeling model may be but is not limited to an LSTM (Long Short Term Memory) model, a Bert (Bidirectional Encoder Representations from Transformer) model, etc. Due to the speaking tone of the speaker in the target audio or the pause time between speaking intervals, there may be situations where punctuation marks are used incorrectly in the corresponding speech recognition text. The sequence labeling model can identify the punctuation error information in each first key sentence and modify it to obtain the corrected first key sentence.

[0075] S104-3. Input each corrected first key sentence into an end-to-end rewriting model respectively to obtain all the target key sentences.

[0076] In this embodiment, the end-to-end rewriting model can be used to convert the colloquial expression part in the input corrected first key sentence into a written expression to obtain the target key sentence. The end-to-end rewriting model can convert the colloquial expression part in the input text sequence into a written expression to obtain an output text sequence, and the expression of the output text sequence is smooth.

[0077] In an alternative embodiment, multiple first key sentences are screened out from the speech recognition text by using a classification model and determined after being scored by multiple evaluation dimensions. Please refer to Figure 4 , correspondingly, step S104-1 may include sub-steps:

[0078] S104-1a. Use a classification model to screen out at least one candidate sentence from the speech recognition text.

[0079] It can be understood that the alternative sentences can represent complete and grammatically correct sentences. The classification model can use the Bert classification model for binary classification to determine whether the beginning and end of the sentence are complete. The Bert classification model can use complete and grammatically correct sentences and sentences with missing words or incorrect sentence breaks at the beginning and end as training materials to train the pre-constructed Bert classification model.

[0080] S104-1b. Obtain the comprehensive score of each alternative sentence based on multiple evaluation dimensions respectively.

[0081] In this embodiment, the multiple evaluation dimensions may include at least one or a combination of the importance degree of the alternative sentence, the matching degree with the field to which the target audio belongs, the relevance with the target audio, and the richness of vocabulary.

[0082] Optionally, for the evaluation dimension of the importance degree of the alternative sentence, a topic model can be used to calculate the importance of the alternative sentence. The topic model can be, but is not limited to, the TextRank algorithm, the TextTeaser algorithm, the LDA (Latent Dirichlet Allocation) algorithm, the LSI (Latent Semantic Indexing) algorithm, and so on.

[0083] It can be understood that in the audio scenario, the audio is classified into different channels according to the category of the audio. All channels are collectively referred to as the full channel, and the channel to which the target audio belongs can also be called the field to which it belongs, that is, the vertical field corresponding to the target audio. Optionally, for the evaluation dimension of the matching degree between the alternative sentence and the field to which the target audio belongs, if there is a sentence pattern expression or keyword in the alternative sentence that is specific to its vertical field, then the score in this dimension will be relatively high; conversely, if there is a sentence pattern expression or keyword in the alternative sentence that is common to the full channel, then the score in this dimension will be relatively low. That is, the matching degree between the alternative sentence and the field to which the target audio belongs depends on whether there is a sentence pattern expression or keyword specific to its field in the alternative sentence.

[0084] Optionally, for the evaluation dimension of the relevance between the alternative sentence and the target audio, it can be judged by combining the entire content of the target audio (i.e., the content of the speech recognition text), as well as the title content and the introduction content. It can be obtained by matching the keywords of the alternative sentence with the keywords in the speech recognition text, the keywords in the title content, and the keywords in the introduction content. In an optional example, the TF-IDF (Term Frequency–Inverse Document Frequency) algorithm can be used to extract the keywords in the title content, the introduction content, and the audio content to form a keyword library. The more words in the alternative sentence trigger the words in the keyword library and the longer the triggered words are, the higher the relevance between the alternative sentence and the target audio is, and the higher the corresponding score is.

[0085] In linguistics, TTR (Text-Type-Ratio) and MTLD (Measure of Textual Lexical Diversity) are commonly used to measure lexical richness. TTR represents the ratio of the total number of words in a document to the number of word types. The essence of MTLD is to calculate the average length of text blocks in a document where TTR > a set threshold. The longer the average length is, the higher the lexical richness can be indicated, and the better the corresponding document quality is. Optionally, for the evaluation dimension of the lexical richness of the alternative sentence, the set threshold can be set to 0.72, and the lexical richness of each alternative sentence can be calculated using TTR and MTLD to obtain the score for this dimension.

[0086] In an optional example, when the number of evaluation dimensions is at least two of the above four dimensions, the comprehensive score can be obtained by weighted averaging the scores of each evaluation dimension. The weight coefficient of each evaluation dimension can be set in advance and is not limited here.

[0087] S104-1c. Determine the first key sentence from at least one alternative sentence according to all the comprehensive scores.

[0088] In this embodiment, all the obtained comprehensive scores can be arranged in descending order, and the alternative sentences with comprehensive scores exceeding a preset value can be selected as the first key sentence. The preset value can be set in advance and is not limited here. In an optional implementation manner, the process of determining the summary candidate sentences from multiple target key sentences is related to the fluency score of each target key sentence and the similarity between each target key sentence and the title information. Please refer to Figure 5 Accordingly, step S105 may include sub-steps:

[0089] S105-1. If it is identified that any one of the target key sentences contains a proper noun error, then in combination with the proper noun information, correct each proper noun error.

[0090] In this embodiment, the proper noun information may be included in the title information and / or introduction information of the target audio. Proper noun errors may include errors in personal names, place names, and proper noun expressions. In an optional example, assuming that a place name used in the title information and introduction information of the target audio is "Huangshan", it may be due to the speaker's pronunciation or inaccurate speech recognition, resulting in the use of errors such as "Huangshan", "Huangsan", and "Huangshan" in multiple target key sentences, which need to be corrected to the correct "Huangshan". It can be understood that this example is only an example. In actual applications, proper noun errors depend on the actual application situation and are not limited here.

[0091] S105-2. Use the GPT2 model to calculate the fluency score of each corrected target key sentence.

[0092] The GPT2 model stands for Generative Pre-training 2.0. While the target key sentence obtained through the end-to-end rewriting model is fluent, the fluency of different target key sentences may vary slightly. The GPT2 model can be used to calculate the confusion degree of each revised target key sentence. This confusion degree can represent the fluency of the revised target key sentence and serve as the fluency score for the revised target key sentence.

[0093] S105-3. Obtain similarity information between each corrected target key sentence and the title information.

[0094] In this embodiment, the keywords of each corrected target key sentence can be matched with the keywords in the title information to obtain similarity information, which can represent the degree of similarity between the corrected target key sentence and the title corresponding to the target audio.

[0095] S105-4. Determine at least one summary candidate sentence from the multiple corrected target key sentences based on the similarity information and the fluency score.

[0096] It is understandable that in order to ensure that the determined abstract candidate sentence is not identical to or too similar to the title, the abstract candidate sentence can be a modified target key sentence that satisfies the fluency score reaching a preset threshold and the similarity information does not exceed the similarity threshold.

[0097] It should be noted that the execution order of the above steps S105-2 and S105-3 is not Figure 5 The figure shows a limitation. In actual application, step S105-2 and step S105-3 can be executed not only in parallel but also in any order.

[0098] In an alternative example, assume that the speech recognition text contains 20 sentences. Ten sentences with complete expressions and correct grammar can be selected from them using a classification model as candidate sentences. Then, based on multiple evaluation dimensions, the comprehensive scores of each candidate sentence are obtained respectively. According to all the comprehensive scores, the top 5 sentences with the highest comprehensive scores are selected from the 10 candidate sentences as the first key sentences. Then, the punctuation error information in the 5 first key sentences is sequentially identified using a sequence annotation model and corrected. Then, the corrected 5 first key sentences are sequentially input into an end-to-end rewriting model to obtain 5 target key sentences. Finally, assume that 2 sentences can be selected from the 5 target key sentences as candidate summary sentences. It should be noted that this example is only for illustration, and the specific situation depends on the actual application and is not limited here.

[0099] In an alternative implementation, the end-to-end rewriting model can be obtained by training using training samples. Correspondingly, the method may further include the steps of:

[0100] S100. Obtain a training sample set and a corresponding set of reference sentences.

[0101] In this embodiment, the training sample set may include multiple training samples, the reference sentence set may include multiple reference sentences, and each training sample corresponds to a reference sentence. Each training sample may contain noise, and the corresponding reference sentence is a complete and smoothly expressed sentence without noise. The noise may represent redundant content such as reduplicated words, modal particles, slobber words, etc. and unsmoothly expressed parts in the sentence.

[0102] S101. Train a pre-established end-to-end rewriting model according to the training sample set and the reference sentence set to obtain a trained end-to-end rewriting model.

[0103] It can be understood that the training samples are input into the pre-established end-to-end rewriting model to obtain an output text sequence, and the output text sequence is compared with the reference sentences to adjust the weight coefficients of each node inside the end-to-end rewriting model. Multiple training samples are input sequentially, and the weight coefficients are continuously adjusted to obtain the final trained end-to-end rewriting model.

[0104] In an alternative example, the end-to-end rewriting model can be the T5-PEGASUS Chinese generative pre-training model.

[0105] In a possible implementation, the training samples and reference sentences in the training sample set and the reference sentence set can be obtained through the following three methods. Correspondingly, S100 may include sub-steps: S100-1. Obtain training audio and the title corresponding to the training audio from an audio database.

[0106] S100-2. Obtain the target sentence in the training audio that matches the title, use the target sentence as the training sample, and use the title as the reference sentence corresponding to the training sample.

[0107] It can be understood that Method 1 is to obtain the training sample in the way of steps S100-1 and S100-2. Among them, the audio can be selected from the audio database as the training audio, and the title of the training audio is used as the reference sentence. Identify the target sentence in the training audio that matches the title, and use the target sentence as the training sample.

[0108] S100-3. Obtain the training sample from the audio database, and use the template to clean the training sample to obtain the reference sentence.

[0109] It can be understood that Method 2 is to obtain the training sample in the way of step S100-3. Among them, the training sample containing noise can be obtained from the audio database, and then the template is used to clean the training sample to obtain the reference sentence.

[0110] S100-4. Obtain the abstract from the open-source abstract task corpus as the reference sentence;

[0111] S100-5. Perform self-supervised data augmentation on the reference sentence to add noise to obtain the training sample.

[0112] It can be understood that Method 3 is to obtain the training sample in the way of steps S100-4 and S100-5. In this method, the reference sentence can be the standard abstract obtained from the open-source abstract task corpus. Then perform self-supervised data augmentation on it, that is, randomly add noise to the reference sentence to obtain the training sample.

[0113] It should be noted that the training sample set can be the set of all training samples obtained by at least one of the above three methods, and the reference sentence set can be the set of all reference sentences obtained by at least one of the above three methods. The execution order of the steps included in the above three methods is not restricted. The steps included in the three methods can be executed in parallel, or the three methods can be executed in any order one after another.

[0114] In a possible implementation, when the abstract candidate sentence matches the user label, the abstract candidate sentence can be pushed to the user. When the keywords in the abstract candidate sentence match the keywords queried by the user, the abstract candidate sentence and its corresponding target audio can be added to the search result list. Further, starting from the audio content, the recommended display of the target audio for the user is realized, and the user search can also perform long-tail search for the audio content.

[0115] Further, for the above audio abstract generation method, please refer to Figure 6, an embodiment of the present invention further provides an audio summary generation device 200, which includes:

[0116] An acquisition module 210, configured to obtain multiple target key sentences from the speech recognition text corresponding to the target audio.

[0117] It can be understood that the target key sentence can represent a complete sentence whose content matches the content of the target audio.

[0118] A screening module 220, configured to select at least one from the multiple target key sentences as a summary candidate sentence.

[0119] It can be understood that the summary candidate sentence can represent a smooth summary content corresponding to the target audio.

[0120] In this embodiment, the acquisition module 210 can be used to execute the above steps S100 - S104 and their sub - steps, and the screening module 220 can be used to execute the above step S105 and its sub - steps. For the relevant content of the acquisition module 210 and the screening module 220, reference can be made to the corresponding specific descriptions above.

[0121] The above - mentioned modules can be one or more integrated circuits configured to implement the above - mentioned method. For example: one or more application - specific integrated circuits (ASICs), or, one or more digital signal processors (DSPs), or, one or more field - programmable gate arrays (FPGAs), etc. Again, when a certain module above is implemented in the form of a processing element dispatching program code, the processing element can be a general - purpose processor, such as a central processing unit (CPU) or other processors that can call program code. Again, these modules can be integrated together and implemented in the form of a system - on - a - chip (SoC).

[0122] Further, based on the above - mentioned audio summary generation method, please refer to Figure 7 , which is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. The electronic device 300 includes a processor 310, a memory 320, and a bus 330. The memory 320 stores machine - readable instructions executable by the processor 310. When the electronic device 300 runs, the processor 310 communicates with the memory 320 through the bus 330, and the processor 310 executes the machine - readable instructions, and when executed, implements the above - mentioned method embodiment.

[0123] The processor 310 can be an integrated circuit chip with signal processing capabilities. The processor 102 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0124] The memory 320 can be used to store software programs or store data. Among them, the memory 320 can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Flash memory, Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electric Erasable Programmable Read-Only Memory (EEPROM), etc.

[0125] It can be understood that Figure 7 the structure shown is only schematic, and the electronic device 300 may also include more or fewer components than those shown Figure 7 in the figure, or have a different configuration from that shown Figure 7 in the figure. Figure 7 Each component shown in the figure can be implemented by hardware, software, or a combination thereof.

[0126] Furthermore, based on the above audio summary generation method, an embodiment of the present invention also provides a readable storage medium on which a computer program is stored. When the computer program is run by a processor, it executes the steps of the above method embodiment. The readable storage medium can be, but is not limited to, various media such as a USB flash drive, a mobile hard disk, ROM, RAM, PROM, EPROM, EEPROM, a FLASH disk, or an optical disc that can store program codes.

[0127] In summary, the embodiments of the present invention provide an audio summary generation method, device, electronic device and readable storage medium, which relate to the field of computer technology. By obtaining multiple target key sentences from the speech recognition text corresponding to the target audio, the target key sentence represents a written complete sentence whose content matches the content of the target audio. Then, at least one is selected from the multiple target key sentences as a summary candidate sentence, and the summary candidate sentence represents the smooth summary content corresponding to the target audio. Its beneficial effect is that the corresponding summary candidate sentence can be automatically generated according to the speech recognition text corresponding to the target audio, eliminating the labor cost and time cost of manually writing a summary for the target audio in the prior art. In addition, the summary candidate sentence is pushed to the user, which realizes the recommendation and display of the target audio to the user based on the audio content, and the summary candidate sentence is used as a user search recall, which also improves the ability to undertake long-tail searches and improves the user experience.

[0128] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. An audio abstract generation method, characterized in that, Including: Extracting multiple first key sentences from the speech recognition text corresponding to the target audio; The first key sentence represents a complete sentence whose content matches the content of the target audio; Using a sequence annotation model, respectively identifying the punctuation error information existing in each of the first key sentences, and correcting each punctuation error information to obtain multiple corrected first key sentences; Respectively inputting each of the corrected first key sentences into an end-to-end rewriting model to obtain multiple target key sentences; The end-to-end rewriting model is used to convert the colloquial expression part in the input text sequence into a written expression; Selecting at least one from the multiple target key sentences as a summary candidate sentence, and the summary candidate sentence represents a smooth summary content corresponding to the target audio; Wherein, the end-to-end rewriting model is trained in the following manner: Obtaining a training sample set and a corresponding benchmark sentence set; the training sample set includes multiple training samples, the benchmark sentence set includes multiple benchmark sentences, and each training sample corresponds to a benchmark sentence; training a pre-established end-to-end rewriting model according to the training sample set and the benchmark sentence set to obtain a trained end-to-end rewriting model; Wherein, the obtaining method of any one of the training samples and its corresponding benchmark sentence is: Obtaining a training audio and the title corresponding to the training audio from an audio database, obtaining a target sentence in the training audio that matches the title, using the target sentence as a training sample, and using the title as the benchmark sentence corresponding to the training sample; and / or, obtaining a training sample from an audio database and cleaning the training sample using a template to obtain a benchmark sentence; and / or, obtaining a summary from an open-source abstract task corpus as a benchmark sentence, and performing self-supervised data augmentation on the benchmark sentence to add noise to obtain a training sample.

2. The method according to claim 1, characterized in that, The step of extracting multiple first key sentences from the speech recognition text corresponding to the target audio includes: Using a classification model to screen out at least one candidate sentence from the speech recognition text; the candidate sentence represents a sentence with complete expression and correct grammar; Based on multiple evaluation dimensions, respectively obtaining the comprehensive score of each candidate sentence; the multiple evaluation dimensions include at least one or a combination of the importance degree of the candidate sentence, the matching degree with the field to which the target audio belongs, the relevance with the target audio, and the lexical richness; Determining the first key sentence from at least one of the candidate sentences according to all the comprehensive scores.

3. The method according to claim 1, wherein The step of selecting at least one from the multiple target key sentences as a summary candidate sentence includes: If it is identified that any one of the target key sentences contains a proper noun error, then combining the proper noun information, correcting each proper noun error; the proper noun information is included in the title information and / or the introduction information of the target audio; Calculating the smoothness score of each corrected target key sentence using the GPT2 model; Obtaining the similarity information between each corrected target key sentence and the title information; Determine at least one candidate abstract sentence from multiple corrected target key sentences according to the similarity information and the fluency score.

4. The method according to claim 1, wherein Before the step of extracting multiple first key sentences from the speech recognition text corresponding to the target audio, the method further includes: Obtain the speech recognition text corresponding to the target audio; Clean the redundant content in the speech recognition text.

5. An audio abstract generation device, characterized in that, Including: An acquisition module, configured to: Extract multiple first key sentences from the speech recognition text corresponding to the target audio; the first key sentence represents a complete sentence whose content matches the content of the target audio; Use a sequence labeling model to identify punctuation error information existing in each first key sentence respectively, and correct each punctuation error information to obtain multiple corrected first key sentences; Input each corrected first key sentence into an end-to-end rewriting model respectively to obtain multiple target key sentences; The end-to-end rewriting model is used to convert the colloquial expression part in the input text sequence into a written expression; A screening module, configured to select at least one from multiple target key sentences as a candidate abstract sentence, and the candidate abstract sentence represents a fluent summary content corresponding to the target audio; Wherein, the end-to-end rewriting model is trained in the following manner: Obtain a training sample set and a corresponding benchmark sentence set; the training sample set includes multiple training samples, the benchmark sentence set includes multiple benchmark sentences, and each training sample corresponds to a benchmark sentence; train a pre-established end-to-end rewriting model according to the training sample set and the benchmark sentence set to obtain a trained end-to-end rewriting model; Wherein, the obtaining method of any training sample and its corresponding benchmark sentence is: Obtain a training audio and the title corresponding to the training audio from an audio database, obtain a target sentence in the training audio that matches the title, use the target sentence as a training sample, and use the title as the benchmark sentence corresponding to the training sample; and / or, obtain a training sample from an audio database, and clean the training sample using a template to obtain a benchmark sentence; and / or, obtain an abstract from an open-source abstract task corpus as a benchmark sentence, and perform self-supervised data augmentation on the benchmark sentence to add noise to obtain a training sample.

6. An electronic device, characterized in that, Including: A memory and a processor, the memory stores machine-readable instructions executable by the processor, and when the electronic device runs, the processor executes the machine-readable instructions to implement the method according to any one of claims 1 to 4.

7. A readable storage medium, characterized in that, The readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method according to any one of claims 1 to 4.

Images