Audio recognition summary generation method and device

Abstract

The invention provides a voice summary generation method and device, and solves the problems of high requirement on voice recognition network environment quality and low voice recognition accuracy in the prior art. The method comprises the following steps: preprocessing audio data acquired from a client, then sequentially performing segmentation, transcription, sensitive word filtering and punctuation mark adding processing, inputting a processed text into a deep learning model, generating a summary through the deep learning model, and feeding back the summary to a user, so that automatic generation of the audio data summary is realized, and the user experience is improved. And smooth user operation experience is provided.

Description

Technical Field

[0001] The present invention relates to the technical field of audio recognition, and in particular to a method and device for generating an audio summary. Background Art

[0002] With the continuous development of deep learning technology, deep learning can realize the automatic learning and training of complex features in audio signals on large-scale data sets, thereby significantly improving the accuracy of audio recognition. In particular, end-to-end audio recognition models, such as Transformer-based models, further simplify the recognition process and improve the efficiency of audio recognition.

[0003] However, when processing real-time audio transcription, existing audio recognition systems have problems such as high requirements for the quality of the audio recognition network environment and low audio recognition accuracy. Summary of the invention

[0004] The present invention provides an audio summary generation method and device to solve the problems existing in the prior art of high requirements on the quality of the audio recognition network environment and low audio recognition accuracy.

[0005] The present invention provides an audio summary generation method, which is applied to a front end, and particularly relates to an audio summary generation method for a user during use, and specifically comprises the following steps:

[0006] Step S1, receiving an audio recognition request triggered by a user through an application;

[0007] Step S2: collecting audio data output by the user;

[0008] Step S3: transmitting the audio data to a back end.

[0009] Preferably, in step S2, the audio output by the audio source (in this application, the "audio source" can be the sound emitted by a device or the sound emitted by a person) is collected through a physical microphone and converted into digital audio data, generally in PCM format, mono, and with a sampling frequency of 16000 Hz.

[0010] Preferably, in step S3, the audio data is transmitted to the backend via webocket technology and the audio data is pre-stored via PCM pre-storage technology (the PCM pre-storage technology refers to pre-storing the audio data in a buffer or storage device for subsequent processing or playback).

[0011] On the other hand, the present invention provides an offline audio summary generation method, which is applied to a backend, and particularly relates to an offline audio summary generation method in a backend audio summary generation process, which specifically comprises the following steps:

[0012] Step S4, receiving audio data, forming data DATA1 and saving the data DATA1;

[0013] Step S5, preprocessing the data DATA1 to form data DATA2;

[0014] Step S6, performing text generation processing on the data DATA2 to form text data DATA3;

[0015] Step S7, generating a content summary of the audio data DATA1 according to the data DATA3;

[0016] Step S8: Transmit the content summary of the audio data DATA1 to the front end.

[0017] Preferably, in step S5, the preprocessing includes one or more processing methods of noise reduction, enhancement and normalization.

[0018] Preferably, in step S6, text generation processing is performed on the data DATA2 to form data DATA3, which specifically includes the following steps:

[0019] Step S601, dividing the data DATA2 according to the timestamps by using the VAD technology, and dividing the pre-processed audio data into N arrays voice_array;

[0020] Step S602, transcribe the array voice_array by using ASR technology to form plain text data data2-1;

[0021] Step S603, performing sensitive word filtering processing on the data data2-1 to form filtered plain text data data2-2;

[0022] Step S604: the plain text data data2-2 is identified and filtered by CAM technology to form plain text data data2-3 corresponding to each audio source (in this application, the "corresponding" means that for audio source A, the audio data outputted by the audio source A is processed to form filtered plain text data);

[0023] Step S605: adding punctuation marks to the plain text data data2-3 to form natural text data DATA3.

[0024] Wherein, A and N are both positive integers.

[0025] Preferably, in step S7, the data DATA3 is input into a deep learning model, and a content summary of the audio data DATA1 is generated by the deep learning model.

[0026] More preferably, the deep learning model includes one of an RNN model, an LSTM model, a GRU model and a Transformer model.

[0027] Preferably, in step S8, the content summary of the audio data DATA1 is transmitted to the front end via webocket technology.

[0028] Preferably, an audio summary generation method, applied to a front end, further includes:

[0029] Step S9: The user verifies the audio data and data DATA3, and retransmits the verified results to the deep learning model as input.

[0030] In the present application, the front end is used to realize human-computer interaction, and the back end is used to realize processing of the interaction results of the human-computer interaction interface; the connection method between the front end and the back end includes wired connection and wireless connection. When the connection method is wired connection, offline generation of audio summary can be realized.

[0031] On the other hand, the present invention also provides an audio summary generating device, which is applied to a front end, and particularly relates to an audio summary generating device during use by a user, and specifically comprises the following modules:

[0032] A recognition request receiving module, used to receive an audio recognition request triggered by a user through an application;

[0033] An audio data collection module, used to collect audio data output by users;

[0034] The audio data transmission module is used to transmit the audio data to a back end.

[0035] Preferably, in the audio data collection module, the audio output by the audio source is collected by a physical microphone and converted into digital audio data, generally in PCM format, mono, and with a sampling frequency of 16000 Hz.

[0036] Preferably, in the audio data transmission module, the audio data is transmitted to the backend through webocket technology and the audio data is pre-stored through PCM pre-storage technology.

[0037] On the other hand, the present invention provides an audio summary generating device, which is applied to a backend, and particularly relates to an audio summary generating device in a backend audio summary generating process, which specifically includes the following modules:

[0038] An audio data receiving module, used for receiving audio data, forming data DATA1 and saving the data DATA1;

[0039] An audio data preprocessing module, used for preprocessing the data DATA1 to form data DATA2;

[0040] A text generation processing module, used for performing text generation processing on the data DATA2 to form text data DATA3;

[0041] A summary generating module, used for generating a content summary of the audio data DATA1 according to the data DATA3;

[0042] The summary transmission module is used to transmit the content summary of the audio data DATA1 to the front end.

[0043] Preferably, in the audio data preprocessing module, the preprocessing includes one or more processing methods of noise reduction, enhancement and normalization.

[0044] Preferably, the text generation processing module specifically includes the following submodules:

[0045] The first submodule of text generation processing is used to segment the data DATA2 according to timestamps by using VAD technology, and to segment the pre-processed audio data into N arrays voice_array;

[0046] The second submodule of text generation processing is used to transcribe the array voice_array through ASR technology to form plain text data data2-1;

[0047] The third submodule of text generation processing is used to perform sensitive word filtering processing on the data data2-1 to form filtered plain text data data2-2;

[0048] The fourth submodule of text generation processing is used to perform recognition and screening processing on the plain text data data2-2 by using CAM technology to form plain text data data2-3 corresponding to each audio source;

[0049] The fifth submodule of text generation processing is used to add punctuation marks to the plain text data data2-3 to form natural text data DATA3.

[0050] Wherein, A and N are both positive integers.

[0051] Preferably, in the summary generation module, the data DATA3 is input into a deep learning model, and a content summary of the audio data DATA1 is generated by the deep learning model.

[0052] More preferably, the deep learning model includes one of an RNN model, an LSTM model, a GRU model and a Transformer model.

[0053] Preferably, in the summary transmission module, the content summary of the audio data DATA1 is transmitted to the front end through webocket technology.

[0054] Preferably, an audio summary generating device, applied to a front end, further comprises:

[0055] The verification module is used for the user to verify the audio data and data DATA3, and retransmit the verified results to the deep learning model as input.

[0056] On the other hand, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for generating an audio summary as described in any one of the above items of the present application is implemented.

[0057] On the other hand, the present invention also provides an electronic device, comprising: a memory storing a computer program; and a processor communicatively connected to the memory, for executing an audio summary generation method described in any one of the above items of the present application when the computer program is called.

[0058] Compared with the prior art, the present invention has the following obvious outstanding substantive features and significant advantages:

[0059] The present invention provides an audio summary generation method and device, which solves the problems of high quality requirements for audio recognition network environment and low audio recognition accuracy in the prior art. The audio data obtained from the front end is pre-processed, and then segmented, transcribed, sensitive word filtered and punctuation added in sequence, and the processed text is input into a deep learning model. The deep learning model generates a summary and feeds it back to the user, thereby realizing the automatic generation of audio data summary and providing a smooth user operation experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings, which constitute part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention. In the accompanying drawings:

[0061] Figure 1 It is a flow chart of an audio summary generation method applied to a front end according to a preferred embodiment of the present invention.

[0062] Figure 2 It is a flow chart of an audio summary generation method applied to a back-end according to a preferred embodiment of the present invention.

[0063] Figure 3 It is a structural schematic diagram of an audio summary generating device applied to a front end according to a preferred embodiment of the present invention.

[0064] Figure 4 It is a structural schematic diagram of an audio summary generating device applied to a back-end according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0065] The present invention provides an audio summary generation method and device. In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0066] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0067] Embodiment 1:

[0068] like Figure 1 As shown, the audio summary generation method described in this embodiment is applied to the front end, and particularly relates to the audio summary generation method of the user during use, and specifically includes the following steps:

[0069] Step S1: receiving an audio recognition request triggered by a user through an application.

[0070] Step S2, collect audio data output by the user; collect the audio output by the audio source (in this embodiment, the audio source is the spoken voice of multiple people) through a physical microphone and convert it into digital audio data, generally in PCM format, mono, and with a sampling frequency of 16000 Hz.

[0071] Step S3, transmitting the audio data to the back end; transmitting the audio data to the back end through webocket technology and pre-storing the audio data through PCM pre-storage technology.

[0072] like Figure 2 As shown, the offline audio summary generation method described in this embodiment is applied to the back end, and particularly relates to the offline audio summary generation method in the back end during the audio summary generation process, and specifically includes the following steps:

[0073] Step S4, receiving audio data, forming data DATA1 and saving the data DATA1. By saving the received data DATA1, subsequent audio data transcription and analysis can be facilitated. For example, a certain audio data was stored a month ago, and no subsequent processing was performed at that time. Now it is necessary to transcribe and analyze what voice content the person or other audio device in the audio data roughly output.

[0074] Step S5, preprocessing the data DATA1 to form data DATA2; wherein the preprocessing includes one or more processing methods of noise reduction, enhancement and normalization (in the present embodiment, the preprocessing is achieved by performing noise reduction and normalization on the data DATA1).

[0075] Step S6, performing text generation processing on the data DATA2 to form text data DATA3.

[0076] Optionally, in step S6, text generation processing is performed on the data DATA2 to form data DATA3, which specifically includes the following steps:

[0077] Step S601, using VAD technology to divide the data DATA2 according to a fixed time interval A, and divide the pre-processed audio data into N arrays voice_array; wherein A and N are both positive integers.

[0078] Step S602, transcribe the array voice_array by using ASR technology to form plain text data data2-1;

[0079] Step S603: Perform sensitive word filtering on the data data2-1 to form filtered plain text data data2-2.

[0080] Step S604: the plain text data data2-2 is identified and screened by CAM technology to form plain text data data2-3 corresponding to each audio source (for example, for Zhang Xsan, the text data formed by transcribing Zhang Xsan's voice data in data2-2 is identified and screened by CAM technology and taken out separately, that is, a label is given to the words spoken by Zhang Xsan, indicating that these words were spoken by Zhang Xsan);

[0081] Step S605: adding punctuation marks to the plain text data data2-3 to form natural text data DATA3.

[0082] Step S7, generating a content summary of the audio data DATA1 according to the data DATA3; wherein, the data DATA3 is input into a deep learning model, and a content summary of the audio data DATA1 is generated by the deep learning model; optionally, the deep learning model includes one of an RNN model, an LSTM model, a GRU model and a Transformer model (in this embodiment, the deep learning model selects a Transformer model).

[0083] Step S8, transmitting the content summary of the audio data DATA1 to the front end; wherein, the content summary of the audio data DATA1 is transmitted to the front end through webocket technology.

[0084] Optionally, an audio summary generation method, applied to a front end, further includes:

[0085] Step S9: The user verifies the audio data and data DATA3, and retransmits the verified results to the deep learning model as input.

[0086] During the implementation of this embodiment, the front-end and back-end realize real-time communication through websocket technology, ensuring that the user's operations can be displayed in real time on the front-end interface, and can also receive real-time feedback and notifications from the back-end.

[0087] As can be seen from the above, this embodiment can realize the local storage of audio data for subsequent audio transcription and analysis, which not only saves the computing power resources required for subsequent audio transcription and analysis, but also, in some application scenarios, if it is necessary to analyze audio data from one month ago, the locally stored audio data can be directly taken out for separate analysis. At the same time, for the situation where multiple people are included in the audio data, it is also possible to annotate the words of each person to indicate which person said these words, and each output text also includes a timestamp, which can achieve the technical effect that a certain person roughly said certain words at a certain time point. Through the method described in this embodiment, one-stop processing from input audio to output audio summary can be directly realized, which is very convenient and fast. The user only needs to operate the front-end device to collect or directly import the audio to obtain the summary of the collected or imported audio data.

[0088] Embodiment 2:

[0089] like Figure 3 As shown, the audio summary generating device described in this embodiment is applied to the front end, and particularly relates to the audio summary generating device used by the user, and specifically includes the following modules:

[0090] The recognition request receiving module is used to receive an audio recognition request triggered by a user through an application.

[0091] The audio data collection module is used to collect audio data output by the user; the audio output by the audio source is collected through a physical microphone and converted into digital audio data, generally in PCM format, mono, and with a sampling frequency of 16000Hz.

[0092] The audio data transmission module is used to transmit the audio data to the back end; wherein the audio data is transmitted to the back end through webocket technology and the audio data is pre-stored through PCM pre-storage technology.

[0093] like Figure 4 As shown, the audio summary generating device described in this embodiment is applied to the back end, and particularly relates to the audio summary generating device in the back end during the audio summary generating process, and specifically includes the following modules:

[0094] The audio data receiving module is used to receive audio data, form data DATA1 and save the data DATA1.

[0095] The audio data preprocessing module is used to preprocess the data DATA1 to form data DATA2; wherein the preprocessing includes one or more processing methods of noise reduction, enhancement and normalization.

[0096] The text generation processing module is used to perform text generation processing on the data DATA2 to form text data DATA3.

[0097] Among them, the text generation processing module specifically includes the following sub-modules:

[0098] The first submodule of text generation processing is used to segment the data DATA2 according to timestamps through VAD technology, and segment the pre-processed audio data into N arrays voice_array; wherein A and N are both positive integers.

[0099] The second submodule of text generation processing is used to transcribe the array voice_array through ASR technology to form plain text data data2-1.

[0100] The third submodule of text generation processing is used to perform sensitive word filtering processing on the data data2-1 to form filtered plain text data data2-2.

[0101] The fourth submodule of text generation processing is used to perform recognition and screening processing on the plain text data data2-2 by using CAM technology to form plain text data data2-3 corresponding to each audio source;

[0102] The fifth submodule of text generation processing is used to add punctuation marks to the plain text data data2-3 to form natural text data DATA3.

[0103] A summary generation module is used to generate a content summary of the audio data DATA1 according to the data DATA3. The data DATA3 is input into a deep learning model, and a content summary of the audio data DATA1 is generated by the deep learning model. More preferably, the deep learning model includes one of an RNN model, an LSTM model, a GRU model, and a Transformer model.

[0104] The summary transmission module is used to transmit the content summary of the audio data DATA1 to the front end; wherein the content summary of the audio data DATA1 is transmitted to the front end through webocket technology.

[0105] Optionally, an audio summary generating device, applied to a front end, further includes:

[0106] The verification module is used for the user to verify the audio data and data DATA3, and retransmit the verified results to the deep learning model as input.

[0107] During the implementation of this embodiment, the front-end and back-end realize real-time communication through websocket technology, ensuring that the user's operations can be displayed in real time on the front-end interface, and can also receive real-time feedback and notifications from the back-end.

[0108] The specific embodiments of the present invention are described in detail above, but they are only examples, and the present invention is not limited to the specific embodiments described above. For those skilled in the art, any equivalent modifications and substitutions made to the present invention are also within the scope of the present invention. Therefore, the equalization changes and modifications made without departing from the spirit and scope of the present invention should be included in the scope of the present invention.

Claims

1. A method for generating audio recognition summary based on deep learning, characterized in that: Applied to the front end, it particularly involves a method for generating an audio summary based on deep learning during user use, specifically including the following steps: Step S1, receiving an audio recognition request triggered by a user through an application; Step S2: collecting audio data output by the user; Step S3: transmitting the audio data to a back end.

2. The method for generating an audio recognition summary based on deep learning according to claim 1, characterized in that: In step S3, the audio data is transmitted to the backend through webocket technology and the audio data is pre-stored through PCM pre-storage technology.

3. A method for generating audio recognition summary based on deep learning, characterized in that: Applied to the backend, in particular, it relates to an offline audio summary generation method based on deep learning in the process of audio summary generation by the backend, which specifically includes the following steps: Step S4, receiving audio data, forming data DATA1 and saving the data DATA1; Step S5, preprocessing the data DATA1 to form data DATA2; Step S6, performing text generation processing on the data DATA2 to form text data DATA3; Step S7, generating a content summary of the audio data DATA1 according to the data DATA3; Step S8: Transmit the content summary of the audio data DATA1 to the front end.

4. The method for generating an audio recognition summary based on deep learning according to claim 3, characterized in that: In step S5, the preprocessing includes one or more processing methods of noise reduction, enhancement and normalization.

5. The method for generating an audio recognition summary based on deep learning according to claim 1, characterized in that: In step S6, text generation processing is performed on the data DATA2 to form data DATA3, which specifically includes the following steps: Step S601, dividing the data DATA2 according to the timestamps by using the VAD technology, and dividing the pre-processed audio data into N arrays voice_array; Step S602, transcribe the array voice_array by using ASR technology to form plain text data data2-1; Step S603, performing sensitive word filtering processing on the data data2-1 to form filtered plain text data data2-2; Step S604: using CAM technology to identify and filter the plain text data data2-2 to form plain text data data2-3 corresponding to each audio source; Step S605, adding punctuation marks to the plain text data data2-3 to form natural text data DATA3; Wherein, A and N are both positive integers.

6. The method for generating an audio recognition summary based on deep learning according to claim 1, characterized in that: In step S7, the data DATA3 is input into a deep learning model, and a content summary of the audio data DATA1 is generated by the deep learning model; Among them, the deep learning model includes one of an RNN model, an LSTM model, a GRU model and a Transformer model.

7. A device for generating audio recognition summary based on deep learning, characterized in that: Applied to the front end, especially to the audio summary generation device based on deep learning during the user's use, specifically including the following modules: A recognition request receiving module, used to receive an audio recognition request triggered by a user through an application; An audio data collection module, used to collect audio data output by users; The audio data transmission module is used to transmit the audio data to a back end.

8. A device for generating audio recognition summary based on deep learning, characterized in that: Applied to the backend, in particular, it relates to an audio summary generation device based on deep learning in the process of generating an audio summary in the backend, specifically including the following modules: An audio data receiving module, used for receiving audio data, forming data DATA1 and saving the data DATA1; An audio data preprocessing module, used for preprocessing the data DATA1 to form data DATA2; A text generation processing module, used for performing text generation processing on the data DATA2 to form text data DATA3; A summary generating module, used for generating a content summary of the audio data DATA1 according to the data DATA3; The summary transmission module is used to transmit the content summary of the audio data DATA1 to the front end.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements a method for generating an audio recognition summary based on deep learning as described in any one of claims 1-2 or claims 3-6.

10. An electronic device, characterized in that: It includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a method for generating an audio recognition summary based on deep learning as described in any one of claims 1-2 or claims 3-6.

Images