Speech transcription method based on deep learning

Abstract

A speech transcription method based on deep learning belongs to the field of speech recognition, and aims to solve the problem of incomplete transcription text caused by inaccurate audio text processing in short time in existing segmented speech text recognition, and is characterized in that a second text is matched with a first text according to an audio time stamp, and the first text is matched with the second text according to the audio time stamp; corresponding texts corresponding to the audios of the audio intervals in the second text to corresponding text positions of the first text, and marking speaking objects for the intervals of the corresponding texts of the first text according to the text interval marks of the second text, the first text of the speaking object in the marked text interval is obtained by voice transcription.

Description

Technical Field

[0001] The present invention belongs to the field of speech recognition and relates to a speech transcription method based on deep learning. Background Art

[0002] When law enforcement agencies or prosecutors handle cases, they usually use law enforcement recorders or monitors to record the law enforcement process. At the same time, law enforcement personnel will record the conversations during the process, and finally form a transcript and archive it. These videos of the law enforcement process and the law enforcement information data recorded in the transcript play an important role in future use. The case facts contained in them play a supporting role in the case and improve the efficiency of case handlers. In addition, the two types of information also complement each other to avoid omissions in law enforcement information data.

[0003] In order to make the information in the transcript more standardized, the recorder is usually required to organize the content of the conversation while recording. This requires high requirements for recorders, who need to have long-term work experience in this field. In addition, recording the law enforcement process will affect the overall progress of law enforcement information data storage, so the generation and storage speed of the process needs to be optimized.

[0004] The overall speech recognition cannot have text partitioning and speaker distinction, resulting in the unreadable transcribed text. Usually, segmented text transcription is used in this field. For example, the U.S. patent application document with application publication number US20230154468A1 discloses a speech recognition device for long-format audio, and specifically discloses segmenting the audio stream into multiple audio segments; identifying multiple speakers in each of the multiple audio segments; performing automatic speech recognition (ASR) on each of the multiple audio segments to generate multiple short segment hypotheses; merging at least a part of the short segment hypotheses into a first merged hypothesis set; inserting a stitching symbol into the first merged hypothesis set, the stitching symbol includes a window change (WC) symbol; and merging the first merged hypothesis set into a first merged hypothesis using a network-based hypothesis splicer. It uses segmentation into multiple audio segments to identify and form a transcribed text, and the defect of this method is that segmented recognition cannot obtain accurate or corresponding audio text for audio with a shorter time compared to overall recognition. Summary of the invention

[0005] In order to solve the problem of inaccurate processing of short-term audio text in existing segmented speech text recognition, resulting in incomplete transcribed text, a speech transcription method based on deep learning in one embodiment of the present invention includes:

[0006] S10. Extracting the audio file from the video file;

[0007] S20. Identify a first text corresponding to the audio in the audio file, the first text is a set of texts corresponding to the audio in the audio file, and the first text includes an audio time mark;

[0008] S30. Classifying different sound features in the audio in the audio file through voiceprint recognition;

[0009] S40. Determine different speaking objects and audio intervals of different speaking objects according to the feature classification;

[0010] S50. Identify a second text corresponding to the audio of the audio interval, the second text is a collection of texts corresponding to the audio of each audio interval, and the second text includes a text interval tag of the text interval corresponding to each audio interval and a speaker tag corresponding to the text interval, and the second text also includes an audio time tag;

[0011] S60. According to the audio time mark, the second text is matched with the first text, and the text corresponding to the audio of each audio interval in the second text is placed in the corresponding text position of the first text. The speaking object of the corresponding text interval of the first text is marked according to the text interval mark of the second text, and the first text of the speaking object of the marked text interval is the file obtained by speech transcription, wherein the audio interval and text interval are corresponding representations of the time interval of the audio time.

[0012] In the deep learning-based speech transcription method in one embodiment of the present invention, in the step S50, no valid text content is identified in the audio interval, and the sound features and feature classification of the audio interval are identified, so that the text interval of this type of text in the second text has no corresponding text content, but the text interval corresponds to the speaker tag, and this type of text is a special text in the second text;

[0013] In step S60, the second text is matched with the first text according to the audio time mark, wherein the special text in the second text corresponds to the corresponding text position in the first text according to the audio time mark, and the speaking object of the corresponding text interval mark of the first text is matched according to the text interval mark of the special text in the second text.

[0014] In a deep learning-based speech transcription method in one embodiment of the present invention, the method for extracting an audio file from a video file in step S10 comprises: for a video file in a law enforcement recorder or a monitor, using an automated editing tool to extract the audio from the video and fixing the format, wherein the extracted audio format is fixed to 16kHz, 16-bit mono audio.

[0015] In the deep learning-based speech transcription method in one embodiment of the present invention, the method of classifying different sound features in the audio in the audio file by voiceprint recognition in step S30 includes:

[0016] Preprocessing the audio signal of the audio in the audio interval;

[0017] Perform MFCC feature extraction on the audio signal of the preprocessed audio interval, convert each frame waveform into a multidimensional vector containing sound information, and obtain a feature vector in the audio of the audio interval;

[0018] The acoustic model classifies the feature vector according to the acoustic features of the speech to obtain a classification of sound features in the audio in the audio file.

[0019] In the deep learning-based speech transcription method of one embodiment of the present invention, the second text corresponding to the audio of the audio interval in step S50 is identified, including

[0020] The acoustic model classifies the acoustic features into phonemes and / or word units;

[0021] The obtained phonemes and / or word units are decoded into a complete sentence through a language model, and the complete sentence is the second text of the text interval corresponding to the audio of the audio interval.

[0022] In the deep learning-based speech transcription method in one embodiment of the present invention, the preprocessing includes any one of frame cutting, pre-emphasis on the high-frequency part of the speech, and increasing the high-frequency resolution of the speech, or a combination thereof.

[0023] An embodiment of the present application also provides an electronic device, comprising: one or more processors, a memory, and one or more programs; wherein the one or more programs are stored in the memory, and the one or more programs include instructions, and when the instructions are executed by the electronic device, the electronic device executes any possible technical solution of the first aspect of the embodiment of the present application.

[0024] An embodiment of the present application also provides a computer-readable storage medium, which includes a computer program. When the computer program runs on an electronic device, the electronic device executes any possible technical solution of the first aspect of the embodiment of the present application.

[0025] Beneficial effects of the present invention:

[0026] On the first aspect, based on the two requirements of manpower and time, the present invention proposes a method for extracting the conversation content of the video in the law enforcement recorder or monitor, and saving the text of the conversation, thereby reducing the demand for manpower and increasing the speed of storing law enforcement information data.

[0027] In the second aspect, the present invention uses speech signal processing and recognition to extract and initially recognize audio in order to recognize text in a video. The present invention also uses voiceprint recognition to classify different sound features in order to distinguish different speakers in the audio.

[0028] On the third aspect, the present invention matches the second text with the first text according to the audio time mark, corresponds the text corresponding to the audio of each audio interval in the second text to the corresponding text position of the first text, marks the speaker of the corresponding text interval of the first text according to the text interval mark of the second text, and the first text of the speaker of the marked text interval is the file obtained by speech transcription, wherein the audio interval and the text interval are corresponding representations of the time interval of the audio time. Thus, the second text of different speakers is matched with the first text obtained by overall recognition, on the one hand, the first text can have the corresponding marks of text partitions and speakers, and the first text with the corresponding marks of text partitions and speakers is used as the final transcription text. On the other hand, the speech that cannot be recognized in the second text of segmented recognition due to reasons such as short audio time is compensated by the first text of overall recognition, avoiding the phenomenon that segmented recognition has some speech that cannot be recognized due to reasons such as short audio time. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a framework diagram of the speech transcription method in an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The present invention relates to speech recognition and natural language processing in the computer field, and specifically to speech signal processing and recognition, voiceprint recognition technology, etc. in speech recognition technology, and text matching in natural language processing technology. The embodiments of the present application are described in detail below with reference to the accompanying drawings, and examples of the embodiments are shown in the accompanying drawings. The present application provides a method, a device, and a computer storage medium to solve the problem of inaccurate processing of audio text for a shorter time in existing segmented speech text recognition, resulting in incomplete transcribed text. Among them, the method, the device, and the computer-readable storage medium are based on the same technical concept, and the principles of solving the problem are similar, so the implementation of each subject can refer to each other, and the repeated parts will not be repeated.

[0031] In the first embodiment, Figure 1 It is a framework diagram of speech transcription method, such as Figure 1 As shown, the speech transcription method comprises the following steps:

[0032] First, for the video files in the law enforcement recorder or monitor, use an automated editing tool to extract the audio from the video and fix the format. For the effect of subsequent audio transcription, the extracted audio format is fixed to 16kHz, 16-bit mono audio, while avoiding loss of audio quality.

[0033] After that, the obtained audio needs to be converted into text. The process of speech signal recognition is generally a process of encoding and decoding. For the input audio signal, it is first preprocessed, including frame cutting, pre-emphasis of the high-frequency part of the speech, and increasing the high-frequency resolution of the speech; then MFCC feature extraction is performed to convert each frame waveform into a multidimensional vector containing sound information, that is, to obtain the feature vector in the audio; then the acoustic features of the speech are classified and corresponded to phonemes and or word units through the acoustic model; finally, the language model is used for decoding, and the obtained words are decoded into a complete sentence, thereby obtaining the speech recognition result. Generally speaking, the above steps are required to recognize a segment of audio, but since the questioning is usually conducted in the form of dialogue during the law enforcement process, it is not enough to simply recognize the text in the audio, and the inability to distinguish the speaker still cannot improve the efficiency of case handling. Therefore, after the overall recognition of the audio, segmented recognition is also required, that is, the entire audio segment is segmented and recognized according to different speakers. The main implementation method of segmented recognition is to distinguish speakers based on their voice features in the audio. By using deep learning models and speech features for classification, the audio intervals of different speakers can be obtained, and then the entire audio can be segmented according to the intervals to obtain many small audio clips, corresponding to the content of each speech of different speakers. After that, voice recognition is performed on each small audio clip to obtain the recognition result of each audio clip and the corresponding speaker. For the recognition of small audio segments, there may be situations where the audio is too short or the speaker's voice is too soft to be recognized, and the whole segment recognition can alleviate these problems. Therefore, for the two types of recognition results, the whole segment recognition and segmented recognition, text matching is performed to obtain the audio transcription result.

[0034] For text matching technology, the main method used is rule-based. First, find the audio with shorter duration in the short audio segment, then match the recognition results of the previous and next segmented audio with the result of the whole segment recognition, so as to obtain the corresponding position of the current short audio segment in the whole audio segment, and use the recognition result of the whole audio segment at this position as the recognition result of the current short audio segment. According to the above method, the results of all the short audio segments in the segment recognition are replaced with the results of the whole segment recognition, so as to obtain the final recognition result that can distinguish different speakers.

[0035] The deep learning-based speech transcription method of this embodiment can alleviate the high manpower requirements for recording conversations during law enforcement, and can also speed up the storage of law enforcement information data, thereby improving overall case handling efficiency.

[0036] In another embodiment, the audio transcription method comprises the following steps:

[0037] Step 1: Convert the input video file and extract the audio. Use the automated editing tool to extract the audio from the .mp4 video file and save it as a 16kHz, 16-bit mono audio file in .wav format.

[0038] Step 2: Perform whole-segment and segment recognition on the extracted audio. Use the audio recognition model trained with Chinese speech to perform whole-segment recognition on the extracted audio and save the recognition results. Then use the voiceprint recognition model to classify the different speakers in the speech. After obtaining the different speakers and their corresponding intervals, divide the whole audio into small audio files. Then perform audio recognition on each audio file to obtain different speakers and their corresponding speech content. Similarly, save the segment recognition results.

[0039] Step 3: Read the saved overall recognition results and segmented recognition results, perform text matching, and obtain the final audio transcription result. According to the duration of the audio in the segmented recognition, select a shorter segment, use the recognition results of the previous and next segments of the segment to match the text in the overall recognition result, and use the text between the two parts of the match as the recognition result of the selected segment. In this way, all short-duration segments are matched to obtain the final audio transcription result.

[0040] Through the above steps, the text content in the video can be obtained more quickly, which helps case handlers organize and record information during the law enforcement process, speeds up the process of storing information data, and improves the overall case handling efficiency.

[0041] In a third embodiment, a speech transcription method based on deep learning includes:

[0042] S10. Extracting the audio file from the video file;

[0043] S20. Identify a first text corresponding to the audio in the audio file, the first text is a set of texts corresponding to the audio in the audio file, and the first text includes an audio time mark;

[0044] S30. Classifying different sound features in the audio in the audio file through voiceprint recognition;

[0045] S40. Determine different speaking objects and audio intervals of different speaking objects according to the feature classification;

[0046] S50. Identify a second text corresponding to the audio of the audio interval, the second text is a collection of texts corresponding to the audio of each audio interval, and the second text includes a text interval tag of the text interval corresponding to each audio interval and a speaker tag corresponding to the text interval, and the second text also includes an audio time tag;

[0047] S60. According to the audio time mark, the second text is matched with the first text, and the text corresponding to the audio of each audio interval in the second text is placed in the corresponding text position of the first text. The speaking object of the corresponding text interval of the first text is marked according to the text interval mark of the second text, and the first text of the speaking object of the marked text interval is the file obtained by speech transcription, wherein the audio interval and text interval are corresponding representations of the time interval of the audio time.

[0048] In the deep learning-based speech transcription method in one embodiment of the present invention, in the step S50, no valid text content is identified in the audio interval, and the sound features and feature classification of the audio interval are identified, so that the text interval of this type of text in the second text has no corresponding text content, but the text interval corresponds to the speaker tag, and this type of text is a special text in the second text;

[0049] In step S60, the second text is matched with the first text according to the audio time mark, wherein the special text in the second text corresponds to the corresponding text position in the first text according to the audio time mark, and the speaking object of the corresponding text interval mark of the first text is matched according to the text interval mark of the special text in the second text.

[0050] In a deep learning-based speech transcription method in one embodiment of the present invention, the method for extracting an audio file from a video file in step S10 comprises: for a video file in a law enforcement recorder or a monitor, using an automated editing tool to extract the audio from the video and fixing the format, wherein the extracted audio format is fixed to 16kHz, 16-bit mono audio.

[0051] In the deep learning-based speech transcription method in one embodiment of the present invention, the method of classifying different sound features in the audio in the audio file by voiceprint recognition in step S30 includes:

[0052] Preprocessing the audio signal of the audio in the audio interval;

[0053] Perform MFCC feature extraction on the audio signal of the preprocessed audio interval, convert each frame waveform into a multidimensional vector containing sound information, and obtain a feature vector in the audio of the audio interval;

[0054] The acoustic model classifies the feature vector according to the acoustic features of the speech to obtain a classification of sound features in the audio in the audio file.

[0055] In the deep learning-based speech transcription method of one embodiment of the present invention, the second text corresponding to the audio of the audio interval in step S50 is identified, including

[0056] The acoustic model classifies the acoustic features into phonemes and / or word units;

[0057] The obtained phonemes and / or word units are decoded into a complete sentence through a language model, and the complete sentence is the second text of the text interval corresponding to the audio of the audio interval.

[0058] In the deep learning-based speech transcription method in one embodiment of the present invention, the preprocessing includes any one of frame cutting, pre-emphasis on the high-frequency part of the speech, and increasing the high-frequency resolution of the speech, or a combination thereof.

[0059] Based on the above embodiments, an embodiment of the present application also provides an electronic device, which includes: one or more processors, a memory, and one or more programs; wherein the one or more programs are stored in the memory, and the one or more programs include instructions, and when the instructions are executed by the electronic device, the electronic device executes the method provided in the above embodiments.

[0060] Based on the above embodiments, the embodiments of the present application further provide a computer storage medium, in which a computer program is stored. When the computer program is executed by a computer, the computer executes the method provided in the above embodiments.

[0061] The storage medium may be any available medium that can be accessed by a computer. For example, but not limited to, a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer.

[0062] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0063] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0064] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0065] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0066] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A speech transcription method based on deep learning, characterized in that: include S10. Extracting the audio file from the video file; S20. Identify a first text corresponding to the audio in the audio file, the first text is a set of texts corresponding to the audio in the audio file, and the first text includes an audio time mark; S30. Classifying different sound features in the audio in the audio file through voiceprint recognition; S40. Determine different speaking objects and audio intervals of different speaking objects according to the feature classification; S50. Identify a second text corresponding to the audio of the audio interval, the second text is a collection of texts corresponding to the audio of each audio interval, and the second text includes a text interval tag of the text interval corresponding to each audio interval and a speaker tag corresponding to the text interval, and the second text also includes an audio time tag; S60. According to the audio time mark, the second text is matched with the first text, and the text corresponding to the audio of each audio interval in the second text is placed in the corresponding text position of the first text. The speaking object of the corresponding text interval of the first text is marked according to the text interval mark of the second text, and the first text of the speaking object of the marked text interval is the file obtained by speech transcription, wherein the audio interval and text interval are corresponding representations of the time interval of the audio time.

2. The deep learning-based speech transcription method according to claim 1, characterized in that: In the step S50, no valid text content is identified in the audio interval, and the sound features and feature classification of the audio interval are identified, so that the text interval of this type of text in the second text has no corresponding text content, but the text interval corresponds to the speaker tag, and this type of text is a special text in the second text; In step S60, the second text is matched with the first text according to the audio time mark, wherein the special text in the second text corresponds to the corresponding text position in the first text according to the audio time mark, and the speaking object of the corresponding text interval mark of the first text is matched according to the text interval mark of the special text in the second text.

3. The deep learning-based speech transcription method according to claim 1 or 2, characterized in that: The method of extracting the audio file from the video file in step S10 includes: For video files in law enforcement recorders or monitors, use automated editing tools to extract the audio from the video and fix the format. The extracted audio format is fixed to 16kHz, 16-bit mono audio.

4. The deep learning-based speech transcription method according to claim 1 or 2, characterized in that: The method for classifying different sound features in the audio in the audio file by voiceprint recognition in step S30 includes: Preprocessing the audio signal of the audio in the audio interval; Perform MFCC feature extraction on the audio signal of the preprocessed audio interval, convert each frame waveform into a multidimensional vector containing sound information, and obtain a feature vector in the audio of the audio interval; The acoustic model classifies the feature vector according to the acoustic features of the speech to obtain a classification of sound features in the audio in the audio file.

5. The deep learning-based speech transcription method according to claim 4, characterized in that: The second text corresponding to the audio of the audio interval in step S50 includes: The acoustic model classifies the acoustic features into phonemes and / or word units; The obtained phonemes and / or word units are decoded into a complete sentence through a language model, and the complete sentence is the second text of the text interval corresponding to the audio of the audio interval.

6. The deep learning-based speech transcription method according to claim 4, characterized in that: The preprocessing includes any one of frame cutting, pre-emphasizing the high frequency part of the speech, and increasing the high frequency resolution of the speech, or a combination thereof.

7. An electronic device, comprising: One or more processors, a memory, and one or more programs; wherein the one or more programs are stored in the memory, and the one or more programs include instructions, and when the instructions are executed by the electronic device, the electronic device executes any one of the methods of claims 1 to 6.

8. A computer-readable storage medium, comprising a computer program, and when the computer program is executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 6.

Images