Method and device for constructing dialect speech recognition model corpus based on broadcast news

Abstract

The invention discloses a dialect speech recognition model corpus construction method and device based on broadcast news, and the method comprises the steps: obtaining a broadcast news video which comprises a first subtitle; performing audio track extraction on the broadcast news video to obtain a broadcast news audio; calculating a mute point list according to the broadcast news audio; calculating a natural voice starting and ending time list according to the mute point list; calculating a key frame time axis according to the broadcast news video; generating an initial corpus data set according to the natural voice starting and ending time list and the key frame time axis; and constructing a dialect speech recognition model corpus according to the initial corpus data set. According to the invention, corpus construction is realized, the efficiency and accuracy are improved, and the corpus management cost is reduced. The method can be widely applied to the technical field of model corpora.

Description

Technical Field

[0001] The present invention relates to the technical field of model corpus, and in particular to a method and device for constructing a dialect speech recognition model corpus based on broadcast news. Background Art

[0002] The existing corpus contains a large amount of speech data in official languages ​​such as Mandarin or English, but the corpus of dialect speech data is small and has not yet been standardized, resulting in low generalization ability of the trained model. The traditional method of corpus production is to produce corpus by combining manual listening and manual annotation, which is inefficient, prone to annotation errors and low accuracy. At the same time, the number of corpora is large and the cost of corpus management is high.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0004] The embodiment of the present invention provides a method and device for constructing a dialect speech recognition model corpus based on broadcast news, which effectively improves efficiency and accuracy and reduces corpus management costs.

[0005] On the one hand, an embodiment of the present invention provides a method for constructing a dialect speech recognition model corpus based on broadcast news, comprising the following steps:

[0006] Acquire a broadcast news video, wherein the broadcast news video has a first subtitle;

[0007] Extracting the audio track from the broadcast news video to obtain the broadcast news audio;

[0008] Calculate a list of silent points based on the broadcast news audio;

[0009] Calculate the start and end time list of natural speech according to the silence point list;

[0010] Calculating a key frame timeline according to the broadcast news video;

[0011] Generate an initial corpus data set according to the natural speech start and end time list and the key frame time axis;

[0012] A dialect speech recognition model corpus is constructed based on the initial corpus data set.

[0013] In some embodiments, the step of calculating a list of quiet points based on the broadcast news audio includes:

[0014] Performing spectrum analysis on the broadcast news audio to obtain the amplitude corresponding to each time point;

[0015] According to the amplitude, calculating the volume;

[0016] The time period during which the volume is less than the first volume threshold is used as the pause duration of the sentence segmentation;

[0017] When the pause duration of the sentence segment is less than the pause duration threshold, the time period is taken as a silence point;

[0018] A plurality of the quiet points are combined to obtain the quiet point list.

[0019] In some embodiments, calculating the natural speech start and end time list according to the silence point list includes:

[0020] Selecting two adjacent quiet points from the quiet point list as a quiet point combination;

[0021] Extracting the end time of the preceding silence point in time order from the silence point combination;

[0022] Extracting the start time of the silence point that is later in time sequence from the silence point combination;

[0023] Calculate the start and end time of the natural speech according to the end time and the start time;

[0024] Combine the multiple natural speech start and end times to obtain the natural speech start and end time list.

[0025] In some embodiments, the step of calculating a key frame timeline according to the broadcast news video includes:

[0026] Decomposing the broadcast news video into frames to obtain a plurality of video frames;

[0027] Set the subtitle position range;

[0028] Extracting a subtitle image from the video frame according to the subtitle position range;

[0029] Binarizing the subtitle image to obtain a binary image;

[0030] Calculating the optical flow change of each pixel in the binary image according to the optical flow function;

[0031] Calculating a key frame sequence according to the optical flow change;

[0032] The key frame timeline is calculated according to the key frame sequence and the video frame rate.

[0033] In some embodiments, generating an initial corpus data set according to the natural speech start and end time list and the key frame time axis includes:

[0034] Extracting a first sentence segment from the natural speech start and end time list;

[0035] Extracting a second sentence segment from the key frame timeline, wherein the sequence number of the second sentence segment is the same as the sequence number of the first sentence segment;

[0036] Calculate a start time difference according to the start time of the first sentence segment and the start time of the second sentence segment;

[0037] Calculate the end time difference according to the end time of the first sentence segment and the end time of the second sentence segment;

[0038] If the start time difference is greater than the time difference threshold or the end time difference is greater than the time difference threshold, the first sentence segment is deleted from the natural speech start and end time list to obtain a corrected time axis;

[0039] Extracting the first subtitle corresponding to each key frame in the correction timeline from the broadcast news video;

[0040] generating a first time-text relationship table according to the corrected time axis and the first subtitle;

[0041] According to the preset cutting time, the broadcast news audio is cut to obtain sub-audio;

[0042] According to the preset cutting time, the first time-text relationship table is cut to obtain sub-sentences;

[0043] Combining the sub-audio and the sub-sentence to obtain a text-audio combination;

[0044] The initial corpus data set is generated according to the multiple text and audio combinations.

[0045] In some embodiments, constructing a dialect speech recognition model corpus based on the initial corpus dataset includes:

[0046] Get the video of the corpus to be recognized;

[0047] Inputting the initial corpus data set into a preset speech recognition model so that the preset speech recognition model is trained to obtain a dialect speech recognition model;

[0048] Extracting the audio to be recognized and a plurality of second subtitles from the corpus video to be recognized;

[0049] Calculate the start and end time list of the speech of the corpus to be recognized according to the audio to be recognized;

[0050] Inputting the audio to be recognized into the dialect speech recognition model to obtain a dialect speech recognition result;

[0051] Generate a second time-text relationship table according to the start and end time list of the speech of the to-be-recognized corpus and the dialect speech recognition result;

[0052] Calculating a target corpus according to the second time-text relationship table and the second subtitles;

[0053] The target corpus is added to the initial corpus data set to obtain the dialect speech recognition model corpus.

[0054] In some embodiments, calculating the volume according to the amplitude includes:

[0055] The volume is calculated according to the amplitude using a volume calculation formula, which is:

[0056]

[0057] Wherein, d is the volume, and F is the amplitude.

[0058] In some embodiments, the expression of the optical flow function is:

[0059] I x (k,l)u+I y (k,l)c+I t (k,l)=0,(k,l)∈W,

[0060] In the formula, I x is the partial derivative of image brightness with respect to the spatial coordinate x-axis, I y is the partial derivative of image brightness with respect to the spatial coordinate y-axis, I t is the partial derivative of image brightness with respect to time, (k, l) is the position of the pixel, u is the optical flow of the pixel in the x-axis direction, v is the optical flow of the pixel in the y-axis direction, and W is the subtitle image.

[0061] On the other hand, an embodiment of the present invention provides a device for constructing a dialect speech recognition model corpus based on broadcast news, comprising:

[0062] The first module is used to obtain a broadcast news video, wherein the broadcast news video has a first subtitle;

[0063] The second module is used to extract the audio track from the broadcast news video to obtain the broadcast news audio;

[0064] The third module is used to calculate a silent point list according to the broadcast news audio;

[0065] A fourth module is used to calculate a natural speech start and end time list according to the silence point list;

[0066] A fifth module is used to calculate a key frame time axis according to the broadcast news video;

[0067] A sixth module is used to generate an initial corpus data set according to the natural speech start and end time list and the key frame time axis;

[0068] The seventh module is used to construct a dialect speech recognition model corpus based on the initial corpus data set.

[0069] In another aspect, an embodiment of the present invention provides a computer device, comprising:

[0070] at least one processor;

[0071] at least one memory for storing at least one program;

[0072] When the at least one program is executed by the at least one processor, the at least one processor implements the described method.

[0073] The beneficial effects of the present invention are as follows:

[0074] The embodiment of the present invention first obtains a broadcast news video, extracts the audio track of the broadcast news video, and obtains the broadcast news audio. Then, according to the broadcast news audio, a list of silence points is calculated, and according to the list of silence points, a list of natural speech start and end times is calculated. Then, according to the broadcast news video, a key frame timeline is calculated, and according to the list of natural speech start and end times and the key frame timeline, an initial corpus data set is generated. Finally, according to the initial corpus data set, a dialect speech recognition model corpus is constructed, thereby realizing corpus construction, improving efficiency and accuracy, and reducing corpus management costs.

[0075] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0077] Figure 1 A flowchart of a method for constructing a dialect speech recognition model corpus based on broadcast news according to an embodiment of the present invention;

[0078] Figure 2An audio waveform and spectrum diagram according to an embodiment of the present invention;

[0079] Figure 3 A schematic diagram of a quiet point according to an embodiment of the present invention;

[0080] Figure 4 A schematic diagram of the start and end time of natural speech in a spectrogram format according to an embodiment of the present invention;

[0081] Figure 5 A schematic diagram of a natural speech start and end time list in a timeline format according to an embodiment of the present invention;

[0082] Figure 6 A schematic diagram of a subtitle image according to an embodiment of the present invention;

[0083] Figure 7 A schematic diagram of a binarized image according to an embodiment of the present invention;

[0084] Figure 8 A schematic diagram of a comparison between a natural speech start and end time list and a key frame time axis according to an embodiment of the present invention;

[0085] Fig. 9 A schematic diagram of a first time character relationship table according to an embodiment of the present invention;

[0086] Fig.10 A schematic diagram of a text and audio combination according to an embodiment of the present invention;

[0087] Fig.11 A schematic diagram of comparing the lengths of text characters according to an embodiment of the present invention;

[0088] Fig.12 A schematic diagram of the overall process of building a corpus according to an embodiment of the present invention;

[0089] Fig.13 A schematic diagram of the structure of a device for constructing a dialect speech recognition model corpus based on broadcast news according to an embodiment of the present invention;

[0090] Fig.14 The figure is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0091] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0092] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0093] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0094] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0095] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0096] Corpus: refers to a large-scale electronic text library that has been scientifically sampled and processed, which contains language materials that have actually appeared in the actual use of the language.

[0097] Among the related technologies, the existing speech recognition technology mainly focuses on the recognition of official languages ​​such as Mandarin and English. At present, the recognition readiness rate can reach more than 95%, because the corpus supporting the training of the speech recognition model has more than 300,000 hours of speech data. However, the industry pays relatively little attention to the speech recognition ability of dialects, and the current recognition accuracy is not high. One of the reasons is that dialect data has not yet been standardized, and there are relatively few large-scale corpus data sets that can be used for training, resulting in weak generalization ability of the model. The dialect corpus data set is not large-scale, mainly due to insufficient investment and lack of standardization. The traditional corpus production method is to produce corpus by manually listening to recordings and manually typing and annotating. The production efficiency per person per day is only 2-3 hours, which is time-consuming and labor-intensive, and the number of corpus is large, and the project management cost is high. At the same time, the situation of audio with few words or many words in the corpus is not easy to detect; long-term repetitive work leads to low corpus accuracy.

[0098] In view of this, this embodiment solves the problem of preprocessing corpus data, replaces the traditional manual annotation method with machine automation processing technology, and uses a comprehensive algorithm combining multimodal recognition such as computer vision processing technology, audio processing technology, and audio-video conversion technology to automatically process the dialect programs broadcast by radio and television stations in batches, thereby forming a high-quality dialect corpus data set that can be used for subsequent training and processing of dialect speech recognition models and dialect speech generation models. Compared with manual methods, it is more efficient, more accurate, and has lower management costs.

[0099] The method for constructing a dialect speech recognition model corpus based on broadcast news provided in the embodiment of the present application relates to the technical field of model corpus. The method for constructing a dialect speech recognition model corpus based on broadcast news provided in the embodiment of the present application can be applied to a terminal, can be applied to a server, or can be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or can be configured as a server cluster or distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other basic cloud computing services. The server can also be a node server in a blockchain network; the software can be an application that implements a dialect speech recognition model corpus construction method based on broadcast news, etc., but is not limited to the above forms.

[0100] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0101] The following is a detailed explanation of the embodiments of the present application in conjunction with the accompanying drawings:

[0102] Figure 1 is an optional flowchart of a method for constructing a dialect speech recognition model corpus based on broadcast news provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S107.

[0103] Step S101: obtaining a broadcast news video, wherein the broadcast news video has a first subtitle;

[0104] Step S102, extracting the audio track from the broadcast news video to obtain the broadcast news audio;

[0105] Step S103, calculating a list of silent points according to the broadcast news audio;

[0106] Step S104, calculating a natural speech start and end time list according to the silence point list;

[0107] Step S105, calculating the key frame time axis according to the broadcast news video;

[0108] Step S106, generating an initial corpus data set according to the natural speech start and end time list and the key frame time axis;

[0109] Step S107: construct a dialect speech recognition model corpus based on the initial corpus data set.

[0110] Steps S101 to S107 shown in the embodiment of the present application implement corpus construction, improve efficiency and accuracy, and reduce corpus management costs.

[0111] In step S101 of some embodiments, a broadcast news video may be obtained through a broadcast station database. The broadcast news video may also be obtained through other methods, not limited thereto. The broadcast news video has a first subtitle. In the first subtitle, the text changes with time. It is understandable that most traditional news broadcast videos have subtitles.

[0112] In some embodiments, in step S102, the audio track of the broadcast news video may be extracted to obtain the broadcast news audio, which may be made into an audio file, generally in wav or mp3 format.

[0113] In some embodiments, in step S103, calculating a list of silent points based on the broadcast news audio may include but is not limited to the following steps:

[0114] Perform spectrum analysis on broadcast news audio to obtain the amplitude corresponding to each time point;

[0115] According to the amplitude, calculate the volume;

[0116] The time period during which the volume is less than the first volume threshold is used as the pause duration of the sentence segmentation;

[0117] When the pause duration of a sentence is less than the pause duration threshold, the time period is regarded as a silent point;

[0118] Combine multiple quiet points to obtain a quiet point list.

[0119] In some embodiments, the broadcast news audio may be first subjected to spectrum analysis to obtain the amplitude corresponding to each time point. It is understandable that a number of broadcast news videos may be manually sampled and converted into broadcast news audio, and then spectrum analysis may be performed to obtain the spectrum analysis result corresponding to each broadcast news audio. Then, the volume is calculated according to the amplitude using the volume calculation formula, where the volume calculation formula is: Where D is the volume and F is the amplitude. The time period when the volume (dB) is less than the first volume threshold is used as the pause duration. For example, the first volume threshold can be set to -50dB. Figure 2 As shown in the figure, the red box indicates the area where the volume (dB) is less than the first volume threshold. The time period that meets the requirement is used as the pause time of the sentence break. When the pause time of the sentence break is less than the pause time threshold, the time period is used as the silent point. For example, the pause time threshold can be set to 0.2 seconds. When the pause time of the sentence break is less than 0.2 seconds, the corresponding time period is used as the silent point. Figure 3 As shown in FIG. 1 , silence points of different durations are included. Finally, multiple silence points are combined to obtain a silence point list. Furthermore, the silence points can be calculated by using the audio waveform and the spectrogram in combination with the duration range of the sentence or the average sentence duration.

[0120] In some embodiments, in step S104, calculating a natural speech start and end time list according to the silence point list may include but is not limited to the following steps:

[0121] Select two adjacent quiet points from the quiet point list as a quiet point combination;

[0122] Extract the end time of the silence point preceding in time order from the silence point combination;

[0123] Extract the start time of the silence point that is later in time sequence from the silence point combination;

[0124] Calculate the start and end time of natural speech according to the end time and start time;

[0125] Combine multiple natural speech start and end times to obtain a natural speech start and end time list.

[0126] In some embodiments, two adjacent silent points can be selected from the silent point list as a silent point combination, that is, the silent point combination contains two adjacent silent points. Then, the end time of the silent point in the previous time sequence is extracted from the silent point combination, and the start time of the silent point in the later time sequence is extracted from the silent point combination. Then, the start and end time of the natural speech is calculated based on the end time and the start time. The start and end time of the natural speech is as follows: Figure 4 As shown in the figure, the figure contains sentences of different lengths. Finally, multiple natural speech start and end times are combined to obtain a natural speech start and end time list, which is displayed in a timeline format as shown in Figure 5 It is understandable that the difference set between the audio and the silence point can be calculated based on the silence point list, that is, the set of differences between the start time of a silence point and the end time of the previous silence point (i.e., the natural speech start and end time list).

[0127] In some embodiments, in step S105, calculating the key frame timeline according to the broadcast news video may include but is not limited to the following steps:

[0128] Decompose the broadcast news video into frames to obtain several video frames;

[0129] Set the subtitle position range;

[0130] Extracting subtitle images from video frames according to the subtitle position range;

[0131] Binarize the subtitle image to obtain a binary image;

[0132] According to the optical flow function, calculate the optical flow change of each pixel in the binary image;

[0133] Calculate the key frame sequence according to the optical flow changes;

[0134] Calculate the key frame timeline based on the key frame sequence and video frame rate.

[0135] In some embodiments, the video can be processed frame by frame by computer vision technology to remove irrelevant color blocks in the video and determine the frame similarity to screen out key frames. The broadcast news video can be first decomposed into frames to obtain a number of video frames and saved as a picture sequence. Then, a subtitle position range is set and subtitle images are extracted from the video frames according to the subtitle position range. The extracted subtitle images are as follows: Figure 6 As shown. Then the subtitle image is binarized to obtain a binary image as shown Figure 7 As shown. Then, according to the optical flow function, the optical flow change of each pixel in the binary image is calculated, and the key frame sequence is calculated according to the optical flow change. For example, the Lucas Kanade algorithm can be used to calculate the optical flow change, and the sequence number of each key frame is recorded and saved as a key frame sequence. Among them, the expression of the optical flow function is: x (k,l)u+I y (k,l)v+I t (k,l)=0,(k,l)∈W,where I x is the partial derivative of the image brightness about the spatial coordinate x-axis, which represents the gradient of the image in the x-axis direction, I y is the partial derivative of the image brightness about the spatial coordinate y-axis, which represents the gradient of the image in the y-axis direction, I t is the partial derivative of the image brightness with respect to time t, indicating the rate of change of the image brightness over time, (k, l) is the position of the pixel, u is the optical flow (speed) of the pixel tracked in the image sequence in the x-axis direction, v is the optical flow (speed) of the pixel tracked in the image sequence in the y-axis direction, and W is the subtitle image. It can be understood that in this embodiment, when the subtitles remain unchanged, I t The optical flow of the subtitle image (i.e., the binarized image) is constant when the subtitle remains unchanged. Therefore, the optical flow method is effective in obtaining key frames and has low computing power consumption. Finally, the key frame timeline is calculated based on the key frame sequence and the video frame rate. For example, the video frame rate can be set to 25fps, and the key frame timeline is calculated by the serial numbers of the previous and next key frames in the key frame sequence.

[0136] In some embodiments, in step S106, generating an initial corpus data set according to the natural speech start and end time list and the key frame time axis may include but is not limited to the following steps:

[0137] Extracting the first sentence segment from the natural speech start and end time list;

[0138] Extracting a second sentence segment from the key frame timeline, wherein the sequence number of the second sentence segment is the same as the sequence number of the first sentence segment;

[0139] Calculate the start time difference according to the start time of the first sentence segment and the start time of the second sentence segment;

[0140] Calculate the end time difference according to the end time of the first sentence segment and the end time of the second sentence segment;

[0141] If the start time difference is greater than the time difference threshold or the end time difference is greater than the time difference threshold, the first sentence segment is deleted from the natural speech start and end time list to obtain a corrected time axis;

[0142] Extracting the first subtitle corresponding to each key frame in the correction timeline from the broadcast news video;

[0143] Generate a first time text relationship table according to the corrected time axis and the first subtitle;

[0144] According to the preset cutting time, the broadcast news audio is cut to obtain sub-audio;

[0145] According to the preset cutting time, the first time text relationship table is cut to obtain sub-sentences;

[0146] Combine the sub-audio and the sub-sentence to obtain a text-audio combination;

[0147] Generate an initial corpus dataset based on multiple text and audio combinations.

[0148] In some embodiments, the first sentence segment can be first extracted from the natural speech start and end time list, and the second sentence segment can be extracted from the key frame timeline, wherein the sequence number of the second sentence segment is the same as the sequence number of the first sentence segment. Then, the start time difference is calculated based on the start time of the first sentence segment and the start time of the second sentence segment, and the end time difference is calculated based on the end time of the first sentence segment and the end time of the second sentence segment. If the start time difference is greater than the time difference threshold or the end time difference is greater than the time difference threshold, the first sentence segment is deleted from the natural speech start and end time list to obtain a corrected timeline. Exemplarily, the time difference threshold can be set to 400 milliseconds. If the start time difference is greater than 400 milliseconds or the end time difference is greater than 400 milliseconds, the first sentence segment is deleted from the natural speech start and end time list to obtain a corrected timeline. The natural speech start and end time list is compared with the key frame timeline. Figure 8 As shown in the figure, the sentence segment with sequence number 3 is marked as filtered, that is, the sentence segment that needs to be deleted. Then extract the first subtitle corresponding to each key frame in the corrected timeline from the broadcast news video, and generate the first time text relationship table according to the corrected timeline and the first subtitle as shown in the figure. Fig. 9Then, according to the preset cutting time, the broadcast news audio is cut to obtain sub-audio, and according to the preset cutting time, the first time text relationship table is cut to obtain sub-sentences, and finally the sub-audio and sub-sentence are combined to obtain a text-audio combination. Based on multiple text-audio combinations, an initial corpus data set is generated. Among them, the text-audio combination is as follows Fig.10 As shown, it includes the audio file name, text and audio duration. For example, the preset cutting duration can be set within the range of 1-12 seconds, and the broadcast news audio and the first time text relationship table can be cut with the same cutting duration to produce a one-to-one corresponding text and audio combination, thereby obtaining the initial corpus data set.

[0149] In some embodiments, in step S107, constructing a dialect speech recognition model corpus based on the initial corpus data set may include but is not limited to the following steps:

[0150] Get the video of the corpus to be recognized;

[0151] Inputting the initial corpus data set into a preset speech recognition model so that the preset speech recognition model is trained to obtain a dialect speech recognition model;

[0152] Extracting the audio to be recognized and a plurality of second subtitles from the video of the corpus to be recognized;

[0153] According to the audio to be recognized, calculate the start and end time list of the speech of the corpus to be recognized;

[0154] Input the audio to be recognized into the dialect speech recognition model to obtain the dialect speech recognition result;

[0155] Generate a second time-text relationship table according to the start and end time list of the speech of the to-be-recognized corpus and the dialect speech recognition result;

[0156] Calculate the target corpus according to the second time text relationship table and the second subtitle;

[0157] Add the target corpus to the initial corpus dataset to obtain the dialect speech recognition model corpus.

[0158] In some embodiments, after a basic dialect speech recognition model is trained with less corpus (30 to 100 hours), the corpus can be further processed using the basic dialect speech recognition model to improve the quality of the corpus, and the model accuracy can be continuously improved by repeating this process. The video of the corpus to be recognized can be obtained first, and the initial corpus data set can be input into the preset speech recognition model so that the preset speech recognition model can be trained to obtain the dialect speech recognition model, and then the audio to be recognized and several second subtitles can be extracted from the video of the corpus to be recognized. According to the audio to be recognized, the start and end time list of the speech of the corpus to be recognized is calculated, and then the audio to be recognized is input into the dialect speech recognition model to obtain the dialect speech recognition result. It can be understood that when the audio to be recognized is converted into text by the basic dialect speech recognition model, there is no need to consider the recognition accuracy, only whether the number of recognized words is consistent with the actual number of words in the audio. If the model can recognize that 1 sound corresponds to 1 word, the recognition is correct. Then, according to the start and end time list of the speech of the corpus to be recognized and the dialect speech recognition result, a second time-text relationship table is generated, and the target corpus is calculated according to the second time-text relationship table and the second subtitle. For example, in the calculation process of the target corpus, optical character recognition (OCR) can be used to obtain the second subtitles, and automatic speech recognition (ASR) can be used for speech recognition. After the dialect speech recognition model converts the audio to be recognized into text, the length of the text characters in the second subtitles is compared with the corresponding length of the text characters in the dialect speech recognition results to correct whether the time axis of the subtitle time is correct. If the length of the text characters in the second subtitle is less than the corresponding length of the text characters in the dialect speech recognition result, it means that the syllables in the speech are more than the text on the image, the time axis is too long, and the corpus has "too many words". The corpus is discarded and not used for this round of speech recognition model training; if the length of the text characters in the second subtitle is greater than the corresponding length of the text characters in the dialect speech recognition result, it means that the syllables in the speech are less than the text on the image, the time axis is too short, and the corpus has "few words". The corpus is discarded and not used for this round of speech recognition model training; if the length of the text characters in the second subtitle is equal to the corresponding length of the text characters in the dialect speech recognition result, it means that the time axis of the corpus is correct and can be retained as a high-precision target corpus. The text character length comparison example is as follows Fig.11 As shown. Finally, the target corpus is added to the initial corpus data set to obtain the dialect speech recognition model corpus. Furthermore, the target corpus can be cut according to the preset cutting time and added to the initial corpus data set to improve the diversity of the initial corpus data set.

[0159] In some embodiments, the overall process of building a corpus is as follows: Fig.12As shown, the broadcast news video can be obtained first, the audio track can be extracted, and the audio file can be made. Then, the silent point recognition can be performed, and the silent point list can be calculated. According to the silent point list, the first time axis (i.e., the natural speech start and end time list) can be calculated. Then, image processing can be performed to calculate the second time axis (i.e., the key frame time axis). The time axis can be corrected using the first time axis and the second time axis to obtain the corrected time axis. Subtitles can be extracted from the image. The first time text relationship table can be generated by combining the corrected time axis and the subtitles to generate an initial corpus data set. Then, the initial corpus data set can be used for model training to obtain a basic dialect speech recognition model. The basic dialect speech recognition model can be used for speech recognition. The second time file relationship table can be generated by combining the speech recognition results. Then, the first time text relationship table and the second time text relationship table can be combined, and the text character length can be compared using ASR and OCR to obtain the target corpus. Finally, the target corpus can be added to the initial corpus data set to obtain a training corpus (i.e., a dialect speech recognition model corpus). Furthermore, the dialect speech recognition model corpus can be used to update the dialect speech recognition model to improve the model accuracy.

[0160] The beneficial effects of implementing the embodiments of the present invention include: the embodiments of the present invention first obtain broadcast news video, extract the audio track of the broadcast news video, obtain broadcast news audio, then calculate the list of silent points based on the broadcast news audio, and calculate the list of natural speech start and end time based on the list of silent points, and then calculate the key frame timeline based on the broadcast news video, and generate the initial corpus data set based on the list of natural speech start and end time and the key frame timeline, and finally construct the dialect speech recognition model corpus based on the initial corpus data set, thereby realizing corpus construction, improving efficiency and accuracy, and reducing corpus management costs. At the same time, this embodiment automatically, efficiently and cost-effectively processes the existing dialect broadcast news materials into dialect corpus data through preprocessing, uses small-scale corpus to train the basic model, and combines optical character recognition and silence detection to greatly improve the accuracy of the corpus, forming a high-quality and high-precision corpus data set. These data sets can be used for the training of dialect speech recognition and dialect speech broadcast models, and are ultimately applied to intelligent interactive scenarios such as dialect recognition and dialect broadcast. In addition, this embodiment can realize automated dialect corpus preprocessing, which is faster, more efficient and less costly than manual annotation. In the process of algorithm optimization, key algorithm optimization is performed on the output corpus content, and optical flow method, binary method and other methods are used to further improve processing efficiency and reduce processing costs while reducing computing power consumption.

[0161] like Fig.13 As shown, the embodiment of the present invention also provides a device for constructing a dialect speech recognition model corpus based on broadcast news, comprising:

[0162] The first module 801 is used to obtain a broadcast news video, where the broadcast news video has a first subtitle;

[0163] The second module 802 is used to extract the audio track from the broadcast news video to obtain the broadcast news audio;

[0164] The third module 803 is used to calculate a silent point list according to the broadcast news audio;

[0165] The fourth module 804 is used to calculate a natural speech start and end time list according to the silence point list;

[0166] The fifth module 805 is used to calculate the key frame time axis according to the broadcast news video;

[0167] The sixth module 806 is used to generate an initial corpus data set according to the natural speech start and end time list and the key frame time axis;

[0168] The seventh module 807 is used to construct a dialect speech recognition model corpus based on the initial corpus data set.

[0169] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0170] like Fig.14 As shown, an embodiment of the present invention further provides a computer device, including:

[0171] at least one processor 901;

[0172] At least one memory 902, used to store at least one program;

[0173] When at least one program is executed by at least one processor, the at least one processor implements Figure 1 The method shown.

[0174] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0175] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for constructing a dialect speech recognition model corpus based on broadcast news, characterized in that: The following steps are involved: Acquire a broadcast news video, wherein the broadcast news video has a first subtitle; Extracting the audio track from the broadcast news video to obtain the broadcast news audio; Calculate a list of silent points based on the broadcast news audio; Calculate the start and end time list of natural speech according to the silence point list; Calculating a key frame timeline according to the broadcast news video; Generate an initial corpus data set according to the natural speech start and end time list and the key frame time axis; A dialect speech recognition model corpus is constructed based on the initial corpus data set.

2. The method according to claim 1, characterized in that The step of calculating a silent point list according to the broadcast news audio includes: Performing spectrum analysis on the broadcast news audio to obtain the amplitude corresponding to each time point; According to the amplitude, calculating the volume; The time period during which the volume is less than the first volume threshold is used as the pause duration of the sentence segmentation; When the pause duration of the sentence segment is less than the pause duration threshold, the time period is taken as a silence point; A plurality of the quiet points are combined to obtain the quiet point list.

3. The method according to claim 1, characterized in that The step of calculating a natural speech start and end time list according to the silence point list includes: Selecting two adjacent quiet points from the quiet point list as a quiet point combination; Extracting the end time of the preceding silence point in time order from the silence point combination; Extracting the start time of the silence point that is later in time sequence from the silence point combination; Calculate the start and end time of the natural speech according to the end time and the start time; Combine the multiple natural speech start and end times to obtain the natural speech start and end time list.

4. The method according to claim 1, characterized in that: The step of calculating the key frame timeline according to the broadcast news video includes: Decomposing the broadcast news video into frames to obtain a plurality of video frames; Set the subtitle position range; Extracting a subtitle image from the video frame according to the subtitle position range; Binarizing the subtitle image to obtain a binary image; Calculating the optical flow change of each pixel in the binary image according to the optical flow function; Calculating a key frame sequence according to the optical flow change; The key frame timeline is calculated according to the key frame sequence and the video frame rate.

5. The method according to claim 1, characterized in that The generating an initial corpus data set according to the natural speech start and end time list and the key frame time axis includes: Extracting a first sentence segment from the natural speech start and end time list; Extracting a second sentence segment from the key frame timeline, wherein the sequence number of the second sentence segment is the same as the sequence number of the first sentence segment; Calculate the start time difference according to the start time of the first sentence segment and the start time of the second sentence segment; Calculate the end time difference according to the end time of the first sentence segment and the end time of the second sentence segment; If the start time difference is greater than the time difference threshold or the end time difference is greater than the time difference threshold, the first sentence segment is deleted from the natural speech start and end time list to obtain a corrected time axis; Extracting the first subtitle corresponding to each key frame in the correction timeline from the broadcast news video; generating a first time-text relationship table according to the corrected time axis and the first subtitle; According to the preset cutting time, the broadcast news audio is cut to obtain sub-audio; According to the preset segmentation duration, the first time-text relationship table is segmented to obtain sub-sentences; Combining the sub-audio and the sub-sentence to obtain a text-audio combination; The initial corpus data set is generated according to the multiple text and audio combinations.

6. The method according to claim 1, characterized in that The step of constructing a dialect speech recognition model corpus based on the initial corpus data set includes: Get the video of the corpus to be recognized; Inputting the initial corpus data set into a preset speech recognition model so that the preset speech recognition model is trained to obtain a dialect speech recognition model; Extracting the audio to be recognized and a plurality of second subtitles from the corpus video to be recognized; Calculate the start and end time list of the speech of the corpus to be recognized according to the audio to be recognized; Inputting the audio to be recognized into the dialect speech recognition model to obtain a dialect speech recognition result; Generate a second time-text relationship table according to the start and end time list of the speech of the to-be-recognized corpus and the dialect speech recognition result; Calculating a target corpus according to the second time-character relationship table and the second subtitles; The target corpus is added to the initial corpus data set to obtain the dialect speech recognition model corpus.

7. The method according to claim 2, characterized in that Calculating the volume according to the amplitude includes: The volume is calculated according to the amplitude using a volume calculation formula, which is: Wherein, D is the volume, and F is the amplitude.

8. The method according to claim 4, characterized in that The expression of the optical flow function is: I x (k,l)u+I y (k,l)c+I t (k,L)=0,(k,L)∈W, In the formula, I x is the partial derivative of image brightness with respect to the spatial coordinate x-axis, I y is the partial derivative of image brightness with respect to the spatial coordinate y-axis, I t is the partial derivative of image brightness with respect to time, (k, l) is the position of the pixel, u is the optical flow of the pixel in the x-axis direction, v is the optical flow of the pixel in the y-axis direction, and W is the subtitle image.

9. A device for constructing a dialect speech recognition model corpus based on broadcast news, characterized in that: include: The first module is used to obtain a broadcast news video, wherein the broadcast news video has a first subtitle; The second module is used to extract the audio track from the broadcast news video to obtain the broadcast news audio; The third module is used to calculate a silent point list according to the broadcast news audio; A fourth module is used to calculate a natural speech start and end time list according to the silence point list; A fifth module is used to calculate a key frame time axis according to the broadcast news video; A sixth module is used to generate an initial corpus data set according to the natural speech start and end time list and the key frame time axis; The seventh module is used to construct a dialect speech recognition model corpus based on the initial corpus data set.

10. A computer device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 8.

Images