Method and device for constructing a dialect speech recognition model corpus based on broadcast news

Abstract

The application discloses a dialect speech recognition model corpus construction method and device based on broadcast news, and the method comprises the following steps: acquiring a broadcast news video, wherein the broadcast news video has a first subtitle; performing audio track extraction on the broadcast news video to obtain broadcast news audio; calculating a mute point list according to the broadcast news audio; calculating a natural speech start and end time list according to the mute point list; calculating a key frame time axis according to the broadcast news video; generating an initial corpus dataset according to the natural speech start and end time list and the key frame time axis; and constructing a dialect speech recognition model corpus according to the initial corpus dataset. The application realizes corpus construction, improves the efficiency and accuracy, and reduces the corpus management cost. The application can be widely applied to the technical field of model corpus.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of model corpus, and in particular to a dialect speech recognition model corpus construction method and device based on broadcast news. BACKGROUND

[0002] Existing corpora have a large amount of official language speech data such as Mandarin or English, but the size of the dialect speech data corpus is small and has not been standardized, resulting in low generalization ability of the trained model. The traditional corpus production method combines manual listening to recordings and manual annotation to produce corpora, which is low in efficiency and prone to annotation errors and omissions, with low accuracy. At the same time, the corpus is large and the corpus management cost is high.

[0003] In summary, the technical problems in the related art need to be improved. SUMMARY

[0004] The present application provides a dialect speech recognition model corpus construction method and device based on broadcast news, which effectively improves efficiency and accuracy and reduces corpus management costs.

[0005] In one aspect, the present application provides a dialect speech recognition model corpus construction method based on broadcast news, comprising the following steps:

[0006] Obtaining a broadcast news video having a first subtitle in the broadcast news video;

[0007] Extracting an audio track from the broadcast news video to obtain a broadcast news audio;

[0008] Calculating a list of silent points from the broadcast news audio;

[0009] Calculating a list of natural speech start and end times from the list of silent points;

[0010] Calculating a key frame timeline from the broadcast news video;

[0011] Generating an initial corpus dataset from the list of natural speech start and end times and the key frame timeline;

[0012] Constructing a dialect speech recognition model corpus from the initial corpus dataset.

[0013] In some embodiments, calculating a list of silent points from the broadcast news audio comprises:

[0014] Performing a spectral analysis on the broadcast news audio to obtain an amplitude corresponding to each time point;

[0015] Calculating a volume from the amplitude;

[0016] The time period in which the volume is less than the first volume threshold is taken as a punctuation pause duration;

[0017] When the punctuation pause duration is less than a pause duration threshold, the time period is taken as a silence point;

[0018] The silence points are combined to obtain a silence point list.

[0019] In some embodiments, the calculating a natural speech start and end time list according to the silence point list comprises:

[0020] Selecting two adjacent silence points from the silence point list as a silence point combination;

[0021] Extracting an end time of a silence point in time sequence in front from the silence point combination;

[0022] Extracting a start time of a silence point in time sequence in back from the silence point combination;

[0023] Calculating a natural speech start and end time according to the end time and the start time;

[0024] Combining a plurality of the natural speech start and end times to obtain the natural speech start and end time list.

[0025] In some embodiments, the calculating a key frame time axis according to the broadcast news video comprises:

[0026] Frame decomposing the broadcast news video to obtain a plurality of video frames;

[0027] Setting a subtitle position range;

[0028] Extracting a subtitle image from the video frames according to the subtitle position range;

[0029] Binaryzation processing the subtitle image to obtain a binary image;

[0030] Calculating an optical flow change of each pixel point in the binary image according to an optical flow function;

[0031] Calculating a key frame sequence according to the optical flow change;

[0032] Calculating the key frame time axis according to the key frame sequence and a video frame rate.

[0033] In some embodiments, the generating an initial corpus dataset according to the natural speech start and end time list and the key frame time axis comprises:

[0034] Extracting a first sentence segment from the natural speech start and end time list;

[0035] extracting a second sentence segment from the keyframe timeline, a sequence number of the second sentence segment being the same as a sequence number of the first sentence segment;

[0036] calculating a start time difference according to a start time of the first sentence segment and a start time of the second sentence segment;

[0037] calculating an end time difference according to an end time of the first sentence segment and an end time of the second sentence segment;

[0038] if the start time difference is greater than a time difference threshold or the end time difference is greater than the time difference threshold, deleting the first sentence segment from the natural speech start and end time list to obtain a corrected timeline;

[0039] extracting a first caption corresponding to each keyframe in the corrected timeline from the broadcast news video;

[0040] generating a first time-text relationship table according to the corrected timeline and the first caption;

[0041] cutting the broadcast news audio according to a preset cutting duration to obtain a sub-audio;

[0042] cutting the first time-text relationship table according to the preset cutting duration to obtain a sub-sentence;

[0043] combining the sub-audio and the sub-sentence to obtain a text-audio combination;

[0044] generating the initial corpus dataset according to a plurality of the text-audio combinations.

[0045] In some embodiments, the constructing a dialect speech recognition model corpus according to the initial corpus dataset comprises:

[0046] obtaining a to-be-recognized corpus video;

[0047] inputting the initial corpus dataset into a preset speech recognition model to enable the preset speech recognition model to be trained to obtain a dialect speech recognition model;

[0048] extracting a to-be-recognized audio and a plurality of second captions from the to-be-recognized corpus video;

[0049] calculating a to-be-recognized corpus speech start and end time list according to the to-be-recognized audio;

[0050] inputting the to-be-recognized audio into the dialect speech recognition model to obtain a dialect speech recognition result;

[0051] According to the start and end time list of the corpus to be identified and the dialect speech recognition result, a second time-text relationship table is generated;

[0052] According to the second time-text relationship table and the second subtitle, a target corpus is calculated;

[0053] The target corpus is added to the initial corpus dataset to obtain the dialect speech recognition model corpus.

[0054] In some embodiments, the volume is calculated according to the amplitude, including:

[0055] The volume is calculated according to the amplitude by a volume calculation formula, the volume calculation formula being:

[0056]

[0057] In the formula, d is the volume, and F is the amplitude.

[0058] In some embodiments, the expression of the optical flow function is:

[0059] I x (k,l)u+I y (k,l)c+I t (k,l)=0,(k,l)∈W,

[0060] In the formula, I x is the partial derivative of the image brightness with respect to the x-axis of the spatial coordinates, I y is the partial derivative of the image brightness with respect to the y-axis of the spatial coordinates, I t is the partial derivative of the image brightness with respect to time, (k,l) is the position of the pixel point, u is the optical flow of the pixel point in the x-axis direction, v is the optical flow of the pixel point in the y-axis direction, and W is the subtitle image.

[0061] In another aspect, the embodiment of the present application provides a dialect speech recognition model corpus construction device based on broadcast news, including:

[0062] A first module is configured to acquire a broadcast news video, wherein the broadcast news video has a first subtitle;

[0063] A second module is configured to extract an audio track from the broadcast news video to obtain a broadcast news audio;

[0064] A third module is configured to calculate a list of mute points according to the broadcast news audio;

[0065] A fourth module is configured to calculate a list of start and end times of natural speech according to the list of mute points;

[0066] a fifth module configured to calculate a key frame time axis according to the broadcast news video;

[0067] a sixth module configured to generate an initial corpus dataset according to the natural speech start and end time list and the key frame time axis;

[0068] a seventh module configured to build a dialect speech recognition model corpus according to the initial corpus dataset.

[0069] In another aspect, an embodiment of the present application provides a computer device, comprising:

[0070] at least one processor;

[0071] at least one memory configured to store at least one program;

[0072] when the at least one program is executed by the at least one processor, the at least one processor implements the method.

[0073] The present application has the following beneficial effects:

[0074] The embodiment of the present application first acquires a broadcast news video, extracts an audio track from the broadcast news video to obtain a broadcast news audio, then calculates a mute point list according to the broadcast news audio, calculates a natural speech start and end time list according to the mute point list, calculates a key frame time axis according to the broadcast news video, generates an initial corpus dataset according to the natural speech start and end time list and the key frame time axis, and finally builds a dialect speech recognition model corpus according to the initial corpus dataset, thereby realizing corpus construction, improving efficiency and accuracy, and reducing corpus management cost.

[0075] Other features and advantages of the present application will be further described in the following description, and will become apparent from the description, or will be learned through practice of the present application. The objects and other advantages of the present application will be realized and achieved by the structure particularly pointed out in the description and the appended drawings. BRIEF DESCRIPTION OF DRAWINGS

[0076] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor based on these drawings.

[0077] Figure 1 a flow chart of a dialect speech recognition model corpus construction method based on broadcast news according to an embodiment of the present application;

[0078] Figure 2An audio waveform and a spectrum diagram for an embodiment of the present application;

[0079] Figure 3 A diagram of a mute point for an embodiment of the present application;

[0080] Figure 4 A diagram of natural speech start and end time in a spectrum diagram format for an embodiment of the present application;

[0081] Figure 5 A diagram of a natural speech start and end time list in a time axis format for an embodiment of the present application;

[0082] Figure 6 A diagram of a subtitle image for an embodiment of the present application;

[0083] Figure 7 A diagram of a binary image for an embodiment of the present application;

[0084] Figure 8 A diagram of a natural speech start and end time list and a key frame time axis for an embodiment of the present application;

[0085] Figure 9 A diagram of a first time text relationship table for an embodiment of the present application;

[0086] Figure 10 A diagram of a text audio combination for an embodiment of the present application;

[0087] Figure 11 A diagram of text character length comparison for an embodiment of the present application;

[0088] Figure 12 A diagram of a whole flow of corpus construction for an embodiment of the present application;

[0089] Figure 13 A structural diagram of a dialect speech recognition model corpus construction device based on broadcast news for an embodiment of the present application;

[0090] Figure 14 A hardware structure diagram of a computer device for an embodiment of the present application. DETAILED DESCRIPTION

[0091] In order to make the purposes, technical solutions and advantages of the present application clearer, further detailed description will be made to the present application in combination with the accompanying drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and not intended to limit the present application. When the following description refers to the accompanying drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with embodiments of the present application. They are merely examples of apparatuses and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0092] It can be understood that the terms "first", "second" and the like used in the present application can be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiments of the present application, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "when" or "in response to determining".

[0093] The terms "at least one", "multiple", "each", "any" and the like used in the present application include one, two or more than two, multiple includes two or more than two, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0094] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as understood by those skilled in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0095] Before the embodiments of the present application are described in detail, first, some nouns and terms involved in the embodiments of the present application are described, and the nouns and terms involved in the embodiments of the present application are applicable to the following explanations.

[0096] Corpus: refers to a large-scale electronic text library processed by scientific sampling and processing, in which real language materials appearing in the actual use of language are stored.

[0097] In the related art, the existing speech recognition technology mainly focuses on the recognition of official languages such as Mandarin and English, and the current recognition rate can reach more than 95%. Because the corpus supporting the speech recognition model training has more than 300,000 hours of voice data. The speech recognition ability of dialects is relatively less concerned by the industry, and the current recognition accuracy is not high. One of the reasons is that the dialect data has not been standardized, and the large-scale corpus data set that can be used for training is relatively small, resulting in weak generalization ability of the model. The dialect corpus data set is not large-scale, mainly due to insufficient investment and non-standardization. The traditional corpus production method is to manually listen to the recording, manually type and mark the audio, and the production efficiency of each person per day is only 2-3 hours, which is time-consuming and labor-intensive. At the same time, it is not easy to detect the situation of less audio and more words in the corpus; long-term repetitive work leads to low accuracy of the corpus.

[0098] Therefore, the embodiment solves the preprocessing problem of corpus data, changes the traditional manual labeling method to machine automatic processing technology, uses a comprehensive algorithm combining computer vision processing technology, audio processing technology, audio-video conversion technology and other multi-modal recognition to automatically process the dialect programs broadcast by radio and television stations in batches, thereby forming a high-quality dialect corpus data set, which can be used for subsequent training of dialect speech recognition model and dialect speech generation model. Compared with the manual method, it is more efficient, more accurate and has lower management cost.

[0099] The broadcast news-based dialect speech recognition model corpus construction method provided by the embodiment of the application relates to the technical field of model corpus. The broadcast news-based dialect speech recognition model corpus construction method provided by the embodiment of the application can be applied to a terminal, can also be applied to a server, and can also be software running in the terminal or the server. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, and the like, but is not limited thereto; the server end can be configured as a standalone physical server, can also be configured as a server cluster or a distributed system composed of multiple physical servers, can also be configured as a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and basic cloud computing services such as big data and artificial intelligence platforms, and the server can also be a node server in a blockchain network; the software can be an application that implements the broadcast news-based dialect speech recognition model corpus construction method, and the like, but is not limited to the above forms.

[0100] The application is operable with numerous general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations that can be suitable for use with the application include personal computers, server computers, handheld or laptop devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The application can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types. The application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in local and remote computer storage media including memory storage devices.

[0101] The embodiments of the application are explained in detail below with reference to the accompanying drawings:

[0102] Figure 1 is an optional flowchart of a dialect speech recognition model corpus construction method based on broadcast news provided by the embodiments of the application, Figure 1 The method in can include but is not limited to steps S101 to S107.

[0103] Step S101, acquiring a broadcast news video, the broadcast news video having a first subtitle;

[0104] Step S102, performing audio track extraction on the broadcast news video to obtain broadcast news audio;

[0105] Step S103, calculating a mute point list according to the broadcast news audio;

[0106] Step S104, calculating a natural speech start and end time list according to the mute point list;

[0107] Step S105, calculating a key frame time axis according to the broadcast news video;

[0108] Step S106, generating an initial corpus data set according to the natural speech start and end time list and the key frame time axis;

[0109] Step S107, constructing a dialect speech recognition model corpus according to the initial corpus data set.

[0110] The steps S101 to S107 shown in the embodiments of the application realize corpus construction, improve efficiency and accuracy, and reduce corpus management cost.

[0111] In step S101 of some embodiments, broadcast news video can be obtained through a radio station database. Broadcast news video can also be obtained through other means, not limited to these. The broadcast news video includes a first subtitle. In the first subtitle, the text changes over time. It is understood that most traditional news broadcast videos include subtitles.

[0112] In some embodiments, in step S102, the audio track of the broadcast news video can be extracted to obtain the broadcast news audio. This audio can then be created as an audio file, typically in WAV or MP3 format.

[0113] In some embodiments, calculating the list of silent points based on the broadcast news audio in step S103 may include, but is not limited to, the following steps:

[0114] Perform spectral analysis on the audio of broadcast news to obtain the amplitude corresponding to each time point;

[0115] Calculate the volume based on the amplitude;

[0116] The duration of the pause is defined as the time period during which the volume is lower than the first volume threshold.

[0117] When the pause duration is less than the pause duration threshold, the time period is used as the silence point;

[0118] Multiple mute points are combined to obtain a list of mute points.

[0119] In some embodiments, spectral analysis can be performed on the broadcast news audio to obtain the amplitude corresponding to each time point. It is understood that this can be achieved by manually sampling several broadcast news videos, converting them into broadcast news audio, and then performing spectral analysis to obtain the spectral analysis results for each broadcast news audio clip. Then, based on the amplitude, the volume is calculated using the volume calculation formula, where the volume calculation formula is: In the formula, D represents volume and F represents amplitude. The time period during which the volume (dB) is less than a first volume threshold is then used as the pause duration. For example, the first volume threshold can be set to -50dB. Figure 2 As shown, the area within the red box represents the region where the volume (dB) is less than the first volume threshold. The time period meeting this requirement is designated as the pause duration. When the pause duration is less than the pause duration threshold, this time period is designated as the mute point. For example, the pause duration threshold can be set to 0.2 seconds. When the pause duration is less than 0.2 seconds, the corresponding time period is designated as the mute point, as shown below. Figure 3 As shown, this includes silence points of varying durations. Finally, multiple silence points are combined to obtain a list of silence points. Furthermore, silence points can be calculated using audio waveforms and spectrograms, combined with the duration range or average duration of the statement.

[0120] In some embodiments, step S104, calculating the start and end time list of natural speech based on the list of silence points, may include, but is not limited to, the following steps:

[0121] Select two adjacent mute points from the list of mute points to form a mute point combination;

[0122] Extract the end time of the mute point that appears earlier in the time sequence from the mute point combination;

[0123] Extract the start time of the mute point that appears later in the time sequence from the mute point combination;

[0124] Calculate the start and end times of natural speech based on the end and start times;

[0125] By combining multiple natural speech start and end times, a list of natural speech start and end times is obtained.

[0126] In some embodiments, two adjacent silence points can be selected from the silence point list as a silence point combination, meaning the silence point combination contains two adjacent silence points. Then, the end time of the silence point that appears earlier in the time sequence is extracted from the silence point combination, and the start time of the silence point that appears later in the time sequence is extracted from the silence point combination. Finally, based on the end time and start time, the start and end times of the natural speech are calculated. The natural speech start and end times are as follows: Figure 4 As shown in the figure, the graph contains sentences of different durations. Finally, the start and end times of multiple natural language sentences are combined to obtain a list of natural language sentence start and end times, which is then displayed in a timeline format as follows: Figure 5 As shown. It can be understood that, based on the list of silence points, the difference between the audio and the silence points can be calculated, which is the set of differences between the start time of a certain silence point and the end time of the previous silence point (i.e., the list of start and end times of natural speech).

[0127] In some embodiments, step S105, calculating the keyframe timeline based on the broadcast news video, may include, but is not limited to, the following steps:

[0128] The broadcast news video is decomposed into frames to obtain several video frames;

[0129] Set the subtitle position range;

[0130] Extract subtitle images from video frames based on the subtitle location range;

[0131] The subtitle image is binarized to obtain a binarized image;

[0132] Calculate the optical flow change of each pixel in the binarized image based on the optical flow function;

[0133] Calculate the keyframe sequence based on optical flow changes;

[0134] Calculate the keyframe timeline based on the keyframe sequence and video frame rate.

[0135] In some embodiments, video can be processed frame by frame using computer vision techniques to remove irrelevant color blocks and determine frame similarity in order to filter out keyframes. For example, broadcast news videos can be first decomposed into several video frames, which are then saved as an image sequence. Then, a subtitle position range is set, and subtitle images are extracted from the video frames based on this range. The extracted subtitle images are shown below. Figure 6 As shown. Then, the subtitle image is binarized to obtain the binarized image as shown. Figure 7 As shown. Then, based on the optical flow function, the optical flow change of each pixel in the binarized image is calculated, and a keyframe sequence is calculated based on the optical flow change. For example, the Lucas-Kanade algorithm can be used to calculate the optical flow change, and the sequence number of each keyframe can be recorded and saved as a keyframe sequence. The expression for the optical flow function is: I x (k,l)u+I y (k,l)v+I t (k,l)=0, (k,l∈W, where I x Let I be the partial derivative of the image brightness with respect to the x-axis, representing the gradient of the image along the x-axis. y Let I be the partial derivative of the image brightness with respect to the y-axis, representing the gradient of the image along the y-axis. t Let I be the partial derivative of image brightness with respect to time t, representing the rate of change of image brightness over time. (k,l) is the position of the pixel, u is the optical flow (velocity) of the tracked pixels in the image sequence along the x-axis, v is the optical flow (velocity) of the tracked pixels in the image sequence along the y-axis, and W is the subtitle image. It can be understood that in this embodiment, when the subtitle remains unchanged, I... t The values ​​u and v remain constant, changing only when the subtitles change. In a binarized subtitle image (i.e., a binarized image), the optical flow is constant when the subtitles remain unchanged. Therefore, using optical flow to acquire keyframes is effective and computationally efficient. Finally, the keyframe timeline is calculated based on the keyframe sequence and the video frame rate. For example, the video frame rate can be set to 25fps, and the keyframe timeline can be calculated using the sequence numbers of the preceding and following keyframes in the keyframe sequence.

[0136] In some embodiments, step S106, generating an initial corpus dataset based on the natural speech start and end time list and the keyframe timeline, may include, but is not limited to, the following steps:

[0137] Extract the first sentence segment from the list of natural speech start and end times;

[0138] extract a second sentence segment from the key frame timeline, the sequence number of the second sentence segment being the same as that of the first sentence segment;

[0139] calculate a start time difference according to the start time of the first sentence segment and the start time of the second sentence segment;

[0140] calculate an end time difference according to the end time of the first sentence segment and the end time of the second sentence segment;

[0141] if the start time difference is greater than a time difference threshold or the end time difference is greater than the time difference threshold, delete the first sentence segment from the natural speech start and end time list to obtain a corrected timeline;

[0142] extract the first caption corresponding to each key frame in the corrected timeline from the broadcast news video;

[0143] generate a first time-text relationship table according to the corrected timeline and the first caption;

[0144] cut the broadcast news audio according to a preset cutting duration to obtain a sub-audio;

[0145] cut the first time-text relationship table according to the preset cutting duration to obtain a sub-sentence;

[0146] combine the sub-audio and the sub-sentence to obtain a text-audio combination;

[0147] generate an initial corpus dataset according to a plurality of text-audio combinations.

[0148] In some embodiments, the first sentence segment can be extracted from the natural speech start and end time list, and the second sentence segment can be extracted from the key frame timeline, wherein the sequence number of the second sentence segment is the same as that of the first sentence segment. Then, the start time difference can be calculated according to the start time of the first sentence segment and the start time of the second sentence segment, and the end time difference can be calculated according to the end time of the first sentence segment and the end time of the second sentence segment. If the start time difference is greater than a time difference threshold or the end time difference is greater than the time difference threshold, the first sentence segment can be deleted from the natural speech start and end time list to obtain a corrected timeline. For example, the time difference threshold can be set to 400 milliseconds, and if the start time difference is greater than 400 milliseconds or the end time difference is greater than 400 milliseconds, the first sentence segment can be deleted from the natural speech start and end time list to obtain a corrected timeline. As shown in FIG. 3, Figure 8 extract the first caption corresponding to each key frame in the corrected timeline from the broadcast news video, and generate a first time-text relationship table according to the corrected timeline and the first caption as shown in FIG. 4. Figure 9The broadcast news audio is cut according to the preset cutting time length to obtain a sub-audio, and the first time text relationship table is cut according to the preset cutting time length to obtain a sub-sentence. Finally, the sub-audio and the sub-sentence are combined to obtain a text-audio combination. According to a plurality of text-audio combinations, an initial corpus dataset is generated. The text-audio combination includes an audio file name, text, and an audio time length, as shown in Figure 10 The preset cutting time length can be set in the range of 1-12 seconds. The broadcast news audio and the first time text relationship table are cut at the same cutting time length to produce a one-to-one corresponding text-audio combination, and then the initial corpus dataset is obtained.

[0149] In some embodiments, in step S107, constructing the dialect speech recognition model corpus according to the initial corpus dataset can include but is not limited to the following steps:

[0150] Obtaining a to-be-recognized corpus video;

[0151] Inputting the initial corpus dataset into a preset speech recognition model to train the preset speech recognition model to obtain a dialect speech recognition model;

[0152] Extracting a to-be-recognized audio and a plurality of second subtitles from the to-be-recognized corpus video;

[0153] According to the to-be-recognized audio, calculating a to-be-recognized corpus speech start and end time list;

[0154] Inputting the to-be-recognized audio into the dialect speech recognition model to obtain a dialect speech recognition result;

[0155] According to the to-be-recognized corpus speech start and end time list and the dialect speech recognition result, generating a second time text relationship table;

[0156] According to the second time text relationship table and the second subtitles, calculating a target corpus;

[0157] Adding the target corpus to the initial corpus dataset to obtain a dialect speech recognition model corpus.

[0158] In some embodiments, after a basic dialect speech recognition model is trained by a small corpus (30-100 hours), the basic dialect speech recognition model is used to further process the corpus to improve the quality of the corpus, and the cycle is repeated to continuously improve the accuracy of the model. First, the to-be-recognized corpus video is obtained, the initial corpus dataset is input into the preset speech recognition model to train the preset speech recognition model, and a dialect speech recognition model is obtained. Then, the to-be-recognized audio and a plurality of second subtitles are extracted from the to-be-recognized corpus video, the to-be-recognized audio is input into the dialect speech recognition model, and a dialect speech recognition result is obtained. It can be understood that the to-be-recognized audio is converted into text by the basic dialect speech recognition model, without considering the recognition accuracy, only considering whether the number of recognized characters is consistent with the actual number of characters in the audio. If the model can recognize 1 audio corresponding to 1 character, the recognition is correct. Then, the second time-text relationship table is generated according to the to-be-recognized corpus speech start and end time list and the dialect speech recognition result, and the target corpus is calculated according to the second time-text relationship table and the second subtitles. Exemplarily, in the calculation process of the target corpus, the second subtitles can be obtained by optical character recognition (OCR), and speech recognition technology (Automatic Speech Recognition, ASR) can be used for speech recognition. After the to-be-recognized audio is converted into text by the dialect speech recognition model, the length of the text characters in the second subtitles is compared with the length of the corresponding text characters in the dialect speech recognition result to correct whether the time axis of the subtitle time is correct. If the length of the text characters in the second subtitles is less than the length of the corresponding text characters in the dialect speech recognition result, it means that the syllable in the speech is more than the text on the image, the time axis is too long, and the corpus is "more words". The corpus is discarded and is not used for the speech recognition model training in this round; if the length of the text characters in the second subtitles is greater than the length of the corresponding text characters in the dialect speech recognition result, it means that the syllable in the speech is less than the text on the image, the time axis is too short, and the corpus is "few words". The corpus is discarded and is not used for the speech recognition model training in this round; if the length of the text characters in the second subtitles is equal to the length of the corresponding text characters in the dialect speech recognition result, it means that the time axis of the corpus is correct, and the corpus can be reserved as a high-precision target corpus. The comparison of the lengths of the text characters is shown in the following table. Figure 11 The target corpus is finally added to the initial corpus dataset to obtain a dialect speech recognition model corpus. More, the target corpus can be cut according to a preset cutting time length and then added to the initial corpus dataset to improve the diversity of the initial corpus dataset.

[0159] In some embodiments, the overall process of building a corpus is as follows Figure 12As shown, the broadcast news video can be acquired first, the audio track extraction is performed, the audio file is made, then the mute point identification is performed, the mute point list is calculated, the first time axis (i.e. the natural speech start and end time list) is calculated according to the mute point list, the image processing is performed, the second time axis (i.e. the key frame time axis) is calculated, the time axis correction is performed by using the first time axis and the second time axis, the corrected time axis is obtained, the subtitle is extracted from the image, the first time-text relationship table is generated by combining the corrected time axis and the subtitle, the initial corpus dataset is generated, then the model training is performed by using the initial corpus dataset, the basic dialect speech recognition model is obtained, the speech recognition is performed by using the basic dialect speech recognition model, the second time-text relationship table is generated by combining the speech recognition result, the first time-text relationship table and the second time-text relationship table are combined, and the target corpus is obtained by using the ASR and OCR compared text character length, and finally the target corpus is added to the initial corpus dataset to obtain the training corpus library (i.e. the dialect speech recognition model corpus library). More, the dialect speech recognition model can be updated by using the dialect speech recognition model corpus library to improve the model precision.

[0160] The beneficial effects of implementing the embodiments of the present application include that the embodiments of the present application first acquire the broadcast news video, perform the audio track extraction on the broadcast news video to obtain the broadcast news audio, then calculate the mute point list according to the broadcast news audio, calculate the natural speech start and end time list according to the mute point list, calculate the key frame time axis according to the broadcast news video, and generate the initial corpus dataset according to the natural speech start and end time list and the key frame time axis, and finally construct the dialect speech recognition model corpus library according to the initial corpus dataset, thereby realizing the corpus library construction, improving the efficiency and accuracy, and reducing the corpus library management cost. Meanwhile, the embodiments automatically, efficiently and at low cost process the existing dialect broadcast news materials into dialect corpus data through preprocessing, train the basic model by using the small-scale corpus, and greatly improve the corpus accuracy by using optical character recognition and mute detection, thereby forming the high-quality and high-precision corpus library dataset. These datasets can be used for training of the dialect speech recognition and the dialect speech broadcast model, and finally applied to the dialect recognition, the dialect broadcast and other intelligent interaction scenes. In addition, the embodiments can realize the automatic dialect corpus preprocessing, and compared with the manual annotation mode, the speed is faster, the efficiency is higher, and the cost is lower. In the optimization process of the algorithm, the key algorithm optimization is performed for the output corpus content, the optical flow method and the binary method are adopted, the processing efficiency is further improved while reducing the algorithm consumption, and the processing cost is reduced.

[0161] As shown in Figure 13 The embodiments of the present application also provide a dialect speech recognition model corpus library construction device based on broadcast news, which comprises:

[0162] The first module 801 is configured to acquire a broadcast news video, wherein the broadcast news video has a first subtitle.

[0163] The second module 802 is configured to perform audio track extraction on the broadcast news video to obtain broadcast news audio.

[0164] The third module 803 is configured to calculate a mute point list according to the broadcast news audio.

[0165] The fourth module 804 is configured to calculate a natural speech start and end time list according to the mute point list.

[0166] The fifth module 805 is configured to calculate a key frame time axis according to the broadcast news video.

[0167] The sixth module 806 is configured to generate an initial corpus dataset according to the natural speech start and end time list and the key frame time axis.

[0168] The seventh module 807 is configured to construct a dialect speech recognition model corpus according to the initial corpus dataset.

[0169] The content in the above method embodiments is applicable to the device embodiments, the device embodiments specifically implement the same functions as the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0170] As shown in Figure 14 , the embodiment of the application further provides a computer device, which comprises:

[0171] at least one processor 901;

[0172] at least one memory 902 configured to store at least one program;

[0173] When the at least one program is executed by the at least one processor, the at least one processor implements the method shown in Figure 1 .

[0174] The content in the above method embodiments is applicable to the device embodiments, the device embodiments specifically implement the same functions as the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0175] The preferred embodiments of the application are described above with reference to the accompanying drawings, and the scope of the right of the application is not limited by this. Any modification, equivalent replacement and improvement made by those skilled in the art without departing from the scope and essence of the application should be within the scope of the right of the application.

Claims

1. A dialect speech recognition model corpus construction method based on broadcast news, characterized in that, The method comprises the following steps: obtaining a broadcast news video having a first subtitle; extracting an audio track from the broadcast news video to obtain a broadcast news audio; calculating a mute point list according to the broadcast news audio; calculating a natural speech start and end time list according to the mute point list; calculating a key frame time axis according to the broadcast news video; generating an initial corpus dataset according to the natural speech start and end time list and the key frame time axis; constructing a dialect speech recognition model corpus according to the initial corpus dataset; wherein the calculation of the key frame time axis according to the broadcast news video comprises: frame decomposition of the broadcast news video to obtain a plurality of video frames; setting a subtitle position range; extracting a subtitle image from the video frames according to the subtitle position range; binaryzation of the subtitle image to obtain a binary image; calculating the optical flow change of each pixel point in the binary image according to an optical flow function; calculating a key frame sequence according to the optical flow change; calculating the key frame time axis according to the key frame sequence and a video frame rate; the generation of the initial corpus dataset according to the natural speech start and end time list and the key frame time axis comprises: extracting a first sentence segment from the natural speech start and end time list; extracting a second sentence segment from the key frame time axis, the serial number of the second sentence segment being the same as that of the first sentence segment; calculating a start time difference according to the start time of the first sentence segment and the start time of the second sentence segment; calculating an end time difference according to the end time of the first sentence segment and the end time of the second sentence segment; if the start time difference is greater than a time difference threshold or the end time difference is greater than the time difference threshold, deleting the first sentence segment from the natural speech start and end time list to obtain a corrected time axis; extracting a first subtitle corresponding to each key frame in the corrected time axis from the broadcast news video; generating a first time-text relationship table according to the corrected time axis and the first subtitle; cutting the broadcast news audio according to a preset cutting duration to obtain a sub-audio; cutting the first time-text relationship table according to the preset cutting duration to obtain a sub-sentence; combining the sub-audio and the sub-sentence to obtain a text-audio combination; generating the initial corpus dataset according to a plurality of the text-audio combinations.

2. The method of claim 1, wherein, the calculation of the mute point list according to the broadcast news audio comprises: spectrum analysis of the broadcast news audio to obtain an amplitude corresponding to each time point; calculating a volume according to the amplitude; regarding a time segment with a volume less than a first volume threshold as a sentence breaking pause duration; regarding the time segment as a mute point when the sentence breaking pause duration is less than a pause duration threshold; combining a plurality of the mute points to obtain the mute point list.

3. The method of claim 1, wherein, the calculation of the natural speech start and end time list according to the mute point list comprises: selecting two adjacent mute points from the mute point list as a mute point combination; extract an end time of a silence point in time sequence in front from the silence point combination; extract a start time of a silence point in time sequence in back from the silence point combination; calculate natural speech start-stop time according to the end time and the start time; combine a plurality of the natural speech start-stop time to obtain the natural speech start-stop time list.

4. The method of claim 1, wherein, The constructing a dialect speech recognition model corpus according to the initial corpus dataset comprises: acquiring a corpus video to be recognized; inputting the initial corpus dataset into a preset speech recognition model to enable the preset speech recognition model to be trained to obtain a dialect speech recognition model; extracting an audio to be recognized and a plurality of second subtitles from the corpus video to be recognized; calculating a corpus speech start-stop time list to be recognized according to the audio to be recognized; inputting the audio to be recognized into the dialect speech recognition model to obtain a dialect speech recognition result; generating a second time-text relationship table according to the corpus speech start-stop time list to be recognized and the dialect speech recognition result; calculating a target corpus according to the second time-text relationship table and the second subtitles; adding the target corpus to the initial corpus dataset to obtain the dialect speech recognition model corpus.

5. The method of claim 2, wherein, The calculating a volume according to the amplitude comprises: calculating a volume according to the amplitude by a volume calculation formula, the volume calculation formula being: ； In the formula, is the volume, is the amplitude.

6. The method of claim 1, wherein, The expression of the optical flow function is: ； wherein is the partial derivative of the image luminance with respect to the spatial coordinate axis, is the partial derivative of the image luminance with respect to the spatial coordinate axis, is the partial derivative of the image luminance with respect to time, is the position of the pixel point, is the optical flow of the pixel point in the axis direction, is the optical flow of the pixel point in the axis direction, is the caption image.

7. A dialectal speech recognition model corpus construction apparatus based on broadcast news, characterized by, comprises: A first module is configured to acquire a broadcast news video, the broadcast news video having a first subtitle. A second module is configured to extract an audio track of the broadcast news video to obtain a broadcast news audio. A third module is configured to calculate a silence point list according to the broadcast news audio. A fourth module is configured to calculate a natural speech start-stop time list according to the silence point list. A fifth module is configured to calculate a key frame time axis according to the broadcast news video. A sixth module is configured to generate an initial corpus dataset according to the natural speech start-stop time list and the key frame time axis. A seventh module is configured to construct a dialect speech recognition model corpus according to the initial corpus dataset. The calculating a key frame time axis according to the broadcast news video comprises: frame decomposing the broadcast news video to obtain a plurality of video frames; setting a subtitle position range; extracting a subtitle image from the video frames according to the subtitle position range; binarizing the subtitle image to obtain a binarized image; calculating an optical flow change of each pixel point in the binarized image according to an optical flow function; calculating a key frame sequence according to the optical flow change; calculating the key frame time axis according to the key frame sequence and a video frame rate; The generating an initial corpus dataset according to the natural speech start-stop time list and the key frame time axis comprises: extracting a first sentence segment from the natural speech start-stop time list; extracting a second sentence segment from the key frame time axis, the serial number of the second sentence segment being the same as that of the first sentence segment; calculating a start time difference according to a start time of the first sentence segment and a start time of the second sentence segment; calculating an end time difference according to an end time of the first sentence segment and an end time of the second sentence segment; if the start time difference is greater than a time difference threshold or the end time difference is greater than the time difference threshold, deleting the first sentence segment from the natural speech start and end time list to obtain a corrected time axis; extracting a first caption corresponding to each key frame in the corrected time axis from the broadcast news video; generating a first time-text relationship table according to the corrected time axis and the first caption; cutting the broadcast news audio according to a preset cutting duration to obtain a sub-audio; cutting the first time-text relationship table according to the preset cutting duration to obtain a sub-sentence; combining the sub-audio and the sub-sentence to obtain a text-audio combination; generating the initial corpus dataset according to a plurality of the text-audio combinations.

8. A computer apparatus, characterized by comprise: at least one processor; at least one memory for storing at least one program; when the at least one program is executed by the at least one processor, the at least one processor implements the method of any one of claims 1-6.

Images