Video speech recognition method, device, equipment and storage medium

Abstract

The present application proposes a video speech recognition method, device, equipment and storage medium, which relates to the field of speech recognition technology. The method includes: processing a video to obtain multiple audio sub-segments and a sequence of image frames corresponding to each audio sub-segment; performing text recognition on the image frame sequence to obtain multiple text results, and processing the multiple text results to obtain multiple keywords; processing each audio sub-segment through a speech recognition model to obtain multiple candidate speech recognition results; determining the target text recognition result of each audio sub-segment based on the multiple candidate speech recognition results and multiple keywords, and obtaining the speech recognition result of the video based on the target text recognition result of each audio sub-segment. Thus, the text recognition result in the image of the video is used to assist the video speech recognition, thereby improving the accuracy of the video speech recognition.

Description

Technical Field

[0001] The present application relates to the field of speech recognition technology, and in particular to a video speech recognition method, apparatus, device and storage medium. Background Art

[0002] At present, video data has the advantages of strong entertainment, rich content, and high user stickiness, and has become one of the main traffic sources for Internet users in recent years. Automatic Speech Recognition (ASR) plays an important role in video content review, video recommendation, etc. by recognizing the spoken content in the video as the corresponding text.

[0003] However, in actual video scenarios, the voice data to be recognized may contain a variety of complex interference factors such as accents, background music and noise, which seriously reduce the effect of video speech recognition.

[0004] In related technologies, manual annotation is usually used to obtain audio data and corresponding labels of a large amount of video scenes, and then these annotated data are used to optimize the model training. However, compared with the annotation of image, text and other data, the annotation cost is often higher because the annotation of audio data requires manual listening at least once. Summary of the invention

[0005] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0006] The present application proposes a video speech recognition method, apparatus, device and storage medium, which assist video speech recognition through text recognition results in video images to improve the accuracy of video speech recognition, and is used to solve the technical problems of inaccurate video speech recognition and relatively high cost in the prior art.

[0007] The first embodiment of the present application provides a video speech recognition method, including:

[0008] Processing the video to obtain a plurality of audio sub-segments and an image frame sequence corresponding to each of the audio sub-segments;

[0009] Performing text recognition on the image frame sequence to obtain a plurality of text results, and processing the plurality of text results to obtain a plurality of keywords;

[0010] Processing each of the audio sub-segments through a speech recognition model to obtain a plurality of candidate speech recognition results;

[0011] A target text recognition result for each of the audio sub-segments is determined according to the multiple candidate speech recognition results and the multiple keywords, and a speech recognition result for the video is obtained according to the target text recognition result for each of the audio sub-segments.

[0012] The video speech recognition method of the embodiment of the present application processes the video to obtain multiple audio sub-segments and a sequence of image frames corresponding to each audio sub-segment; performs text recognition on the image frame sequence to obtain multiple text results, and processes the multiple text results to obtain multiple keywords; processes each audio sub-segment through a speech recognition model to obtain multiple candidate speech recognition results; determines the target text recognition result of each audio sub-segment based on the multiple candidate speech recognition results and multiple keywords, and obtains the speech recognition result of the video based on the target text recognition result of each audio sub-segment. Thus, the text recognition result in the video image assists the video speech recognition, thereby improving the accuracy of the video speech recognition.

[0013] The second aspect of the present application provides a video speech recognition device, including:

[0014] A first acquisition module is used to process the video to acquire a plurality of audio sub-segments and an image frame sequence corresponding to each of the audio sub-segments;

[0015] A recognition module, used for performing text recognition on the image frame sequence to obtain multiple text results;

[0016] A second acquisition module, used to process the multiple text results to obtain multiple keywords;

[0017] A processing module, used for processing each of the audio sub-segments through a speech recognition model to obtain a plurality of candidate speech recognition results;

[0018] A determination module, configured to determine a target text recognition result for each of the audio sub-segments according to the multiple candidate speech recognition results and the multiple keywords;

[0019] The third acquisition module is used to obtain the speech recognition result of the video according to the target text recognition result of each of the audio sub-segments.

[0020] The video speech recognition device of the embodiment of the present application processes the video to obtain multiple audio sub-segments and a sequence of image frames corresponding to each audio sub-segment; performs text recognition on the image frame sequence to obtain multiple text results, and processes the multiple text results to obtain multiple keywords; processes each audio sub-segment through a speech recognition model to obtain multiple candidate speech recognition results; determines the target text recognition result of each audio sub-segment based on the multiple candidate speech recognition results and the multiple keywords, and obtains the speech recognition result of the video based on the target text recognition result of each audio sub-segment. Thus, the text recognition result in the video image assists the video speech recognition, thereby improving the accuracy of the video speech recognition.

[0021] The third aspect embodiment of the present application proposes a server, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the video speech recognition method proposed in the first aspect embodiment of the present application is implemented.

[0022] The fourth aspect embodiment of the present application proposes a non-temporary computer-readable storage medium storing a computer program, which, when executed by a processor, implements the video speech recognition method proposed in the first aspect embodiment of the present application.

[0023] The fifth aspect embodiment of the present application proposes a computer program product. When the instruction processor in the computer program product executes, the video speech recognition method proposed in the first aspect embodiment of the present application is executed.

[0024] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0026] Figure 1 A flowchart of a video speech recognition method provided in Example 1 of the present application;

[0027] Figure 2 A flowchart of a video speech recognition method provided in Embodiment 2 of the present application;

[0028] Figure 3 This is an example diagram of audio silence detection in an embodiment of the present application;

[0029] Figure 4 This is an example diagram of text recognition performed by the ORC algorithm of an embodiment of the present application;

[0030] Figure 5 A schematic diagram of the structure of a video speech recognition device provided in Embodiment 8 of the present application;

[0031] Figure 6 A schematic diagram of the structure of a video speech recognition device provided in Embodiment 9 of the present application;

[0032] Figure 7 A block diagram of an exemplary server suitable for implementing the embodiments of the present application is shown. DETAILED DESCRIPTION

[0033] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0034] In practical applications, automatic speech recognition is used to identify the spoken content in a video as the corresponding text, which is quite common in the fields of video content review and video recommendation. However, manual annotation is usually used to obtain audio data and corresponding labels of a large number of video scenes, and then these annotated data are used to optimize the model training. The annotation cost is relatively high, and video recognition is difficult and inefficient.

[0035] In response to the above problems, the present application proposes a video speech recognition method, which processes a video to obtain multiple audio sub-segments and an image frame sequence corresponding to each audio sub-segment; performs text recognition on the image frame sequence to obtain multiple text results, and processes the multiple text results to obtain multiple keywords; processes each audio sub-segment through a speech recognition model to obtain multiple candidate speech recognition results; determines a target text recognition result for each audio sub-segment based on the multiple candidate speech recognition results and the multiple keywords, and obtains the speech recognition result of the video based on the target text recognition result of each audio sub-segment.

[0036] Therefore, the text recognition results in the video image assist the video speech recognition, thereby improving the accuracy of the video speech recognition.

[0037] The following describes the video speech recognition method, apparatus, device and storage medium of the embodiments of the present application with reference to the accompanying drawings.

[0038] Figure 1 A flowchart of the video speech recognition method provided in Example 1 of the present application.

[0039] The embodiment of the present application takes the video and speech recognition method being configured in a video and speech recognition device as an example. The video and speech recognition device can be applied to any device so that the device can perform the video and speech recognition function.

[0040] like Figure 1 As shown, the video speech recognition method may include the following steps:

[0041] Step 101: Process the video to obtain multiple audio sub-segments and an image frame sequence corresponding to each audio sub-segment.

[0042] In the embodiments of the present application, the video can be understood as the video to be subjected to voice recognition, which can be obtained according to the application scenario, such as the video uploaded by the relevant terminal, the video shot in real time, etc., and the setting is selected according to the application scenario.

[0043] In the embodiment of the present application, it can be understood that the video can be a video with audio data continuously for a period of time, or it can be audio data of an interval time period. The video can be processed in a corresponding manner according to different scenes to obtain multiple audio sub-segments and an image frame sequence corresponding to each audio sub-segment. An example is described as follows:

[0044] In a first example, a video is processed to obtain audio data and image data, silence detection is performed on the audio data to obtain multiple audio sub-segments; and an image frame sequence corresponding to each audio sub-segment is obtained from the image data.

[0045] In the second example, the video is processed to obtain audio data and image data, the audio data is cropped according to a preset time period to obtain multiple audio sub-segments; and an image frame sequence corresponding to each audio sub-segment is obtained from the image data.

[0046] In the embodiment of the present application, the image frame sequence corresponding to each audio sub-segment can be understood as a collection of multiple images corresponding to a segment of audio.

[0047] Step 102: perform text recognition on the image frame sequence to obtain a plurality of text results, and process the plurality of text results to obtain a plurality of keywords.

[0048] In the embodiment of the present application, performing text recognition on an image frame sequence and obtaining multiple text results can be understood as performing text recognition on each frame image in the image frame sequence and obtaining corresponding text results. There are many ways to perform text recognition on an image frame sequence and obtain multiple text results, and examples are as follows:

[0049] In the first example, optical character recognition is performed on each image frame in the image frame series to obtain multiple text results.

[0050] In the second example, each frame of an image frame series is processed by a trained neural network to obtain multiple text results; wherein the trained neural network has the ability to perform text recognition on images through training samples.

[0051] In the embodiment of the present application, there are many ways to process multiple text results and obtain multiple keywords, and examples are as follows:

[0052] In the first example, each text result is segmented to obtain multiple segmented words, and the multiple segmented words are filtered to obtain multiple keywords.

[0053] In the second example, each text result is matched with a text template to obtain multiple keywords, wherein the text model is preset.

[0054] Step 103: Process each audio sub-segment through a speech recognition model to obtain multiple candidate speech recognition results.

[0055] Step 104, determining a target text recognition result for each audio sub-segment based on the multiple candidate speech recognition results and the multiple keywords, and obtaining a speech recognition result for the video based on the target text recognition result for each audio sub-segment.

[0056] In an embodiment of the present application, the speech recognition model is a model that is pre-trained based on samples and has the ability to recognize speech. Each audio sub-segment is input into the speech recognition model for processing, and multiple candidate speech recognition results can be obtained. For example, audio sub-segment A is input into the speech recognition model for processing, and candidate speech recognition results "how are you", "what's your number" and "how are you" are obtained.

[0057] Furthermore, there are many ways to determine the target text recognition result of each audio sub-segment based on multiple candidate speech recognition results and multiple keywords, and examples are given below.

[0058] As an example, the recognition probability of each candidate speech recognition text is obtained, the statistical probability and the first coefficient of each candidate speech recognition text are obtained, the contribution probability and the second coefficient of multiple keywords corresponding to each candidate speech recognition text are obtained, calculations are performed based on the recognition probability, statistical probability, first coefficient, contribution probability and second coefficient of each candidate speech recognition text, the accuracy of each candidate speech recognition text is obtained, and the target text recognition result of each audio sub-segment is determined from multiple candidate speech recognition results based on the accuracy of each candidate speech recognition text.

[0059] As another example, the accuracy and corresponding coefficient of each candidate speech recognition text and the coefficients of multiple keywords are obtained, and weighted summation is performed. The target text recognition result of each audio sub-segment is determined from the multiple candidate speech recognition results based on the weighted summation result.

[0060] Furthermore, multiple target text recognition results of multiple audio sub-segments are combined and spliced ​​to obtain the speech recognition result of the video.

[0061] The video speech recognition method of the embodiment of the present application processes the video to obtain multiple audio sub-segments and a sequence of image frames corresponding to each audio sub-segment; performs text recognition on the image frame sequence to obtain multiple text results, and processes the multiple text results to obtain multiple keywords; processes each audio sub-segment through a speech recognition model to obtain multiple candidate speech recognition results; determines the target text recognition result of each audio sub-segment based on the multiple candidate speech recognition results and multiple keywords, and obtains the speech recognition result of the video based on the target text recognition result of each audio sub-segment. Thus, the text recognition result in the video image assists the video speech recognition, thereby improving the accuracy of the video speech recognition.

[0062] Figure 2 A flowchart of the video speech recognition method provided in Example 2 of the present application.

[0063] like Figure 2 As shown, the video speech recognition method may include the following steps:

[0064] Step 201, process the video, obtain audio data and image data, perform silence detection on the audio data, obtain multiple audio sub-segments, and obtain an image frame sequence corresponding to each audio sub-segment from the image data.

[0065] In an embodiment of the present application, the video may be sampled and processed, for example, the video may be sampled and processed according to an image sampling frequency to obtain image data, and the video may be sampled and processed according to an audio signal sampling frequency to obtain audio data.

[0066] For example, given a video data V = {P, S}, where P = {I 1 ,I 2 ,...,I N} is the image data in the video data, I n is the nth frame image in the image data, n∈[1,N], and the image sampling frequency is f P , S = {x 1 ,x 2 ,...,x M} is the audio data in the video data, x mis the mth audio signal in the audio data, m∈[1,M], and the sampling frequency of the audio signal is f S In this application, for example, the sampling frequency of the image is f P =30Hz, the sampling frequency of the audio signal is f S =16kHz.

[0067] Furthermore, silence detection is performed on the audio data to obtain multiple audio sub-segments. It can be understood that usually in the audio of a video, there are mostly silent or non-human voice parts. In order to improve the accuracy and efficiency of speech recognition, the present application uses a silence detection algorithm to detect and delete the silent parts in the audio data, cut a longer audio into multiple sub-segments, and record the start and end time of each speech segment.

[0068] For example, Figure 3 As shown, in order to prevent the background music and other factors from causing the silence detection algorithm to be unable to completely truncate the audio and thus obtain a longer sub-segment, a fixed duration threshold T can be set to forcibly cut the audio with a duration greater than the threshold T into multiple segments of duration T. In the present application, for example, the fixed duration threshold T = 10s.

[0069] Specifically, the sequence of audio sub-segments obtained after silence detection for a given audio data segment is denoted as X 1 ,X 2 ,...,X K , where each audio sub-segment X k The start and end times are t sk and t ek , k∈[1,K]. In the present application, for example, WebRTC (Web Real-Time Communication) is used to perform audio silence detection.

[0070] In an embodiment of the present application, there are many ways to obtain an image frame sequence corresponding to each audio sub-segment from image data. As an example scenario, the start time and end time corresponding to each audio sub-segment are obtained, the start frame image is determined according to the ratio of the start time and the image sampling frequency, and the end frame image is determined according to the ratio of the end time to the image sampling frequency. The interval frame image is determined from the image data according to the start frame image and the end frame image, and the image is extracted from the interval frame image at a preset frequency to obtain an image frame sequence corresponding to each audio sub-segment.

[0071] Specifically, since the sampling frequencies of image data and audio data are inconsistent, and the actual image subtitles change frequency in the image stream is low, for each audio sub-segment X k , extract the corresponding image frame sequence Y from the image datak , where the start and end frame numbers of the image frame sequence are and in and In this application, for example, the image frame sequence Y k The sampling frequency is set to 1 Hz, that is, in the entire image data Frame and Between frames, every f P Frame sampling a picture, forming each audio sub-segment X k The corresponding image frame sequence Y k .

[0072] Step 202, performing optical character recognition on each image frame in the image frame series to obtain multiple text results, performing word segmentation processing on each text result to obtain multiple word segments, filtering the multiple word segments to obtain multiple keywords.

[0073] In the embodiment of the present application, the text in the image is recognized by an optical character recognition (OCR) algorithm. The present application adopts a general OCR algorithm based on a neural network to recognize the text in the image frame sequence Y. k Recognize the text in each frame of the image and obtain multiple text results L 1 ,L 2 ,...,L R ,like Figure 4 As shown, in this application, the commonly used CRNN (Convolutional Recurrent Neural Network) is used as the OCR algorithm to perform text recognition in images.

[0074] Furthermore, each text result is segmented to obtain the segmentation list W corresponding to each text result. 1 ,W 2 ,...,W R In this application, for example, L 1 The text result in the image obtained by the OCR algorithm is "Hello, here are the subtitles of the video." After word segmentation, we can get the word segmentation list W 1 for {"Hello", ","Here", "Yes", "Video", "Of", "Subtitles", "."}.

[0075] Among them, considering that there may be punctuation marks, single characters, numbers, letters, etc. in the word segmentation list, it is difficult to assist speech recognition well. Therefore, the punctuation marks and single characters in the word segmentation list can be removed. In addition, since the results of OCR recognition cannot be completely correct or special characters are replaced, after word segmentation, single-character results will appear. Therefore, it can effectively reduce the negative impact on video speech recognition caused by incomplete OCR information.

[0076] For example, taking "Hello, this is the subtitle of the video." as an example, in some videos, the subtitle may replace "视" with "s", and the recognition result is "Hello, this is the subtitle of s video". After word segmentation, the word segmentation list {"Hello", ",", "this", "is", "s", "video", "of", "the", "subtitle", "."} can be obtained. After removing single characters, "s video" will not be used for speech recognition, thus avoiding the misleading information caused by misspelled words.

[0077] Step 203, process each of the audio sub-fragments through a speech recognition model to obtain multiple candidate speech recognition results, obtain the recognition probability of each candidate speech recognition text, and obtain the statistical probability and the first coefficient of each candidate speech recognition text.

[0078] Step 204, obtain the contribution probability and the second coefficient of multiple keywords corresponding to each candidate speech recognition text, and perform calculations based on the recognition probability, statistical probability, first coefficient, contribution probability, and second coefficient of each candidate speech recognition text to obtain the accuracy rate of each candidate speech recognition text.

[0079] Step 205, determine the target text recognition result of each audio sub-fragment from the multiple candidate speech recognition results according to the accuracy rate of each candidate speech recognition text, and obtain the speech recognition result of the video according to the target text recognition result of each audio sub-fragment.

[0080] In the embodiments of the present application, obtaining the recognition probability of each candidate speech recognition text can be understood as the probability that each candidate speech recognition text is correctly recognized through the speech recognition model. The statistical probability and the first coefficient of each candidate speech recognition text can be understood as the naturalness of each candidate speech recognition text, that is, the probability that the expression conforms to the rules, etc., and the corresponding first coefficient can be set according to the application scenario. The contribution probability and the second coefficient of multiple keywords corresponding to each candidate speech recognition text can be understood as the contribution degree of the multiple keywords to the recognition result, and the corresponding second coefficient can be set according to the application scenario.

[0081] In the embodiments of the present application, speech recognition refers to recognizing the corresponding text result from an audio signal. The speech recognition model in the embodiments of the present application mainly includes an acoustic model f A and a language model fL There are two parts. The acoustic model is mainly used to extract features from the audio signal, and the language model is mainly used to make the speech recognition model output relatively natural language text results. Therefore, in the speech recognition process, by decoding, in the solution space, the audio features obtained by the acoustic model are combined with the language model to search for an optimal output text sequence, as follows:

[0082]

[0083] Where b is the candidate speech recognition text obtained by speech recognition, B is the text result solution space of speech recognition, and f is A (b|X k ) is the recognition probability of the acoustic model, f L (b) is the statistical probability of text b, f H (b,W 1 ,W 2 ,...,W R ) is the contribution probability of the keyword in text b, and the first coefficient α and the second coefficient β are used to configure the weights of the language model and the keyword respectively.

[0084] It should be noted that when there are no subtitles in the video, it can be directly processed by the speech recognition model.

[0085] The video speech recognition method of the embodiment of the present application processes a video to obtain audio data and image data, performs silence detection on the audio data, obtains multiple audio sub-segments, obtains an image frame sequence corresponding to each audio sub-segment from the image data, performs optical character recognition on each frame of the image frame series, obtains multiple text results, performs word segmentation processing on each text result to obtain multiple word segments, filters the multiple word segments, obtains multiple keywords, processes each of the audio sub-segments through a speech recognition model, obtains multiple candidate speech recognition results, obtains the recognition probability of each candidate speech recognition text, obtains the statistical probability and the first coefficient of each candidate speech recognition text, obtains the contribution probability and the second coefficient of multiple keywords corresponding to each candidate speech recognition text, calculates according to the recognition probability, statistical probability, first coefficient, contribution probability and second coefficient of each candidate speech recognition text, obtains the accuracy of each candidate speech recognition text, determines the target text recognition result of each audio sub-segment from the multiple candidate speech recognition results according to the accuracy of each candidate speech recognition text, and obtains the speech recognition result of the video according to the target text recognition result of each audio sub-segment. Therefore, in order to address the problems of high difficulty and high annotation cost of video speech recognition technology, the accuracy of speech recognition in video scenes is improved by using text information in images as an aid, avoiding the retraining and optimization of the speech recognition model. Only on the basis of the existing speech recognition model, text recognition technology is used to extract text information in the video as an aid, and ultimately the speech recognition accuracy of videos with subtitles is improved efficiently and at low cost.

[0086] In order to implement the above embodiment, the present application also proposes a video speech recognition device.

[0087] Figure 5 This is a schematic diagram of the structure of the video speech recognition device provided in Example 5 of the present application.

[0088] like Figure 5 As shown, the video speech recognition device 500 may include: a first acquisition module 510 , a recognition module 520 , a second acquisition module 530 , a processing module 540 , a determination module 550 and a third acquisition module 560 .

[0089] The first acquisition module 510 is used to process the video to acquire multiple audio sub-segments and an image frame sequence corresponding to each audio sub-segment.

[0090] The recognition module 520 is used to perform text recognition on the image frame sequence to obtain multiple text results.

[0091] The second acquisition module 530 is used to process the multiple text results to obtain multiple keywords.

[0092] The processing module 540 is used to process each audio sub-segment through a speech recognition model to obtain multiple candidate speech recognition results.

[0093] The determination module 550 is used to determine the target text recognition result of each audio sub-segment according to the multiple candidate speech recognition results and the multiple keywords.

[0094] The third acquisition module 560 is used to acquire the speech recognition result of the video according to the target text recognition result of each audio sub-segment.

[0095] Further, in a possible implementation of the embodiment of the present application, see Figure 6 ,exist Figure 5 Based on the illustrated embodiment, the first acquisition module 510 includes: a processing unit 511 , a detection unit 512 and an acquisition unit 513 .

[0096] The processing unit 511 is used to process the video and obtain audio data and image data.

[0097] The detection unit 512 is configured to perform silence detection on the audio data to obtain a plurality of audio sub-segments.

[0098] The acquisition unit 513 is configured to acquire, from the image data, an image frame sequence corresponding to each of the audio sub-segments.

[0099] Furthermore, in a possible implementation method of an embodiment of the present application, the acquisition unit 513 is specifically used to: obtain the start time and the end time corresponding to each of the audio sub-segments; determine the start frame image according to the ratio of the start time and the image sampling frequency, and determine the end frame image according to the ratio of the end time to the image sampling frequency; determine the interval frame image from the image data according to the start frame image and the end frame image, and extract the image from the interval frame image at a preset frequency to obtain an image frame sequence corresponding to each of the audio sub-segments.

[0100] Furthermore, in a possible implementation of the embodiment of the present application, the recognition module 520 is specifically configured to: perform optical character recognition on each image frame in the image frame series to obtain the multiple text results.

[0101] Furthermore, in a possible implementation of the embodiment of the present application, the second acquisition module 530 is specifically used to: perform word segmentation processing on each of the text results to obtain multiple word segments; and filter the multiple word segments to obtain the multiple keywords.

[0102] Furthermore, in a possible implementation method of the embodiment of the present application, the determination module 550 is specifically used to: obtain the recognition probability of each candidate speech recognition text; obtain the statistical probability and the first coefficient of each candidate speech recognition text; obtain the contribution probability and the second coefficient of the multiple keywords corresponding to each candidate speech recognition text; calculate according to the recognition probability, the statistical probability, the first coefficient, the contribution probability and the second coefficient of each candidate speech recognition text to obtain the accuracy of each candidate speech recognition text; determine the target text recognition result of each audio sub-segment from the multiple candidate speech recognition results according to the accuracy of each candidate speech recognition text.

[0103] It should be noted that the above explanation of the embodiment of the video speech recognition method is also applicable to the video speech recognition device of this embodiment, and will not be repeated here.

[0104] The video speech recognition device of the embodiment of the present application processes the video to obtain multiple audio sub-segments and a sequence of image frames corresponding to each audio sub-segment; performs text recognition on the image frame sequence to obtain multiple text results, and processes the multiple text results to obtain multiple keywords; processes each audio sub-segment through a speech recognition model to obtain multiple candidate speech recognition results; determines the target text recognition result of each audio sub-segment based on the multiple candidate speech recognition results and the multiple keywords, and obtains the speech recognition result of the video based on the target text recognition result of each audio sub-segment. Thus, the text recognition result in the video image assists the video speech recognition, thereby improving the accuracy of the video speech recognition.

[0105] In order to implement the above embodiments, the present application also proposes a server, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the video speech recognition method proposed in the above embodiments of the present application is implemented.

[0106] In order to implement the above embodiments, the present application also proposes a non-temporary computer-readable storage medium storing a computer program, which, when executed by a processor, implements the video speech recognition method proposed in the above embodiments of the present application.

[0107] In order to implement the above embodiments, the present application also proposes a computer program product. When the instruction processor in the computer program product is executed, the video speech recognition method proposed in the above embodiments of the present application is executed.

[0108] Figure 7 A block diagram of an exemplary server suitable for implementing the embodiments of the present application is shown. Figure 7The server 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0109] like Figure 7 As shown, the server 12 is in the form of a general purpose computing device. The components of the server 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that connects various system components (including the system memory 28 and the processing unit 16).

[0110] The bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor or a local bus using any of a variety of bus structures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnection (PCI) bus.

[0111] The server 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the server 12, including volatile and non-volatile media, removable and non-removable media.

[0112] The memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The server 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 7 not shown, usually called a "hard drive"). Although Figure 7Not shown in the figure, a disk drive for reading and writing a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing a removable non-volatile optical disk (e.g., a compact disc read only memory (CD-ROM), a digital versatile disc read only memory (DVD-ROM), or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the various embodiments of the present application.

[0113] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in the memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0114] The server 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), one or more devices that enable a user to interact with the server 12, and / or any device that enables the server 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the server 12 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 20. Figure 7 As shown, the network adapter 20 communicates with other modules of the server 12 via the bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the server 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0115] The processing unit 16 executes various functional applications and data processing by running the programs stored in the system memory 28, such as implementing the video speech recognition method mentioned in the above embodiment.

[0116] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0117] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0118] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0119] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0120] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0121] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0122] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0123] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A video speech recognition method, characterized in that: include: Processing the video to obtain a plurality of audio sub-segments and an image frame sequence corresponding to each of the audio sub-segments; Performing text recognition on the image frame sequence to obtain a plurality of text results, and processing the plurality of text results to obtain a plurality of keywords; Processing each of the audio sub-segments through a speech recognition model to obtain a plurality of candidate speech recognition results; Obtaining a recognition probability of each candidate speech recognition text, wherein the recognition probability represents a probability that each candidate speech recognition text is correctly recognized by a speech recognition model; Acquire a statistical probability and a first coefficient of each candidate speech recognition text, wherein the statistical probability and the first coefficient represent the naturalness of each candidate speech recognition text; Acquire contribution probabilities and second coefficients of the multiple keywords corresponding to each candidate speech recognition text, wherein the contribution probabilities and second coefficients represent contribution degrees of the multiple keywords to the recognition result; Calculating according to the recognition probability, the statistical probability, the first coefficient, the contribution probability and the second coefficient of each candidate speech recognition text to obtain the accuracy of each candidate speech recognition text; Determine a target text recognition result for each audio sub-segment from the plurality of candidate speech recognition results according to the accuracy of each candidate speech recognition text; According to the target text recognition result of each of the audio sub-segments, the speech recognition result of the video is obtained.

2. The method according to claim 1, characterized in that The step of processing the video to obtain a plurality of audio sub-segments and an image frame sequence corresponding to each of the audio sub-segments includes: Process the video to obtain audio data and image data; Performing silence detection on the audio data to obtain multiple audio sub-segments; An image frame sequence corresponding to each of the audio sub-segments is acquired from the image data.

3. The method according to claim 2, characterized in that The acquiring, from the image data, a sequence of image frames corresponding to each of the audio sub-segments comprises: Obtain the start time and end time corresponding to each of the audio sub-segments; Determine a start frame image according to a ratio of the start time to the image sampling frequency, and determine an end frame image according to a ratio of the end time to the image sampling frequency; An interval frame image is determined from the image data according to the start frame image and the end frame image, and images are extracted from the interval frame image at a preset frequency to obtain an image frame sequence corresponding to each of the audio sub-segments.

4. The method according to claim 1, characterized in that The performing text recognition on the image frame sequence to obtain a plurality of text results includes: Optical character recognition is performed on each image frame in the image frame series to obtain the multiple text results.

5. The method according to claim 1, characterized in that The processing of the multiple text results to obtain multiple keywords includes: Perform word segmentation processing on each of the text results to obtain multiple word segments; The multiple word segments are filtered to obtain the multiple keywords.

6. A video speech recognition device, characterized in that: include: A first acquisition module is used to process the video to acquire a plurality of audio sub-segments and an image frame sequence corresponding to each of the audio sub-segments; A recognition module, used for performing text recognition on the image frame sequence to obtain multiple text results; A second acquisition module, used to process the multiple text results to obtain multiple keywords; A processing module, used for processing each of the audio sub-segments through a speech recognition model to obtain a plurality of candidate speech recognition results; A determination module, configured to determine a target text recognition result for each of the audio sub-segments according to the multiple candidate speech recognition results and the multiple keywords; A third acquisition module, configured to acquire a speech recognition result of the video according to a target text recognition result of each of the audio sub-segments; The determining module is specifically used for: Obtaining a recognition probability of each candidate speech recognition text, wherein the recognition probability represents a probability that each candidate speech recognition text is correctly recognized by a speech recognition model; Acquire a statistical probability and a first coefficient of each candidate speech recognition text, wherein the statistical probability and the first coefficient represent the naturalness of each candidate speech recognition text; Acquire contribution probabilities and second coefficients of the multiple keywords corresponding to each candidate speech recognition text, wherein the contribution probabilities and second coefficients represent contribution degrees of the multiple keywords to the recognition result; Calculating according to the recognition probability, the statistical probability, the first coefficient, the contribution probability and the second coefficient of each candidate speech recognition text to obtain the accuracy of each candidate speech recognition text; The target text recognition result of each audio sub-segment is determined from the multiple candidate speech recognition results according to the accuracy of each candidate speech recognition text.

7. The device according to claim 6, characterized in that The first acquisition module includes: A processing unit, used to process the video and obtain audio data and image data; A detection unit, configured to perform silence detection on the audio data to obtain a plurality of audio sub-segments; An acquisition unit is used to acquire an image frame sequence corresponding to each of the audio sub-segments from the image data.

8. The device according to claim 7, characterized in that The acquisition unit is specifically used for: Obtain the start time and end time corresponding to each of the audio sub-segments; Determine a start frame image according to a ratio of the start time to the image sampling frequency, and determine an end frame image according to a ratio of the end time to the image sampling frequency; An interval frame image is determined from the image data according to the start frame image and the end frame image, and images are extracted from the interval frame image at a preset frequency to obtain an image frame sequence corresponding to each of the audio sub-segments.

9. The device according to claim 6, characterized in that The identification module is specifically used for: Optical character recognition is performed on each image frame in the image frame series to obtain the multiple text results.

10. The device according to claim 6, characterized in that The second acquisition module is specifically used for: Perform word segmentation processing on each of the text results to obtain multiple word segments; The multiple word segments are filtered to obtain the multiple keywords.

11. A server, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the program, the video speech recognition method as described in any one of claims 1 to 5 is implemented.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the video speech recognition method as described in any one of claims 1 to 5 is implemented.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the video speech recognition method according to any one of claims 1 to 5 is implemented.

Images