Speech recognition system and speech recognition method

Abstract

To provide a speech recognition system which can increase the accuracy of phrase recognition in video speech recognition.SOLUTION: In a speech recognition system, a beat delimitation recognition unit 203 performs beat delimitation processing of, on the basis of a temporal change of feature points around mouth lips in a speech video, determining whether it is in a beat delimited state or a normal state, for each time at a prescribed interval (e.g., for each frame). A speech section extraction and length normalization unit 204 performs beat normalization processing of normalizing the length of beats included in one phrase of video on the basis of the result of beat delimitation processing. A speech recognition unit 205 inputs one phrase of video having undergone the normalization processing into a prescribed recognition model and performs recognition processing of recognizing the phase uttered by a speaker.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a speech recognition system that recognizes phrases spoken by a speaker based on an image of the speaker that includes the speaker's lips. [Background technology]

[0002] Research is being conducted on video speech recognition, which uses the shape and movement of the lips captured in a video of the speaker to recognize spoken words, phrases, and sentences. Audio speech recognition, which recognizes words and sentences from spoken audio, is used as an input method for computer systems, web services, and automated telephone answering systems, but it cannot be used in noisy environments or situations where it is important to avoid the content of speech being overheard by others. Video speech recognition recognizes speech from the shape and movement of the lips captured in a video of a speaker speaking toward an imaging device such as a camera. Using video speech recognition makes it possible to perform speech recognition even in environments or situations where audio speech recognition cannot be used. Furthermore, speech recognition is possible even in systems that have an imaging device such as a camera, even if they do not have a sound collection device such as a microphone.

[0003] The following are examples of conventional technologies in the technical field related to the present invention. For example, Non-Patent Document 1 discloses a speech recognition technology that inputs a video of a speaker's lips. In the speech recognition technology of Non-Patent Document 1, a partial area of ​​the video that includes the speaker's lips is input to a recognition model for a certain period of time to perform recognition. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Joon Son Chung and Andrew Zisserman, “Lip Reading in the Wild”, [online], [Retrieved December 13, 2022], Internet <URL:https: / / ora.ox.ac.uk / objects / uuid:c3238375-ec8b-4ecd-9543-8b179a6b74ba / files / mba51cd1dfb915238c9cec82fec506631> Summary of the Invention [Problem to be solved by the invention]

[0005] It is common for a speaker to not pronounce multiple mora in a phrase at the same time, and for some mora to be longer or shorter than others. However, conventional video speech recognition does not adequately address this issue, and so if the lengths of the mora in a phrase used as training data when generating the recognition model differ, the recognition model output is likely to be overlooked or misrecognized as a different phrase.

[0006] Furthermore, if the length of the video input to the recognition model is set to a fixed time (for example, 1.0 second), if the speaker speaks the phrase to be recognized for longer than the set time, part of the phrase will be missing from the input to the recognition model, making it more likely that the recognition model will miss or mistakenly recognize a different phrase as its output.Furthermore, if the speaker speaks the phrase to be recognized for shorter than the set time, it will also be more likely that the recognition model will miss or mistakenly recognize a different phrase as its output.

[0007] The present invention has been made in view of the above-described conventional circumstances, and has an object to provide a speech recognition system that can improve the accuracy of phrase recognition in video speech recognition. [Means for solving the problem]

[0008] In order to achieve the above object, a speech recognition system according to one aspect of the present invention is configured as follows. That is, the speech recognition system recognizes phrases spoken by a speaker based on video including the speaker's lips, and is characterized by comprising: a beat segmentation process that determines whether the speaker is in a beat segmentation state or a normal state at predetermined time intervals based on temporal changes in feature points of the speaker's lips in the video; a beat normalization process that normalizes the length of the beats contained in the video for one phrase based on the results of the beat segmentation process; and a recognition process that inputs the video for one phrase that has undergone the beat normalization process into a predetermined recognition model to recognize the phrases spoken by the speaker.

[0009] Here, the above-mentioned speech recognition system may further include a video length normalization process that normalizes the length of the video for one phrase that has undergone the meter normalization process so that it matches the appropriate length of the input video for the recognition model, and the recognition process may be configured to input the video for one phrase that has undergone the meter normalization process and the video length normalization process into the recognition model to recognize the phrase spoken by the speaker.

[0010] Furthermore, if there is a section in the normal state with fewer frames than a preset minimum number of frames N1 after the beat delimiter process has been performed, each frame in that section may be modified to a beat delimiter state.

[0011] Furthermore, if there is a section in the beat-delimited state with fewer frames than a preset minimum number of frames N2 after the beat-delimited process has been performed, each frame in that section may be corrected to the normal state.

[0012] Furthermore, the beat normalization process can be configured to normalize the length of the beats contained in one phrase's worth of video by comparing the length of each beat in one phrase's worth of video with a representative value for the length of the beats in one phrase's worth of video, thinning out frames for sections with beats longer than the representative value, and inserting frames for sections with beats shorter than the representative value.

[0013] In addition, the video length normalization process can be configured to normalize the length of one phrase's video by thinning frames if the length of one phrase's video is longer than the appropriate length of the input video, and by inserting frames if the length of one phrase's video is shorter than the appropriate length of the input video.

[0014] A speech recognition method according to another aspect of the present invention is configured as follows. That is, in the speech recognition method for recognizing phrases uttered by a speaker based on video including the speaker's lips, a server that receives the video of the speaker executes a beat delimitation step for determining whether the state is beat-delimited or normal at predetermined time intervals based on temporal changes in feature points of the speaker's lips in the video; a beat normalization step for normalizing the length of beats included in the video of one phrase based on the result of the beat delimitation step; and a recognition step for inputting the video of one phrase that has undergone the beat normalization step into a predetermined recognition model to recognize the phrases uttered by the speaker. [Effects of the Invention]

[0015] According to the present invention, it is possible to provide a speech recognition system that can improve the accuracy of phrase recognition in video speech recognition. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a diagram illustrating an example of the configuration of an utterance recognition system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram illustrating an outline of a process executed by the server in FIG. [Figure 3] FIG. 10 is a diagram illustrating an example of a flowchart of processing by a beat boundary recognition unit. [Figure 4] FIG. 10 is a diagram illustrating an example of a flowchart of the processing of an utterance part extraction / length normalization unit. [Figure 5] FIG. 10 is a diagram illustrating a method for determining the speech duration of a phrase. [Figure 6] 10A and 10B are diagrams showing example images before and after normalizing the length of a beat by shortening it; [Figure 7] 10A and 10B are diagrams showing example images before and after normalizing the length of a beat by extending it; [Figure 8] 1A and 1B are diagrams illustrating example videos before and after the video length is normalized by shortening. [Figure 9] 1A and 1B are diagrams showing example images before and after normalizing the length of the image by stretching; DETAILED DESCRIPTION OF THE INVENTION

[0017] An embodiment of the present invention will be described with reference to the drawings. An example of the configuration of an utterance recognition system according to an embodiment of the present invention is shown in Fig. 1. The utterance recognition system shown in Fig. 1 includes a server 101, an image capturing unit 103, and a notification unit 104.

[0018] The imaging means 103 captures an image of the speaker 102 and generates video data of the image. The server 101 acquires the video data of the speaker 102 from the imaging means 103 and executes a program for recognizing spoken phrases based on the video data. When the server 101 detects that the speaker 102 is uttering a specific phrase, the notification means 104 notifies a predetermined notification destination of the information. Examples of the notification destination include the user 105, other processes executed on the server 101, and devices or systems outside the system.

[0019] The server 101 may be a device designed specifically for this system, or may be a general-purpose server, a personal computer, a smart device, or any other device capable of executing the program according to the present invention. The server 101 has an interface for receiving information from the imaging means 103 and an interface for transmitting information to the notification means 104. The server 101 may further have an interface for acting on other processes, files, etc. running on the server 101. In addition to a processor such as a CPU (Central Processing Unit), the server 101 may also include auxiliary processors such as a GPU (Graphic Processing Unit), FPGA (Field Programmable Gate Array), DSP (Digital Signal Processor), or ASIC (Application Specific Integrated Circuit).

[0020] The imaging means 103 is a means for capturing an image including the face of the speaker 102. As the imaging means 103, a camera that captures an image using a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor), or the like can be used. The imaging means 102 may be installed in a fixed manner on a building or outdoors, or in a movable manner using pan, tilt, and zoom and installed on a building or outdoors. The imaging means 102 may also be fixed to a tablet terminal or kiosk terminal used by the speaker 102, which can be used as the server 101, facing the speaker 102.

[0021] The notification means 104 is a means for notifying a predetermined notification destination (such as the user 105 or another device or system) when the server 101 detects that the speaker 102 is uttering a specific phrase. To notify the user 105, the notification means 104 may use, for example, a monitor that displays a video showing the spoken phrase in text, a projector, a head-mounted display, an LED (Light Emitting Diode) display or electronic paper that displays text, or a speaker that outputs recorded or synthesized speech of the spoken phrase or a corresponding phrase. To notify other devices, the notification means 104 may output, for example, text information of the spoken phrase or a corresponding phrase, image information including text or graphics of the spoken phrase or a corresponding phrase, or audio information of the spoken phrase or a corresponding phrase. To notify other devices or systems, the notification means 104 may call an internal API (Application Programming Interface) or an external API, or output to a serial or network.

[0022] Next, an overview of the processing executed on server 101 will be described with reference to Fig. 2. Server 101 has a video acquisition unit 201, a lip portion extraction unit 202, a meter boundary recognition unit 203, a speech portion extraction / length normalization unit 204, and an utterance recognition unit 205. These processing units are realized, for example, by server 101 having hardware resources such as a memory and a processor, reading from the memory a program relating to each function of the present invention and executing it with the processor.

[0023] The video acquisition unit 201 acquires video data transmitted from the imaging means 103 to the server 101, converts the video data into a format suitable for subsequent processing, and outputs the video data as a speaker video 211. The format of the acquired video information can be, for example, a table format that allows reference to information about each screen and each pixel in at least the dimensions of width, height, time, or frame. Dimensions such as color and depth may also be added to this table, and metadata about the server 101 and the imaging means 103 may also be added.

[0024] The lip portion extraction unit 202 receives the speaker video 211 obtained by the video acquisition unit 201 as input, extracts an area of ​​the screen so that the lip portion of the speaker shown in the video is included on the screen, and performs processing to output the extracted area as the lip portion video 212.

[0025] The beat boundary recognition unit 203 performs processing to recognize beat changes in a phrase spoken by a speaker, using as input the lip portion image 212 obtained by the lip portion extraction unit 202. In general, the beat boundary recognition unit 203 analyzes the lip portion image 212 to recognize that the speaker is pronouncing a boundary portion before or after a beat in a phrase made up of one or more beats, and outputs information on whether or not a beat boundary state is present (i.e., whether or not a normal state is present) at predetermined time intervals (for example, every frame).

[0026] Lip part video 212 obtained by lip part extraction unit 202 and information on whether or not there is a beat boundary for each time (frame) obtained by beat boundary recognition unit 203 are input to speech part extraction and length normalization unit 204. Based on the input information, speech part extraction and length normalization unit 204 recognizes the duration of speech for one phrase, cuts out video for that period from lip part video 212, and then performs processing to normalize the length of the beats of the cut-out video for one phrase and the overall length of that video.

[0027] The speech recognition unit 205 receives the video obtained by the speech part extraction / length normalization unit 204 as input, recognizes the spoken phrase using a recognition model prepared in advance, and outputs the recognition result phrase 213 or a predetermined code corresponding to it. The number of frames of the input video is assumed to be a fixed value (for example, 30 or 60 frames per 1.0 second).

[0028] Here, we will explain the phrases and mora that are the targets of processing by the mora segment recognition unit 203 and the speech portion extraction and length normalization unit 204. The lip portion image 212 input to the mora segment recognition unit 203 contains one or more phrases spoken by the speaker 102. Each phrase is made up of one or more mora. In Japanese, there is one mora for each character, excluding yōon sounds, or one for every two yōon sounds. For example, there is one mora for each character in the phrase "arigatou" (thank you).

[0029] Next, a detailed processing procedure of the lip portion extraction unit 202 will be described. The lip portion extraction unit 202 performs facial feature point extraction processing on the speaker video 211. For the facial feature point extraction processing, a library such as Openpose can be used. Of the results of the facial feature point extraction processing, feature points corresponding to the periphery of the lip area are set as lip-periphery feature points. The lip portion extraction unit 202 cuts out the screen of the speaker video 211 so that all of these lip-periphery feature points are included in the output video, and outputs the cutout as the lip portion video 212. Alternatively, instead of the processing after facial feature point extraction, the lip portion extraction unit 202 may perform face detection processing on the speaker video 211, cut out the speaker video 211 so that part of the detected area (for example, a region whose height is the lower 40 percent and whose width is the same as the input video) is included in the output video, and output the cutout as the lip portion video 212. For face detection, for example, a library such as a Haar Cascade extractor in OpenCV can be used.

[0030] Next, the detailed processing procedure of the beat boundary recognition unit 203 will be described with reference to Fig. 3. The beat boundary recognition unit 203 is configured to execute the processing shown in steps S101 to S102 in Fig. 3, and after determining whether the state is a beat boundary state or a normal state from the lip portion image 212, corrects the determination if it is estimated to be incorrect. Each step will be described in detail below.

[0031] In step S101, it is determined whether each frame of the input lip portion image 212 is in a beat-delimiter state or a normal state. The beat-delimiter state is a state in which the speaker's mouth shape is between beats, or when the shape of the lips changes depending on the start or end of an utterance, and indicates a beat-delimiter timing. On the other hand, the normal state is a state in which the speaker is not in a beat-delimiter state.

[0032] A specific method for determining each state will be described. First, the positions of the lip-periphery feature points extracted by the lip portion extraction unit 202 are obtained, and then, for a pair of feature points, one above the lips and one below the lips, or for multiple pairs selected in a similar manner, the absolute value of the rate of change per time of the distance between the feature points is calculated. Then, if the absolute value of the rate of change is greater than a threshold value T1 (for example, 10 pixels per frame when the screen size is 150 pixels square), the state is determined to be a beat-delimiter state; otherwise, the state is determined to be normal. In this way, the beat-delimiter recognition unit 203 identifies beat-delimiter timings, which indicate changes in beats within a phrase, based on the temporal changes in the lip-periphery feature points extracted by the lip portion extraction unit 202.

[0033] Here, the rate of change may be the difference in distance between feature points between the input frame and the previous frame, or the average of the differences in distance between feature points between the input frame and the previous frame for multiple frames, or may be calculated by other methods. Furthermore, the lip-periphery feature points may be extracted using a method different from that used by the lip portion extraction unit 202.

[0034] Furthermore, instead of determining based on the distance between the feature points, a method may be used in which a beat-delimited state is determined when the rate of change per time of the area of ​​a figure enveloping the lip-periphery feature points is greater than a threshold, and a normal state is determined when it is smaller. Alternatively, a method may be used in which a beat-delimited state is determined when the rate of change per time of the aspect ratio (width / height) of the figure is greater than a threshold, and a normal state is determined when it is smaller. Alternatively, the determination may be made using a recognition model that inputs one or more frames of images using machine learning such as deep learning or SVM (Support Vector Machine) and determines whether one or more frames are in a beat-delimited state or a normal state.

[0035] Step S102 is a step for detecting and correcting the judgment result of step S101 if it is estimated to be an erroneous judgment. The erroneous judgment to be detected is when the speaker's lips are actually in a beat-delimiter state over multiple frames but an intermediate frame is recognized as a normal state, or when the speaker's lips are recognized as a beat-delimiter state in a frame when they are not actually in a beat-delimiter state.

[0036] A specific method for detecting and correcting the erroneous determination will be described. If there is a normal state section with fewer frames than a predetermined minimum number N1 (e.g., three frames) between a section in a beat-delimited state and the section immediately following it in a beat-delimited state, this is detected as an erroneous determination, and each frame of that section is corrected to a beat-delimited state. Also, if there is a section in a beat-delimited state with fewer frames than a predetermined minimum number N2 (e.g., two frames), this is detected as an erroneous determination, and each frame of that section is corrected to a normal state.

[0037] Note that even if the number of frames in a section in a beat-delimiter state is less than the minimum number of frames N2 after obtaining information on the feature points used for the determination in the beat-delimiter recognition unit 203, if the absolute value of the rate of change per time of the distance between feature points corresponding to frames in the section is greater than threshold value T2 (where T2>T1), the frame may remain in the beat-delimiter state. Furthermore, if the accuracy of the positions of the lip-periphery feature points obtained by the lip portion extraction unit 202 is high and it can be accurately determined in step S101 that the lips of the actual speaker are in a beat-delimiter state, the processing of step S102 may be omitted. Furthermore, if the number of frames per time in the lip portion video 212 is small and the minimum number of frames N1 or N2 used in the determination is one frame, some or all of the processing of step S102 may be omitted.

[0038] Next, the detailed processing procedure of the speech part extraction / length normalization unit 204 will be described with reference to Fig. 4. The speech part extraction / length normalization unit 204 is configured to execute the processing shown in steps S103 to S111 in Fig. 4, by extracting the lip part video 212 for each phrase, normalizing the length of the beats that make up each phrase, and then normalizing the overall length of each phrase. Each step will be described in detail below.

[0039] Step S103 is a step of determining the speech duration for each phrase included in the lip portion image 212. The speech duration of the phrase is determined using the beat boundary state output from the beat boundary recognition unit 203.

[0040] A method for determining the duration of speech of a phrase will be described with reference to FIG. First, the beginning of the first beat-delimiter state section in the lip portion image 212 is set as the speech start time of the phrase. Next, speech of the phrase is continued until a beat-delimiter state section followed by a normal state section longer than reference time T3 (e.g., 1.0 second) is detected. Reference time T3 is a threshold value used to determine whether the phrase has ended based on the elapsed time of the normal state section. Next, the speech end time of the phrase is set as the point when reference time T4 (e.g., 0.5 seconds) has elapsed from the beginning of the detected beat-delimiter state section (shown in FIG. 5(a)). Reference time T4 represents the maximum time for speaking one beat. However, if the lip portion image 212 ends before a beat-delimiter state section followed by a normal state section longer than reference time T3 is detected, the time of the end is set as the speech end time.

[0041] The time from the speech start time to the speech end time determined by the above method is calculated as the speech duration of the phrase. After the speech end time of the phrase is determined, if lip part image 212 continues, the beginning of the next beat-delimited section is set as the speech start time of the next phrase, and the speech end time is determined by the above method, and the speech duration is calculated repeatedly.

[0042] Here, when multiple phrases are spoken, the speech duration of one phrase determined by the above method may be divided into two speech durations. The speech duration is divided to prevent two phrases from being erroneously determined as one phrase when speakers often extend the beat at the end of a phrase.

[0043] Specifically, for a phrase longer than the maximum speech time T5 (e.g., 1.5 seconds), a first beat-delimiter state section that ends just before the point at which the maximum speech time T5 has elapsed from the speech start time of the phrase, a first normal state section that follows the first beat-delimiter state section, a second beat-delimiter state section, and a second normal state section are identified. If the first normal state section is longer than the second normal state section by a ratio R1 (e.g., 2.0 times) or more, it is determined that the two phrases have been erroneously classified as one phrase, and the resulting phrase is divided into two phrases. That is, the end time of the speech duration of the first phrase is determined to be the point at which a reference time T4 (e.g., 0.5 seconds) has elapsed from the beginning of the first beat-delimiter state section, or a point earlier than the beginning of the second beat-delimiter state section, and the start time of the speech duration of the second phrase is determined to be the point at which the beginning of the second beat-delimiter state section is earlier than the beginning of the second beat-delimiter state section (shown in FIG. 5(b)).

[0044] If the speech duration of one phrase cannot be divided into two speech durations using the above method, the maximum speech duration T5 is modified to be longer and the method is performed again. The above method may be performed only when the condition that the rate of change per unit time of the distance between feature points measured in step S101 at any time is equal to or greater than (or less than) a threshold value is satisfied.

[0045] Step S104 is a step of extracting an utterance video of one phrase from the lip portion video 212 in accordance with the start time and end time of the utterance duration of the phrase determined in step S103.

[0046] Step S105 is a step of obtaining the start time, end time, and length of each beat included in the speech video of one phrase extracted in step S104, and further obtaining a representative value of the beat length within the phrase. Specifically, within the speech duration of the phrase, the period from the beginning of a section in a beat-delimited state to the beginning of the section in the immediately following beat-delimited state (however, if there is no corresponding time point, the end of the speech duration of the phrase) is considered to be one beat, and the start time and end time of the beat are obtained for each beat that makes up the phrase. The length of a beat is the time from the start time of the beat to the end time of the beat. The representative value of the beat length within a phrase can be the average or median of the beat lengths within the phrase.

[0047] Furthermore, if a beat is detected that is longer than the reference time T4 and shorter than the reference time T3 used for the determination in step S103 (for example, 0.5 seconds or more and less than 1.0 second), it can be determined that two consecutive beats with the same mouth shape have been detected as one beat because it is longer than the maximum time for one beat. In this case, the detected beat may be divided into two beats at a time (for example, the midpoint) at a predetermined ratio between its start time and end time.

[0048] Step S106 is a step for determining whether the length of each beat in the phrase acquired in step S105 is larger than the representative value of the beat length in the phrase.

[0049] Step S107 is a step for normalizing the length of beats determined in step S106 to be longer than the representative value of the beats in the phrase by thinning out frames to shorten the length of the beats so that it matches the representative value. The frame thinning may involve deleting frames selected at equal intervals from the video of the beat section, replacing two adjacent frames in the video of the beat section with one frame that is the average of the images of those frames, or other methods of changing the length of the video. Note that step S107 is executed on the condition that the length of the beat is at least a ratio R2 (e.g., 1.5) times the representative value of the beat length in the phrase; if this condition is not met, the length of the beat does not need to be changed.

[0050] Step S108 normalizes the length of a beat determined in step S106 to be shorter than the representative value of the beats in the phrase by inserting a frame to extend the length of the beat so that it matches the representative value. The length of the beat can be extended by inserting a frame, for example, by repeating a frame (e.g., the final frame) at a certain point in the video of the beat section, by repeating frames selected at equal intervals from the video of the beat section multiple times at each frame point, by inserting a frame between two adjacent frames in the video of the beat section that is an average of the images of those frames, or by any other method of changing the length of the video. Note that step S008 is executed on the condition that the length of the beat is equal to or less than the ratio R3 (e.g., 0.75) of the representative value of the beat length in the phrase; if this condition is not met, the length of the beat need not be changed.

[0051] Step S109 is a step for determining whether the length of the utterance video (mora-normalized video) after steps S107 and S108 is larger or smaller than the appropriate video length of the input video for utterance recognition unit 205. The appropriate video length of the input video for utterance recognition unit 205 is set in advance based on the characteristics of the recognition model used in utterance recognition unit 205, etc., so as to obtain good phrase recognition performance.

[0052] Step S110 is a step for normalizing the length of the speech video to be input to speech recognition unit 205 by shortening the video by frame thinning so that the length of the beat-normalized video matches the appropriate video length of the input video to speech recognition unit 205. To shorten the length of the speech video by frame thinning, a method of deleting frames selected at equal intervals from the speech video may be used, a method of replacing two adjacent frames in the speech video with one frame that is the average of the images of those frames and omitting frames may be used, or another method of changing the length of the video may be used.

[0053] Step S111 is a step for normalizing the length of the speech video to be input to speech recognition unit 205 by extending the video by inserting frames so that the length of the beat-normalized video matches the appropriate video length of the input video to speech recognition unit 205. To extend the length of the speech video by inserting frames, a method of repeating a frame at a certain point in time of the speech video (for example, the final frame) may be used, a method of repeating frames selected at equal intervals from the speech video multiple times at each frame point in time may be used, a method of inserting one frame between two adjacent frames in the speech video that is an average of the images of those frames may be used, or another method of changing the length of the video may be used.

[0054] Next, the processing of the speech recognition unit 205 will be described in detail. The speech recognition unit 205 receives the speech video output from the speech part extraction / length normalization unit 204 as input, and outputs a phrase represented by the speech or a code corresponding thereto. The speech video input to the speech recognition unit 205 is input to a recognition model that receives a video of a certain length as input and outputs a phrase recognition result.

[0055] The recognition model outputs, as a phrase recognition result, a class corresponding to a phrase to be recognized, a class corresponding to a phrase not to be recognized, or a class in a state where there is no utterance. The structure of the recognition model may be an SVM (Support Vector Machine), a CNN (Convolutional Neural Network), an LSTM (Long Short Term Memory), or any other recognition model.

[0056] The recognition model may be generated by deep learning so that when a video of a correct utterance is input, a corresponding utterance class is output, or may define the correspondence between an input vector and an utterance class. For example, when a video of speaker 102 saying "thank you" is input, the recognition model outputs a class corresponding to the phrase "thank you" if the recognition is successful. The output of the recognition model is input to an output conversion unit that converts it into an output of the utterance recognition unit 205, and by converting it into, for example, a recognition phrase 213 in a character string format, the character string "thank you" is output from the utterance recognition unit 205. This output conversion unit may convert the input (output of the recognition model) into a state in which there is no code or utterance corresponding to the phrase and output it, or may convert it into a code corresponding to a state in which a phrase not to be recognized is spoken and output it, or may output the output of the recognition model without conversion.

[0057] Next, the processing of step S107 will be described in detail with reference to FIG. 6. FIG. 6(a) shows an example video before the mora lengths are normalized. In FIG. 6(a), the length of the mora in which "a" is spoken in the video of the phrase "thank you" is 10 frames, and the lengths of all other mora are 5 frames. If the median is used as the representative value of the mora length, the specific normalization process is to shorten the length of the mora in which "a" is spoken to 5 frames. When shortening by deleting frames selected at equal intervals from the video of the mora section, for example, the 2nd, 4th, 6th, 8th, and 10th frames out of the 10 frames in which "a" is spoken are deleted, thereby shortening the length of the mora in which "a" is spoken to 5 frames. As a result, the lengths of all the mora in the phrase "thank you" are normalized to 5 frames, as shown in FIG. 6(b), which shows an example video after the mora lengths have been normalized.

[0058] Next, the processing of step S108 will be described in detail with reference to FIG. 7. FIG. 7(a) shows an example of a video before the mora lengths are normalized. In FIG. 7(a), the length of the mora in which "a" is spoken in the video of the phrase "thank you" is 3 frames, and the lengths of the other mora are all 5 frames. If the median is used as the representative value of the mora lengths, the specific normalization process is to extend the length of the mora in which "a" is spoken to 5 frames. When using a method in which frames selected at equal intervals from the video of the mora section are repeated multiple times at each frame point, for example, the first and third frames of the three frames in which "a" is spoken are repeated twice each to extend the length of the mora in which "a" is spoken to 5 frames. As a result, the lengths of all the mora in which "a" is spoken are normalized to 5 frames, as shown in FIG. 7(b), an example of a video after the mora lengths have been normalized.

[0059] Next, the processing of step S110 will be described in detail with reference to FIG. 8. FIG. 8(a) shows an example video before the length of the speech video is normalized. In FIG. 8(a), the total length of the speech video for the phrase "Thank you" is 40 frames. Here, the appropriate video length of the video input to the speech recognition unit 114 is 30 frames. The specific normalization process involves shortening the length of the speech video by 10 frames to achieve the appropriate video length. When shortening the speech video by deleting frames selected at equal intervals from the speech video, for example, frames 4, 8, 12, 16, 20, 24, 28, 32, 36, and 40 of the 40 frames constituting the speech video are deleted, thereby shortening the total length of the speech video to 30 frames. As a result, the total length of the speech video is normalized to 30 frames, as shown in FIG. 8(b), which shows an example video after the length of the speech video has been normalized.

[0060] Next, the processing of step S111 will be described in detail with reference to FIG. 9. FIG. 9(a) shows an example video before the length of the speech video is normalized. In FIG. 9(a), the total length of the speech video for the phrase "Thank you" is 20 frames. Here, the appropriate video length of the input video to the speech recognition unit 114 is set to 30 frames. The specific normalization process involves extending the length of the speech video by 10 frames to achieve the appropriate video length. When extending the speech video by repeating a frame at a certain point in time, for example, the 20th frame of the 20 frames constituting the input video is repeated 11 times to extend the overall length of the speech video to 30 frames. As a result, the overall length of the speech video is normalized to 30 frames, as shown in FIG. 9(b), which shows an example video after the length of the speech video has been normalized.

[0061] In the above description, both the process of normalizing the length of the beats in the phrases (steps S105 to S108) and the process of normalizing the length of the speech video (steps S109 to S111) are performed, but it is also possible to perform only one of them and omit the other. Note that the above description is an example of a specific embodiment, and the present invention is not limited to this.

[0062] As described above, the speech recognition system of this example is configured such that: the meter segmentation recognition unit 203 performs meter segmentation processing to determine whether the speech video is in a meter segmentation state or a normal state at predetermined time intervals (e.g., every frame) based on temporal changes in lip-related feature points in the speech video; the speech part extraction / length normalization unit 204 performs meter normalization processing to normalize the length of the meter contained in the video for one phrase based on the results of the meter segmentation processing; and the speech recognition unit 205 inputs the video for one phrase that has undergone meter normalization processing into a predetermined recognition model and performs recognition processing to recognize the phrase uttered by the speaker. In this way, normalizing the length of the meter contained in the video for one phrase before inputting it into the recognition model can improve the recognition accuracy of the phrase uttered by the speaker. As a result, it is possible to reduce the number of misrecognitions and oversights of the phrase uttered by the speaker.

[0063] Furthermore, the speech recognition system of this example is configured so that the speech part extraction / length normalization unit 204 further performs a video length normalization process to normalize the length of the video for one phrase that has undergone the morae normalization process so that it matches the appropriate length of the input video for the recognition model, and the speech recognition unit 205 inputs the video for one phrase that has undergone the morae normalization process and the video length normalization process into a predetermined recognition model and performs a recognition process to recognize the phrase uttered by the speaker. In this way, by normalizing the length of the video for one phrase and then inputting it into the recognition model, the recognition accuracy of the phrase uttered by the speaker can be further improved.

[0064] Furthermore, with the speech recognition system of this example, when generating a recognition model to be used for video speech recognition, it is not necessary to consider how to deal with input video in which parts of phrases are missing or input video in which the length of syllables contained in the video is longer or shorter than the other mora, which makes it easier to generate a recognition model to be used for video speech recognition.

[0065] The speech recognition system of this example can be applied to various systems. For example, by applying the speech recognition system of this example to a surveillance system, it is possible to recognize the speech of a target person in an environment where sound cannot be collected and notify the user, who is a surveillance officer. Furthermore, by applying the speech recognition system of this example to a smart device, it is possible for a user to face the camera and input information using only lip movements without speaking.

[0066] Although the embodiments of the present invention have been described above, these embodiments are merely illustrative and do not limit the technical scope of the present invention. The present invention can take on various other embodiments, and various modifications such as omissions and substitutions can be made without departing from the spirit of the present invention. These embodiments and modifications thereof are included in the scope and spirit of the invention described in this specification, etc., and are included in the invention described in the claims and their equivalents.

[0067] Furthermore, the present invention can be provided not only as devices such as those described above or as systems composed of these devices, but also as methods executed by these devices, programs for realizing the functions of these devices using a processor, and storage media for storing such programs in a computer-readable manner. [Industrial Applicability]

[0068] The present invention can be used in a speech recognition system that recognizes phrases spoken by a speaker based on an image of the speaker that includes the speaker's lips. [Explanation of symbols]

[0069] 101: Server, 102: Speaker, 103: Shooting means, 104: Notification means, 105: User, 201: Video acquisition unit, 202: Lip part extraction unit, 203: Beat division recognition unit, 204: Speech part extraction and length normalization unit, 205: Speech recognition unit, 211: Speaker video, 212: Lip part video, 213: Recognition phrase

Claims

1. 1. A speech recognition system for recognizing a phrase spoken by a speaker based on an image including the speaker's lips, a beat delimiting process for determining whether the state is a beat delimiting state or a normal state at each predetermined time interval based on a temporal change in the feature points of the lip portion in the video of the speaker; a beat normalization process for normalizing the length of beats included in a video of one phrase based on the result of the beat delimitation process; a recognition process for inputting the video of one phrase that has undergone the morae normalization process into a predetermined recognition model and recognizing the phrase spoken by the speaker.

2. 2. The speech recognition system of claim 1, a video length normalization process for normalizing the length of the video for one phrase that has undergone the meter normalization process so that the length matches an appropriate length of the input video for the recognition model; In the recognition process, the video of one phrase that has undergone the meter normalization process and the video length normalization process is input into the recognition model to recognize the phrase spoken by the speaker.

3. 2. The speech recognition system of claim 1, a speech recognition system characterized in that, after the meter segmentation process, if there is a section in a normal state with a number of frames less than a predetermined minimum number of frames N1, each frame in the section is corrected to a meter segmentation state.

4. 2. The speech recognition system of claim 1, a speech recognition system characterized in that, after the beat delimitation process, if there is a section in a beat delimitation state with a number of frames less than a predetermined minimum number of frames N2, each frame in the section is corrected to a normal state.

5. 2. The speech recognition system of claim 1, the beat normalization process compares the length of each beat in the video of one phrase with a representative value of beat lengths in the video of one phrase, and normalizes the lengths of the beats included in the video of one phrase by thinning out frames for sections with beats longer than the representative value and inserting frames for sections with beats shorter than the representative value.

6. 3. The speech recognition system of claim 2, In the video length normalization process, frames are thinned out if the length of the video for one phrase is longer than the appropriate length of the input video, and frames are inserted if the length of the video for one phrase is shorter than the appropriate length of the input video, thereby normalizing the length of the video for one phrase.

7. 1. A speech recognition method for recognizing a phrase spoken by a speaker based on an image including the speaker's lips, comprising: A server that receives the video of the speaker a beat delimiting step of determining whether the state is a beat delimiting state or a normal state at each predetermined time interval based on a temporal change in the feature point of the lip portion in the video of the speaker; a beat normalization step of normalizing the length of a beat included in a video of one phrase based on the result of the beat delimitation step; a recognition step of inputting the video of one phrase that has passed through the morae normalization step into a predetermined recognition model and recognizing the phrase uttered by the speaker.

Images