Speech recognition method, device, vehicle, system and storage medium

Abstract

The invention discloses a voice recognition method and device, a vehicle, a system and a storage medium, and the method comprises the steps: collecting the video information of a preset position when the audio information is collected; identifying the collected video information so as to determine a video frame in which the mouth of a person is opened in the video information; determining a target audio frame corresponding to the appearing video frame in which the mouth of the person is opened; generating an effective audio containing the target audio frame; and performing voice recognition on the effective audio to obtain a recognition result corresponding to the effective audio. According to the scheme provided by the invention, the audio information is combined with the video information, only the audio when the mouth of a person is opened is reserved, the interference of voice emitted by other sound sources is eliminated, and the accuracy of voice recognition is improved.

Description

Technical Field

[0001] The present application relates to the field of speech recognition technology, and in particular to a speech recognition method, device, vehicle, system and storage medium. Background Art

[0002] Existing voice activities determine whether someone is speaking by detecting audio modal information. However, when there is other voice interference, for example, when there is an electronic device playing a video in the cockpit, the voice of the person in the video can easily be marked as voice. Therefore, in the existing technology, it is impossible to eliminate the interference of other sound sources and it is impossible to accurately determine whether there is a real interaction intention. Summary of the invention

[0003] The present application provides a speech recognition method, device, vehicle, system and storage medium for eliminating interference from other sound sources and improving the accuracy of speech recognition.

[0004] The present application provides a speech recognition method, comprising:

[0005] When audio information is collected, video information of a preset position is collected;

[0006] Identify the collected video information to determine the video frame in which the mouth of the person is open in the video information;

[0007] Determine a target audio frame corresponding to the video frame in which the mouth of the person appears open;

[0008] Generate valid audio including the target audio frame;

[0009] Perform speech recognition on the valid audio to obtain a recognition result corresponding to the valid audio.

[0010] The beneficial effects of the present application are: when audio information is collected, video information of a preset position is collected, the collected video information is identified to determine the video frame in which the mouth of the person appears in the video information, and then the target audio frame corresponding to the video frame in which the mouth of the person appears is determined, and effective audio containing the target audio frame is generated, and then speech recognition is performed on the effective audio to obtain a recognition result corresponding to the effective audio, and speech recognition is performed on the effective audio to obtain a recognition result corresponding to the effective audio. This solution combines audio information with video information, thereby retaining only the audio when the mouth of the person is open, eliminating the interference of speech from other sound sources, and improving the accuracy of speech recognition.

[0011] In one embodiment, the step of identifying the collected video information to determine a video frame in which a person's mouth is open appears in the video information, includes:

[0012] Extracting video frames from the video information;

[0013] Determining whether a human face appears in the video frame;

[0014] When a human face appears in the video frame, determining a mouth area in the human face;

[0015] Comparing the mouth region of the human face with the mouth region of a pre-stored video frame in a closed mouth state;

[0016] When the difference between the mouth region in the human face and the mouth region in the pre-stored video frame in the closed mouth state is higher than a preset difference, the video frame is determined to be a video frame in which the mouth of the person appearing in the video information is open.

[0017] In one embodiment, the step of identifying the collected video information to determine a video frame in which a person's mouth is open appears in the video information, includes:

[0018] Inputting the video information into the trained opening judgment model;

[0019] Obtaining a judgment result of whether a person's mouth is open in each video frame of the video information output by the trained mouth opening judgment model;

[0020] A video frame in which a person's mouth is open appearing in the video information is determined according to the judgment result.

[0021] In one embodiment, the method for creating the trained opening judgment model includes:

[0022] Construct an opening judgment model;

[0023] Using a first audio frame, a first video frame corresponding to the first audio frame, and open and closed mouth labels corresponding to the first audio frame and the first video frame as a training set to train the mouth opening judgment model to obtain a trained mouth opening judgment model;

[0024] Inputting a second audio frame and a second video frame corresponding to the second audio frame into the trained opening judgment model as a test set;

[0025] Obtaining the open and closed mouth recognition result corresponding to the test set output by the trained opening judgment model;

[0026] Comparing the open and closed mouth recognition results corresponding to the test set with the open and closed mouth labels corresponding to the test set;

[0027] When the comparison result indicates that the match degree between the open and closed mouth recognition result and the open and closed mouth labels corresponding to the test set is greater than a preset match degree, it is determined that the training of the opening judgment model is completed.

[0028] In one embodiment, determining the target audio frame corresponding to the video frame in which the mouth of the character appears open includes:

[0029] When a video frame with a person's mouth open appears in the video information, the audio state is set to a speech state, and the audio frames collected during the speech state are used as target audio frames corresponding to the video frame with the person's mouth open;

[0030] During the duration of the speech state, when the video information switches from an open state to a closed state, determining the duration of the closed state;

[0031] When the duration of the closed-mouth state is greater than a preset duration, the voice state is switched to the silent state, and the audio information collected after the closed-mouth state is deleted.

[0032] In one embodiment, generating valid audio including the target audio frame comprises:

[0033] The target audio frames in the same speech state and all audio frames between the same group of target audio frames are combined into valid audio.

[0034] In one embodiment, generating valid audio including the target audio frame comprises:

[0035] determining a frame interval between adjacent target audio frames;

[0036] Determine that the target audio frames whose frame interval is less than a preset interval are the same group of target audio frames;

[0037] The target audio frames in the same group and all audio frames between the target audio frames in the same group are combined into valid audio.

[0038] In one embodiment, the method further comprises:

[0039] Determine that the target audio frames whose frame interval is greater than the preset interval are target audio frames of different groups;

[0040] A deletion operation is performed on the audio frames between the different groups of target audio frames.

[0041] The present application also provides a speech recognition device, comprising:

[0042] A collection module, used to collect video information at a preset position when audio information is collected;

[0043] A first recognition module is used to recognize the collected video information to determine the video frame with the mouth of the person open appearing in the video information;

[0044] A determination module, used to determine a target audio frame corresponding to the video frame in which the mouth of the character appears open;

[0045] A generating module, used for generating valid audio including the target audio frame;

[0046] The second recognition module is used to perform speech recognition on the valid audio to obtain a recognition result corresponding to the valid audio.

[0047] In one embodiment, the first identification module includes:

[0048] An extraction submodule, used for extracting video frames from the video information;

[0049] A judgment submodule, used to judge whether a human face appears in the video frame;

[0050] A first determination submodule, configured to determine a mouth area in a human face when a human face appears in the video frame;

[0051] A comparison submodule, used for comparing the mouth area in the human face with the mouth area of ​​a pre-stored video frame in a closed mouth state;

[0052] The second determination submodule is used to determine that the video frame is a video frame with an open mouth of a person appearing in the video information when the difference between the mouth area in the face and the mouth area of ​​the pre-stored video frame in a closed mouth state is higher than a preset difference.

[0053] In one embodiment, the first identification module includes:

[0054] An input submodule, used for inputting the video information into the trained opening judgment model;

[0055] An acquisition submodule, used for acquiring a judgment result of whether a person's mouth is open in each video frame of the video information output by the trained mouth opening judgment model;

[0056] The third determination submodule is used to determine the video frame with the person's mouth open appearing in the video information according to the judgment result.

[0057] In one embodiment, the method for creating the trained opening judgment model includes:

[0058] Construct an opening judgment model;

[0059] Using a first audio frame, a first video frame corresponding to the first audio frame, and open and closed mouth labels corresponding to the first audio frame and the first video frame as a training set to train the mouth opening judgment model to obtain a trained mouth opening judgment model;

[0060] Inputting a second audio frame and a second video frame corresponding to the second audio frame into the trained opening judgment model as a test set;

[0061] Obtaining the open and closed mouth recognition result corresponding to the test set output by the trained opening judgment model;

[0062] Comparing the open and closed mouth recognition results corresponding to the test set with the open and closed mouth labels corresponding to the test set;

[0063] When the comparison result indicates that the match degree between the open and closed mouth recognition result and the open and closed mouth labels corresponding to the test set is greater than a preset match degree, it is determined that the training of the opening judgment model is completed.

[0064] In one embodiment, the determining module includes:

[0065] A submodule is provided for setting the audio state to a speech state when a video frame with a person's mouth open appears in the video information, and using the audio frames collected during the speech state as target audio frames corresponding to the video frame with the person's mouth open;

[0066] The fourth determination submodule is used to determine the duration of the closed state when the video information switches from the open state to the closed state during the continuous speech state;

[0067] The switching submodule is used to switch from the voice state to the silent state when the duration of the closed-mouth state is greater than a preset duration, and to delete the audio information collected after the closed-mouth state.

[0068] In one embodiment, the generating module is further used to:

[0069] The target audio frames in the same speech state and all audio frames between the same group of target audio frames are combined into valid audio.

[0070] In one embodiment, the generating module comprises:

[0071] a fifth determination submodule, configured to determine a frame interval between adjacent target audio frames;

[0072] A sixth determining submodule, configured to determine that the target audio frames whose frame interval is less than a preset interval are the same group of target audio frames;

[0073] The combining submodule is used to combine the target audio frames in the same group and all the audio frames between the target audio frames in the same group into valid audio.

[0074] In one embodiment, the generating module further includes:

[0075] A seventh determining submodule, configured to determine target audio frames having a frame interval greater than the preset interval as target audio frames of different groups;

[0076] The deleting module is used to perform a deleting operation on the audio frames between the different groups of target audio frames.

[0077] The present application also provides a speech recognition system, comprising:

[0078] at least one processor; and,

[0079] a memory communicatively connected to the at least one processor; wherein,

[0080] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to implement the speech recognition method described in any of the above embodiments.

[0081] The present application also provides a computer-readable storage medium. When the instructions in the storage medium are executed by a processor corresponding to the speech recognition system, the speech recognition system can implement the speech recognition method recorded in any of the above embodiments.

[0082] The present application also provides a vehicle, comprising:

[0083] The speech recognition device as described in the above embodiment;

[0084] or

[0085] A speech recognition system as described in the above embodiment.

[0086] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings.

[0087] The technical solution of the present application is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] The accompanying drawings are used to provide a further understanding of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings:

[0089] Figure 1 This is a flow chart of a speech recognition method in one embodiment of the present application;

[0090] Figure 2 This is a structural diagram of a speech recognition device in an embodiment of the present application;

[0091] Figure 3 This is a schematic diagram of the hardware structure of a speech recognition system in one embodiment of the present application. DETAILED DESCRIPTION

[0092] The preferred embodiments of the present application are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application and are not used to limit the present application.

[0093] Figure 1 Flow chart of a speech recognition method in one embodiment of the present application. Figure 1 As shown, the method can be implemented as the following steps S101-S105:

[0094] In step S101, when audio information is collected, video information of a preset position is collected;

[0095] In step S102, the collected video information is identified to determine the video frames in which the mouth of the person is open appearing in the video information;

[0096] In step S103, a target audio frame corresponding to the video frame in which the mouth of the character appears open is determined;

[0097] In step S104, generating effective audio including the target audio frame;

[0098] In step S105, speech recognition is performed on the valid audio to obtain a recognition result corresponding to the valid audio.

[0099] In the present application, in order to eliminate interference from other sound sources, when audio information is collected, video information at a preset position is collected. Among them, the audio information can be windowed, framed, and acoustic features extracted from the audio sampling points through the voice VAD module to obtain fbank (FilterBank, an audio processing method that performs nonlinear processing on audio in a manner similar to the human ear to improve the performance of speech recognition) features of one frame every 10ms, and then predicted whether each frame is speech through neural network modeling. After it is determined that the audio information has been collected, in order to determine whether it is a voice emitted by a person in the car, the video information of the preset position is collected. Among them, the preset position can be a pre-set position, such as a driving seat; it can also be a position in the car where there are people determined according to the in-car environment. For example, the position where a passenger is present can be determined based on video information, and the position where a passenger is present can also be determined based on a seat sensor, which is not limited in this application.

[0100] The collected video information is identified to determine the video frame in which the mouth of the person appears in the video information. For example, the video frame in which the face image appears in the video information can be compared with the pre-stored video frame in the closed state. When the difference degree of the comparison is higher than the preset difference degree, the video frame is determined to be a video frame in which the mouth of the person appears. The coordinates of the key points of the video frame or the mouth area collected by the camera in the image can also be used as visual features through the visual VAD module, and the visual features are synchronized with the voice features through the resampling module, and then the subsequent audio information and video information are input into the trained opening judgment model to predict whether the passenger speaks in each video frame. It should be noted that since the visual frame rate is affected by the system computing load and usually floats within a certain range, and the frame rate of the voice frame is basically fixed, it is necessary to consider how to align the voice frame and the video frame in the decoding logic of the VAD. The present application adopts the method of resampling the visual signal and aligning it with the voice frame. The specific method is to calculate the frame data that needs to be interpolated according to the fixed frame rate, and interpolate the average of the previous and next frames. For example, if the frame rate is fixed at 10ms, the interval between the previous picture Vt and the next picture Vt+1 is 113ms, and 10 frames need to be inserted in between, each frame: Vi=i×(V t+1 -V t ) / 11. Since some pronunciations do not require opening the mouth, such as "hmm", and the mouth shapes of each audio are different, the training of the opening judgment model in one embodiment of the present application uses the DFSMN network for supervised training of binary classification modeling (open mouth, closed mouth). The training data is the speech frame and the resampled visual key points, as well as the frame-level opening and closing mouth labels. The input is the mouth area key points after resampling. Each state of each layer depends on the dependency information of the previous 5 to 10 frames and the next 1 to 2 frames. Generally speaking, the longer the dependency information, the more accurate the model, but the higher the response delay, so a 6-layer DFSMN can be used for opening and closing mouth detection. The output of the model is whether the mouth is open.

[0101] Determine the target audio frame corresponding to the video frame in which the mouth of the person appears open. In this embodiment, when a video frame in which the mouth of the person appears open appears in the video information, the audio state is set to the speech state, and the audio frames collected during the speech state are used as the target audio frames corresponding to the video frame in which the mouth of the person appears open. During the speech state, when the video information switches from the open state to the closed state, the duration of the closed state is determined. When the duration of the closed state is greater than the preset duration, the speech state is switched to the silent state, and the audio information collected after the closed state is deleted.

[0102] Generate valid audio including the target audio frame. Since the voice information sent by the occupants may be interrupted, the present embodiment combines the target audio frames in the same voice state and all the audio frames between the same group of target audio frames to form valid audio. The same voice state refers to the voice state within a complete switch of the execution subject from "silent state→voice state→silent state". The same group of target audio frames refers to the target audio frames whose frame interval between adjacent target audio frames is less than the preset interval. For example, the time of the silent state between adjacent target audio frames can be determined as the frame interval between adjacent target audio frames; or the number of audio frames between adjacent target audio frames can be determined as the frame interval. Determine the target audio frames whose frame interval is less than the preset interval as the same group of target audio frames. Determine the target audio frames whose interval time is less than the preset time interval, or the target audio frames whose interval frame number is less than the preset number. Combine the target audio frames of the same group and all the audio frames between the same group of target audio frames into valid audio. When the frame interval between adjacent target audio frames is greater than the preset interval, the adjacent target audio frames are determined to be target audio frames of different groups. When the current target audio frame is combined, the target audio frames in a different group from the current target audio frame are deleted.

[0103] Perform speech recognition on the valid audio to obtain a recognition result corresponding to the valid audio.

[0104] The beneficial effects of the present application are: when audio information is collected, video information of a preset position is collected, the collected video information is identified to determine the video frame in which the mouth of the person appears in the video information, and then the target audio frame corresponding to the video frame in which the mouth of the person appears is determined, and effective audio containing the target audio frame is generated, and then speech recognition is performed on the effective audio to obtain a recognition result corresponding to the effective audio, and speech recognition is performed on the effective audio to obtain a recognition result corresponding to the effective audio. This solution combines audio information with video information, thereby retaining only the audio when the mouth of the person is open, eliminating the interference of speech from other sound sources, and improving the accuracy of speech recognition.

[0105] In one embodiment, the above step S102 may be implemented as the following steps A1-A5, including:

[0106] In step A1, extracting video frames from the video information;

[0107] In step A2, it is determined whether a human face appears in the video frame;

[0108] In step A3, when a human face appears in the video frame, a mouth area in the human face is determined;

[0109] In step A4, the mouth region of the human face is compared with the mouth region of the pre-stored video frame in the closed mouth state;

[0110] In step A5, when the difference between the mouth area in the human face and the mouth area in the pre-stored video frame in the closed mouth state is higher than a preset difference, the video frame is determined to be a video frame with an open mouth of the person appearing in the video information.

[0111] In this embodiment, a video frame in the video information is first extracted, and then the image in the video frame is detected to determine whether a face appears in the video frame. When a face appears in the video frame, the mouth area in the face is determined, that is, in the picture captured by the camera, the mouth area in the image is extracted by identifying the key points of the mouth area.

[0112] The mouth region of the face is compared with the mouth region of the pre-stored video frame in the closed mouth state. For example, the feature vector of the mouth can be directly extracted from the mouth region image of the face, and after feature vector preprocessing such as translation, stretching, etc., it is compared with the feature vector of the mouth region of the pre-stored video frame in the closed mouth state to calculate the similarity between the two.

[0113] When the difference between the mouth region in the face and the mouth region of the pre-stored video frame in the closed mouth state is higher than a preset difference, the video frame is determined to be a video frame with an open mouth of the person appearing in the video information. For example, when the similarity is lower than a preset similarity value, it is considered that the difference between the current mouth region and the mouth region in the closed mouth state is large, and it is determined to be a video frame with an open mouth.

[0114] In this embodiment, the collected video information is compared with the pre-stored closed-mouth state video frame to determine whether the current video frame is in the closed-mouth state, and further determine whether the current person in the car has uttered a voice message.

[0115] In one embodiment, the above step S102 may be implemented as the following steps B1-B3:

[0116] In step B1, the video information is input into the trained opening judgment model;

[0117] In step B2, a judgment result of whether a person's mouth is open is obtained in each video frame of the video information output by the trained mouth opening judgment model;

[0118] In step B3, the video frames in which the mouth of the person is open appearing in the video information are determined according to the judgment result.

[0119] In this embodiment, the video information is input into a trained mouth opening judgment model, which can be a neural network model, for example, a DFSMN network model is constructed, the input is a feature vector collected from key points of the mouth area, and the output of the model is whether the mouth is open.

[0120] Obtain a judgment result of whether a person's mouth is open in each video frame in the video information output by the trained opening judgment model; in this embodiment, video frames are extracted from the video information, and the trained opening judgment model detects each video frame in the video information one by one to determine whether a person's mouth is open in each video frame.

[0121] Finally, a video frame in which the mouth of a person is open in the video information is determined according to the judgment result.

[0122] In one embodiment, the process of creating the trained opening judgment model can be implemented as the following steps C1-C6:

[0123] In step C1, an opening judgment model is constructed;

[0124] In step C2, the mouth opening judgment model is trained using the first audio frame, the first video frame corresponding to the first audio frame, and the open and closed mouth labels corresponding to the first audio frame and the first video frame as a training set to obtain a trained mouth opening judgment model;

[0125] In step C3, the second audio frame and the second video frame corresponding to the second audio frame are input into the trained opening judgment model as a test set;

[0126] In step C4, the open and closed mouth recognition result corresponding to the test set output by the trained opening judgment model is obtained;

[0127] In step C5, the open and closed mouth recognition results corresponding to the test set are compared with the open and closed mouth labels corresponding to the test set;

[0128] In step C6, when the comparison result indicates that the matching degree between the open and closed mouth recognition result and the open and closed mouth labels corresponding to the test set is greater than a preset matching degree, it is determined that the training of the opening judgment model is completed.

[0129] In this embodiment, first, an opening judgment model is constructed. The opening judgment model may be a neural network model. For example, a DFSMN network model is constructed, the input of which is a feature vector collected from key points of the mouth area, and the output of the model is whether the mouth is opened.

[0130] Secondly, the first audio frame, the first video frame corresponding to the first audio frame, and the open and closed mouth labels corresponding to the first audio frame and the first video frame are used as training sets to train the opening judgment model to obtain the trained opening judgment model. Since some pronunciations do not require opening, such as "hmm", and the mouth shapes of each audio are different, in this embodiment, the first audio frame, the first video frame corresponding to the first audio frame, and the corresponding open and closed mouth labels are used as training sets. For example, the training set is used to perform supervised training on the DFSMN network for binary classification modeling (open, closed).

[0131] Then, the second audio frame and the second video frame corresponding to the second audio frame are input into the trained mouth opening judgment model as a test set, and the open and closed mouth recognition result corresponding to the test set output by the trained mouth opening judgment model is obtained.

[0132] Finally, by comparing the open and closed mouth recognition results corresponding to the test set with the open and closed mouth labels corresponding to the test set, when the open and closed mouth recognition results are consistent with the open and closed mouth labels corresponding to the test set, it is determined that the recognition is successful; otherwise, it is determined that the recognition fails. When the comparison result indicates that the match between the open and closed mouth recognition results and the open and closed mouth labels corresponding to the test set is greater than the preset match, it is determined that the opening judgment model training is completed. Among them, the match is the ratio of successfully recognized test samples to the total test samples, and the preset match is a preset ratio value, such as 98%.

[0133] In one embodiment, the above step S103 may be implemented as the following steps D1-D3:

[0134] In step D1, when a video frame with a person's mouth open appears in the video information, the audio state is set to a speech state, and the audio frames collected during the speech state are used as target audio frames corresponding to the video frame with the person's mouth open;

[0135] In step D2, during the duration of the speech state, when the video information switches from the open state to the closed state, the duration of the closed state is determined;

[0136] In step D3, when the duration of the closed-mouth state is greater than a preset duration, the voice state is switched to the silent state, and the audio information collected after the closed-mouth state is deleted.

[0137] In this embodiment, when a video frame with a person's mouth open appears in the video information, the audio state is set to a speech state, and the audio frames collected during the speech state are used as the target audio frames corresponding to the video frame with the person's mouth open. In the prior art, the usual solution for judging whether there is speech in a period of time is state transition, that is, silent state -> speech state -> silent state. The probability output of the speech model is used to set the threshold, and after the state jump, the complete speech start point and end point can be obtained. In order to eliminate the interference of other sound sources, the present application combines visual judgment with audio state transition. When a video frame with a person's mouth open appears in the video information, the audio state is set to a speech state. At the same time, the audio frames in the audio information of the car content personnel speaking are extracted based on the video information, and the audio frames collected during the speech state are used as the target audio frames corresponding to the video frame with the person's mouth open. It should be noted that this solution proposes that when jumping from the silent state to the voice state, in order to further confirm visually whether the person in the car is speaking, a certain detection time can also be set. For example, if the detection results of the video frames for a period of time (such as 300ms) are that the person in the car is in a speaking state, the voice state is converted to the voice state, otherwise the state is reset to the silent state. Similarly, during the continuation of the voice state, when the video information switches from the open state to the closed state, the duration of the closed state is determined. When the duration of the closed state is greater than the preset duration, the voice state is switched to the silent state, and the audio information collected after the closed state is deleted.

[0138] In one embodiment, the above step S104 may be implemented as the following steps, including:

[0139] The target audio frames in the same speech state and all audio frames between the same group of target audio frames are combined into valid audio.

[0140] Since the voice information sent by the occupants may be interrupted, the present embodiment combines the target audio frames in the same voice state and all the audio frames between the same group of target audio frames to form a valid audio. The same voice state refers to the voice state within a complete switch of the execution subject from "silent state → voice state → silent state". The same group of target audio frames refers to the target audio frames whose frame interval between adjacent target audio frames is less than the preset interval.

[0141] In one embodiment, the above step S104 may be implemented as the following steps E1-E3, including:

[0142] In step E1, a frame interval between adjacent target audio frames is determined;

[0143] In step E2, it is determined that the target audio frames whose frame interval is less than the preset interval are the same group of target audio frames;

[0144] In step E3, the target audio frames in the same group and all audio frames between the target audio frames in the same group are combined into valid audio.

[0145] In this embodiment, the frame interval between adjacent target audio frames is determined. Specifically, the silence time between adjacent target audio frames can be determined as the frame interval between adjacent target audio frames; or the number of audio frames between adjacent target audio frames can be determined as the frame interval.

[0146] Determine that the target audio frames whose frame interval is less than the preset interval are the same group of target audio frames. Since the occupants may pause when sending voice information, etc. Therefore, the target audio frames whose frame interval is less than the preset interval are identified as the same group of target audio frames, for example, the target audio frames whose interval time is less than the preset time interval, or the target audio frames whose interval frame number is less than the preset number.

[0147] Finally, the target audio frames in the same group and all the audio frames between the target audio frames in the same group are combined into valid audio.

[0148] In one embodiment, the method can also be implemented as the following steps F1-F2:

[0149] In step F1, target audio frames having a frame interval greater than the preset interval are determined as target audio frames of different groups;

[0150] In step F2, a deletion operation is performed on the audio frames between the different groups of target audio frames.

[0151] In this embodiment, when the frame interval between adjacent target audio frames is greater than a preset interval, it is determined that the adjacent target audio frames are target audio frames of different groups.

[0152] When the current target audio frame is combined, the target audio frames in a different group from the current target audio frame are deleted.

[0153] Figure 2 FIG. 1 is a structural diagram of a speech recognition device in an embodiment of the present application. Figure 2 As shown, the device comprises:

[0154] The acquisition module 201 is used to acquire video information of a preset position when the audio information is acquired;

[0155] A first recognition module 202 is used to recognize the collected video information to determine the video frame where the mouth of the person is open in the video information;

[0156] A determination module 203 is used to determine a target audio frame corresponding to the video frame in which the mouth of the character appears open;

[0157] A generating module 204, configured to generate valid audio including the target audio frame;

[0158] The second recognition module 205 is used to perform speech recognition on the valid audio to obtain a recognition result corresponding to the valid audio.

[0159] In one embodiment, the first identification module includes:

[0160] An extraction submodule, used for extracting video frames from the video information;

[0161] A judgment submodule, used to judge whether a human face appears in the video frame;

[0162] A first determination submodule, configured to determine a mouth area in a human face when a human face appears in the video frame;

[0163] A comparison submodule, used for comparing the mouth area in the human face with the mouth area of ​​a pre-stored video frame in a closed mouth state;

[0164] The second determination submodule is used to determine that the video frame is a video frame with an open mouth of a person appearing in the video information when the difference between the mouth area in the face and the mouth area of ​​the pre-stored video frame in a closed mouth state is higher than a preset difference.

[0165] In one embodiment, the first identification module includes:

[0166] An input submodule, used for inputting the video information into the trained opening judgment model;

[0167] An acquisition submodule, used for acquiring a judgment result of whether a person's mouth is open in each video frame of the video information output by the trained mouth opening judgment model;

[0168] The third determination submodule is used to determine the video frame with the person's mouth open appearing in the video information according to the judgment result.

[0169] In one embodiment, the method for creating the trained opening judgment model includes:

[0170] Construct an opening judgment model;

[0171] Using a first audio frame, a first video frame corresponding to the first audio frame, and open and closed mouth labels corresponding to the first audio frame and the first video frame as a training set to train the mouth opening judgment model to obtain a trained mouth opening judgment model;

[0172] Inputting a second audio frame and a second video frame corresponding to the second audio frame into the trained mouth opening judgment model as a test set;

[0173] Obtaining the open and closed mouth recognition result corresponding to the test set output by the trained opening judgment model;

[0174] Comparing the open and closed mouth recognition results corresponding to the test set with the open and closed mouth labels corresponding to the test set;

[0175] When the comparison result indicates that the match degree between the open and closed mouth recognition result and the open and closed mouth labels corresponding to the test set is greater than a preset match degree, it is determined that the training of the opening judgment model is completed.

[0176] In one embodiment, the determining module includes:

[0177] A submodule is provided for setting the audio state to a speech state when a video frame with a person's mouth open appears in the video information, and using the audio frames collected during the speech state as target audio frames corresponding to the video frame with the person's mouth open;

[0178] The fourth determination submodule is used to determine the duration of the closed state when the video information switches from the open state to the closed state during the continuous speech state;

[0179] The switching submodule is used to switch from the voice state to the silent state when the duration of the closed-mouth state is greater than a preset duration, and to delete the audio information collected after the closed-mouth state.

[0180] In one embodiment, the generating module is further used to:

[0181] The target audio frames in the same speech state and all audio frames between the same group of target audio frames are combined into valid audio.

[0182] In one embodiment, the generating module comprises:

[0183] a fifth determination submodule, configured to determine a frame interval between adjacent target audio frames;

[0184] A sixth determining submodule, configured to determine that the target audio frames whose frame interval is less than a preset interval are the same group of target audio frames;

[0185] The combining submodule is used to combine the target audio frames in the same group and all the audio frames between the target audio frames in the same group into valid audio.

[0186] In one embodiment, the generating module further includes:

[0187] A seventh determining submodule, configured to determine target audio frames having a frame interval greater than the preset interval as target audio frames of different groups;

[0188] The deleting module is used to perform a deleting operation on the audio frames between the different groups of target audio frames.

[0189] Figure 3 FIG. 1 is a schematic diagram of the hardware structure of a speech recognition system in an embodiment of the present application. Figure 3 As shown, the speech recognition system includes:

[0190] at least one processor 320; and,

[0191] A memory 304 in communication with the at least one processor 320; wherein,

[0192] The memory 304 stores instructions that can be executed by the at least one processor 320, and the instructions are executed by the at least one processor 320 to implement the speech recognition method described in any of the above embodiments.

[0193] Reference Figure 3 The speech recognition system 300 may include one or more of the following components: a processing component 302 , a memory 304 , a power component 306 , a multimedia component 308 , an audio component 310 , an input / output (I / O) interface 312 , a sensor component 314 , and a communication component 316 .

[0194] Processing component 302 generally controls the overall operation of speech recognition system 300. Processing component 302 may include one or more processors 320 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, processing component 302 may include one or more modules to facilitate the interaction between processing component 302 and other components. For example, processing component 302 may include a multimedia module to facilitate the interaction between multimedia component 308 and processing component 302.

[0195] The memory 304 is configured to store various types of data to support the operation of the speech recognition system 300. Examples of such data include instructions for any application or method operating on the speech recognition system 300, such as text, pictures, videos, etc. The memory 304 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0196] The power supply assembly 306 provides power to various components of the speech recognition system 300. The power supply assembly 306 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the vehicle control system 300.

[0197] The multimedia component 308 includes a screen that provides an output interface between the speech recognition system 300 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 308 may also include a front camera and / or a rear camera. When the speech recognition system 300 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0198] The audio component 310 is configured to output and / or input audio signals. For example, the audio component 310 includes a microphone (MIC), and when the speech recognition system 300 is in an operation mode, such as an alarm mode, a recording mode, a speech recognition mode, and a speech output mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 304 or sent via the communication component 316. In some embodiments, the audio component 310 also includes a speaker for outputting audio signals.

[0199] I / O interface 312 provides an interface between processing component 302 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0200] The sensor assembly 314 includes one or more sensors for providing various aspects of state assessment for the speech recognition system 300. For example, the sensor assembly 314 may include a sound sensor. In addition, the sensor assembly 314 may detect the on / off state of the speech recognition system 300, the relative positioning of the components, such as the display and keypad of the speech recognition system 300, and the sensor assembly 314 may also detect the running state of the speech recognition system 300 or a component of the speech recognition system 300, the orientation of the speech recognition system 300 or the acceleration / deceleration and the temperature change of the speech recognition system 300. The sensor assembly 314 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 314 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 314 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, a material accumulation thickness sensor or a temperature sensor.

[0201] The communication component 316 is configured to enable the speech recognition system 300 to provide the ability to communicate with other devices and cloud platforms in a wired or wireless manner. The speech recognition system 300 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 316 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 316 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0202] In an exemplary embodiment, the speech recognition system 300 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to execute the speech recognition method described in any of the above embodiments.

[0203] The present application also provides a computer-readable storage medium. When the instructions in the storage medium are executed by a processor corresponding to the speech recognition system, the speech recognition system can implement the speech recognition method recorded in any of the above embodiments.

[0204] The present application also provides a vehicle, comprising:

[0205] The speech recognition device as described in the above embodiment;

[0206] or

[0207] A speech recognition system as described in the above embodiment.

[0208] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0209] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0210] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0211] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0212] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A speech recognition method, characterized in that: include: When audio information is collected, video information of a preset position is collected; Identify the collected video information to determine the video frame in which the mouth of the person is open in the video information; Determine a target audio frame corresponding to the video frame in which the mouth of the person appears open; Generate valid audio including the target audio frame; Perform speech recognition on the valid audio to obtain a recognition result corresponding to the valid audio.

2. The method according to claim 1, characterized in that The step of identifying the collected video information to determine a video frame in which a person's mouth is open and appears in the video information includes: Extracting video frames from the video information; Determining whether a human face appears in the video frame; When a human face appears in the video frame, determining a mouth area in the human face; Comparing the mouth region of the human face with the mouth region of a pre-stored video frame in a closed mouth state; When the difference between the mouth region in the human face and the mouth region in the pre-stored video frame in the closed mouth state is higher than a preset difference, the video frame is determined to be a video frame in which the mouth of the person appearing in the video information is open.

3. The method according to claim 1, characterized in that The step of identifying the collected video information to determine a video frame in which a person's mouth is open and appears in the video information includes: Inputting the video information into the trained opening judgment model; Obtaining a judgment result of whether a person's mouth is open in each video frame of the video information output by the trained mouth opening judgment model; A video frame in which a person's mouth is open appearing in the video information is determined according to the judgment result.

4. The method according to claim 3, characterized in that The method for creating the trained opening judgment model includes: Construct an opening judgment model; Using a first audio frame, a first video frame corresponding to the first audio frame, and open and closed mouth labels corresponding to the first audio frame and the first video frame as a training set to train the mouth opening judgment model to obtain a trained mouth opening judgment model; Inputting a second audio frame and a second video frame corresponding to the second audio frame into the trained mouth opening judgment model as a test set; Obtaining the open and closed mouth recognition result corresponding to the test set output by the trained opening judgment model; Comparing the open and closed mouth recognition results corresponding to the test set with the open and closed mouth labels corresponding to the test set; When the comparison result indicates that the match degree between the open and closed mouth recognition result and the open and closed mouth labels corresponding to the test set is greater than a preset match degree, it is determined that the training of the opening judgment model is completed.

5. The method according to claim 1, characterized in that The step of determining a target audio frame corresponding to the video frame in which the mouth of the person appears open comprises: When a video frame with a person's mouth open appears in the video information, the audio state is set to a speech state, and the audio frames collected during the speech state are used as target audio frames corresponding to the video frame with the person's mouth open; During the duration of the speech state, when the video information switches from an open state to a closed state, determining the duration of the closed state; When the duration of the closed-mouth state is greater than a preset duration, the voice state is switched to the silent state, and the audio information collected after the closed-mouth state is deleted.

6. The method according to claim 1, characterized in that The generating of valid audio including the target audio frame comprises: determining a frame interval between adjacent target audio frames; Determine that the target audio frames whose frame interval is less than a preset interval are the same group of target audio frames; The target audio frames in the same group and all audio frames between the target audio frames in the same group are combined into valid audio.

7. The method according to claim 6, characterized in that The method comprises: Determine that the target audio frames whose frame interval is greater than the preset interval are target audio frames of different groups; A deletion operation is performed on the audio frames between the different groups of target audio frames.

8. A speech recognition device, characterized in that: include: A collection module, used to collect video information at a preset position when audio information is collected; A first recognition module is used to recognize the collected video information to determine the video frame with the mouth of the person open appearing in the video information; A determination module, used to determine a target audio frame corresponding to the video frame in which the mouth of the character appears open; A generating module, used for generating valid audio including the target audio frame; The second recognition module is used to perform speech recognition on the valid audio to obtain a recognition result corresponding to the valid audio.

9. A speech recognition system, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to implement the speech recognition method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor corresponding to the speech recognition system, the speech recognition system can implement the speech recognition method as described in any one of claims 1 to 7.

11. A vehicle, characterized in that: include: The speech recognition device as claimed in claim 8; or The speech recognition system as claimed in claim 9.

Images