control device

Abstract

To improve the accuracy of speech recognition. [Solution] The control device 20 can detect voices emitted by one or more occupants riding in the vehicle VH and control the functions of the vehicle VH based on the detected voices. When the control device 20 detects a voice in the vehicle VH, it includes a control unit 21 that determines whether to accept the voice as a command to control a function based on the results of analyzing images taken inside the vehicle VH.

Description

Technical Field

[0001] This disclosure relates to a control device.

Background Art

[0002] Patent Document 1 discloses a voice recognition control system that controls an in-vehicle device using the voice of a vehicle occupant as an input signal.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the vehicle interior, in addition to the voice of the occupant attempting a voice operation, noises such as environmental sounds and the voices of multiple occupants may enter the microphone. In this case, it is difficult to determine which occupant is attempting to perform an operation. Therefore, there has been room for improvement in the accuracy of voice recognition.

[0005] An object of this disclosure is to improve the accuracy of voice recognition.

[0006] A control device according to an embodiment of this disclosure is a voice control device capable of detecting voices emitted by one or more occupants in a vehicle and controlling the functions of the vehicle based on the detected voices, and includes a control unit that, when detecting a voice in the vehicle, determines whether to accept the voice as a command for controlling the function based on the result of analyzing an image of the interior of the vehicle taken by the vehicle.

Advantages of the Invention

[0007] According to this disclosure, the accuracy of voice recognition is improved.

Brief Description of the Drawings

[0008] [Figure 1] This is a block diagram showing the configuration of a voice control system according to an embodiment of the present disclosure. [Figure 2] This is a flowchart showing the operation of the control device according to the embodiment of this disclosure. [Modes for carrying out the invention]

[0009] The embodiments of this disclosure will be described below with reference to the figures.

[0010] In each figure, identical or corresponding parts are denoted by the same reference numerals. In the description of this embodiment, the description of identical or corresponding parts will be omitted or simplified as appropriate.

[0011] Referring to Figure 1, the configuration of the voice control system 10 according to this embodiment will be described.

[0012] The voice control system 10 comprises a control device 20 and an imaging device 30. The control device 20 and the imaging device 30 are mounted on the vehicle VH. The control device 20 and the imaging device 30 are connected to each other so as to be able to communicate with each other, for example, via an in-vehicle network such as CAN or a dedicated line.

[0013] The control device 20 can detect voices emitted by one or more occupants riding in the vehicle VH and control the functions of the vehicle VH based on the detected voices. The control device 20 includes a microcomputer equipped with a processor, memory, and input / output interfaces. This microcomputer, also called an ECU, controls various functions of the vehicle VH. "ECU" is an abbreviation for Electronic Control Unit.

[0014] The imaging device 30 is an in-vehicle camera that photographs the interior of the vehicle VH. The in-vehicle camera may be, for example, a driver monitoring camera, an interior camera, an electronic mirror such as an electronic rearview mirror, or a drive recorder. These may be used individually or in combination. Alternatively, the in-vehicle camera may be a terminal device such as a smartphone held by the occupant P, as long as it is capable of photographing the interior of the vehicle VH.

[0015] Vehicle VH is any type of automobile, such as a gasoline car, diesel car, hydrogen car, HEV, PHEV, BEV, or FCEV. "HEV" is an abbreviation for hybrid electric vehicle. "PHEV" is an abbreviation for plug-in hybrid electric vehicle. "BEV" is an abbreviation for battery electric vehicle. "FCEV" is an abbreviation for fuel cell electric vehicle. In this embodiment, vehicle VH is a private car, but is not limited to this, and may be a taxi or a bus, etc. Vehicle VH may be driven by a driver, or it may be an AV with any level of automated driving. "AV" is an abbreviation for autonomous vehicle. In this embodiment, vehicle VH has one or more occupants P. Occupants P include occupants P1 and P2.

[0016] In Figure 1, the control device 20 is configured as an on-board control device mounted on the vehicle VH. However, the control device 20 may be located outside the vehicle VH and connected to the vehicle VH via a network such as the Internet.

[0017] The outline of this embodiment will be described with reference to Figure 1.

[0018] The control device 20 can detect the voice emitted by one or more passengers P riding in the vehicle VH, and control the functions of the vehicle VH based on the detected voice. When the control device 20 detects voice in the vehicle VH, it acquires an image of the interior of the vehicle VH. The image is captured by the imaging device 30. The control device 20 analyzes the acquired image and determines whether to accept the detected voice as a command for controlling the functions of the vehicle VH based on the analysis result.

[0019] According to this embodiment, the speech situation of the passenger P can be detected by the in-vehicle camera and utilized for the operation in voice recognition. For example, based on the image captured by the in-vehicle camera, it is possible to determine the passenger who is trying to perform an operation. As a result, the accuracy of voice recognition can be improved.

[0020] Referring to FIG. 2, the configuration of the control device 20 according to this embodiment will be described.

[0021] The control device 20 includes a control unit 21, a storage unit 22, a communication unit 23, an input unit 24, and an output unit 25.

[0022] The control unit 21 includes at least one processor, at least one programmable circuit, at least one dedicated circuit, or any combination thereof, and executes predetermined processing.

[0023] The storage unit 22 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of at least two of these. The storage unit 22 stores data used for the operation of the control device 20 and data obtained by the operation of the control device 20.

[0024] The communication unit 23 includes at least one communication interface. The communication unit 23 receives data used for the operation of the control device 20 and transmits data obtained by the operation of the control device 20. In this embodiment, the communication unit 23 communicates with the imaging device 30.

[0025] The input unit 24 includes at least one input interface. In this embodiment, the input unit 24 is an audio sensor such as a microphone. Instead of being provided in the control device 20, the input unit 24 may be connected to the control device 20 as an external input device located inside the vehicle. The microphone, as an audio sensor, may be installed in multiple locations inside the vehicle to capture not only the voice of the occupant P but also ambient sounds.

[0026] The output unit 25 includes at least one output interface, which may be, for example, a display or a speaker. The output unit 25 outputs data obtained by the operation of the control device 20. Instead of being integrated into the control device 20, the output unit 25 may be connected to the control device 20 as an external output device.

[0027] The functions of the control device 20 are realized by executing the program according to this embodiment on the processor acting as the control unit 21.

[0028] Referring to Figure 2, the operation of the control device 20 according to this embodiment will be described. This operation corresponds to the control method according to this embodiment. That is, the control method according to this embodiment includes steps S1 to S6 shown in Figure 2. Hereinafter, each step in the flowchart will be identified by S and a number.

[0029] In S1, the control unit 21 of the control device 20 detects a voice V emitted by the occupant P of the vehicle VH. The voice V may be detected by any procedure, but for example, it can be detected by the following procedure: The voice sensor, acting as the input unit 24, picks up the voice V emitted by the occupant P. The input unit 24 transmits the picked-up voice V as a voice signal to the control unit 21. The control unit 21 acquires the voice signal transmitted by the input unit 24. The process in S1 is repeated until voice V is detected. When voice V is detected in S1, the process in S2 is performed.

[0030] In S2, the control unit 21 of the control device 20 acquires an image M of the interior of the vehicle VH. In this embodiment, "image" includes video. Image M may be acquired by any procedure, but for example, it may be acquired by the following procedure. First, the imaging device 30, acting as an in-vehicle camera, takes an image of the interior of the vehicle with occupants P1 and P2 inside, acting as occupants P. The imaging device 30 transmits the captured image data to the control device 20. The control unit 21 receives the image data transmitted from the imaging device 30 via the communication unit 23. The control unit 21 acquires the image indicated by the received image data as image M. In this embodiment, image M shows occupants P1 and P2.

[0031] In S3, the control unit 21 of the control device 20 identifies the speaker T. Specifically, the control unit 21 analyzes the image M acquired in S2 and identifies which of the crew members P1 and P2 shown in image M is the speaker T. The identification of the speaker T can be performed using any procedure, but for example, the speaker T can be identified by combining facial recognition technology with lip movement analysis. For example, the control unit 21 compares the lip movements of crew members P1 and P2 shown in image M and identifies the one whose lips are moving when speaking as the speaker T.

[0032] In S4, the control unit 21 of the control device 20 determines whether at least one of the orientation of the speaker T's face, facial expression, and movements identified in S3 satisfies a pre-set condition. In this embodiment, the orientation of the speaker T's face, facial expression, and movements are collectively referred to as face information. The conditions can be arbitrarily set as long as they allow for the estimation of the likelihood that the speaker T is performing voice control. The control unit 21 further analyzes the image M acquired in S2 and makes a determination based on the face information of the speaker T identified in S3. For example, the condition for "face orientation" can be set to "facing forward". In this case, the control unit 21 makes a determination based on the "face orientation" indicated by the face information of the speaker T. The control unit 21 determines that the condition is met if it determines that the speaker T's face is facing forward. The control unit 21 may further determine that there is a high probability that the speaker T is performing voice control if it determines that the speaker T's face is facing the direction of the microphone or the device that is the target of voice control. Furthermore, a condition for "facial expression" can be set as "facial expression indicating conscious speaking." "Facial expression indicating conscious speaking" may include, for example, expressions such as "frowning" or "widening the eyes." In this case, the control unit 21 determines that the condition is met if it determines that the "facial expression" shown in the speaker T's facial information corresponds to one of these "facial expressions indicating conscious speaking." Alternatively, a condition for "action" can be set as "specific hand movement (gesture)." "Specific hand movement" may include, for example, actions such as "raising a hand," "pointing," "spreading a hand," or "waving a hand." In this case, the control unit 21 determines that the condition is met if it determines that the "action" shown in the speaker T's facial information corresponds to one of these "specific hand movements."

[0033] As one modification of this embodiment, the control unit 21 of the control device 20 may, in S4, convert the voice V1 of speaker T into text and decide whether to accept the voice V as a command based on the text. For example, the control unit 21 may perform natural language processing on the textified voice V1 of speaker T to detect keywords and / or phrases related to a specific command contained in the text, and then decide whether to accept the voice V as a command by evaluating their frequency of appearance, context, etc. The control unit 21 may accept the voice V as a command if, for example, command phrases such as "Turn off the air conditioner" or "Play music" are detected as keywords and / or phrases related to the command. Alternatively, the control unit 21 may determine whether the text matches a predefined command and accept the voice V as a command if the match rate is above a threshold. According to this modification, since the decision on whether to accept the voice V as a command is made based on both the image and the voice, the command recognition accuracy is improved. For example, since it is possible to cross-check whether the speaker's actions and the content of the voice V match, misrecognition can be suppressed.

[0034] If the conditions are met in S4, the process in S6 is executed. If the conditions are not met in S4, the process terminates.

[0035] As another modification of this embodiment, the processing in S5 may be performed before S6. Specifically, the control unit 21 of the control device 20 may remove as noise any audio V' other than the voice V1 of speaker T identified in S3 from the audio V detected in S1. "Noise" includes at least ambient sounds from outside the vehicle VH, conversations of people around speaker T, or background sounds inside the vehicle. Ambient sounds are, for example, the sound of rain and wind outside. Background sounds are, for example, the sound of the vehicle VH running, the sound of the car radio, etc. Noise removal may be performed by any procedure. For example, the control unit 21 collects the audio signal from the input unit 24, analyzes the collected audio signal, and separates the audio V' other than the voice V1 of speaker T as background noise. Next, the control unit 21 removes the background noise using, for example, a noise cancellation technique such as spectral subtraction. The control unit 21 may also remove noise by utilizing machine learning algorithms or the like. By identifying the speaker T from image M, it is possible to recognize when speaker T spoke (at the start and end of speech), thus accurately obtaining speaker T's voice V1 from the audio signal. Furthermore, by removing noise, speaker T's voice V1 can be made clearer, further improving the accuracy of speech recognition.

[0036] In S6, the control unit 21 of the control device 20 receives the voice V detected in S1 as a command to control the functions of the vehicle VH. That is, the control unit 21 performs voice control based on the voice V. Specifically, the control unit 21 detects the voice V as a command and controls the functions installed in the vehicle VH based on the detected voice V. For example, the control unit 21 analyzes the voice V using a voice recognition engine and converts its contents into text data. Next, it compares the text data with a pre-programmed command list and identifies the corresponding command. This command may include, for example, adjusting the temperature of the air conditioner, setting the navigation system, or operating the audio system. Based on the identified command, the control unit 21 transmits a signal to the corresponding function or device to control the relevant vehicle function and executes the command.

[0037] According to this embodiment, the likelihood of whether the occupant P of the vehicle VH is requesting voice control can be estimated from the direction of the occupant's face, facial expression, mouth movements, etc., and misrecognition can be suppressed by deciding whether or not to implement voice control based on the estimation. Therefore, the accuracy of voice recognition is improved.

[0038] In this embodiment, it was explained that after detecting sound in S1, image acquisition (S2) and speaker identification (S3) are performed. However, steps S2 and S3 may be performed continuously, for example. Ultimately, it is sufficient to determine whether to accept the detected sound as a command to control a function based on the results of analyzing the image taken inside the vehicle's interior. Also, in this embodiment, the noise removal process in S5 was explained as being performed after determining in S4 whether to accept the sound V as a command. However, the control unit 21 may first perform the process in S5 prior to S4 to remove noise other than the speaker T's voice V1 from the sound V, and then perform the processes in S4 and S6. This configuration makes it possible to more accurately determine the speaker T's command. Specifically, by first removing noise from the sound V, a clear sound signal is obtained, and the risk of misrecognition in subsequent voice recognition processing can be reduced. Furthermore, by removing noise in advance, the processing load on the control unit 21 is reduced, and it is expected that the real-time response performance will be improved. As a result, voice operations inside the vehicle VH will be performed more smoothly and accurately.

[0039] This disclosure is not limited to the embodiments described above. For example, multiple blocks described in the block diagram may be combined, or a single block may be divided. Instead of executing multiple steps described in the flowchart in chronological order as described, they may be executed in parallel or in a different order, depending on the processing capacity of the device performing each step, or as necessary. Other modifications are possible without departing from the spirit of this disclosure. [Explanation of symbols]

[0040] 10. Voice control system 20 Control device 21 Control Unit 22 Memory section 23 Communications Department 24 Input section 25 Output section 30 Imaging device VH Vehicle

Claims

1. A control device capable of detecting voices emitted by one or more occupants in a vehicle and controlling the functions of the vehicle based on the detected voices, A control device comprising a control unit that, upon detecting sound in the vehicle, determines whether to accept the sound as a command for controlling the function, based on the results of analyzing images taken inside the vehicle's interior.

2. The control device according to claim 1, wherein the control unit analyzes the image to identify the speaker from among the one or more occupants, and decides to accept the voice as the command if at least one of the orientation of the identified speaker's face, facial expression, and movement satisfies a preset condition.

3. The control device according to claim 2, wherein the control unit converts the speaker's voice into text and, based on the text, determines whether to accept the voice as the command.

4. The control device according to claim 2 or 3, wherein the control unit removes from the sound any sound other than the speaker's voice as noise.

5. The control device according to claim 4, wherein the noise includes at least ambient sounds from outside the vehicle, conversations of people around the speaker, or background sounds inside the vehicle.

Images