Vehicle control system and vehicle control method

Abstract

The present invention provides a vehicle control system and a vehicle control method. The vehicle control system includes a portable terminal and a vehicle-mounted device that wirelessly communicates with the portable terminal. The vehicle-mounted device includes: a first voice recognition unit that recognizes voice; a control unit that performs control of the vehicle corresponding to the voice recognized in the first voice recognition unit. The portable terminal includes: a second voice recognition unit that recognizes voice; a determination unit that determines the control of the vehicle corresponding to the voice recognized in the second voice recognition unit; an instruction unit that instructs the control unit to perform the determined control of the vehicle; and a notification unit that, when the control of the vehicle corresponding to the voice recognized in the second voice recognition unit cannot be determined by the determination unit, performs notification based on at least the working status of the first voice recognition unit.

Description

Technical Field

[0001] The present disclosure relates to a vehicle control system and a vehicle control method. Background Art

[0002] U.S. Patent Application Publication No. 2017 / 0308689 discloses a portable device having a microphone capable of communicating with a vehicle, and discloses a technology for outputting a command to the vehicle to execute a predetermined control based on a voice instruction of an occupant.

[0003] Here, there is also a case where the vehicle side is equipped with a voice recognition device as an onboard device. When the occupant intends to speak to the onboard device but the portable device also performs voice recognition, if the command is a command that can be handled by the onboard device side but cannot be handled by the portable device side, the portable device will feedback the content that it cannot be handled to the occupant, causing discomfort to the occupant. Summary of the invention

[0004] An object of the present disclosure is to provide a vehicle control system and a vehicle control method that appropriately perform notification in consideration of voice recognition on the vehicle-mounted device side and voice recognition in a portable terminal.

[0005] Means for solving problems

[0006] The first method is a vehicle control system including a portable terminal and a vehicle-mounted device that wirelessly communicates with the portable terminal, wherein the vehicle-mounted device includes: a first voice recognition unit that recognizes voice; a control unit that performs control of the vehicle corresponding to the voice recognized in the first voice recognition unit, and the portable terminal includes: a second voice recognition unit that recognizes voice; a determination unit that determines the control of the vehicle corresponding to the voice recognized in the second voice recognition unit; an indication unit that controls the control unit to execute the determined control of the vehicle; and a notification unit that, when the control of the vehicle corresponding to the voice recognized in the second voice recognition unit cannot be determined by the determination unit, performs a notification based on at least the working status of the first voice recognition unit.

[0007] In the vehicle control system of the first mode, in the vehicle-mounted device, the first voice recognition unit recognizes the voice, and the control unit executes the control of the vehicle corresponding to the voice recognized in the first voice recognition unit. In addition, in the portable terminal, the second voice recognition unit recognizes the voice, the determination unit determines the control of the vehicle corresponding to the voice recognized in the second voice recognition unit, and the instruction unit instructs the control unit to execute the determined control of the vehicle. In addition, the notification unit performs notification according to at least the working state of the first voice recognition unit of the vehicle-mounted device when the control of the vehicle corresponding to the voice recognized in the second voice recognition unit cannot be determined by the determination unit. Thus, in the case where the control of the vehicle corresponding to the voice recognized by the portable terminal cannot be determined, the situation where the control of the vehicle cannot be determined is notified according to the working state of the voice recognition of the vehicle-mounted device. That is, the notification can be appropriately performed in consideration of the voice recognition on the vehicle-mounted device side and the voice recognition in the portable terminal. In addition, determining the control of the vehicle corresponding to the voice means determining the control content of the vehicle instructed by the voice. Furthermore, executing notification means providing notification of feedback for the recognized speech.

[0008] The second vehicle control system is such that, in the first vehicle control system, the notification unit stops the notification when the control of the vehicle cannot be determined by the determination unit and when the recognition of the voice performed by the first voice recognition unit is in progress. Thus, even when the control of the vehicle corresponding to the voice recognized by the portable terminal cannot be determined, when the voice recognition of the vehicle-mounted device is in progress, the notification of the failure to determine the control of the vehicle can be set not to be performed.

[0009] The vehicle control system of the third aspect is that, in the vehicle control system of the first aspect or the second aspect, the determination unit does not determine the control of the vehicle when the recognition of the voice performed by the first voice recognition unit is in operation. Thus, when the voice recognition of the vehicle-mounted device is in operation, it is possible to set the control of the vehicle corresponding to the voice recognized by the portable terminal not to be determined and to stop the notification corresponding to the voice recognized by the portable terminal.

[0010] The vehicle control system of the fourth aspect is that in any one of the vehicle control systems of the first aspect to the third aspect, the control unit preferentially executes the control of the vehicle corresponding to the voice recognized by the first voice recognition unit over the control of the vehicle determined by the determination unit of the portable terminal. Thus, even when both the control of the vehicle determined by the portable terminal and the control of the vehicle corresponding to the voice recognized by the vehicle-mounted device exist, the control of the vehicle can be appropriately executed.

[0011] A fifth aspect of the vehicle control system is a vehicle control system in which, in any one of the first to third aspects, the control unit determines whether a person is riding in the vehicle based on the information obtained in the vehicle-mounted device, and when it is determined that the person is not riding in the vehicle, the control of the vehicle determined by the determination unit of the portable terminal is prioritized over the control of the vehicle corresponding to the voice recognized by the first voice recognition unit. Thus, even when both the control of the vehicle determined by the portable terminal and the control of the vehicle corresponding to the voice recognized by the vehicle-mounted device are present, the control of the vehicle can be appropriately performed.

[0012] The vehicle control system of the sixth aspect is that in any one of the vehicle control systems of the first aspect to the third aspect, when the control unit determines that a person is not in the vehicle based on the information obtained in the portable terminal, the control of the vehicle determined by the determination unit of the portable terminal is preferentially executed over the control of the vehicle corresponding to the voice recognized by the first voice recognition unit. Thus, even when both the control of the vehicle determined by the portable terminal and the control of the vehicle corresponding to the voice recognized by the vehicle-mounted device exist, the control of the vehicle can be appropriately executed.

[0013] The vehicle control system of the seventh aspect is that in any one of the vehicle control systems of the first aspect to the sixth aspect, when the speech recognized by the second speech recognition unit contains a specific character string, the control unit preferentially executes the control of the vehicle determined by the determination unit of the portable terminal over the control of the vehicle corresponding to the speech recognized by the first speech recognition unit. Thus, even when both the control of the vehicle determined by the portable terminal and the control of the vehicle corresponding to the speech recognized by the vehicle-mounted device exist, the control of the vehicle can be appropriately executed.

[0014] The eighth mode is a vehicle control method in a vehicle control system including a portable terminal and a vehicle-mounted device that wirelessly communicates with the portable terminal, wherein the method includes the following processing, namely: a first voice recognition unit of the vehicle-mounted device recognizes voice; a control unit of the vehicle-mounted device performs control of the vehicle corresponding to the voice recognized in the first voice recognition unit; a second voice recognition unit of the portable terminal recognizes voice; a determination unit of the portable terminal determines the control of the vehicle corresponding to the voice recognized in the second voice recognition unit; an indication unit of the portable terminal instructs the control unit to perform the determined control of the vehicle; and a notification unit of the portable terminal performs notification at least according to the working status of the first voice recognition unit when the control of the vehicle corresponding to the voice recognized in the second voice recognition unit cannot be determined by the determination unit.

[0015] In the vehicle control method of the eighth mode, in the vehicle-mounted device, the first voice recognition unit recognizes the voice, and the control unit executes the control of the vehicle corresponding to the voice recognized in the first voice recognition unit. In addition, in the portable terminal, the second voice recognition unit recognizes the voice, the determination unit determines the control of the vehicle corresponding to the voice recognized in the second voice recognition unit, and the instruction unit instructs the control unit to execute the determined control of the vehicle. In the case where the control of the vehicle corresponding to the voice recognized in the second voice recognition unit cannot be determined by the determination unit, the notification unit of the portable terminal at least performs notification based on the working state of the first voice recognition unit. Thus, in the case where the control of the vehicle corresponding to the voice recognized by the portable terminal cannot be determined, the situation where the control of the vehicle cannot be determined is notified based on the working state of the voice recognition of the vehicle-mounted device. That is, it is possible to appropriately perform notification in consideration of the voice recognition on the vehicle-mounted device side and the voice recognition in the portable terminal.

[0016] Effects of the Invention

[0017] According to the present disclosure, it is possible to appropriately perform notification in consideration of the voice recognition on the vehicle-mounted device side and the voice recognition in the portable terminal. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Exemplary embodiments of the present disclosure are described in detail based on the following drawings, in which:

[0019] Figure 1 It is a diagram showing a configuration example of a vehicle control system according to an embodiment of the present disclosure.

[0020] Figure 2 It is a diagram showing functional blocks of a vehicle control system according to an embodiment of the present disclosure.

[0021] Figure 3A , Figure 3B It is a sequence diagram showing the processing involved in the first embodiment of the present disclosure.

[0022] Figure 4A , Figure 4B It is a sequence diagram showing the processing involved in the second embodiment of the present disclosure.

[0023] Figure 5A , Figure 5B It is a sequence diagram showing the processing involved in the third embodiment of the present disclosure.

[0024] Figure 6 This is a sequence diagram showing an example of the process according to the fifth embodiment of the present disclosure.

[0025] Figure 7 It is a sequence diagram showing another example of the process according to the fifth embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] [First embodiment]

[0027] Figure 1 It is a diagram showing a configuration example of a vehicle control system according to the first embodiment.

[0028] (structure)

[0029] The vehicle control system 100 includes a terminal device 10 worn on the ear of a user as an example, a portable terminal 20 held by the user, and an onboard device 30 mounted on a vehicle. The terminal device 10 can wirelessly communicate with the onboard device 30 when the distance from the onboard device 30 is within a fixed value, that is, within a predetermined range 500. In addition, the portable terminal 20 is, for example, a portable terminal device such as a smartphone, and can wirelessly communicate with the terminal device 10 when the distance from the terminal device 10 is within a fixed value. The predetermined range 500 is, for example, a range of about 10 meters or less from the onboard device 30.

[0030] exist Figure 2 , an example of the functional modules of the vehicle control system 100 is shown. The terminal device 10 includes a sound playback unit 11 that plays sound to the user's ear canal, a sound pickup unit 13 that picks up the user's voice, and a terminal communication unit 14 that performs wireless communication with a portable terminal 20 and a vehicle-mounted device 30, and may also include various buttons and acceleration sensors. In addition, the terminal device 10 includes a terminal control unit 15 that controls the operation of these various parts. The terminal communication unit 14 performs wireless communication using Bluetooth (a registered trademark), for example.

[0031] In addition, the terminal device 10 includes a user authentication unit and a first authentication unit. The user authentication unit performs user authentication based on the acoustic reflection sound in the ear. The first authentication unit performs authentication with the vehicle-mounted device 30 through wireless communication when entering a predetermined range 500 in which wireless communication with the vehicle-mounted device 30 is possible. After the authentication of the user and the authentication with the vehicle-mounted device 30 are successful, the terminal communication unit 14 sends the data of the picked-up voice to the portable terminal 20.

[0032] The portable terminal 20 includes a portable control unit 21. The portable control unit 21 includes a second voice recognition unit 22 that recognizes the voice picked up by the terminal device 10, and a determination unit 23 that determines the control of the vehicle according to the recognized voice.

[0033] In addition, the portable control unit 21 includes an indication unit 24 and a notification unit 25, wherein the indication unit 24 instructs the body ECU 34 of the vehicle-mounted device 30 described later to execute the determined vehicle control, and the notification unit 25 is notified by the terminal device 10 according to the working status of the first voice recognition unit 31 described later when the determination cannot be made by the determination unit 23.

[0034] The determination unit 23 does not determine the control of the vehicle corresponding to the recognized voice when the voice recognition performed by the first voice recognition unit 31 is in operation. Therefore, the notification unit 25 does not notify the feedback corresponding to the voice recognized by the second voice recognition unit 22 when the voice recognition performed by the first voice recognition unit 31 is in operation.

[0035] In addition, the portable terminal 20 includes a portable communication unit 26 for implementing communication with the terminal device 10, the vehicle-mounted device 30, and a server (not shown), an input unit for accepting input from a user, and a display unit for displaying notifications to the user. Alternatively, the second voice recognition unit 22 may communicate with the server to recognize the picked-up voice. The portable communication unit 26 implements wireless communication with the terminal device 10 and the vehicle-mounted device 30 using, for example, Bluetooth (a registered trademark).

[0036] The vehicle-mounted device 30 includes a first voice recognition unit 31 for recognizing a voice picked up by the terminal device 10, a vehicle-mounted communication unit 32 for communicating with the terminal device 10 and the portable terminal 20, a vehicle-mounted control unit 33 for controlling the vehicle corresponding to the recognized voice, and a vehicle-mounted sensor 36. The vehicle-mounted control unit 33 includes a body ECU 34 and a control target device 35. The control target device 35 includes, for example, a power window ECU for controlling a power window, a sliding sunroof ECU for controlling a sliding sunroof, a power tailgate ECU for controlling a power tailgate, or a power sliding door ECU for controlling a power sliding door.

[0037] The first voice recognition unit 31 recognizes the voice picked up by the terminal device 10 and implements multimedia control of the navigation system. In addition, the first voice recognition unit 31 implements wireless communication with the terminal device 10 using, for example, Bluetooth (registered trademark) and communicates with the body ECU 34 using the vehicle-mounted LAN (Local Area Network).

[0038] The in-vehicle communication unit 32 performs wireless communication with the terminal device 10 and the portable terminal 20 using, for example, Bluetooth (registered trademark), and performs communication with the first voice recognition unit 31 using the in-vehicle LAN.

[0039] The in-vehicle device 30 includes a second authentication unit that communicates with the first authentication unit to perform authentication with the terminal device 10 .

[0040] The terminal control unit 15, the portable control unit 21, and the vehicle-mounted control unit 33 each include a CPU that executes a control processing program and a storage unit that appropriately stores information used in the control processing. The control processing program is pre-stored in a non-temporary recording medium that is readable by a computer. In addition, the portable communication unit 26 and the vehicle-mounted communication unit 32 can also perform wireless communication with an external server via a network such as the Internet.

[0041] (deal with)

[0042] Figure 3A , Figure 3B 1 is a sequence diagram for explaining the processing executed by the terminal device 10, the portable terminal 20, and the vehicle-mounted device 30. Figure 3A , Figure 3B The processing contents will be described.

[0043] First, the terminal device 10 performs ear canal authentication. Ear canal authentication can be performed when the terminal device 10 is worn on the ear, can be performed periodically thereafter, or can be performed before the terminal device 10 performs a certain action each time. As an example, the user authentication unit of the terminal device 10 plays the sound for authentication from the sound playback unit 11 to the external auditory canal, and the sound pickup unit 13 obtains its echo, and authenticates the user by comparing the characteristics of the echo with the characteristics of the echo of a regular user who has been pre-registered. In the event of a failed authentication, subsequent processing is not performed. In addition, the sound for authentication can be either a sound in the audible frequency band or a sound in the non-audible frequency band such as ultrasound. In the case of a sound in the audible frequency band, the user can know the execution of the authentication process through the sound. In addition, in the case of a sound in the non-audible frequency band, the user can also be notified by a voice such as "authentication will be performed" before the authentication process.

[0044] and, Figure 3A , Figure 3B The processing is started, for example, when the portable terminal 20 reaches a predetermined range 500 within which wireless communication with the vehicle-mounted device 30 is possible.

[0045] In step S101, wireless communication is performed between the portable communication unit 26 of the portable terminal 20 and the vehicle-mounted communication unit 32 of the vehicle-mounted device 30, and communication is established. At this time, authentication is performed by checking whether the identifier of the terminal device 10 is consistent with the regular identifier pre-registered in the vehicle-mounted device 30. Alternatively, authentication may be performed by checking whether the identifier of the vehicle-mounted device 30 is consistent with the regular identifier pre-registered in the terminal device 10. In the case of authentication failure, the subsequent processing is not performed.

[0046] When the authentication is successful, the terminal device 10 drives the sound emitting unit 11 to notify the user. The user can thereby know that the terminal device 10 has entered the range where the vehicle-mounted device 30 can communicate. The notification method is not limited, and the terminal device 10 may also be equipped with a vibrator and notify through vibration.

[0047] In step S102, wireless communication is performed and established between the terminal communication unit 14 of the terminal device 10 and the portable communication unit 26 of the portable terminal 20. In step S103, communication is performed and established between the first voice recognition unit 31 of the vehicle-mounted device 30 and the vehicle-mounted communication unit 32.

[0048] In step S104, the vehicle-mounted device 30 determines whether the first voice recognition unit 31 has been activated. If the first voice recognition unit 31 has been activated, the process proceeds to step S105.

[0049] In step S105 , the first voice recognition unit 31 of the vehicle-mounted device 30 notifies the terminal communication unit 14 of the terminal device 10 that the first voice recognition unit 31 has been activated via the vehicle-mounted communication unit 32 and the portable communication unit 26 of the portable terminal 20 .

[0050] In step S106, the terminal device 10 determines whether there is already a notification indicating that the first voice recognition unit 31 has been activated. If there is already a notification indicating that the first voice recognition unit 31 has been activated, the process proceeds to step S107.

[0051] In step S107 , the terminal device 10 disconnects the communication established with the portable communication unit 26 of the portable terminal 20 .

[0052] In step S108, the terminal device 10 is connected to the first voice recognition unit 31 of the vehicle-mounted device 30 through wireless communication. In step S109, communication is implemented between the terminal communication unit 14 of the terminal device 10 and the first voice recognition unit 31 of the vehicle-mounted device 30, and communication is established.

[0053] Then, when the user speaks a wake-up word (WuW) and a voice command for instructing control of the vehicle, in step S110 , the sound pickup unit 13 of the terminal device 10 transmits data of the picked-up voice to the first voice recognition unit 31 .

[0054] In step S111 , the first voice recognition unit 31 of the vehicle-mounted device 30 recognizes the picked-up voice, converts it into a command, and transmits it to the body ECU 34 .

[0055] In step S112 , the body ECU 34 receives the command from the first voice recognition unit 31 , and transfers the command to the control target device 35 .

[0056] In step S113 , the controlled device 35 receives a command from the body ECU 34 and executes control according to the command.

[0057] In step S114 , the controlled device 35 feeds back the execution result of step S113 to the body ECU 34 .

[0058] In step S116 , the body ECU 34 forwards the feedback ID corresponding to the feedback in step S114 to the first voice recognition unit 31 .

[0059] In step S117 , the first voice recognition unit 31 converts the voice data into voice data corresponding to the feedback ID, and transmits the voice data to the terminal device 10 .

[0060] In step S118, the terminal device 10 outputs a voice corresponding to the voice data through the sound emitting unit 11. Then, the user receives a voice that provides feedback on the execution result of the above step S113.

[0061] (Summary of the First Embodiment)

[0062] In the vehicle control system of this embodiment, even if the control of the vehicle corresponding to the voice recognized by the portable terminal cannot be determined, the situation that the control of the vehicle cannot be determined is not notified when the voice recognition of the vehicle-mounted device is in operation. That is, it is possible to appropriately notify in consideration of the voice recognition on the vehicle-mounted device side and the voice recognition in the portable terminal.

[0063] In addition, when the user rides in a vehicle wearing the terminal device 10 and turns on the ignition switch, the first voice recognition unit 31 of the vehicle-mounted device 30 operates. At this time, when the user vocalizes that he wants to instruct the control of the vehicle by voice, voice recognition will be implemented in both the portable terminal 20 and the vehicle-mounted device 30. This is because there is no means for monitoring the working status of each other in both the portable terminal 20 and the vehicle-mounted device 30, nor is there a means for mutual mediation. In this embodiment, the portable terminal 20 detects the activation of the first voice recognition unit 31 of the vehicle-mounted device 30, and switches to connect the terminal device 10 and the first voice recognition unit 31 of the vehicle-mounted device 30. As a result, voice recognition is implemented in both the portable terminal 20 and the vehicle-mounted device 30, so that inconvenience to the user can be suppressed.

[0064] [Second embodiment]

[0065] In the first embodiment, the terminal device 10 picks up the voice. In contrast, in the second embodiment, the microphones of the terminal device 10 and the vehicle-mounted device 30 pick up the voice respectively, and the picked-up voice is recognized and converted into a command in parallel in the portable terminal 20 and the vehicle-mounted device 30, which is different from the first embodiment. The differences from the first embodiment are described below. In addition, the same symbols are marked for the same structures, and the description is omitted.

[0066] (structure)

[0067] The determination unit 23 of the mobile terminal 20 determines the control of the vehicle according to the recognized voice, regardless of whether the voice recognition by the first voice recognition unit 31 is in operation.

[0068] When the determination unit 23 fails to determine the control of the vehicle and the first voice recognition unit 31 is in operation for voice recognition, the notification unit 25 does not provide feedback indicating that the control of the vehicle corresponding to the recognized voice cannot be determined.

[0069] The vehicle control unit 33 of the vehicle-mounted device 30 performs the control of the vehicle corresponding to the voice recognized by the first voice recognition unit 31 with priority over the control of the vehicle determined by the determination unit 23 of the portable terminal 20 .

[0070] (deal with)

[0071] Figure 4A , Figure 4B 1 is a sequence diagram for explaining the processing executed by the terminal device 10, the portable terminal 20, and the vehicle-mounted device 30. Figure 4A , Figure 4B The processing content will be described below. This processing is started when the portable terminal 20 reaches a predetermined range 500 in which wireless communication with the vehicle-mounted device 30 is possible, for example.

[0072] In step S201 , wireless communication is performed between the portable communication unit 26 of the portable terminal 20 and the vehicle-mounted communication unit 32 of the vehicle-mounted device 30 , and communication is established.

[0073] In step S202, wireless communication is performed and established between the terminal communication unit 14 of the terminal device 10 and the portable communication unit 26 of the portable terminal 20. In step S203, communication is performed and established between the first voice recognition unit 31 of the vehicle-mounted device 30 and the vehicle-mounted communication unit 32.

[0074] Then, when the user speaks a wake-up word and a voice command for controlling the vehicle, in step S204 , the sound pickup unit 13 of the terminal device 10 transmits data of the picked-up voice to the second voice recognition unit 22 of the portable terminal 20 .

[0075] In step S205 , the first voice recognition unit 31 of the vehicle-mounted device 30 recognizes the picked-up voice and converts it into a command. If the command conversion is successful, the command is sent to the body ECU 34 .

[0076] In step S206 , the body ECU 34 receives a command from the portable terminal 20 .

[0077] In parallel with the above-mentioned step S205 , in step S207 , the microphone of the vehicle-mounted device 30 picks up a voice, and the first voice recognition unit 31 recognizes the picked-up voice, converts it into a command, and transmits it to the body ECU 34 .

[0078] In step S208 , the body ECU 34 receives a command from the first voice recognition unit 31 .

[0079] In step S209, the body ECU 34 determines whether it has received both the command from the portable terminal 20 and the command from the first voice recognition unit 31. If it has received both the command from the portable terminal 20 and the command from the first voice recognition unit 31, in step S210, the body ECU 34 discards the command from the portable terminal 20 and forwards the command from the first voice recognition unit 31 to the controlled device 35.

[0080] On the other hand, when only one of the command from the portable terminal 20 and the command from the first voice recognition unit 31 is received, the body ECU 34 transfers the received command to the control target device 35 in step S211 .

[0081] In step S212 , the controlled device 35 receives a command from the body ECU 34 and executes control according to the command.

[0082] In step S213 , the controlled device 35 feeds back the execution result of the above-mentioned step S212 to the body ECU 34 .

[0083] In step S214, the body ECU 34 determines whether it is receiving a command from either the portable terminal 20 or the first voice recognition unit 31. In the case where the command is being received only from the first voice recognition unit 31 or is being received from both the portable terminal 20 and the first voice recognition unit 31, in step S215, the body ECU 34 forwards the feedback ID indicating the feedback of step S213 to the first voice recognition unit 31.

[0084] In step S216, the first voice recognition unit 31 converts the voice into voice data corresponding to the feedback ID, and outputs voice corresponding to the voice data using the speaker of the vehicle-mounted device 30. Then, the user receives voice that provides feedback on the execution result of step S212.

[0085] When receiving a command only from the portable terminal 20 , the body ECU 34 transfers the feedback ID indicating the feedback in the above-mentioned step S213 to the vehicle communication unit 32 in step S217 .

[0086] In step S218 , the vehicle-mounted communication unit 32 converts the feedback ID forwarded from the body ECU 34 into radio waves, and transmits the radio waves to the portable terminal 20 through wireless communication.

[0087] Furthermore, when the conversion of the command fails in the above-mentioned step S205 , in step S219 , the portable terminal 20 generates a feedback ID indicating that the conversion of the command has failed.

[0088] In step S220, the portable terminal 20 determines whether the first voice recognition unit 31 is in operation. For example, if the communication in step S203 is established, it is determined that the first voice recognition unit 31 is in operation, and in step S221, the portable terminal 20 discards the feedback ID generated in step S219. On the other hand, if the communication in step S203 is not established, it is determined that the first voice recognition unit 31 is not in operation, and the process moves to step S222.

[0089] In step S222 , the portable terminal 20 converts the voice data into voice data corresponding to the feedback ID, and transmits the voice data to the terminal device 10 .

[0090] In step S223, the terminal device 10 outputs a voice corresponding to the voice data through the sound emitting unit 11. Then, the user receives a voice that gives feedback on the execution result of the above step S212 or a voice that gives feedback on the failure of conversion of the command.

[0091] (Summary of Second Embodiment)

[0092] In the vehicle control system of the present embodiment, the control of the vehicle determined by the determination unit 23 of the portable terminal 20 is preferentially performed in accordance with the control of the vehicle corresponding to the voice recognized by the first voice recognition unit 31 of the vehicle-mounted device 30. Thus, even in the case where both the control of the vehicle determined by the portable terminal 20 and the control of the vehicle corresponding to the voice recognized by the vehicle-mounted device 30 exist, the control of the vehicle can be appropriately performed.

[0093] In this embodiment, the voice recognition of the portable terminal 20 and the voice recognition of the first voice recognition unit 31 of the vehicle-mounted device 30 are operated in parallel, so that the control of the vehicle corresponding to the voice recognized by the first voice recognition unit 31 of the vehicle-mounted device 30 is preferentially performed. In this way, the voice recognition is implemented in both the portable terminal 20 and the vehicle-mounted device 30, so that the inconvenience to the user can be suppressed.

[0094] Furthermore, by implementing feedback corresponding to the voice command in consideration of the execution state of the control of the vehicle corresponding to the voice recognized by the first voice recognition unit 31 of the vehicle-mounted device 30, it is possible to demonstrate that the control of the vehicle has been correctly performed.

[0095] In addition, in the case of a voice command that is not prepared on the portable terminal 20 side, if the first voice recognition unit 31 of the vehicle-mounted device 30 is in operation, the feedback ("〇〇 cannot work") corresponding to the voice recognized by the portable terminal 20 can be stopped. For example, the voice command prepared on the vehicle-mounted device 30 side is a command related to the operation of the navigation system, the operation of the power window, and the operation of the sliding sunroof, and the voice command prepared on the portable terminal 20 side is a command related to the operation of the power window, the operation of the sliding sunroof, the operation of the door lock, the operation of the electric tailgate, and the operation of the electric sliding door. In this case, even if the voice command is a command related to the operation of the navigation system that is not prepared on the portable terminal 20 side, as long as the first voice recognition unit 31 of the vehicle-mounted device 30 is in operation, the feedback ("〇〇 cannot work") corresponding to the voice recognized by the portable terminal 20 can be stopped.

[0096] [Third Embodiment]

[0097] In the second embodiment, the control of the vehicle determined by the vehicle-mounted device 30 is preferentially executed. In contrast, in the third embodiment, when it is determined that a person is not riding in the vehicle, the control of the vehicle determined by the portable terminal 20 is preferentially executed, which is different from the second embodiment. The differences from the second embodiment are described below. In addition, the same symbols are marked for the same structures, and the description is omitted.

[0098] (structure)

[0099] The vehicle-mounted control unit 33 of the vehicle-mounted device 30 determines whether a person is riding in the vehicle. If it is determined that the person is not riding in the vehicle, the control of the vehicle determined by the determination unit 23 of the portable terminal 20 is prioritized over the control of the vehicle corresponding to the voice recognized by the first voice recognition unit 31.

[0100] Specifically, the vehicle-mounted control unit 33 determines the position of the person using the vehicle-mounted sensor 36 and determines whether the person is inside or outside the vehicle. For example, the vehicle-mounted sensor 36 may be an occupant monitor, a seat sensor, a seat belt buckle, or an intruder sensor to determine whether the person is inside or outside the vehicle.

[0101] Alternatively, the door sensor and the seat sensor are used to determine whether the person is inside or outside the vehicle. For example, when the door sensor detects that the door has changed from an open state to a closed state, and the seat sensor detects that the person is not sitting, the person is determined to be outside the vehicle. In addition, when the door sensor detects that the door is in an open state, and the seat sensor detects that the person is not sitting, the person is determined to be outside the vehicle. In addition, when the door sensor detects that the door is in a closed state, and the seat sensor detects that the person is sitting, and the ignition switch is on, the person is determined to be inside the vehicle.

[0102] Alternatively, the vehicle power is turned on / off and whether the vehicle is parked to determine whether the person is inside or outside the vehicle. For example, if the vehicle power is turned on and the shift lever is in the parking position, it is determined that the person is likely to be outside the vehicle. In addition, if the vehicle power is turned on and the shift lever is in a position other than the parking position, it is determined that the person is inside the vehicle.

[0103] When the vehicle-mounted control unit 33 simultaneously receives the vehicle control command corresponding to the voice recognized by the first voice recognition unit 31 and the vehicle control command determined by the determination unit 23 of the portable terminal 20, if it is determined that the person is inside the vehicle, the vehicle-mounted control unit 33 discards the vehicle control command determined by the determination unit 23 of the portable terminal 20 and executes the vehicle control command corresponding to the voice recognized by the first voice recognition unit 31. On the other hand, when the vehicle-mounted control unit 33 simultaneously receives the vehicle control command corresponding to the voice recognized by the first voice recognition unit 31 and the vehicle control command determined by the determination unit 23 of the portable terminal 20, if it is determined that the person is outside the vehicle, the vehicle-mounted control unit 33 discards the vehicle control command corresponding to the voice recognized by the first voice recognition unit 31 and executes the vehicle control command determined by the determination unit 23 of the portable terminal 20.

[0104] Alternatively, when determining that a person is in the vehicle cabin, the vehicle-mounted control unit 33 stops the second voice recognition unit 22 of the portable terminal 20 .

[0105] Figure 5A , Figure 5B 1 is a sequence diagram for explaining the processing executed by the terminal device 10, the portable terminal 20, and the vehicle-mounted device 30. Figure 5A , Figure 5B The processing content will be described below. This processing is started when the portable terminal 20 reaches a predetermined range 500 within which wireless communication with the vehicle-mounted device 30 is possible, for example. Figure 4A , Figure 4B The same processes are denoted by the same symbols, and the description thereof will be omitted.

[0106] First, execute the same Figure 4A , Figure 4B The same processing is performed as in steps S201 to S209.

[0107] Then, in step S301, the body ECU 34 determines whether a person is riding in the vehicle using the vehicle-mounted sensor 36. If it is determined that a person is riding in the vehicle, in step S210, the body ECU 34 discards the command from the portable terminal 20 and forwards the command from the first voice recognition unit 31 to the control target device 35.

[0108] On the other hand, when it is determined that the person is not riding in the vehicle, in step S302 , the body ECU 34 discards the command from the first voice recognition unit 31 and transfers the command from the portable terminal 20 to the controlled device 35 .

[0109] Then, execute the same Figure 4A , Figure 4B The same processing is performed as in steps S211 to S223.

[0110] (Summary of the Third Embodiment)

[0111] According to the vehicle control system of this embodiment, even when there are both control of the vehicle determined by the portable terminal 20 and control of the vehicle corresponding to the voice recognized by the vehicle-mounted device 30, the vehicle control can be appropriately executed. In particular, even when the vehicle power is turned on but the vehicle-mounted sensor 36 determines that a person is not riding in the vehicle, the control of the vehicle determined by the portable terminal 20 can be executed preferentially.

[0112] [Fourth Embodiment]

[0113] In the third embodiment, the vehicle-mounted sensor 36 is used to determine whether a person is riding in the vehicle. In contrast, in the fourth embodiment, the information obtained by the terminal device 10 is used to determine whether a person is riding in the vehicle, which is different from the third embodiment. The differences from the third embodiment are described below. In addition, the same symbols are marked on the same structures, and the description is omitted.

[0114] (structure)

[0115] The sound pickup unit 13 of the terminal device 10 transmits the picked-up data to the vehicle-mounted device 30 via the portable terminal 20. The vehicle-mounted control unit 33 of the vehicle-mounted device 30 determines whether a person is riding in the vehicle based on the data picked up by the sound pickup unit 13 of the terminal device 10, and when it is determined that the person is not riding in the vehicle, the control of the vehicle determined by the determination unit 23 of the portable terminal 20 is prioritized over the control of the vehicle corresponding to the voice recognized by the first voice recognition unit 31.

[0116] Specifically, the vehicle-mounted control unit 33 determines the position of the person based on the data picked up, and determines whether the person is inside or outside the vehicle. For example, when the sound of the door being closed, the wind pressure when the door is closed, or the pressure change when the door is closed is detected based on the data picked up, it is determined that the person is inside the vehicle. In addition, when the music when the navigation system is activated or the ETC activation sound is detected as a characteristic sound when riding based on the data picked up, it is determined that the person is inside the vehicle. In addition, when the negative pressure when the door is opened is detected based on the data picked up, it is determined that the person is outside the vehicle.

[0117] In addition, when a sensor for detecting motion (for example, an acceleration sensor) is provided in the terminal device 10, the terminal device 10 sends the data of the sensor to the vehicle-mounted device 30 via the portable terminal 20. The following method may also be adopted, that is, the vehicle-mounted control unit 33 of the vehicle-mounted device 30 determines that a person is riding in a vehicle when a motion when sitting is detected based on the sensor data received from the terminal device 10, and determines that a person is not riding in a vehicle when a motion when leaving the seat is detected. For example, when up and down movements corresponding to walking, sudden downward movements, rotational movements, and stable movements are detected in sequence based on the sensor data received from the terminal device 10, it is determined that the person is in the vehicle. In addition, when stable movements, sudden upward movements, and up and down movements corresponding to walking are detected in sequence based on the sensor data received from the terminal device 10, it is determined that the person is outside the vehicle.

[0118] In addition, since other structures and processes of the vehicle control system of the fourth embodiment are the same as those of the third embodiment, the description thereof will be omitted.

[0119] (Summary of the Fourth Embodiment)

[0120] According to the vehicle control system of this embodiment, even in the case where there are both control of the vehicle determined by the portable terminal 20 and control of the vehicle corresponding to the voice recognized by the vehicle-mounted device 30, the vehicle control can be appropriately executed. In particular, even in the case where it is determined that a person is not riding in the vehicle based on the information obtained by the terminal device 10 although the vehicle power is turned on, the vehicle control determined by the portable terminal 20 can be preferentially executed.

[0121] [Fifth Embodiment]

[0122] In the third and fourth embodiments, when it is determined that a person is not on the vehicle, the control of the vehicle determined by the determination unit 23 of the portable terminal 20 is preferentially executed. In contrast, in the fifth embodiment, when the voice recognized by the portable terminal 20 contains a specific character string, the control of the vehicle determined by the determination unit 23 of the portable terminal 20 is preferentially executed, which is different from the third and fourth embodiments. The differences from the third embodiment are described below. In addition, the same symbols are marked on the same structures, and the description is omitted.

[0123] (structure)

[0124] The vehicle-mounted control unit 33 of the vehicle-mounted device 30 determines whether a specific character string is contained in the voice recognized by the portable terminal 20. When it is determined that the specific character string is contained, the control of the vehicle corresponding to the voice recognized by the first voice recognition unit 31 is performed, thereby giving priority to the control of the vehicle determined by the determination unit 23 of the portable terminal 20.

[0125] Specifically, when the ignition switch is off and the voice recognized by the portable terminal 20 contains a specific character string (for example, "OK AAA") as a wake-up word, the vehicle control unit 33 of the vehicle-mounted device 30 starts the vehicle control according to the voice, and plays the voice feedback for confirmation through the portable terminal 20 and the terminal device 10.

[0126] In addition, when the ignition switch is on and the voice recognized by the portable terminal 20 contains a specific character string (for example, "AAA outside") as a wake-up word, the vehicle control unit 33 of the vehicle-mounted device 30 starts the control of the vehicle according to the voice, and plays the voice feedback for confirmation via the portable terminal 20 and the terminal device 10.

[0127] In addition, when the ignition switch is off, the vehicle-mounted control unit 33 of the vehicle-mounted device 30 converts a specific character string (for example, "open the tailgate") as a voice command in the voice recognized by the portable terminal 20 into a command to open the tailgate and sends it to the body ECU 34 for execution.

[0128] At this time, when the vehicle-mounted control unit 33 simultaneously receives a vehicle control command corresponding to the voice recognized by the first voice recognition unit 31 and a vehicle control command determined by the determination unit 23 of the portable terminal 20, the vehicle-mounted control unit 33 discards the vehicle control command corresponding to the voice recognized by the first voice recognition unit 31.

[0129] In addition, when the voice recognized by the portable terminal 20 contains a specific character string as a voice command (for example, "open the tailgate from the outside"), the vehicle-mounted control unit 33 of the vehicle-mounted device 30 converts it into a command to open the tailgate and sends it to the body ECU 34 for execution.

[0130] At this time, when the vehicle-mounted control unit 33 simultaneously receives a vehicle control command corresponding to the voice recognized by the first voice recognition unit 31 and a vehicle control command determined by the determination unit 23 of the portable terminal 20, the vehicle-mounted control unit 33 discards the vehicle control command corresponding to the voice recognized by the first voice recognition unit 31.

[0131] (deal with)

[0132] Figure 6 FIG. 1 is a sequence diagram for explaining an example of processing executed by the terminal device 10, the portable terminal 20, and the vehicle-mounted device 30. Figure 6 The processing content will be described. This processing is started when the portable terminal 20 reaches a predetermined range 500 in which wireless communication with the vehicle-mounted device 30 is possible, for example. In addition, the case where the ignition switch is off will be described.

[0133] First, execute the same Figure 4A , Figure 4B The same processing is performed as in steps S201 to S203.

[0134] Then, when the user says the wake-up word "OK AAA", in step S500, the sound pickup unit 13 of the terminal device 10 sends the picked-up voice data to the second voice recognition unit 22 of the portable terminal 20, the second voice recognition unit 22 of the portable terminal 20 recognizes the picked-up voice, and the determination unit 23 converts it into a command and sends it to the body ECU34.

[0135] In step S501, the body ECU 34 receives a command from the portable terminal 20. The body ECU 34 transmits a feedback ID indicating that the system has been started to the portable terminal 20.

[0136] In step S502, the portable terminal 20 converts the voice data into voice data corresponding to the feedback ID, and transmits the voice data to the terminal device 10. The terminal device 10 outputs the voice "Pi" corresponding to the voice data through the sound emitting unit 11.

[0137] Next, when the user says the voice command "open the tailgate", in step S503, the sound pickup unit 13 of the terminal device 10 sends the picked up voice data to the second voice recognition unit 22 of the portable terminal 20, the second voice recognition unit 22 of the portable terminal 20 recognizes the picked up voice, and the determination unit 23 converts it into a command and sends it to the body ECU34.

[0138] In step S504, the body ECU 34 receives the command from the portable terminal 20, and transfers the received command to the control target device 35. The control target device 35 receives the command from the body ECU 34, and executes control according to the command.

[0139] In the above Figure 6 In the processing, when the ignition switch is on, even if the voice command "open the rear tailgate" is spoken, the body ECU 34 will determine that the command received from the portable terminal 20 is a command that does not work when the ignition switch is on, and send a feedback ID indicating that it does not correspond to the voice command to the portable terminal 20. The portable terminal 20 converts the voice data corresponding to the feedback ID and sends the voice data to the terminal device 10. The terminal device 10 outputs a voice corresponding to the voice data (for example, "not supported") through the sound playback unit 11.

[0140] Figure 7 This is a sequence diagram for explaining another example of the processing executed by the terminal device 10, the portable terminal 20, and the vehicle-mounted device 30. Figure 7 The processing content is explained. This processing is started, for example, when the portable terminal 20 reaches a predetermined range 500 in which wireless communication with the vehicle-mounted device 30 is possible. In addition, a case where the ignition switch is turned on and the user holding the terminal device 10 and the portable terminal 20 is outside the vehicle is explained. In addition, it is assumed that a sensor (for example, an acceleration sensor) for detecting movement is provided in the terminal device 10.

[0141] First, execute the same Figure 4A , Figure 4B The same processing is performed as in steps S201 to S203.

[0142] Then, when the user says the wake-up word "OK AAA", in step S510, the sound pickup unit 13 of the terminal device 10 sends the picked-up voice data to the second voice recognition unit 22 of the portable terminal 20, the second voice recognition unit 22 of the portable terminal 20 recognizes the picked-up voice, and the determination unit 23 converts it into a command and sends it to the body ECU34.

[0143] In step S511, the body ECU 34 receives a command from the portable terminal 20. The body ECU 34 transmits a feedback ID indicating confirmation of the operation from outside the vehicle compartment to the portable terminal 20.

[0144] In step S512, the portable terminal 20 converts the voice data into voice data corresponding to the feedback ID, and transmits the voice data to the terminal device 10. The terminal device 10 outputs a voice corresponding to the voice data, “Is it an external operation?”, through the sound emitting unit 11.

[0145] Next, when the user answers "yes" or nods, in step S513, the sound pickup unit 13 of the terminal device 10 sends the picked up voice data to the second voice recognition unit 22 of the portable terminal 20. The second voice recognition unit 22 of the portable terminal 20 recognizes the picked up voice, and the determination unit 23 converts it into a command and sends it to the body ECU34.

[0146] Alternatively, the sensor of the terminal device 10 transmits the sensor data to the portable terminal 20 , and the determination unit 23 determines whether a nodding action has been made. If it is determined that a nodding action has been made, it converts the data into a command and transmits it to the body ECU 34 .

[0147] In step S514 , the body ECU 34 receives the command from the portable terminal 20 , and transmits a feedback ID indicating that the system has been started to the portable terminal 20 .

[0148] In step S515, the portable terminal 20 converts the voice data into voice data corresponding to the feedback ID, and transmits the voice data to the terminal device 10. The terminal device 10 outputs the voice "Pi" corresponding to the voice data through the sound emitting unit 11.

[0149] Next, when the user says the voice command "open the tailgate", in step S516, the sound pickup unit 13 of the terminal device 10 sends the picked up voice data to the second voice recognition unit 22 of the portable terminal 20, the second voice recognition unit 22 of the portable terminal 20 recognizes the picked up voice, and the determination unit 23 converts it into a command and sends it to the body ECU34.

[0150] In step S517, the body ECU 34 receives the command from the portable terminal 20, and transfers the received command to the control target device 35. The control target device 35 receives the command from the body ECU 34, and executes control according to the command.

[0151] (Summary of the Fifth Embodiment)

[0152] According to the vehicle control system of the present embodiment, when the speech recognized by the portable terminal 20 includes a specific character string, the vehicle control unit 33 of the vehicle-mounted device 30 preferentially executes the control of the vehicle determined by the determination unit 23 of the portable terminal 20 over the control of the vehicle corresponding to the speech recognized by the first speech recognition unit 31 of the vehicle-mounted device 30. Thus, even when both the control of the vehicle determined by the portable terminal 20 and the control of the vehicle corresponding to the speech recognized by the vehicle-mounted device 30 are present, the control of the vehicle can be appropriately executed.

[0153] In particular, even when the ignition switch is turned on, when the speech recognized by the portable terminal 20 includes a specific character string, the control of the vehicle specified by the specifying unit 23 of the portable terminal 20 can be preferentially executed.

[0154] Furthermore, a wake-up word and a voice command that prioritize execution in accordance with voice recognized on the portable terminal 20 side can be prepared for the case where the ignition switch is on, thereby distinguishing from the case where the ignition switch is off.

[0155] [Remark]

[0156] In addition, various processors other than the CPU can also be used to perform various processes that the CPU reads the software (program) and performs in the above-mentioned embodiment. As the processor in this case, the following special circuit can be illustrated, and the special circuit is a processor with a circuit structure specially designed for performing specific processes such as PLD (Programmable Logic Device) and ASIC (Application Specific Integrated Circuit) that can change the circuit structure after manufacturing such as FPGA (Field-Programmable Gate Array). In addition, each process can be performed by one of these various processors, or each process can be performed by a combination of two or more processors of the same type or different types (for example, a combination of multiple FPGAs, and CPUs and FPGAs, etc.). In addition, more specifically, the hardware structure of these various processors is a circuit formed by combining circuit elements such as semiconductor elements.

[0157] In addition, in the above-mentioned embodiment, the program is pre-stored (installed) in a non-temporary storage medium readable by a computer. For example, in the ECU, the program is pre-stored in the ROM. However, it is not limited to this, and each program may also be stored in a storage device such as a HDD (Hard Disk Drive) and an SSD (Solid State Disk). In addition, each program may also be provided in the form of being recorded in a non-temporary storage medium such as a CD-ROM (Compact Disc Read Only Memory), a DVD-ROM (Digital Versatile Disc Read Only Memory), and a USB (Universal Serial Bus) memory. In addition, each program may also be provided in the form of being downloaded from an external device via a network.

[0158] The process flow described in the above embodiment is also an example, and unnecessary steps may be deleted, new steps may be added, or the process order may be replaced within a scope that does not deviate from the gist.

Claims

1. A vehicle control system comprising a portable terminal and a vehicle-mounted device for wireless communication with the portable terminal, wherein: The vehicle-mounted device comprises: A first speech recognition unit, which recognizes speech; a control unit that performs control of the vehicle according to the voice recognized by the first voice recognition unit, The portable terminal comprises: A second speech recognition unit, which recognizes speech; a determination unit that determines control of the vehicle corresponding to the voice recognized by the second voice recognition unit; an instruction unit configured to instruct the control unit to execute the determined control of the vehicle; a notification unit configured to perform notification based on at least an operating state of the first voice recognition unit when the control of the vehicle corresponding to the voice recognized by the second voice recognition unit cannot be determined by the determination unit, The notification unit stops the notification when the determination unit fails to determine the control of the vehicle and when the first voice recognition unit is in the process of recognizing the voice, The determination unit does not determine control of the vehicle when the first voice recognition unit is performing voice recognition.

2. The vehicle control system according to claim 1, wherein: The control unit preferentially performs control of the vehicle corresponding to the voice recognized by the first voice recognition unit over control of the vehicle determined by the determination unit of the portable terminal.

3. The vehicle control system according to claim 1, wherein: The control unit determines whether a person is riding in a vehicle based on the information obtained from the vehicle-mounted device, and if it is determined that the person is not riding in the vehicle, priority is given to controlling the vehicle determined by the determination unit of the portable terminal over controlling the vehicle corresponding to the voice recognized by the first voice recognition unit.

4. The vehicle control system according to claim 1, wherein: When the control unit determines that a person is not in a vehicle based on the information obtained from the portable terminal, the control unit prioritizes control of the vehicle determined by the determination unit of the portable terminal over control of the vehicle corresponding to the voice recognized by the first voice recognition unit.

5. The vehicle control system according to any one of claims 1 to 4, wherein: When the voice recognized by the second voice recognition unit includes a specific character string, the control unit preferentially executes control of the vehicle specified by the specifying unit of the portable terminal over control of the vehicle corresponding to the voice recognized by the first voice recognition unit.

6. A vehicle control method, which is a vehicle control method in a vehicle control system including a portable terminal and a vehicle-mounted device that wirelessly communicates with the portable terminal, wherein: This includes the following processing, namely: The first speech recognition unit of the vehicle-mounted device recognizes the speech; The control unit of the vehicle-mounted device performs control of the vehicle according to the voice recognized by the first voice recognition unit; The second voice recognition unit of the portable terminal recognizes the voice; The determination unit of the portable terminal determines control of the vehicle corresponding to the voice recognized by the second voice recognition unit; the instruction unit of the portable terminal instructs the control unit to execute the determined control of the vehicle; The notification unit of the portable terminal performs notification based on at least the operating state of the first voice recognition unit when the control of the vehicle corresponding to the voice recognized by the second voice recognition unit cannot be determined by the determination unit. The notification unit stops the notification when the determination unit fails to determine the control of the vehicle and when the first voice recognition unit is in the process of recognizing the voice, The determination unit does not determine control of the vehicle when the first voice recognition unit is performing voice recognition.

Images