Information processing method, information processing device, and program

Abstract

An information processing device according to the present disclosure comprises at least one processor. The at least one processor is configured to acquire data representing a first voice of a first user registered as a user who controls a mobile body, the voice being acquired by a plurality of microphones. The at least one processor is configured to output position information indicating a seat occupied by the first user on the basis of the direction of the first voice, the seat being one of a plurality of seats arranged in the passenger space of the mobile body.

Description

Information processing method, information processing device, and program

[0001] The present disclosure relates to an information processing method, an information processing device, and a program.

[0002] When controlling the operation (driving control) of a vehicle (mobile object) by voice, it is necessary to properly recognize the voice uttered by the driver and to operate the vehicle in accordance with the driver's voice alone. Conventionally, the seating arrangement and the driver's seat (driver's seat) in a vehicle have been fixed. Therefore, it has been easy to obtain the driver's voice data because the driver's position can be determined by detecting whether the driver is sitting in the driver's seat.

[0003] For example, Patent Document 1 discloses a technology for detecting the presence or absence of a person or a seated person in a space such as the passenger compartment of an automobile using sound.

[0004] International Publication No. 2021 / 256303

[0005] For example, as autonomous driving becomes more advanced, seating layouts where the driver's seat is not fixed are also expected, allowing the driver to freely choose where to sit within the passenger space.However, in seating layouts where the driver's seat is not fixed, even if seat occupancy detection is performed, there is a problem in that it is not possible to determine which seat is the driver.

[0006] One of the problems that the present disclosure aims to solve is to grasp the position of a driver who controls driving by voice in a seating layout where the driver's seat is not fixed.

[0007] An information processing device according to the present disclosure includes at least one processor configured to acquire data of a first voice of a first user to be registered as a user controlling a moving object, the data being acquired by a plurality of microphones, and output position information indicating which of a plurality of seats arranged in a passenger space of the moving object the first user is sitting in, based on a direction of the first voice.

[0008] FIG. 1 is a diagram showing an example of the configuration of a vehicle equipped with a display device according to the first embodiment. FIG. 2 is a diagram showing an example of the hardware configuration of an information processing device that realizes each device equipped in the vehicle according to the first embodiment. FIG. 3 is a flowchart showing an example of the flow of a driver registration process executed in the display device according to the first embodiment. FIG. 4 is a diagram showing an example of a driver registration screen displayed on the display device according to the first embodiment. FIG. 5 is a flowchart showing an example of the flow of a voice direction calculation process executed in the display device according to the first embodiment. FIG. 6 is a diagram showing an example of a driver position presentation screen displayed on the display device according to the first embodiment. FIG. 7 is a flowchart showing an example of the flow of a vehicle control process executed in the display device according to the first embodiment. FIG. 8 is a flowchart showing an example of the flow of a voice reception process executed in the display device according to the first embodiment. FIG. 9 is a diagram showing an example of an operation screen during autonomous driving displayed on the display device according to the first embodiment. FIG. 10 is a flowchart showing an example of the flow of a position display update process executed in the display device according to the second embodiment. FIG. 11 is a diagram showing an example of a driver position presentation screen displayed on the display device according to the second embodiment. FIG. 12 is a flowchart showing an example of the flow of a position display update process executed in the display device according to the third embodiment. FIG. 13 is a flowchart showing an example of the flow of a driver change process executed in the display device according to the fourth embodiment.

[0009] (1) An information processing method according to the present disclosure is an information processing method executed by at least one processor in an information processing device including the at least one processor, the information processing method acquiring data of a first voice of a first user to be registered as a user controlling a moving object, the first voice data being acquired by a plurality of microphones, and outputting position information indicating which of a plurality of seats arranged in a passenger space of the moving object the first user is sitting in, based on a direction of the first voice data.

[0010] According to this configuration, even in a seating layout where the driver's seat is not fixed, the position of the driver who controls driving by voice can be determined based on the direction of the user's voice. Therefore, according to the above configuration, even in a seating layout where the driver's seat is not fixed, the accuracy of sound collection can be improved for the voice of the driver who controls driving by voice. This improvement in sound collection accuracy contributes to appropriate recognition of the driver's voice, i.e., realizing improved safety in voice-based driving control. Furthermore, since the occupants of the vehicle can easily know where they are seated and are recognized as the driver by the vehicle, a sense of security and high convenience can be provided.

[0011] (2) The information processing method described in (1) above acquires data on second voices of users in the passenger space of the vehicle acquired by the multiple microphones, determines whether the second voices are the voices of the first user, and if the second voices are the voices of the first user, determines whether the position of the first user calculated based on the first voice indicated by the output position information matches the position of the first user calculated based on the direction of the second voice.

[0012] According to this configuration, it is possible to determine whether the recognized driver (first user) position matches the driver position calculated based on the direction of the second voice of the driver contained in the acquired voice within the passenger space. Therefore, in a seating layout in which the driver's seat is not fixed, it is possible to grasp, for example, whether the user recognized as the driver has moved from his / her seat. This makes it possible to achieve appropriate sound collection according to the position of the user recognized as the driver. Furthermore, it is possible to easily allow the occupant of the mobile body to understand where the occupant is seated and recognized as the driver by the mobile body, and to easily understand that the occupant's movement is recognized, thereby providing a sense of security and high convenience.

[0013] (3) In the information processing method described in (2) above, when the position of the first user calculated based on the first voice indicated by the output position information matches the position of the first user calculated based on the direction of the second voice, the moving object is controlled based on the second voice.

[0014] According to this configuration, the moving object can be controlled using voice obtained by appropriate sound collection according to the user's position as recognized as a driver. Therefore, in controlling the moving object by voice, the accuracy of voice recognition can be improved, unintended control can be suppressed, and user convenience can be improved.

[0015] (4) An information processing method according to the present disclosure is an information processing method executed by at least one processor in an information processing device including at least one processor, the information processing method comprising: acquiring first voice data of a first user registered as a user controlling a moving object, the first voice data being acquired by a plurality of microphones; acquiring second voice data of a user in a passenger space of the moving object, the second voice data being acquired by the plurality of microphones; determining whether the second voice is the voice of the first user; if the second voice is the voice of the first user, determining whether a position of the first user calculated based on a direction of the first voice coincides with a position of the first user calculated based on a direction of the second voice, the position being one of a plurality of seats arranged in the passenger space of the moving object; and, if the position of the first user based on the first voice coincides with the position of the first user based on the direction of the second voice, controlling the moving object based on the second voice.

[0016] According to this configuration, even in a seating layout where the driver's seat is not fixed, the position of the driver performing voice-based driving control can be determined based on the direction of the user's voice. Furthermore, it is possible to determine whether the recognized driver (first user) position matches the driver position calculated based on the direction of the driver's second voice contained in the acquired voice within the passenger space. Furthermore, it is possible to control the moving object using voice obtained by appropriate sound collection according to the user's position being recognized as the driver. Therefore, in voice-based control of the moving object, the accuracy of voice recognition can be improved, unintended control can be suppressed, and user convenience can be improved.

[0017] (5) In the information processing method described in (3) or (4) above, the control of the moving body is control related to the operation of the moving body.

[0018] According to this configuration, it is possible to control the driving of a mobile object using voice obtained by appropriate sound collection according to the user's position as the driver. Therefore, in voice-based driving control of a mobile object, it is possible to improve the accuracy of voice recognition and suppress a decrease in safety due to unintended control.

[0019] (6) In the information processing method described in (3) or (4) above, the control of the mobile body is control related to content provided in the mobile body, and the content includes first content that realizes a function related to driving the mobile body and second content that realizes a function unrelated to driving the mobile body.

[0020] According to this configuration, it is possible to use voice obtained by appropriate sound collection according to the user's position as a driver to control the driving of a mobile object and other controls related to driving, such as entertainment, etc. Therefore, in various voice-based controls of a mobile object, the accuracy of voice recognition can be improved, unintended control can be suppressed, and user convenience can be improved.

[0021] (7) In the information processing method described in any one of (2) to (6) above, if the position of the first user calculated based on the first voice does not match the position of the first user calculated based on the direction of the second voice, a confirmation message regarding the position of the first user is output.

[0022] According to this configuration, in a seating layout where the driver's seat is not fixed, it is possible to grasp the seat movement or possibility of the user being recognized as the driver. Therefore, it is possible to notify the user that there is a risk that appropriate sound collection according to the user's position being recognized as the driver may not be possible, that is, that the accuracy of voice recognition may be reduced, or unintended control may occur or increase, resulting in a decrease in user convenience.

[0023] (8) In the information processing method described in any one of (2) to (7) above, when the position of the first user calculated based on the first voice does not match the position of the first user calculated based on the direction of the second voice, the information processing method disables control of the moving body based on the second voice, including control of driving the moving body and control of first content provided by the moving body that realizes a function related to driving the moving body, and enables control of second content provided by the moving body that realizes a function not related to driving the moving body.

[0024] According to this configuration, when it is not possible to appropriately collect the driver's voice, such as when the driver is moving, the voice control can be disabled for the driving control and the first content control that have a (large) impact on the movement (driving) of the mobile object, while being enabled for the entertainment-related second content control that has no (small) impact. Therefore, it is possible to ensure user convenience while suppressing a decrease in safety due to the occurrence or increase of unintended controls.

[0025] (9) In the information processing method according to any one of (1) to (8), the directivity of the sound acquisition by the plurality of microphones is adjusted in accordance with the sound direction of the first sound.

[0026] According to this configuration, in a seating layout in which the driver's seat is not fixed, it is possible to achieve appropriate sound collection according to the position of the user recognized as the driver.

[0027] (10) An information processing method described in any one of (2) to (9) above, which directly or indirectly cites (2) or (4) above, adjusts the directionality of sound acquisition by the multiple microphones to match the sound direction of the second sound when the position of the first user calculated based on the first sound does not match the position of the first user calculated based on the direction of the second sound.

[0028] With this configuration, even if the driver moves in a seating layout where the driver's seat is not fixed, the driver's movements can be followed and appropriate sound recording can be achieved according to the position of the user recognized as the driver after the movement.

[0029] (11) The information processing method described in any one of (2) to (10) above, which directly or indirectly cites (2) or (4) above, outputs, when the position of the first user calculated based on the first voice does not match the position of the first user calculated based on the direction of the second voice, position information indicating which seat of multiple seats arranged in the passenger space of the moving body the first user is in, based on the direction of the second voice.

[0030] With this configuration, even if the driver moves in a seating layout where the driver's seat is not fixed, the position of the driver who controls driving by voice after moving can be determined based on the direction of the user's voice, and appropriate sound collection according to the position after moving, i.e., sound collection that follows the driver's movement, can be realized. Furthermore, since the occupants of the vehicle can easily understand that they are recognized as the driver even after moving seats, a sense of security and high convenience can be provided.

[0031] (12) In the information processing method described in (11) above, when the position information based on the second sound is being output, data of the second sound newly acquired by the plurality of microphones is acquired, and it is determined whether the position of the first user indicated by the output position information matches the position of the first user calculated based on the direction of the newly acquired second sound.

[0032] According to this configuration, even after the driver moves in a seating layout in which the driver's seat is not fixed, it is possible to continue to know whether the user recognized as the driver has moved in his or her seat.

[0033] (13) The information processing method described in any one of (1) to (12) above acquires third audio data relating to the first user's seat movement within the passenger space of the moving body, acquired by the plurality of microphones, and outputs position information indicating which of the plurality of seats arranged within the passenger space of the moving body the first user is sitting in based on the direction of the third audio.

[0034] According to this configuration, the driver's seat movement can be grasped based on at least the voice input operation.

[0035] (14) The information processing method described in any one of (1) to (13) above includes outputting position information indicating the position of the first user calculated based on the first voice before the start of autonomous driving of the moving body.

[0036] According to this configuration, in a seating layout in which the driver's seat is not fixed, the position of the user recognized as the driver can be grasped before autonomous driving begins.

[0037] (15) The information processing method described in any one of (2) to (14) above, which directly or indirectly cites (2) or (4) above, determines whether the position of the first user calculated based on the first voice matches the position of the first user calculated based on the direction of the second voice after the start of autonomous driving of the moving body.

[0038] According to this configuration, in a seating layout in which the driver's seat is not fixed, it is possible to determine whether the position of the user recognized as the driver has changed (whether the driver has moved) after the start of automatic driving of the vehicle.

[0039] (16) The information processing method described in any one of (1) to (15) above acquires, after the start of autonomous driving of the mobile body, data of a fourth voice regarding a change of driver of the mobile body of a second user other than the first user who is registered as a user controlling the mobile body acquired by the plurality of microphones, and outputs position information indicating which seat of a plurality of seats arranged in the passenger space of the mobile body the second user is sitting in based on the direction of the fourth voice.

[0040] According to this configuration, in a seating layout where the driver's seat is not fixed, even if the driver is changed after autonomous driving starts, the position of the user recognized as the new driver can be grasped. Also, the occupants of the mobile body can easily understand where the occupant sitting in the mobile body is recognized as the new driver, and can easily understand that the change has been recognized, thereby providing a sense of security and high convenience.

[0041] (17) The information processing method described in any one of (2) to (16) above, which directly or indirectly cites (2) or (4) above, when the second voice is not the voice of the first user, disables control of the mobile body based on the second voice, including control of driving the mobile body and control of first content provided by the mobile body that realizes a function related to driving the mobile body, and enables control of second content provided by the mobile body that realizes a function not related to driving the mobile body.

[0042] According to this configuration, for passengers other than the driver, voice control can be disabled for driving control and first content control that have a (large) impact on the movement (driving) of the vehicle, while being enabled for entertainment-related second content control that has no (small) impact. Therefore, it is possible to ensure user convenience while suppressing a decrease in safety due to the occurrence or increase of unintended control.

[0043] (18) The information processing method described in any one of (1) to (17) above outputs position information indicating the position of the first user calculated based on at least the first voice to the display, thereby displaying on the display a seating layout indicating the arrangement of the multiple seats within the passenger space of the moving body and the position in the seating layout of the user controlling the moving body.

[0044] This configuration allows the occupants of the vehicle to easily understand where they are seated and are recognized as the driver by the vehicle, providing a sense of security and high convenience.

[0045] (19) An information processing device according to the present disclosure includes at least one processor configured to acquire data of a first voice of a first user to be registered as a user controlling a moving body, the data being acquired by a plurality of microphones, and to output position information indicating which seat among a plurality of seats arranged in a passenger space of the moving body the first user is sitting in, based on a direction of the first voice.

[0046] According to this configuration, even in a seating layout where the driver's seat is not fixed, the position of the driver who controls driving by voice can be determined based on the direction of the user's voice. Therefore, according to the above configuration, even in a seating layout where the driver's seat is not fixed, the accuracy of sound collection can be improved for the voice of the driver who controls driving by voice. This improvement in sound collection accuracy contributes to appropriate recognition of the driver's voice, i.e., realizing improved safety in voice-based driving control. Furthermore, since the occupants of the vehicle can easily know where they are seated and are recognized as the driver by the vehicle, a sense of security and high convenience can be provided.

[0047] (20) A program according to the present disclosure causes a computer to acquire data of a first voice of a first user to be registered as a user controlling a moving body, the data being acquired by a plurality of microphones, and output location information indicating which seat among a plurality of seats arranged in a passenger space of the moving body the first user is sitting in, based on the direction of the first voice.

[0048] According to this configuration, even in a seating layout where the driver's seat is not fixed, the position of the driver who controls driving by voice can be determined based on the direction of the user's voice. Therefore, according to the above configuration, even in a seating layout where the driver's seat is not fixed, the accuracy of sound collection can be improved for the voice of the driver who controls driving by voice. This improvement in sound collection accuracy contributes to appropriate recognition of the driver's voice, i.e., realizing improved safety in voice-based driving control. Furthermore, since the occupants of the vehicle can easily know where they are seated and are recognized as the driver by the vehicle, a sense of security and high convenience can be provided.

[0049] Hereinafter, with reference to the drawings, embodiments of an information processing device, a display device, a vehicle, an information processing method, a display method, a program, and a recording medium according to the present disclosure will be described.

[0050] In the description of the present disclosure, components having the same or substantially the same functions as those described above with respect to the previously-mentioned drawings may be given the same reference numerals, and descriptions thereof may be omitted as appropriate. Furthermore, even when the same or substantially the same parts are shown, the dimensions and proportions may be different depending on the drawing. Furthermore, for example, in order to ensure the visibility of the drawings, reference numerals may be given to only the main components in the description of each drawing, and reference numerals may not be given to components having the same or substantially the same functions as those described above with respect to the previously-mentioned drawings.

[0051] In the description of the present disclosure, components having the same or substantially the same functions may be distinguished by adding an alphanumeric character to the end of the reference symbol. Alternatively, when multiple components having the same or substantially the same functions are not distinguished, they may be collectively described by omitting the alphanumeric character at the end of the reference symbol.

[0052] In the description of the present disclosure, a vehicle having a seating layout in which the driver's seat is not fixed and configured to be able to execute voice-based driving control in autonomous driving is exemplified. Note that the technology according to the present disclosure may be realized as information processing for grasping the position of a speaker in any space having a seating layout in which the speaker's seat (speaking position) is not fixed, not limited to a passenger space (cabin) of a moving body such as a vehicle interior.

[0053] Here, the passenger space (cabin) of a moving body refers to a space (internal space) within the moving body, equivalent to the passenger compartment of a vehicle. In other words, the passenger space of a moving body is a space provided inside the moving body, and is a space in which at least one occupant (user), such as a driver or passenger traveling by the moving body, can ride or board. Note that, for example, seats for occupants are arranged in the passenger space of a moving body according to the present disclosure, but this is not limited thereto. For example, instead of or in addition to seats, other seat structures or signs may be arranged that specify or define the position of the occupant within the passenger space. In other words, in the present disclosure, a "seat" may refer to a position that the occupant can take within the passenger space of the moving body. Furthermore, in the present disclosure, a "seat layout" may refer to information indicating the positions that the occupant can take within the passenger space of the moving body.

[0054] Here, the mobile body may be any type of vehicle, such as an electric vehicle (EV) driven by a motor, an automobile driven by an engine (internal combustion engine), or a hybrid vehicle driven by both an engine and a motor. The mobile body may be, for example, a passenger car, truck, motorcycle, or other automobile (vehicle), but may also be an electric bicycle, electric kick scooter, electric wheelchair, construction machine, agricultural machine, ship, train, airplane, or the like. The mobile body may be configured to be autonomously mobile, or may be configured to be mobile in response to direct or remote operation by a driver. Here, the "movement" of the mobile body is achieved by autonomous or heteronomic control (driving control), and may be expressed as "driving." In this disclosure, vehicles are mainly used as examples of mobile bodies. Therefore, the "travel" of a vehicle in this disclosure is an example of the movement (driving) of a mobile body.

[0055] First Embodiment Fig. 1 is a diagram showing an example of the configuration of a vehicle 1 equipped with a display device 3 according to a first embodiment. As shown in Fig. 1, the vehicle 1 has the display device 3, a driving control device 5, and a content control device 6. The display device 3 is communicably connected to each of the driving control device 5 and the content control device 6 via an in-vehicle network including a CAN (Controller Area Network), Ethernet (registered trademark), USB (Universal Serial Bus (registered trademark), etc. in the vehicle 1.

[0056] The display device 3 outputs various information such as notifications and warnings, and content information to the occupants of the vehicle 1. As an example, the display device 3 is detachably disposed in a position visible to the occupants seated in the seats 103 (see FIG. 4 ), such as on the dashboard or console of the vehicle 1. Note that the display device 3 may be fixed to a predetermined position in the vehicle 1.

[0057] The display device 3 outputs location information indicating which of the multiple seats 103 arranged in the passenger compartment (inside the vehicle 1) the first user P1 is sitting in, based on the direction of the first voice of the first user P1 who is registered as the user controlling the vehicle 1.

[0058] For example, the display device 3 displays the driver's position in the seat layout based on the position information on the display 331 (see FIG. 4). Note that the display device 3 is not limited to outputting the position information via the display 331, and may output the position information to an external device via the I / F 84 (see FIG. 2). In other words, the display device 3 according to the embodiment does not need to have the display 331, and may be realized as a position detection device (information processing device) that identifies (calculates) the position in the vehicle of the first user P1 as the driver based on the direction of the first voice of the first user P1, and outputs the position information to an external information processing device such as an external display.

[0059] Furthermore, for example, the display device 3 adjusts the directivity of voice acquisition (sound collection) by the microphone 321 (see FIG. 4 ) based on the position of the first user P1 who will be the driver. Note that the display device 3 according to the embodiment does not need to have the microphone 321, and may be realized as a voice acquisition assistance device (information processing device) that identifies the position in the vehicle of the first user P1 who will be the driver, and outputs position information and control information for adjusting the directivity to an external information processing device that controls the external microphone.

[0060] As shown in FIG. 1 , the display device 3 includes a control unit 31 , an acquisition unit 32 , an output unit 33 , a communication unit 34 , and a storage unit 35 .

[0061] The control unit 31 realizes each function of the display device 3 , including a voice separation unit 311 , a speaker identification unit 312 , and a direction calculation unit 313 .

[0062] The audio separation unit 311 performs audio separation processing on audio data acquired by the microphone 321 of the acquisition unit 32. The audio separation processing may be performed by any method. For example, the audio separation unit 311 separates the audio acquired by the multiple microphones 321 into audio for each sound source by linear filtering. The audio separation unit 311 may also separate the audio acquired by a single microphone 321 into audio for each sound source by non-linear filtering.

[0063] The speaker identification unit 312 performs a speaker identification process based on the voice data of each sound source separated by the voice separation unit 311. The speaker identification process may be performed by any method. For example, the speaker identification unit 312 compares the voice data of each sound source separated by the voice separation unit 311 with the voice data of a registered occupant (e.g., first user P1) stored in the memory unit 35, thereby identifying whether the acquired voice is the voice of the registered occupant.

[0064] The direction calculation unit 313 performs a sound direction calculation process to calculate the arrival direction of the target sound based on the sound data of the target sound source separated by the sound separation unit 311. Note that the sound direction calculation process may be performed by any method. For example, the direction calculation unit 313 may calculate the arrival direction of the target sound based on the time difference or sound pressure level between sounds acquired by the multiple microphones 321, or may calculate the arrival direction of the target sound based on the directivity of a single microphone 321 and the sound pressure level at that microphone 321.

[0065] The acquisition unit 32 has a plurality of microphones 321 and a camera 322 (see FIG. 4 ). The plurality of microphones 321 and the camera 322 may be components of the vehicle 1, that is, may be provided outside the display device 3.

[0066] The output unit 33 has a display 331 and a speaker 332 (see FIG. 4). The display 331 and the speaker 332 may be components of the vehicle 1, that is, may be provided outside the display device 3. As the display 331, a display device such as a liquid crystal display (LCD), an organic electroluminescence (EL) display, or a projector can be used as appropriate.

[0067] As an example, the display 331, under the control of the control unit 31, displays various screens such as a driver registration screen (see Figure 4), a driver position presentation screen (see Figure 6), and an operation screen during automatic driving (see Figure 9).

[0068] The communication unit 34 is a communication circuit that communicates with the outside of the display device 3. The communication unit 34 is a communication circuit that communicates with the outside of the display device 3. A communication circuit for wired or wireless communication can be used as the communication unit 34 as appropriate. A communication circuit for wireless communication can be used as appropriate as a communication circuit compatible with various standards such as 3G, 4G, 5G, 6G, Wi-Fi (registered trademark), Bluetooth (registered trademark), and infrared communication.

[0069] As an example, the communication unit 34 transmits to the driving control device 5 voice data of control (driving control) regarding the driving of the vehicle 1 from the first user P1 as the driver, or the content of speech (control content) based on the voice data.

[0070] As an example, the communication unit 34 transmits to the driving control device 5 the operation results of each user P1, P2 on the driving control (driving control) performed on the operation screen (see Figure 9) during automatic driving.

[0071] As an example, the communication unit 34 transmits voice data from each user P1, P2 regarding control (content control) of various contents provided in the vehicle 1 or speech content (control content) based on the voice data to the content control device 6. Note that the control content may be the result of an operation on the display screen.

[0072] As an example, the communication unit 34 receives data on various types of content to be provided in the vehicle 1 from the content control device 6. For example, the communication unit 34 receives data on driving camera video 411a (see FIG. 9 ) obtained by an onboard camera of the vehicle 1. For example, the communication unit 34 receives data on navigation (driving position) video 411b (see FIG. 9 ) that shows route guidance and the driving position of the vehicle 1.

[0073] The storage unit 35 is a storage medium or storage device that stores programs and parameters, data being processed, and data resulting from processing, related to each process executed by the display device 3.

[0074] As an example, the storage unit 35 stores voice information that stores voice data of a registered occupant (for example, the first user P1).

[0075] As an example, the storage unit 35 stores location information indicating the location where the display device 3 is located. The location of the display device 3 may be input by the user, or may be calculated based on the reception strength of radio waves from the display device 3 at an access point inside the vehicle. For example, when the location of the display device 3 is input by the user, the input screen may display the positions of mounting parts that are prepared in advance inside the vehicle and that detachably support the display device 3, so that the user can select them as possible locations for placement. The location information may also be information regarding the positional relationship with each seat in the vehicle.

[0076] As an example, the memory unit 35 stores seating layout information indicating the arrangement of multiple seats arranged inside the vehicle 1. The seating layout information is, for example, registered in advance. Note that, for example, in cases where the seating arrangement can be changed by the user, or where there are seats where the driver is not seated, such as when a child seat is secured, the seating layout information may be editable by the user or may be additionally registered as a new seating layout. In other words, the seating layout information is, for example, information indicating seats where the driver may possibly be seated, but may also include information indicating seats where the driver may not be seated. If information excluding some seats is included, the accuracy of acquiring the driver's voice data can be further improved.

[0077] Note that two or more components included in the display device 3 according to the above-described embodiment may be configured as an integrated unit. For example, two or more of the functions of the display device 3, including the voice separation unit 311, the speaker identification unit 312, and the direction calculation unit 313, may be integrated and realized as a single function. Furthermore, part of the functional configuration and hardware configuration of the display device 3 may be realized by an information processing device external to the display device 3.

[0078] For example, the voice separation unit 311, the speaker identification unit 312, and the direction calculation unit 313 may be realized by an information processing device external to the display device 3. In this case, the control unit 31 may perform the voice separation process, the speaker identification process, and the voice direction calculation process using a machine learning model in which parameters are determined so as to output a voice separation result, a speaker identification result, and voice direction information in response to input voice data. As this machine learning model, any machine learning model such as a CNN (Convolutional Neural Network) can be used as appropriate depending on the process. Of course, each process may be performed using the machine learning model in the display device 3.

[0079] The cruise control device 5 autonomously controls the driving of the vehicle, such as steering, acceleration / deceleration, and shifting (gear ratio), in accordance with control information from the display device 3 based on driving control voices and operations on the operation screen, or in accordance with the output of on-board sensors. Here, the control of the driving of the vehicle according to the embodiment is an example of control of the driving (movement) of a mobile object. The autonomous driving control by the cruise control device 5 is an example of automated driving. This automated driving may be autonomous driving with a dynamic surrounding environment that involves route planning, or may be route-memory type autonomous driving with a static surrounding environment, such as automated parking.

[0080] The content control device 6 controls various types of content provided in the vehicle 1. Specifically, the content control device 6 controls various in-vehicle infotainment (IVI) functions and in-vehicle entertainment functions, such as a car navigation function, a location information providing service function, an Internet connection function, and a multimedia playback function, in accordance with control information from the display device 3 based on control voices from the driver or passengers and operations on the operation screen.

[0081] Here, the various contents provided in the vehicle 1 include first content that realizes functions related to the driving of the vehicle 1. The functions realized by this first content are functions that affect or have a large impact on the driving of the vehicle 1, among various in-vehicle infotainment functions and in-vehicle entertainment functions. As an example, the function realized by the first content is a function related to specifying or changing a destination, waypoints, or route plan in a car navigation function. As an example, the function realized by the first content is a function related to disabling an internet connection function during autonomous driving.

[0082] Furthermore, the various contents provided in the vehicle 1 include second content that realizes functions unrelated to the driving of the vehicle 1. The functions realized by this second content are functions among various in-vehicle infotainment functions and in-vehicle entertainment functions that do not affect or have a small impact on the driving of the vehicle 1. As an example, the function realized by the second content is a car navigation function related to the display of route guidance, such as displaying information on planned routes and stopovers, estimated arrival times, and traffic congestion information. As an example, the function realized by the second content is an Internet connection function, such as a search function, a download function, or a connection function that provides data communication to a mobile device such as a passenger's smartphone. As an example, the function realized by the second content is a multimedia playback function, such as music playback or video playback.

[0083] Two or more devices included in the vehicle 1 according to the above-described embodiment may be integrated. Furthermore, part of the functional configuration and hardware configuration of each device of the vehicle 1 may be realized by an information processing device external to the vehicle 1.

[0084] 2 is a diagram showing an example of the hardware configuration of an information processing device 8 that realizes each device (the display device 3, the driving control device 5, and the content control device 6) mounted on the vehicle according to the first embodiment. Note that each device mounted on the vehicle 1, including the display device 3, the driving control device 5, and the content control device 6, may be realized by an information processing device 8 shared with other devices, or each device may be realized by a different information processing device 8. For example, the information processing device 8 that realizes the display device 3 may be configured integrally with a car navigation device.

[0085] The information processing device 8 is a device that comprehensively controls the operations of each device that it implements. The information processing device 8 may be implemented by a computer such as an ECU (Electronic Control Unit) provided inside the vehicle 1, a DCU (Domain Control Unit) such as a CDC (Cockpit Domain Controller) that integrates multiple ECUs, or an OBU (On Board Unit). Alternatively, the information processing device 8 may be an external computer installed near the dashboard of the vehicle 1.

[0086] 2, the information processing device 8 has a processor 81, a main storage device 82, an auxiliary storage device 83, and an I / F (interface) 84. The processor 81, the main storage device 82, the auxiliary storage device 83, and the I / F 84 are interconnected by a bus or the like, and have a hardware configuration using a typical computer. Note that each component of the information processing device 8 may be realized by a combination of two or more components.

[0087] The processor 81 is, for example, at least one CPU (Central Processing Unit). The processor 81 executes a program, for example, to comprehensively control the operation of the information processing device 8 and realize each function of the information processing device 8.

[0088] As an example, in an information processing device 8 that realizes the display device 3, a processor 81 loads a program stored in an auxiliary storage device 83 into a main storage device 82 and executes it, thereby realizing each function of the display device 3, including a voice separation unit 311, a speaker identification unit 312, and a direction calculation unit 313 illustrated in Figure 1.

[0089] 1 illustrates only the functions necessary for explaining the main parts of the present embodiment, but the functions of each device in the vehicle 1 are not limited to these. In addition, some or all of the functions of each device in the vehicle 1 may be realized by dedicated hardware circuits.

[0090] The processor 81 according to the embodiment is an example of at least one processor in the information processing device 8. As the at least one processor, at least one other processor may be used instead of or in addition to the CPU. As the other processor, various processors such as a CPU, a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), or a dedicated arithmetic circuit realized by an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array) can be used as appropriate.

[0091] The main storage device 82 is, for example, a RAM (Random Access Memory). The main storage device 82 temporarily stores data necessary for various processes performed by the processor 81. The main storage device 82 according to the embodiment is an example of at least one memory in the information processing device 8.

[0092] The auxiliary storage device 83 is, for example, a read-only memory (ROM). The auxiliary storage device 83 stores programs, parameters, and the like that realize various processes performed by the processor 81. The auxiliary storage device 83 according to the embodiment is an example of at least one memory in the information processing device 8. Note that, as the auxiliary storage device 83, various storage media and storage devices such as a hard disk drive (HDD), a solid state drive (SSD), and flash memory can be used as appropriate instead of or in addition to the ROM.

[0093] As an example, in the information processing device 8 that realizes the display device 3 , the main storage device 82 and the auxiliary storage device 83 realize the storage unit 35 .

[0094] The I / F 84 is an interface for implementing input / output functions, connecting external devices, and / or communicating with the outside. The I / F 84 may be an output interface for connecting or implementing an output device that outputs audio, images, or video, an input interface for connecting or implementing an input device that acquires user input, or an interface that functions as one of these devices. As output devices, various displays such as LCD displays, organic EL displays, and projectors, as well as speakers, may be used as appropriate. As input devices, keyboards, mice, touch panels, microphones, and the like may be used as appropriate. The I / F 84 may transmit and receive information to and from other ECUs installed in the vehicle 1 via an in-vehicle network within the vehicle 1, or may communicate with an information processing device outside the vehicle 1 via a network such as the Internet.

[0095] As an example, in an information processing device 8 that realizes a display device 3, the I / F 84 realizes an input interface (acquisition unit 32) that connects or realizes an input device, an output interface (output unit 33) that connects or realizes an output device, and a communication interface (communication unit 34) for communicating with the outside.

[0096] An example of the operation of the vehicle 1 according to the embodiment will be described below with reference to the drawings. Note that the processing described below is an example, and it is possible to change the processing order, delete some processing, or add other processing.

[0097] 3 is a flowchart showing an example of the flow of a driver registration process executed in the display device 3 according to the first embodiment. The flow in FIG. 3 is executed, for example, before the start of autonomous driving (before autonomous driving).

[0098] First, the control unit 31 of the display device 3 acquires data of the driver's voice (S101).

[0099] As an example, the control unit 31 displays a driver registration screen on the display 331 of the output unit 33. FIG. 4 is a diagram showing an example of the driver registration screen displayed on the display device 3 according to the first embodiment. The driver registration screen includes a seating layout display showing the arrangement of the plurality of seats 103a to 103f provided in the vehicle 1. In the example of FIG. 4, a seating layout is illustrated in which each of the plurality of seats 103a to 103f is arranged along a side window 101 of the vehicle 1, with the occupant sitting facing the opposing window 101. Specifically, in the example of FIG. 4, from the front window 101a to the rear window 101f of the vehicle 1, the seats 103a to 103c are arranged on the side of the left windows 101b and 101c of the vehicle 1, and the seats 103d to 103f are arranged on the side of the right windows 101d and 101e. That is, the seats 103a to 103c are seats where the occupant faces the right windows 101d and 101e, respectively, and the seats 103d to 103f are seats where the occupant faces the left windows 101b and 101c, respectively.

[0100] The display 331 that displays the driver's registration screen is placed on the side of the front window 101a as shown in FIG. 4, but it may also be placed in the center of the vehicle between the seats 103a to 103c or on the side of the rear window 101f.

[0101] For example, a passenger (user) to be registered as a driver presses the setting button 401 to start the registration process and speaks into the microphone 321 of the acquisition unit 32. At this time, the user is seated in the seat 103 to be used as the driver's seat. The display device 3 is placed at a position stored in the storage unit 35 as placement information. Then, the control unit 31 acquires data of the driver's voice acquired by the microphone 321, for example, in response to the setting button 401 being pressed. Note that acquisition of the driver's voice data may be triggered by input of a predetermined voice.

[0102] The control unit 31 also stores the acquired driver voice data as driver voice information in the storage unit 35, thereby registering the occupant (user) who spoke as the driver (S102).

[0103] Thereafter, the control unit 31 executes a voice direction calculation process (S103) for the acquired driver voice data.

[0104] FIG. 5 is a flowchart showing an example of the flow of a voice direction calculation process executed in the display device 3 according to the first embodiment. In the voice direction calculation process for acquired driver voice data, the control unit 31 calculates the arrival angle (voice direction) of the acquired driver voice (S201). Then, the control unit 31 adjusts the directivity of voice acquisition by the microphone 321 to match the calculated voice direction of the driver voice (S202). This adjustment includes, for example, increasing the sensitivity (gain) with respect to the calculated voice direction. Alternatively, this adjustment includes, for example, decreasing the sensitivity (gain) with respect to directions other than the calculated voice direction. Note that, if the microphone 321 is movable, this adjustment may also be a change to deflect the microphone 321 to match the calculated voice direction of the driver voice. After adjusting the microphone directivity, the flow in FIG. 5 ends.

[0105] After the voice direction calculation process for the acquired driver voice data is completed, the control unit 31 displays the driver's position based on the calculated voice direction (S104). Here, the display of the driver's position is an example of outputting position information indicating the recognized driver's position. As an example, the control unit 31 identifies the recognized driver's seating position based on the voice direction calculated using the driver voice data, the arrangement information of the display device 3, and the seat layout information.

[0106] 6 is a diagram showing an example of a driver position presentation screen displayed on the display device 3 according to the first embodiment. As shown in FIG. 6, the driver position presentation screen presents the seating position in the seating layout of the recognized driver, a first user P1, to a user in the vehicle cabin. Specifically, the presentation screen in FIG. 6 is a screen presenting that the first user P1, who is seated in seat 103b among the multiple seats 103a to 103f arranged in the vehicle cabin of the vehicle 1, is recognized as the driver.

[0107] After the driver's position is displayed, the flow of FIG. 3 ends.

[0108] In this way, the display device 3 of this embodiment outputs position information indicating the seating position of the recognized driver in the seating layout of the vehicle 1 based on the direction of the driver voice for registration acquired during the driver registration process executed before autonomous driving.

[0109] 7 is a flowchart showing an example of the flow of a vehicle control process executed in the display device 3 according to the first embodiment. The flow in FIG. 7 is executed, for example, after the start of autonomous driving, at a predetermined interval or each time an utterance is detected.

[0110] First, the control unit 31 executes a voice reception process (S301).

[0111] FIG. 8 is a flowchart showing an example of the flow of a voice reception process executed in the display device 3 according to the first embodiment. The control unit 31 acquires data of in-vehicle voices collected by the microphone 321 of the acquisition unit 32 (S401). The control unit 31 performs a voice separation process on the acquired in-vehicle voice data to separate the in-vehicle voice into voices for each sound source (S402). The control unit 31 then determines whether any of the separated in-vehicle voices (second voices) is a voice registered as the driver (S403). If the voice is a registered voice (S403: Yes), the control unit 31 executes a voice direction calculation process (S103) for the driver voice separated from the in-vehicle voice. On the other hand, if the voice is not a registered voice (S403: No), the processes in FIGS. 7 and 8 end.

[0112] After calculating the voice direction of the driver's voice included in the acquired in-vehicle voice, the control unit 31 determines whether the calculated voice direction matches the voice direction of the driver's voice when the driver was registered (S302). This determination is made to determine whether the driver's position calculated based on the direction of the driver's voice included in the acquired in-vehicle voice matches the driver's position displayed on the display 331.

[0113] If the voice directions match (S302: Yes), the control unit 31 transmits control information based on the driver's voice included in the acquired in-vehicle voice (second voice) to the driving control device 5 or the content control device 6 depending on the control target (S303). In other words, the control information transmitted in the processing of this step is information for controlling the driving of the vehicle 1 or the first content or the second content.

[0114] On the other hand, if the voice directions do not match (S302: No), the control unit 31 displays confirmation information on the display 331 notifying the driver that the driver's location is different from that at the time of registration (S304). The display of this confirmation information is an example of outputting a confirmation message regarding the driver's location. The confirmation information may also be output through a speaker. Thereafter, the control unit 31 transmits control information based on the acquired in-vehicle voice (second voice) to the content control device 6 (S303). Note that in the processing of this step, control information for controlling the traveling of the vehicle 1 is not transmitted. In other words, the control information transmitted in the processing of this step is only information for controlling the second content.

[0115] Thus, in the vehicle control process executed during autonomous driving, if the voice direction matches between the acquired in-vehicle voice (second voice) and the registered driver voice, the display device 3 according to this embodiment controls the driving of the vehicle 1 or controls the first content or the second content provided by the vehicle 1, based on the driver voice included in the acquired in-vehicle voice. On the other hand, if the voice direction does not match, the display device 3 according to this embodiment presents the user with confirmation information regarding the driver's position, and enables control of the vehicle 1 based on the acquired in-vehicle voice (second voice) to control the second content provided by the vehicle 1, but disables control of the driving of the vehicle 1 and control of the first content.

[0116] 8, if it is determined that the voice is not a registered voice, the control unit 31 may execute the process of S303 in the same manner as when it is determined that the voice direction does not match in the process of S302. In other words, the control unit 31 may enable control of the vehicle 1 based on unregistered voice included in the acquired in-vehicle voice (second voice) by enabling control of the second content provided in the vehicle 1 and disabling control of the vehicle 1's running and control of the first content.

[0117] FIG. 9 is a diagram showing an example of an operation screen during automatic driving, which is displayed on the display device 3 according to the first embodiment.

[0118] 9 , the operation screen during autonomous driving includes an image display area 411 for displaying a driving camera image 411a and a navigation (driving position) image 411b. Here, the driving camera image 411a is, for example, an image of the surroundings of the vehicle 1 acquired by at least one in-vehicle camera (not shown) of the vehicle 1. The navigation (driving position) image 411b is, for example, an image provided by the navigation function of the vehicle 1, including a display of the current position of the vehicle 1 and route guidance (planned route).

[0119] The operation screen during autonomous driving illustrated in FIG. 9 also includes driving operation buttons 421 for controlling the driving of the vehicle 1. In the example of FIG. 9, the driving operation buttons 421 include a brake (stop) button 421a, an accelerator (start) button 421b, steering buttons 421c and 421d, and driving direction buttons 423e and 423f. The brake (stop) button 421a is a driving operation button 421 for instructing the vehicle 1 to brake. The accelerator (start) button 421b is a driving operation button 421 for instructing the vehicle 1 to accelerate. The steering button 421c is a driving operation button 421 for instructing the vehicle to steer left. The steering button 421d is a driving operation button 421 for instructing the vehicle to steer right. The driving direction button 423e is a driving operation button 421 for instructing a shift for forward driving. The driving direction button 423f is a driving operation button 421 for instructing a shift for reverse driving.

[0120] For example, the driver of the vehicle 1, while watching the image displayed in the image display area 411, speaks (voice-based driving control) to control the driving of the vehicle 1. At this time, the control unit 31 transmits control information for controlling the driving of the vehicle 1 to the driving control device 5 in accordance with the driver's voice included in the in-vehicle voice, as described above with reference to Figures 7 and 8.

[0121] For example, while viewing the image displayed in the image display area 411, the occupant of the vehicle 1 operates the driving operation button 421 (non-audio driving control) to control the driving of the vehicle 1. At this time, the control unit 31 transmits control information to the driving control device 5 to control the driving of the vehicle 1 in accordance with the result of the operation of the driving operation button 421.

[0122] As described above, the information processing of this embodiment outputs location information indicating which of the multiple seats 103 arranged in the passenger compartment (inside the vehicle) of the vehicle 1 the first user P1 is in, based on the direction of the first voice of the first user P1 who is registered as the user controlling the vehicle 1.

[0123] Generally, when voice-controlled driving is used to operate an autonomous vehicle, it is necessary to properly recognize the driver's voice and operate the vehicle in accordance with the driver's voice alone. Conventionally, the seating arrangement and the driver's seat in an automobile have been fixed. Therefore, it has been easy to obtain the driver's voice data because the driver's position can be determined by detecting whether the driver is sitting in the driver's seat.

[0124] In this context, as autonomous driving of automobiles progresses, it is expected that the arrangement of seats (seat layout) inside the automobile cabin will be freely set. For example, as autonomous driving becomes more advanced, seat layouts in which occupants do not necessarily face the direction of travel of the automobile are also expected. Furthermore, as autonomous driving becomes more advanced, for example, if vehicle operation related to driving can be controlled by voice, seat layouts in which the driver's seat is not fixed will also be expected, allowing the driver to freely choose where to sit. Furthermore, for example, in a seat layout in which the driver's seat is not fixed, it is expected that operating devices such as a steering wheel, accelerator pedal, and brake pedal will not be located, or that the driver will sit in a seat where no operating devices are located.

[0125] However, in seating layouts where the driver's seat is not fixed, there is a problem in that even if seating detection is performed, it is not possible to determine which seat is the driver.If the driver's position in the vehicle cannot be determined, the accuracy of sound collection decreases, and there is a risk that the driver's voice will not be properly recognized, resulting in incorrect driving control, leaving room for improvement in terms of safety in voice-based driving control.

[0126] Furthermore, from the perspective of the vehicle occupants, it was unclear which seat the autonomous driving control device recognized as the driver, leaving room for improvement in terms of their sense of security and convenience.

[0127] On the other hand, according to the information processing of this embodiment, for example, before autonomous driving, the position of the driver can be determined based on the direction of the voice of the first user P1 who will be the driver.

[0128] Furthermore, the information processing according to this embodiment displays the identified position of the driver in the seating layout on the display 331. This allows the occupants of the vehicle 1 to easily understand where the occupant is recognized as the driver by the vehicle 1. In other words, the information processing according to this embodiment can provide the occupants of the vehicle 1 with a sense of security and high convenience.

[0129] Furthermore, the information processing according to the present embodiment adjusts the directionality of voice acquisition by the microphone 321 based on the determined driver's position. This allows for sound collection according to the position of the first user P1 who will be the driver, improves the accuracy of voice acquisition, and improves safety in voice-based driving control.

[0130] In the present embodiment, the information processing is exemplified as determining whether the voice directions match and performing driving control and / or content control based on the determination result, but is not limited to this. For example, the information processing according to the embodiment may be configured to perform driving control if it is determined that the speaker is the driver based on the speaker identification result.

[0131] As an example, the speaker identification unit 312 may identify whether the acquired voice is the voice of a registered passenger (driver) by comparing the voice data of the occupant included in the in-vehicle voice (second voice) acquired in the process of S301 of FIG. 7 with the voice data of the driver voice acquired in the process of S101 of FIG. 3 and registered in the process of S102. Furthermore, when the acquired voice is the voice of the registered driver, the control unit 31 may transmit control information based on the acquired driver voice to the driving control device 5 or the content control device 6 depending on the control target, in the same way as when the voice direction matches (S302: Yes in FIG. 7). On the other hand, when the acquired voice is not the voice of the registered driver, the control unit 31 may display confirmation information on the display 331 or transmit control information based on the acquired in-vehicle voice (second voice) to the content control device 6, in the same way as when the voice direction does not match (S302: No in FIG. 7).

[0132] In the present embodiment, the information processing is exemplified as outputting position information indicating which of the multiple seats 103 arranged in the cabin (inside) of the vehicle 1 the first user P1 is sitting in. However, the present invention is not limited to this. In other words, the processing for outputting position information is not a required configuration and may be omitted.

[0133] Other embodiments of the present disclosure will be described below with reference to the drawings. In the following description of each embodiment, differences will be mainly described, and descriptions of contents that overlap with the above-described contents will be omitted as appropriate.

[0134] Second Embodiment In the information processing according to the above-described embodiment, the adjustment of microphone directivity and the display of the driver's position may be updated in accordance with the voice direction of the driver's voice.

[0135] 10 is a flowchart showing an example of the flow of a location display update process executed in the display device 3 according to the second embodiment. The flow in FIG. 10 is executed, for example, before or after the start of autonomous driving, at a predetermined interval or each time an utterance is detected. Here, differences from the flow in FIG. 7 will be mainly described.

[0136] After executing the voice reception process (S301), the control unit 31 adjusts the directivity of voice acquisition by the microphone 321 to match the voice direction calculated for the driver voice included in the in-vehicle voice (second voice) (S501), in the same manner as the process of S202 in Fig. 5. Thereafter, the control unit 31 determines whether the voice direction of the driver voice included in the newly acquired in-vehicle voice matches the voice direction of the driver voice used to identify the driver position displayed on the display 331 (S302).

[0137] If the voice direction does not match (S302: No), the control unit 31 identifies the current driver's position using the voice direction of the driver's voice contained in the in-vehicle voice, similar to the processing of S104 in Figure 3, and updates the display of the driver's position (S502).

[0138] 11 is a diagram showing an example of a driver position display screen displayed on the display device according to the second embodiment. Fig. 11 illustrates a case where a first user P1 (driver) who was seated in seat 103e when the driver was registered or when the previous location display update process was performed has moved to seat 103a. In this case, the control unit 31 displays, on the display 331, a display screen indicating that the first user P1 seated in seat 103a is recognized as the driver.

[0139] In addition, the control unit 31 may indicate on the presentation screen that the user recognized as the driver has been changed from the first user P1 seated in seat 103e to the first user P1 seated in seat 103a.

[0140] As an example, the presentation screen may display the seating position of the user who was recognized as the driver before the update, as illustrated in Fig. 11. In this case, the display mode of the first user P1 before and after the update, such as color, darkness, or shape, may be different.

[0141] As an example, the presentation screen may display an icon such as an arrow indicating movement, as illustrated in FIG.

[0142] 11, the control unit 31 may display the position of a second user P2, who is another passenger (co-passenger) of the driver, in addition to the first user P1 who is recognized as the driver, on the presentation screen in the same manner as the driver (first user P1). In this case, the first user P1 and the second user P2 may be displayed in different ways, such as in color, darkness, or shape.

[0143] After the display of the driver's position is updated, or if the voice direction matches (S302: Yes), the flow of FIG. 10 ends.

[0144] In this way, the information processing according to this embodiment adjusts the directivity of the microphone 321 and changes the position indication on the display 331 in accordance with the position based on the direction of the driver's voice contained in the in-vehicle voice (second voice), for example, after the start of autonomous driving. This allows the position of the driver who controls driving by voice to be properly grasped even if the driver of the vehicle 1 moves his / her seat within the vehicle. Furthermore, the user can easily understand that the seat movement has been recognized.

[0145] Third Embodiment In the information processing according to the above-described embodiments, adjustment of microphone directivity and display of the driver's position may be updated in response to an input operation by a passenger.

[0146] 12 is a flowchart showing an example of the flow of a location display update process executed in the display device 3 according to the third embodiment. The flow in FIG. 12 is executed, for example, before the start of autonomous driving. Here, differences from the flow in FIG. 3 will be mainly described.

[0147] The control unit 31 acquires movement information indicating seat movement (S601).

[0148] As an example, movement information indicating seat movement is acquired by receiving an input operation by the occupant on the display device 3 .

[0149] For example, the control unit 31 may receive voice input of movement information indicating the movement of the occupant's seat. For example, the control unit 31 may acquire the movement information indicating the movement of the seat by acquiring data of a voice (third voice) that is picked up by the microphone 321 and indicates that the driver's seat has moved.

[0150] For example, the control unit 31 may display an input screen for movement information or an input field on each of the display screens described above. For example, the control unit 31 may display a seating layout that allows the user to select the seat 103 after movement. For example, the control unit 31 may display a seating layout that allows the user to select the seat 103 after movement by dragging and dropping an icon representing the first user P1 to the destination seat 103. In these cases, for example, the control unit 31 may acquire the position of the first user P1 displayed at that time as the seat after movement when the confirm button 402 is pressed. Note that the display device 3 may be provided with a physical button for inputting movement information that realizes the operation of selecting the seat 103.

[0151] As an example, movement information indicating seat movement is acquired by the camera 322 of the display device 3. For example, the control unit 31 may detect the movement of the driver by performing image processing on the image of the interior of the vehicle acquired by the camera 322. Any processing may be used as appropriate for this image processing, such as processing for person detection such as face recognition or pupil detection, or processing for object detection. Note that a camera external to the display device 3, such as a camera mounted on the vehicle 1, may be used as the camera for acquiring the movement information.

[0152] The following description will be given taking as an example a case where the driver, seated in the seat 103 after the seat has been moved, inputs the movement information indicating the seat movement by voice.

[0153] The control unit 31 executes a voice direction calculation process (S103) for the acquired data of the voice of the movement information (third voice). Thereafter, the control unit 31 identifies the current driver's position using the voice direction of the voice of the movement information, similar to the process of S104, and updates the display of the driver's position (S502). At this time, the control unit 31 may adjust the directivity of voice acquisition by the microphone 321 to match the voice direction of the voice of the movement information, similar to the process of S202 in FIG. 5.

[0154] In this way, the information processing according to this embodiment adjusts the directivity of the microphone 321 and changes the position display on the display 331 based on movement information indicating seat movement, for example, before autonomous driving. This allows the position of the driver who controls driving by voice to be properly determined even if the driver is seated in a seat different from that at the time of registering voice information or the previous time of riding. Furthermore, the user can easily understand that they are recognized as the driver in their current seating position. Note that the technology according to this embodiment can be appropriately applied to the vehicle 1 according to each of the above-mentioned embodiments.

[0155] Fourth Embodiment In the information processing according to the above-described embodiments, the driver may be able to be changed after the start of autonomous driving.

[0156] Fig. 13 is a flowchart showing an example of the flow of driver change processing executed in the display device 3 according to the fourth embodiment. The flow in Fig. 13 is executed, for example, after the start of autonomous driving, when the driver change button 403 is pressed. Here, differences from the flow in Fig. 3 will mainly be described.

[0157] The control unit 31 acquires driver change information indicating a driver change (S701).

[0158] The driver change information indicating a driver change may be obtained by voice input, as with the movement information in the third embodiment, or may be obtained based on the results of operations on the screen or physical buttons of the display device 3, or may be obtained by a camera on the display device 3 or vehicle 1.

[0159] After the driver change information indicating the driver change is acquired, the control unit 31 executes the driver registration process of FIG. 3 regarding the voice (fourth voice) of the new driver (S702).

[0160] In addition, when the driver after the changeover, while seated in seat 103, makes a voice input of changeover information indicating a driver change, the control unit 31 may acquire the acquired voice data of the changeover information (fourth voice) as the driver's voice (first voice).

[0161] In this way, the information processing according to this embodiment adjusts the directivity of the microphone 321 and displays the position of the new driver on the display 331, for example, after the start of autonomous driving, based on driver change information indicating a change in driver. This allows the same effect as in the first embodiment to be obtained even when the driver is changed. Note that the technology according to this embodiment can be appropriately applied to the vehicle 1 according to each of the above-described embodiments.

[0162] In each of the above-described embodiments, "is it A?" refers to at least one of "is A" and "is not A." In other words, in each of the above-described embodiments, the determination of "is A" may be realized by determining "is A," or by determining "is not A," or by determining both of these.

[0163] The programs executed by each device of the vehicle 1 in each of the above-described embodiments may be provided by being recorded in an installable or executable file format on a computer-readable recording medium (Computer Program Product) such as a CD-ROM, FD, CD-R, or DVD.

[0164] The programs executed by the devices of the vehicle 1 in each of the above-described embodiments may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. The programs executed by the devices of the vehicle 1 in each of the above-described embodiments may be provided or distributed via a network such as the Internet.

[0165] Furthermore, the programs executed by the devices of the vehicle 1 in each of the above-described embodiments may be configured to be provided in a state where they are pre-installed in a ROM or the like.

[0166] According to at least one of the embodiments described above, in a seating layout in which the driver's seat is not fixed, it is possible to grasp the position of the driver who controls driving by voice.

[0167] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, as well as within the scope of the invention described in the claims and their equivalents.

[0168] 1 Vehicle 101 Window 103 Seat 3 Display device 31 Control unit 311 Voice separation unit 312 Speaker identification unit 313 Direction calculation unit 32 Acquisition unit 321 Microphone 322 Camera 33 Output unit 331 Display 332 Speaker 34 Communication unit 35 Memory unit 401 Setting button 402 Confirmation button 403 Driver change button 411 Image display area 411a Driving camera image 411b Navigation (driving position) image 421 Driving operation button 421a Brake (stop) button 421b Accelerator (start) button 421c, 421d Steering button 423e, 423f Driving direction button 5 Driving control device 6 Content control device 81 Processor 82 Main memory device 83 Auxiliary memory device 84 I / F P1 First user (driver) P2 Second user (passenger)

Claims

1. An information processing method executed by at least one processor in an information processing device having the processor, the information processing method comprising: acquiring first voice data of a first user to be registered as a user controlling a mobile object, the first voice data being acquired by multiple microphones; and outputting location information indicating which of multiple seats arranged in the passenger space of the mobile object the first user is sitting in, based on the direction of the first voice.

2. The information processing method of claim 1, further comprising: acquiring data on a second voice of a user in the passenger space of the vehicle acquired by the plurality of microphones; determining whether the second voice is the voice of the first user; and, if the second voice is the voice of the first user, determining whether the position of the first user calculated based on the first voice indicated by the output position information matches the position of the first user calculated based on the direction of the second voice.

3. An information processing method as described in claim 2, wherein, when the position of the first user calculated based on the first voice indicated by the output position information matches the position of the first user calculated based on the direction of the second voice, the moving body is controlled based on the second voice.

4. An information processing method executed by at least one processor in an information processing device having the at least one processor, comprising: acquiring first voice data of a first user to be registered as a user controlling a moving object, acquired by a plurality of microphones; acquiring second voice data of a user in a passenger space of the moving object, acquired by the plurality of microphones; determining whether the second voice is the voice of the first user; if the second voice is the voice of the first user, determining whether the position of the first user calculated based on the direction of the first voice, which is one of a plurality of seats arranged in the passenger space of the moving object, matches the position of the first user calculated based on the direction of the second voice; and if the position of the first user based on the first voice matches the position of the first user based on the direction of the second voice, controlling the moving object based on the second voice.

5. The information processing method according to claim 3 or 4, wherein the control of the mobile object is control related to the operation of the mobile object.

6. An information processing method as described in claim 3 or claim 4, wherein the control of the mobile body is control regarding content provided in the mobile body, and the content includes first content that realizes a function related to driving the mobile body and second content that realizes a function not related to driving the mobile body.

7. An information processing method as described in claim 2 or claim 4, wherein if the position of the first user calculated based on the first voice does not match the position of the first user calculated based on the direction of the second voice, a confirmation message regarding the position of the first user is output.

8. An information processing method as described in claim 2 or claim 4, wherein, when the position of the first user calculated based on the first voice does not match the position of the first user calculated based on the direction of the second voice, the control of the mobile body based on the second voice is made inoperable, including control related to driving the mobile body and control related to first content provided by the mobile body that realizes a function related to driving the mobile body, and is made operable, and control related to second content provided by the mobile body that realizes a function not related to driving the mobile body is made operable.

9. The information processing method according to claim 1 or claim 4, further comprising adjusting the directionality of the sound acquisition by the plurality of microphones in accordance with the sound direction of the first sound.

10. An information processing method as described in claim 2 or claim 4, wherein, when the position of the first user calculated based on the first sound does not match the position of the first user calculated based on the direction of the second sound, the directionality of sound acquisition by the multiple microphones is adjusted to match the sound direction of the second sound.

11. An information processing method as described in claim 2 or claim 4, wherein, if the position of the first user calculated based on the first voice does not match the position of the first user calculated based on the direction of the second voice, position information indicating which of a plurality of seats arranged in the passenger space of the moving body the first user is sitting in is output based on the direction of the second voice.

12. The information processing method of claim 11, further comprising, when the position information based on the second audio is being output, acquiring data of the second audio newly acquired by the plurality of microphones, and determining whether the position of the first user indicated by the output position information matches the position of the first user calculated based on the direction of the newly acquired second audio.

13. An information processing method as described in claim 1 or claim 4, which acquires third audio data relating to the first user's seat movement within the passenger space of the mobile body acquired by the multiple microphones, and outputs position information indicating which seat of multiple seats arranged within the passenger space of the mobile body the first user is sitting in based on the direction of the third audio.

14. The information processing method according to claim 1, further comprising outputting location information indicating the location of the first user calculated based on the first voice before the start of autonomous driving of the mobile body.

15. An information processing method as described in claim 2 or claim 4, wherein after the start of autonomous driving of the moving body, it is determined whether the position of the first user calculated based on the first voice matches the position of the first user calculated based on the direction of the second voice.

16. An information processing method as described in claim 1 or claim 4, further comprising: acquiring fourth voice data regarding a change of driver of the mobile body of a second user other than the first user, who is registered as a user controlling the mobile body captured by the plurality of microphones after the start of automatic driving of the mobile body; and outputting location information indicating which of a plurality of seats arranged in the passenger space of the mobile body the second user is sitting in based on the direction of the fourth voice.

17. An information processing method as described in claim 2 or claim 4, wherein, if the second voice is not the voice of the first user, the control of the mobile body based on the second voice is made inoperable, including control related to the driving of the mobile body and control related to first content provided by the mobile body that realizes functions related to the driving of the mobile body, and is made operable, and control related to second content provided by the mobile body that realizes functions not related to the driving of the mobile body is made operable.

18. The information processing method described in claim 1 or claim 4, wherein the information processing device further comprises a display, and by outputting position information indicating the position of the first user calculated based on at least the first voice to the display, a seating layout indicating the arrangement of the multiple seats within the passenger space of the mobile body and the position in the seating layout of the user controlling the mobile body are displayed on the display.

19. An information processing device comprising at least one processor configured to acquire data of a first voice of a first user to be registered as a user controlling a mobile object, acquired by multiple microphones, and output location information indicating which of multiple seats arranged in the passenger space of the mobile object the first user is sitting in based on the direction of the first voice.

20. A program for causing a computer to execute the following steps: acquire first voice data of a first user to be registered as a user controlling a mobile object, acquired by multiple microphones; and output location information indicating which of multiple seats arranged in the passenger space of the mobile object the first user is sitting in, based on the direction of the first voice.

Images