Control device, control method, and control program

Abstract

To provide a control device, a control method, and a control program for sharing a voice among a plurality of users without an input operation.SOLUTION: In a voice output system 100, a control device 1 includes: a detection part 13 for detecting a prescribed behavior of a user existing in an output destination space of a directional speaker 3; a specification part 14 for specifying a sound source of interest out of sound sources 2 and directional speakers to be controlled in accordance with the prescribed behavior; and an output control part 15 for outputting a voice emitted from the specified sound source from each of the specified directional speakers.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a control device that causes a sound generated by a sound source to be output from a sound output device. [Background technology]

[0002] There is a technology that allows multiple users present in a given space to hear different sounds by outputting sounds emitted from different sound sources from multiple sound output devices provided in the given space. One example of the technology is a technology in which directional speakers are provided in each seat in a vehicle, and each passenger sitting in each seat hears a sound emitted from a different sound source.

[0003] Based on such a technique, Patent Document 1 discloses a technique for sharing one sound source among multiple audio output devices. Specifically, Patent Document 1 discloses a technique for selecting multiple seats in a vehicle by operating a graphical user interface displayed on a touch panel, and sharing one sound source among speakers provided in each of the multiple seats. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2009-147814 A Summary of the Invention [Problem to be solved by the invention]

[0005] The technology of Patent Document 1 requires input operations into a graphical user interface to share audio sources. For users who are accustomed to sharing video, for example, by bringing a smartphone screen close to the other person, performing the above input operations each time is cumbersome.

[0006] Furthermore, in the technique of Patent Document 1, a screen for displaying the above-mentioned graphical user interface must be provided.

[0007] An object of one aspect of the present invention is to enable a plurality of users to share audio without performing an input operation. [Means for solving the problem]

[0008] In order to solve the above problems, the control device of aspect 1 of the present invention is a control device that outputs sound emitted by a sound source from a plurality of sound output devices, wherein the sound source can be set for each of the sound output devices, and includes a detection unit that detects predetermined words and actions of a user present in the output destination space of the sound output device, an identification unit that identifies a sound source of interest among the sound sources and the sound output device to be controlled in accordance with the detected predetermined words and actions, and an output control unit that outputs the sound emitted by the identified sound source from each of the identified sound output devices.

[0009] According to the above configuration, a sound source of interest and an audio output device to be controlled are identified according to a predetermined behavior performed by a user, and the sound emitted by the sound source is output from the audio output device. In other words, when a user performs a predetermined behavior, the sound emitted from the sound source of interest is output from the audio output device to which the sound source is set and the audio output device to be controlled. This allows a user to share the sound emitted from the sound source with others by a simple method of simply performing a predetermined behavior.

[0010] A control device according to a second aspect of the present invention is based on the above-mentioned first aspect, and the detection section may detect a predetermined action of the user as the predetermined speech or behavior from an image captured by an imaging device.

[0011] According to the above configuration, a predetermined action of a user is captured and the predetermined action is detected as a predetermined speech or behavior. This allows the user to share the sound emitted from the sound source with others just by performing the predetermined action. For example, the sound emitted from the sound source can be shared with others by a simple gesture that is predetermined as the predetermined action.

[0012] A control device according to a third aspect of the present invention is in the above-mentioned second aspect, wherein the detection section detects the predetermined action based on a movement of a body part of the user included in the image.

[0013] According to the above configuration, a predetermined action is detected from the movement of an imaged body part of the user, so that the user can share the sound that he or she is listening to with others by moving a predetermined part of his or her body. Here, the movement of a body part includes, for example, the movement of the arms and hands, as well as the movement of the eyes (in other words, the movement of the gaze).

[0014] A control device according to aspect 4 of the present invention, in aspect 3 above, may be configured such that the movement of the body part includes a movement of moving the position of an information terminal operating as a sound source along a predetermined path, and the identification unit may identify the information terminal as the sound source of interest.

[0015] According to the above configuration, the sound emitted by the sound source is shared among multiple users by moving the position of the information terminal operating as the sound source along a predetermined path. The predetermined path does not need to be complicated, and typically, a simple path from the sharing source user to the sharing destination user is sufficient. Therefore, the user can share the sound emitted by the sound source with others by a simple operation.

[0016] A control device according to aspect 5 of the present invention, in aspect 4 above, is such that the specified route is a route that leads from a user present in the output destination space of an audio output device in which the information terminal is set as a sound source to a user present in the output destination space of an audio output device in which the information terminal is not set as a sound source, and the identification unit may identify an audio output device in which the information terminal is not set as a sound source as the control target.

[0017] According to the above configuration, by moving an information terminal operating as a sound source closer to another person, the sound emitted by the sound source is output from the sound output device that allows the other person to hear the sound, thereby realizing sound sharing.

[0018] A control device according to aspect 6 of the present invention is such that, in aspect 5 above, the detection unit further detects the line of sight of a user to which the information terminal is brought close, and the identification unit identifies the sound source of interest and the control object if the line of sight is directed toward the information terminal, and does not need to identify the sound source of interest and the control object if the line of sight is not directed toward the information terminal.

[0019] According to the above configuration, when another person looks at the approaching information terminal, the audio output device that allows the other person to listen to the audio outputs the audio emitted by the audio source, thereby realizing audio sharing. This allows audio sharing to be realized through a series of actions that are usually performed when the other person agrees to audio sharing proposed by the user, so the user can naturally realize audio sharing. Furthermore, audio sharing will not be performed if the other person does not look at the information terminal, so audio sharing that the other person does not want can be prevented.

[0020] A control device according to aspect 7 of the present invention, in any one of aspects 1 to 4 above, may be configured such that the detection unit detects a predetermined utterance uttered by the user as the predetermined speech or action from a voice input to a voice input device.

[0021] According to the above configuration, a predetermined utterance of a user is acquired, and the predetermined utterance is detected as a predetermined speech and behavior. This allows the user to share the sound emitted from the sound source with others just by making the predetermined utterance. For example, by making an utterance including a simple term determined in advance as the predetermined utterance, the sound emitted from the sound source can be shared with others.

[0022] A control device according to aspect 8 of the present invention is such that, in aspect 7 above, the predetermined utterance includes an indication of providing a sound source, and the detection unit further detects behavior or words by a second user different from the user who uttered the predetermined utterance after the predetermined utterance is uttered, and if the behavior or words are positive, the identification unit identifies the sound source of audio being output into a space in which the user who uttered the predetermined utterance is present as the sound source of interest, and identifies at least an audio output device that outputs audio into the space in which the second user is present as the object to be controlled, and if the behavior or words are negative, does not identify the sound source of interest and the object to be controlled.

[0023] According to the above configuration, if a user makes a speech to provide a sound source, i.e., to offer to share sound, and the second user makes a positive speech and deed, sound sharing between the two users is realized. This allows the two users to share the sound emitted from a sound source in an exchange that occurs when sharing sound emitted from the sound source. Also, if the second user makes a negative speech and deed, sound sharing is not performed, so that sound sharing that the second user does not want can be prevented.

[0024] A control device according to aspect 9 of the present invention is such that, in aspect 7 or 8 above, the specified utterance includes a desire to enjoy a sound source, and the detection unit further detects behavior or words of a third user other than the user who uttered the specified utterance after the specified utterance is uttered, and if the behavior or words are positive, the identification unit identifies the sound source of the sound being output into the space in which the third user is present as the sound source of interest, and identifies at least an audio output device that outputs sound into the space in which the user who uttered the specified utterance is present as the object to be controlled, and if the behavior or words are negative, does not identify the sound source of interest and the object to be controlled.

[0025] According to the above configuration, if a user expresses a desire to enjoy a sound source, i.e., expresses a desire to share sound, and the third user expresses positive behavior, sound sharing between the two users is realized. This allows the two users to share the sound emitted from a sound source in an exchange that occurs when sharing sound emitted from the sound source. Also, if the third user expresses negative behavior, sound sharing is not performed, so that sound sharing that the third user does not want can be prevented.

[0026] The control device of aspect 10 of the present invention is, in any of aspects 1 to 9 above, wherein the detection unit further detects a second predetermined behavior of the user indicating an end, different from the predetermined behavior, and the output control unit may terminate the output of the sound emitted by the identified sound source from each of the identified audio output devices when the second predetermined behavior is detected.

[0027] According to the above configuration, the output of the sound emitted from the specified sound source from each of the specified sound output devices is terminated in response to the second predetermined speech or behavior performed by the user. This allows the user who is sharing the sound to end the sound sharing by simply performing the second predetermined speech or behavior. Thus, the sound sharing can be ended without the user performing any input operation.

[0028] In the control device of aspect 11 of the present invention, in aspect 10 above, the output control unit may, upon the termination, reset the sound source for each of the identified audio output devices to the sound source that was set immediately before control by the output control unit began.

[0029] According to the above configuration, when the audio sharing ends, the audio source that was set immediately before the control by the output control unit started is reset to the audio output device that was the control target. This allows the user who was sharing the audio to relisten to the audio that he or she was listening to before the audio sharing ends.

[0030] A control device according to aspect 12 of the present invention is any one of aspects 1 to 11 above, wherein each of the audio output devices has directionality in the direction in which the audio is emitted, and is configured so as to prevent the formation of a sound field in which multiple audio outputs from different audio output devices reach.

[0031] According to the above configuration, a sound field in which a plurality of sounds output from different sound output devices reach is not formed in the output destination space of the sound output device, so a user in one of the formed sound fields will hear the sound output from one of the sound output devices. In other words, if a user in one of the formed sound fields wants to hear a sound different from the sound he or she is listening to, the user needs to move to another sound field. Here, according to the above configuration, when a user performs a predetermined speech or action, a sound emitted from a target sound source is output from the sound output device in which the sound source is set and the target sound output device to be controlled. As a result, the user can listen to the sound reaching the other sound field without moving to the other sound field where the sound emitted from the target sound source reaches by performing a predetermined speech or action.

[0032] The audio output control method of aspect 13 of the present invention is a method for controlling audio output by a control device that causes audio emitted by a sound source to be output from a plurality of audio output devices, wherein the sound source can be set for each of the audio output devices, and includes a detection step for detecting predetermined words and actions of a user present in the output destination space of the audio output device, an identification step for identifying a sound source of interest among the sound sources and the audio output device to be controlled in accordance with the detected predetermined words and actions, and an output control step for outputting the audio emitted by the identified sound source from each of the identified audio output devices.

[0033] According to the above configuration, the same effects as those of the control device according to the first aspect are achieved.

[0034] The control device according to each aspect of the present invention may be realized by a computer. In this case, the control program of the control device that realizes the control device by the computer by operating the computer as each part (software element) of the control device, and the computer-readable recording medium on which it is recorded, also fall within the scope of the present invention. Effect of the Invention

[0035] According to one aspect of the present invention, audio can be shared among a plurality of users without requiring input operations. [Brief description of the drawings]

[0036] [Figure 1] 2 is a diagram showing an example of a configuration of a main part of a control device included in the audio output system according to the first embodiment of the present invention. FIG. [Diagram 2] 2 is a top view of a vehicle to which the audio output system shown in FIG. 1 is applied. [Diagram 3] 2 is a diagram showing a specific example of sound source settings stored in the control device shown in FIG. 1. [Figure 4] 2 is a flowchart showing an example of the flow of a sharing process executed by the control device shown in FIG. 1. [Diagram 5] FIG. 11 is a diagram showing a specific example of a sharing start behavior performed in a vehicle interior space. [Figure 6] FIG. 11 is a diagram showing another specific example of a sharing start speech / behavior performed in the vehicle interior space. [Figure 7] FIG. 13 is a diagram showing yet another specific example of a sharing start speech or action performed in the vehicle interior space. [Figure 8] 10 is a flowchart showing an example of the flow of a sharing end process executed by a control device according to a second embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0037] [Embodiment 1] Hereinafter, one embodiment of the present invention will be described in detail.

[0038] (Outline of the audio output system) 1 is a diagram showing an example of a configuration of a main part of a control device 1 included in an audio output system 100 according to this embodiment. The audio output system 100 is a system that outputs audio to a predetermined space.

[0039] As shown in FIG. 1, the audio output system 100 includes a control device 1, sound sources 2A to 2N, directional speakers 3A to 3N (audio output devices), a camera 4 (imaging device), and a microphone 5 (audio input device).

[0040] The directional speakers 3A to 3N are a plurality of audio output devices that output audio to a predetermined space. The notation "3A to 3N" indicates that there are a plurality of directional speakers, and is not intended to limit the number of directional speakers included in the audio output system 100. In addition, hereinafter, when there is no need to distinguish between the directional speakers 3A to 3N, they will be referred to as directional speaker 3. The directional speaker 3 is an audio output device that has directivity in the direction from which the audio is emitted. In addition, each of the directional speakers 3A to 3N is provided in a predetermined space so that a sound field in which the outputted audio reaches is not formed. In other words, a user present in the predetermined space will hear the audio output from any one of the directional speakers 3.

[0041] In this embodiment, a directional speaker 3 is described as an example of an audio output device having a function of allowing each user to hear audio, but other audio output devices having the same function may be applied instead of the directional speaker 3.

[0042] The sound sources 2A to 2N are a plurality of audio output devices that emit sounds. The notation "2A to 2N" indicates that there are a plurality of sound sources, and this notation is not intended to limit the number of sound sources included in the audio output system 100. The sound source included in the audio output system 100 may be one. In addition, hereinafter, when there is no need to distinguish between the sound sources 2A to 2N, they will be referred to as sound source 2. Each of the sound sources 2A to 2N is set to one of a plurality of directional speakers 3A to 3N. Specifically, each of the sound sources 2A to 2N is connected to one of the plurality of directional speakers 3A to 3N via the control device 1 by wire or wirelessly. As a result, the sound emitted by the sound source 2 is output from the directional speaker 3 connected to the sound source 2 via the control device 1. In this embodiment, the sound source 2 and the control device 1, and the control device 1 and the directional speaker 3 are wirelessly connected using Bluetooth (registered trademark) as an example. The connection means is not limited to Bluetooth, and may be other wireless connection such as WiFi (registered trademark), or may be a wired connection.

[0043] As described above, each of the directional speakers 3A to 3N is provided in a predetermined space so as not to form a sound field in which the output sounds reach. For this reason, a user who wants to listen to a sound emitted by a certain sound source 2 usually needs to be present in a sound field in which the sound output from the directional speaker 3 to which the sound source 2 is set reaches.

[0044] The sound source 2 may be an information terminal that can be brought into the predetermined space, or may be a sound source that is installed in the predetermined space in advance. Specific examples of information terminals include, but are not limited to, smartphones, tablet terminals, laptop computers, portable game consoles, portable music players, portable DVD players, and Blu-ray disc players. In the example of FIG. 1, at least the sound source 2A is illustrated as an information terminal.

[0045] The camera 4 is an imaging device that captures an image within a predetermined space. The camera 4 transmits data of the captured image (a typical example is a video) to the control device 1.

[0046] The microphone 5 is a voice input device that acquires voice generated within a specified space. The microphone 5 converts the acquired voice into a voice signal and transmits the voice signal to the control device 1. Hereinafter, the acquisition of voice by the microphone 5 may be referred to as "sound collection."

[0047] The control device 1 controls the audio output in a predetermined space. Specifically, the control device 1 sets one of the directional speakers 3A to 3N for each of the sound sources 2A to 2N, and causes the audio emitted by each of the sound sources 2A to 2N to be output from the set directional speaker 3. As an example, based on a signal for pairing with one of the directional speakers 3A to 3N received from the sound source 2, the control device 1 sets the sound source 2 that transmitted the signal to the directional speaker 3 indicated by the signal. Then, the control device 1 receives audio data from the sound source 2, and transmits the audio data to the directional speaker 3 to which the sound source 2 is set. As a result, the control device 1 causes the audio emitted by the sound source 2 to be output from the directional speaker 3 to which the sound source 2 is set.

[0048] In addition, the control device 1 detects a predetermined speech and behavior of a user present in a predetermined space, and identifies a noteworthy sound source among the sound sources 2A to 2N and a directional speaker 3 to be controlled among the directional speakers 3A to 3N according to the predetermined speech and behavior, and outputs the sound emitted by the identified sound source from each of the identified directional speakers 3. That is, the control device 1 outputs the sound emitted by the sound source 2 identified based on the user's predetermined speech and behavior from the directional speaker 3 identified based on the predetermined speech and behavior in addition to the directional speaker 3 to which the sound source 2 is set. In other words, the control device 1 can share the sound emitted by the identified sound source 2 with multiple users present in different output destination spaces based on the user's predetermined speech and behavior. Hereinafter, the predetermined speech and behavior will be referred to as a "sharing start speech and behavior."

[0049] Here, the sharing start behavior is at least one of a predetermined action performed by a user and a predetermined utterance uttered by a user. Typical examples of the predetermined behavior are, but are not limited to, the movements of body parts such as the face, eyes (gaze), arms, and hands. In this embodiment, the sharing start behavior is a behavior performed by at least two users. Typically, the sharing start behavior according to this embodiment is a behavior performed by a user to start sharing audio, and then another user performs a behavior to agree to sharing the audio. In addition, hereinafter, a user who performs a behavior to start sharing audio may be referred to as a "user who has initiated a sharing start behavior". In addition, a user who performs a behavior to agree to sharing audio may be referred to as a "user who has established a sharing start behavior". In addition, details of the control device 1 will be described later.

[0050] With the above configuration, the control device 1 can allow a plurality of users to share audio emitted from one sound source 2 in response to a sharing start action performed by the user. That is, a user who wants to share audio emitted from one sound source 2 can share the audio by performing a sharing start action without performing an input operation on a user interface for audio sharing (for example, a button operation on the sound source 2 or the directional speaker 3, etc.). In other words, the user can share audio in a simple manner without feeling the bother of the above input operation.

[0051] (Example of application of voice input system) 2 is a diagram showing a vehicle 200 to which the audio output system 100 is applied, as viewed from above. That is, in the example shown in FIG.

[0052] As shown in FIG. 2, four directional speakers 3A to 3D are provided in vehicle interior space 210. In the example shown in FIG. 2, directional speakers 3A to 3D are provided in the headrests of the respective seats. Specifically, directional speaker 3A is provided in the headrest of driver's seat 20A. directional speaker 3B is provided in the headrest of passenger seat 20B. directional speaker 3C is provided in the headrest of rear seat 20C. directional speaker 3D is provided in the headrest of rear seat 20D.

[0053] The directional speakers 3A to 3D have directivity in the direction of sound emission so that only the passengers sitting in the seats where the speakers are installed can hear the output sound. Specifically, the intensity of the sound waves output from each of the directional speakers 3A to 3D is strong in the forward direction of the vehicle 200 and weak in other directions. As a result, the sound output from the directional speaker 3A can be heard only by the passenger (driver) sitting in the driver's seat 20A. The sound output from the directional speaker 3B can be heard only by the passenger sitting in the passenger seat 20B. The sound output from the directional speaker 3C can be heard only by the passenger sitting in the rear seat 20C. The sound output from the directional speaker 3D can be heard only by the passenger sitting in the rear seat 20D. By arranging the directional speakers 3 in this way, the passengers of the vehicle 200 can hear the sound they want to hear.

[0054] The number and installation positions of the directional speakers 3 provided in the vehicle interior space 210 are not limited to the example shown in FIG.

[0055] As shown in FIG. 2, two cameras 4A and 4B and four microphones 5A to 5D are provided in the vehicle interior space 210. In the example shown in FIG. 2, the camera 4A is provided in front of the passenger seat 20B and near the glove box, and captures images of the driver's seat 20A and the passenger seat 20B. As a result, the passengers sitting in the driver's seat 20A and the passenger seat 20B are captured by the camera 4A. The camera 4B is provided on the back of the passenger seat 20B, and captures images of the rear seats 20C and 20D. As a result, the passengers sitting in the rear seats 20C and 20D are captured by the camera 4B.

[0056] Cameras 4A and 4B transmit video data that allows identification of whether the camera 4 that captured the video is camera 4A or 4B to control device 1. As an example, cameras 4A and 4B transmit camera identification information for identifying themselves to control device 1 in association with the video data.

[0057] The number and installation positions of the cameras 4 provided in the vehicle interior space 210 are not limited to the example in FIG. 2. The installation positions of the cameras 4 may be any positions that can capture images of the body parts (typically, the upper body) of the occupants to be imaged, and for example, the camera 4A may be provided near the area between the steering wheel and the glove box. In addition, when cameras with narrow imaging ranges are used, each of the four cameras 4 may be provided in front of each seat. In addition, for example, if the camera 4A provided near the area between the steering wheel and the glove box can capture images of the body parts of the occupants sitting in the rear seats 20C and 20D to be imaged, the camera 4B may not be provided in the vehicle 200.

[0058] 2, microphones 5A to 5D are provided in front of each seat with their sound collecting portion facing the corresponding seat. Specifically, microphone 5A is provided near the steering wheel in front of driver's seat 20A with its sound collecting portion facing the driver's seat 20A. Microphone 5B is provided near the glove box in front of passenger seat 20B with its sound collecting portion facing the passenger seat 20B. Directivity Microphone 5C is provided on the back of the driver's seat 20A in front of the rear seat 20C with its sound collecting portion facing the rear seat 20C. Microphone 5Dis provided on the back of the passenger seat 20B in front of the rear seat 20D with its sound collecting portion facing the rear seat 20D.

[0059] The microphones 5A to 5D according to the present embodiment have directivity in their sound collection range so that they can collect only the voices emitted by the passengers sitting in the seats to which the sound collection units are directed. Specifically, the microphones 5A to 5D have high sound collection sensitivity in the range from the sound collection units to the seats to which the sound collection units are directed (i.e., in front of the sound collection units), and low sound collection sensitivity in other ranges. As a result, the microphone 5A collects only the voices emitted by the passengers sitting in the driver's seat 20A among the passengers of the vehicle 200. The microphone 5B collects only the voices emitted by the passengers sitting in the passenger seat 20B among the passengers of the vehicle 200. The microphone 5C collects only the voices emitted by the passengers sitting in the rear seats 20C among the passengers of the vehicle 200. The microphone 5C collects only the voices emitted by the passengers sitting in the rear seats 20D among the passengers of the vehicle 200.

[0060] The microphones 5A to 5D transmit audio signals that enable identification of which of the microphones 5A to 5D has collected the sound to the control device 1. As an example, the microphones 5A to 5D transmit microphone identification information for identifying the microphones 5A to 5D to the control device 1 in association with the audio signal.

[0061] The number and installation positions of the microphones 5 provided in the vehicle interior space 210 are not limited to the example in Fig. 2. For example, if the direction from which the sound is emitted can be identified, one microphone whose sound collection range is the entire vehicle interior space 210 may be provided in an appropriate position of the vehicle 200 (for example, the position of point O in Fig. 2). Also, in Fig. 2, an example is shown in which the microphones 5 are provided in front of each seat to show the correspondence between the microphones 5 and the seats, but it is preferable that the microphones 5 are provided in a position that does not collect the sound output from the directional speaker 3 but only collects the speech of the passenger sitting in each seat.

[0062] 2 includes an information terminal that can be brought into the vehicle interior space 210, and an in-vehicle sound source that is installed in advance in the vehicle interior space 210. Specific examples of the in-vehicle sound source include, but are not limited to, an in-vehicle radio, an in-vehicle television, and a car audio system.

[0063] In addition, in this embodiment, the control device 1 is described as being provided in the vehicle 200. The installation position of the control device 1 in the vehicle 200 is not particularly limited as long as it is a position where the control device 1 can be communicatively connected to the sound source 2, the directional speaker 3, the camera 4, and the microphone 5. The control device 1 may be provided in a position different from the vehicle 200 as long as it is communicatively connected to the sound source 2, the directional speaker 3, the camera 4, and the microphone 5. For example, the control device 1 may be provided in a server device on a network communicatively connected to the vehicle 200.

[0064] (Details of control device 1) 1 again, the control device 1 will be described in detail. The control device 1 includes a video recognition unit 11, a voice recognition unit 12, a detection unit 13, an identification unit 14, an output control unit 15, a storage unit 16, and a communication unit 17. In the following, an example in which the voice output system 100 according to this embodiment is applied to a vehicle 200 shown in FIG. 2 will be described. That is, in the following description, the user of the voice output system 100 is a passenger of the vehicle 200.

[0065] The storage unit 16 stores various data used by the control device 1. As shown in Fig. 1, the storage unit 16 stores at least a sound source setting 161. Note that the storage unit 16 may be realized as an external storage device accessible by the control device 1.

[0066] The communication unit 17 receives information from an information terminal operating as the sound source 2. An example of the information is information regarding the orientation of the information terminal (hereinafter, referred to as "orientation information"). The orientation information will be described later. The communication unit 17 may have a function of transmitting information to the information terminal. The communication unit 17 may also transmit and receive information to and from a sound source 2 other than an information terminal.

[0067] Video recognition unit 11 performs recognition processing on the video captured by camera 4. Video recognition unit 11 includes a movement recognition unit 111, a gaze recognition unit 112, and a terminal recognition unit 113. Video recognition unit 11 outputs the video recognition result to detection unit 13 together with camera identification information associated with the acquired video data.

[0068] The movement recognition unit 111 recognizes the movement of the body parts of the occupant shown in the video. Note that the recognition of the movement of the body parts can be realized using existing technology, and therefore a detailed description thereof will be omitted here. As an example, the movement recognition unit 111 recognizes the movement of the body parts of the occupant using a motion estimation model for estimating the movement of the body parts from video data showing the body parts. The motion estimation model may be stored in the storage unit 16.

[0069] The gaze recognition unit 112 recognizes the gaze of the occupant shown in the video, that is, the direction in which the occupant is looking. Since gaze recognition can be realized using existing technology, detailed description will be omitted here. As an example, the gaze recognition unit 112 recognizes the gaze of the occupant using a gaze estimation model for estimating the gaze from video data showing the eyes. The gaze estimation model may be stored in the storage unit 16, or may be stored in another device communicably connected to the control device 1.

[0070] The terminal recognition unit 113 recognizes the movement and orientation of the information terminal shown in the video. Note that this recognition in the information terminal can be realized using existing technology, so detailed explanations will be omitted here. As an example, the terminal recognition unit 113 recognizes the movement and orientation of the information terminal using a terminal estimation model for estimating the movement and orientation of the information terminal from video data in which the information terminal is shown. The terminal estimation model may be stored in the storage unit 16, or may be stored in another device connected to the control device 1 so as to be able to communicate with it.

[0071] Furthermore, the terminal recognition unit 113 may recognize the orientation of the information terminal based on orientation information received in real time from the information terminal via the communication unit 17. The orientation information is at least one of inertial information and radio wave strength information. The inertial information is, for example, information measured by an inertial sensor included in the information terminal. The inertial sensor is, for example, but is not limited to, a 3-axis acceleration sensor, a 3-axis angular velocity sensor, a 3-axis direction sensor, or a 9-axis sensor consisting of a combination of these. The radio wave strength information is information indicating the signal strength (RSSI: Received Signal Strength Indicator) of a wireless connection such as Bluetooth.

[0072] The camera 4 has functions necessary for recognizing the movement of body parts, the line of sight, and the movement and orientation of the information terminal. For example, when an infrared image is necessary for line of sight recognition, the camera 4 has a function as an infrared camera.

[0073] The voice recognition unit 12 performs voice recognition processing on the voice collected by the microphone 5. Specifically, the voice recognition unit 12 converts the voice signal acquired from the microphone 5 into text data, and outputs the text data to the detection unit 13. Note that the voice recognition processing can be realized using existing technology, so detailed description will be omitted here. As an example, the voice recognition unit 12 performs the voice recognition processing using a voice recognition model. The voice recognition model includes an acoustic model, a pronunciation dictionary, and a language model, and the voice recognition unit 12 inputs a voice signal that has been subjected to acoustic analysis into the voice recognition model to obtain a recognition result, that is, text data converted from the voice signal. The voice recognition model may be generated by an existing deep learning technology. The voice recognition model may be stored in the storage unit 16, or may be stored in another device that is communicably connected to the control device 1.

[0074] The voice recognition unit 12 judges whether the collected voice is a speech or not based on the text data obtained as the voice recognition result. If it is judged to be a speech, the voice recognition unit 12 outputs the text data to the detection unit 13 together with microphone identification information associated with the acquired voice signal. If it is judged to be not a speech, the voice recognition unit 12 ends the process without outputting the text data to the detection unit 13. The voice recognition unit 12 may acquire the judgment result of whether it is a speech or not from a voice recognition model together with the voice recognition result. In other words, the judgment of whether it is a speech or not may be performed by the voice recognition model. Moreover, a voice that is not a speech is, for example, a noise generated in the vehicle interior space 210, but is not limited thereto.

[0075] The above-mentioned "other devices communicably connected to the control device 1" may be realized as, for example, a cloud server.

[0076] The detection unit 13 detects sharing start speech and behavior of an occupant present in the output destination space of the directional speaker 3. The output destination space is a space to which the directional speaker 3 outputs sound. That is, the detection unit 13 according to this embodiment detects sharing start speech and behavior of an occupant present in the vehicle interior space 210.

[0077] The detection unit 13 detects a predetermined action of the occupant as a sharing start speech or behavior from the image captured by the camera 4. The predetermined action is at least a part of the sharing start speech or behavior. Specifically, the detection unit 13 determines whether or not a predetermined action of the occupant is being performed in the video based on the recognition result of the video captured by the camera 4 acquired from the video recognition unit 11. If the detection unit 13 determines that the predetermined action is being performed, it outputs the acquired recognition result and camera identification information to the identification unit 14. On the other hand, if the detection unit 13 determines that the predetermined action is not being performed, it ends the detection process of the sharing start speech or behavior. Specific examples of the predetermined actions detected by the detection unit 13 will be described later.

[0078] Furthermore, the detection unit 13 detects a predetermined utterance uttered by the occupant as a sharing start utterance from the voice input to the microphone 5. The predetermined utterance is at least a part of the sharing start utterance. Specifically, the detection unit 13 determines whether the voice is a predetermined utterance based on text data, which is the recognition result of the voice input to the microphone 5 and is acquired from the voice recognition unit 12. If the detection unit 13 determines that the voice is a predetermined utterance, it outputs the acquired text data and microphone identification information to the identification unit 14. On the other hand, if the detection unit 13 determines that the utterance is not a predetermined utterance, it ends the detection process of the sharing start utterance. Specific examples of the predetermined utterance detected by the detection unit 13 will be described later.

[0079] In this embodiment, the sharing start behavior consists of a plurality of predetermined actions (not including speech) performed in a predetermined order, or a plurality of predetermined utterances (not including actions) uttered in a predetermined order. That is, the detection unit 13 outputs the video recognition result and camera identification information, or the voice recognition result (text data) and microphone identification information to the specification unit 14. The predetermined order refers to a predetermined action or utterance for starting audio sharing followed by a predetermined action or utterance for accepting audio sharing.

[0080] The identification unit 14 identifies a sound source of interest among the sound sources 2 and the directional speaker 3 to be controlled, in accordance with the detected sharing start speech or behavior. The identification unit 14 includes a sound source identification unit 141 and a speaker identification unit 142.

[0081] The sound source identifying unit 141 identifies the sound source of interest according to the detected sharing start speech. Specifically, the sound source identifying unit 141 acquires the video recognition result and camera identification information, or the voice recognition result (text data) and microphone identification information from the detection unit 13, and identifies the sound source of interest based on the acquired data and the sound source setting 161 stored in the storage unit 16.

[0082] Fig. 3 is a diagram showing a specific example of the sound source setting 161. The sound source setting 161 is information indicating a combination of a sound source 2 and a directional speaker 3 that outputs a sound emitted by the sound source 2. Fig. 3 shows an example in which the sound source setting 161 is a database in a table format, but the data format of the sound source setting 161 is not limited to this example.

[0083] 3 shows sound source settings 161A and 161B as examples of the sound source setting 161. The sound source setting 161A is the sound source setting 161 before the execution of a process by the identification unit 14, and the sound source setting 161B is the sound source setting 161 after the execution of the process.

[0084] The "Speaker" column stores information indicating the directional speaker 3 provided in the vehicle interior space 210. In the example of FIG. 3, information indicating each of the directional speakers 3A to 3D is stored in each record. Hereinafter, the information stored in the "Speaker" column may be referred to as "speaker information." The "Sound source" column stores information indicating the sound source 2 that is causing each directional speaker 3 to output sound. Hereinafter, the information stored in the "Sound source" column may be referred to as "sound source information."

[0085] According to the sound source setting 161A, the directional speaker 3A outputs the sound generated by the sound source 2A, which is a car radio. The directional speaker 3B outputs the sound generated by the sound source 2B, which is a portable game console. The directional speaker 3C outputs the sound generated by the sound source 2C, which is a smartphone. The directional speaker 3D outputs the sound generated by the sound source 2D, which is a smartphone. The "Sharing Control" column and the "Pre-Sharing Sound Source" column will be described later.

[0086] Referring again to FIG. 1, the sound source identification unit 141 will be described. First, a case will be described in which the identification unit 14 acquires the recognition result of the image and the camera identification information as a result of the occupant performing the sharing start behavior consisting of a predetermined action. The sound source identification unit 141 identifies the camera 4 that captured the image based on the acquired camera identification information. Next, the sound source identification unit 141 identifies the space in which the occupant who started the sharing start behavior exists based on the identification result of the camera 4 and the recognition result of the image. That space is, that is, the output destination space of the directional speaker 3 that outputs the sound that the occupant who started the sharing start behavior is listening to. In this embodiment, the sound source identification unit 141 identifies the space by identifying the seat in which the occupant who started the sharing start behavior is sitting. Next, the sound source identification unit 141 identifies the directional speaker 3 provided in that seat based on the identification result of the seat. Next, the sound source identification unit 141 identifies the sound source 2 set in the directional speaker 3 based on the identification result of the directional speaker 3 and the sound source setting 161.

[0087] For example, if the passenger sitting in the back seat 20D is the user who initiated the sharing start behavior, and the passenger sitting in the back seat 20C is the user who established the sharing start behavior, the sound source identification unit 141 identifies the camera 4 that captured the video as the camera 4B, and identifies the seat in which the passenger who is the user who initiated the sharing start behavior is sitting as the back seat 20D. Next, the sound source identification unit 141 identifies the directional speaker 3D provided in the back seat 20D, and identifies the sound source 2D set in the directional speaker 3D based on the sound source setting 161A.

[0088] Next, a case will be described in which the identification unit 14 acquires the voice recognition result and the microphone identification information as a result of the occupant making a sharing start speech consisting of a predetermined speech. The sound source identification unit 141 identifies the microphone 5 that collected the sound based on the acquired microphone identification information. Next, the sound source identification unit 141 identifies the seat where the occupant who started the sharing start speech is sitting based on the identification result of the microphone 5 and the voice recognition result, thereby identifying the space in which the occupant exists. Next, the sound source identification unit 141 identifies the directional speaker 3 provided in the seat based on the identification result of the seat. Next, the sound source identification unit 141 identifies the sound source 2 set in the directional speaker 3 based on the identification result of the directional speaker 3 and the sound source setting 161.

[0089] For example, if the passenger sitting in the rear seat 20D is the user who initiated the sharing start behavior, and the passenger sitting in the rear seat 20C is the user who established the sharing start behavior, the sound source identification unit 141 identifies the microphone 5 that collected the sound as the microphone 5C and the microphone 5D, and identifies the seat in which the passenger who is the user who initiated the sharing start behavior is sitting as the rear seat 20D. Next, the sound source identification unit 141 identifies the directional speaker 3D provided in the rear seat 20D, and identifies the sound source 2D set in the directional speaker 3D based on the sound source setting 161A.

[0090] The speaker identification unit 142 identifies the directional speaker 3 to be controlled according to the detected sharing start speech and behavior. Specifically, the speaker identification unit 142 acquires the video recognition result and camera identification information, or the voice recognition result and microphone identification information from the detection unit 13, and identifies the directional speaker 3 to be controlled based on the acquired data.

[0091] First, a case will be described in which the identification unit 14 acquires the recognition result of the image and the camera identification information as a result of the occupant performing the sharing start behavior consisting of a predetermined action. The speaker identification unit 142 identifies the camera 4 that captured the image based on the acquired camera identification information. Next, the speaker identification unit 142 identifies the seat in which the occupant, who is the user who established the sharing start behavior, is sitting based on the identification result of the camera 4 and the recognition result of the image. That is, the speaker identification unit 142 identifies the occupant who performed the predetermined action to accept sharing among the occupants in the image, and identifies the seat in which the occupant is sitting. Next, based on the identification result of the seat, the speaker identification unit 142 identifies the directional speaker 3 provided in the seat as the control target.

[0092] For example, if the passenger sitting in back seat 20D is the user who initiated the sharing start behavior, and the passenger sitting in back seat 20C is the user who established the sharing start behavior, speaker identification unit 142 identifies camera 4B as the camera that captured the video, and identifies the seat in which the passenger who established the sharing start behavior is sitting as back seat 20C. Next, speaker identification unit 142 identifies directional speaker 3C provided in back seat 20C as the control target.

[0093] Next, a case will be described in which the identification unit 14 acquires the voice recognition result and the microphone identification information as a result of the occupant making a sharing start speech consisting of a predetermined utterance. The speaker identification unit 142 identifies the microphone 5 that collected the sound based on the acquired microphone identification information. Next, the speaker identification unit 142 identifies the seat in which the occupant, who is the user who established the sharing start speech, is sitting based on the identification result of the microphone 5 and the voice recognition result. That is, the speaker identification unit 142 identifies the occupant who made the predetermined utterance to accept sharing, and identifies the seat in which the occupant is sitting. Next, based on the seat identification result, the speaker identification unit 142 identifies the directional speaker 3 provided in the seat as the control target.

[0094] For example, if the passenger sitting in the back seat 20D is the user who initiated the sharing start behavior, and the passenger sitting in the back seat 20C is the user who established the sharing start behavior, the speaker identification unit 142 identifies the microphone 5 that collected the sound as the microphone 5C and the microphone 5D, and identifies the seat in which the passenger who is the user who established the sharing start behavior is sitting as the back seat 20C. Next, the speaker identification unit 142 identifies the directional speaker 3C provided in the back seat 20C as the control target.

[0095] The identification unit 14 updates the "sound source" column of the sound source setting 161 based on the identified sound source 2 and directional speaker 3. For example, when an occupant sitting in the rear seat 20D is a user who has initiated a sharing start behavior, and an occupant sitting in the rear seat 20C is a user who has established a sharing start behavior, the sound source 2D and the directional speaker 3C are identified as described above. Based on this identification result, the identification unit 14 changes the sound source 2C that was associated with the directional speaker 3C in the sound source setting 161A to the sound source 2D as shown in the sound source setting 161B. The identification unit 14 notifies the output control unit 15 of the update of the sound source setting 161.

[0096] Furthermore, the identification unit 14 updates the "shared control" and "pre-shared sound source" columns of the sound source setting 161. The "shared control" column stores a flag indicating whether or not the directional speaker 3 is identified as a control target. In the example of FIG. 3, "0" indicates that the directional speaker 3 is not identified as a control target, and "1" indicates that the directional speaker 3 is identified as a control target, but the flag indicating whether or not the directional speaker 3 is identified as a control target is not limited to this. Hereinafter, the flag may be referred to as a shared control flag.

[0097] For example, if an occupant sitting in rear seat 20D is the user who initiated the sharing start behavior, and an occupant sitting in rear seat 20C is the user who established the sharing start behavior, the identification unit 14 changes the sharing control flag from "0" to "1" in a record including speaker information for directional speaker 3C.

[0098] The "sound source before sharing" column stores information indicating the sound source 2 that was set immediately before control was started for the directional speaker 3 that the identification unit 14 has identified as the control target. That is, the "sound source before sharing" column stores sound source information that was stored in the "sound source" column before the identification of the directional speaker 3 to be controlled by the identification unit 14, triggered by this identification. Note that hereinafter, the information stored in the "sound source before sharing" column may be referred to as "sound source information before sharing".

[0099] For example, if an occupant sitting in the rear seat 20D is the user who initiated the sharing start behavior, and an occupant sitting in the rear seat 20C is the user who established the sharing start behavior, pre-sharing sound source information indicating the sound source 2C is stored in the record including the speaker information of the directional speaker 3C. Also, pre-sharing sound source information indicating the sound source 2D is stored in the record including the speaker information of the directional speaker 3D.

[0100] In addition, if a sound source 2 is not set for a directional speaker 3 identified by the identification unit 14 as a control target immediately before control begins, pre-shared sound source information is not stored in a record including speaker information for the directional speaker 3.

[0101] In addition, in the example of sound source setting 161B shown in FIG. 3, for a record including speaker information of directional speaker 3A and a record including speaker information of directional speaker 3B, these directional speakers 3 are not identified as control targets, so the identification unit 14 does not update the pre-shared sound source information for these records.

[0102] Based on the sound source setting 161, the output control unit 15 outputs the sound emitted by the sound source 2 from one of the directional speakers 3. That is, the output control unit 15 outputs the sound emitted by the sound source 2 identified by the identification unit 14 from each of the directional speakers 3 identified by the identification unit 14. In the case of the example of the sound source setting 161B shown in FIG. 3, upon receiving a notification from the identification unit 14, the output control unit 15 refers to the sound source setting 161B and outputs the sound emitted by the sound source 2D from the directional speaker 3D in addition to the directional speaker 3D. This allows the sound emitted by the sound source 2D to be shared between the passengers sitting in the rear seats 20C and 20D.

[0103] Note that the method of outputting the sound emitted by the sound source 2 identified by the identification unit 14 from the directional speaker 3 identified by the identification unit 14 is not particularly limited. For example, the output control unit 15 may cancel the current pairing between the directional speaker 3 and the sound source 2, and pair the identified sound source 2 with the identified directional speaker 3. In addition, the output control unit 15 may output the sound emitted by the identified sound source 2 from the identified directional speaker 3 by transmitting audio data from the directional speaker 3 with which the identified sound source 2 is paired to the identified directional speaker 3.

[0104] (Sharing process flow) 4 is a flowchart showing an example of the flow of the sharing process executed by the control device 1. In step S1, the video recognition unit 11 acquires video data from the camera 4. In step S2, the video recognition unit 11 recognizes the movement of the body parts of the occupant shown in the video, and the movement and orientation of the information terminal. The movement of the body parts includes the movement of the occupant's line of sight. The video recognition unit 11 outputs the result of this recognition process and camera identification information associated with the acquired video data to the detection unit 13.

[0105] In step S3, the voice recognition unit 12 acquires a voice signal from the microphone 5. In step S4, the voice recognition unit 12 performs voice recognition and speech determination, and when it is determined that the voice indicated by the acquired voice signal is a speech, outputs to the detection unit 13 the voice recognition result and microphone identification information associated with the acquired voice signal.

[0106] In step S5 (detection step), the detection unit 13 detects a sharing start behavior. The detection unit 13 determines whether or not a sharing start behavior has been performed based on the acquired video recognition result and voice recognition result. If it is determined that a sharing start behavior has been performed (YES in step S5), the detection unit 13 outputs the recognition result of the acquired video and camera identification information, or the voice recognition result and microphone identification information to the identification unit 14, and the sharing process proceeds to step S6. If it is determined that a sharing start behavior has not been performed (NO in step S5), the sharing process returns to step S1.

[0107] In step S6 (identification step), the sound source identification unit 141 identifies the shared sound source 2. Specifically, the identification unit 14 identifies the sound source 2 based on the data acquired from the detection unit 13 and the sound source setting 161.

[0108] In step S7 (identification step), the speaker identification unit 142 identifies the directional speaker 3 to be controlled. Specifically, the speaker identification unit 142 identifies the directional speaker 3 based on data acquired from the detection unit 13. The identification unit 14 updates the sound source setting 161 based on the identified sound source 2 and directional speaker 3, and notifies the output control unit 15 of the update.

[0109] In step S8 (output control step), the output control unit 15 causes the identified directional speaker 3 to output the sound emitted by the identified sound source 2. Specifically, upon receiving a notification from the identification unit 14, the output control unit 15 refers to the sound source setting 161. In the sound source setting 161, the identified sound source 2 is associated with the identified directional speaker 3 by updating the sound source setting 161, so the output control unit 15 can cause the identified directional speaker 3 to output the sound emitted by the identified sound source 2. This ends the sharing process.

[0110] It should be noted that, when no voice is generated in the vehicle interior space 210, the processes of steps S3 and S4 are omitted. Also, when the voice recognition unit 12 determines in step S4 that the voice generated in the vehicle interior space 210 is not speech, the voice recognition unit 12 does not transmit the voice recognition result and the microphone identification information to the detection unit 13.

[0111] (Example of sharing-initiating behavior 1) FIG. 5 is a diagram showing a specific example of a sharing start behavior performed in the vehicle interior space 210. The sharing start behavior shown in FIG. 5 is composed of a predetermined action, and includes a movement of a body part of the occupant that moves the position of the information terminal operating as the sound source 2 along a predetermined path. Specifically, the sharing start behavior shown in FIG. 5 is composed of the following predetermined actions. Note that the sound source 2D shown in FIG. 5 is a smartphone equipped with a display unit. (1) An occupant 90D present in the output destination space of the directional speaker 3D in which the sound source 2D is set brings the sound source 2D closer to an occupant 90C present in the output destination space of the directional speaker 3C in which the sound source 2D is not set. (2) The occupant 90D looks at the display unit of the sound source 2D. (3) The occupant 90C looks at the display unit of the sound source 2D.

[0112] Here, the output destination spaces of directional speakers 3C and 3D are formed in rear seats 20C and 20D, respectively. In addition, the order of the predetermined actions (2) and (3) is not particularly limited as long as they are performed after the predetermined action (1).

[0113] The detection unit 13 detects the sharing start speech and behavior by detecting the following points from the video recognition result: (A) The movement of the hand of the occupant 90D holding the sound source 2D is a movement approaching the occupant 90C. (B) The display unit of the sound source 2D is facing the side of the occupants 90C and 90D. (C) The gazes of the occupants 90C and 90D are directed toward the sound source 2D.

[0114] The detection unit 13 performs the above detection of (A) from the recognition result of the movement recognition unit 111 and the recognition result of the terminal recognition unit 113 (recognition result of the movement of the information terminal). The detection unit 13 performs the above detection of (B) from the recognition result of the terminal recognition unit 113 (recognition result of the orientation of the information terminal). The detection unit 13 performs the above detection of (C) from the recognition result of the line of sight recognition unit 112.

[0115] When the detection unit 13 detects all of the above (A) to (C), it determines that the sharing start speech or behavior has been detected, and outputs the video recognition result and the camera identification information to the identification unit 14. On the other hand, when the detection unit 13 does not detect any of the above (A) to (C), it determines that the sharing start speech or behavior has not been detected, and does not output the video detection result and the camera identification information to the identification unit 14. For example, when the line of sight of the occupant 90C is not directed toward the sound source 2D, it does not output the video recognition result and the camera identification information to the identification unit 14. As a result, when all of the above (A) to (C) are detected, the identification unit 14 identifies the sound source 2 of interest and the directional speaker 3 to be controlled based on the video recognition result and the camera identification information. On the other hand, when the identification unit 14 does not detect any of the above (A) to (C), it does not identify the sound source 2 of interest and the directional speaker 3 to be controlled. That is, when the occupant 90D turns the display unit of the sound source 2D towards himself and brings it closer to the occupant 90C, if the line of sight of the occupant 90C is directed toward the sound source 2D, the identification unit 14 identifies the sound source 2 of interest and the directional speaker 3 to be controlled. On the other hand, if the line of sight of the occupant 90C is not directed toward the sound source 2D, the identification unit 14 does not identify the sound source 2 of interest and the directional speaker 3 to be controlled.

[0116] The predetermined actions constituting the sharing start speech and behavior are not limited to the above predetermined actions (1) to (3). For example, the predetermined actions may be only the above actions (1) and (3), that is, may not include the action of the occupant 90D looking at the display unit of the information terminal. In this example, the detection unit 13 detects the following (B') and (C') instead of the above (B) and (C). (B') The display unit of the sound source 2D is facing the occupant 90D side. (C') It is detected that the line of sight of the occupant 90D is directed toward the sound source 2D.

[0117] Alternatively, the above-mentioned (2) action in the predetermined action constituting the sharing start speech and behavior may be an action of looking at something other than the information terminal. For example, the above-mentioned (2) action may be an action of the above-mentioned 90D looking at the occupant 90C. In this example, the detection unit 13 detects the above-mentioned (A) and (B'), as well as the fact that the line of sight of the occupant 90D is directed toward the occupant 90C.

[0118] The sharing start behavior shown in Fig. 5 is, in other words, the behavior of the occupants 90C and 90D when the occupant 90D, listening to the sound emitted by the sound source 2D through the directional speaker 3C, encourages the occupant 90C to listen to the sound, and the occupant 90C agrees. That is, the occupants of the vehicle 200 can output the sound emitted by the same sound source 2 from the directional speaker 3 provided in the seat where each occupant is present, by a general behavior performed when sharing sound among multiple people, thereby realizing sound sharing among the occupants. As a result, the occupants of the vehicle 200 can realize sound sharing without performing an input operation on a user interface for sound sharing, and can realize sound sharing without feeling the bother of the above-mentioned input operation.

[0119] (Example of sharing-initiating behavior 2) Fig. 6 is a diagram showing another specific example of a sharing start speech / behavior performed in vehicle interior space 210. Note that, regarding the specific example shown in Fig. 6, the same parts as those in the specific example shown in Fig. 5 have already been described, and therefore will not be described again here.

[0120] The sharing start utterances shown in Fig. 6 are made up of predetermined utterances, and the predetermined utterances include an utterance to the effect that a sound source will be provided, and an utterance that responds affirmatively to the utterance. Specifically, the sharing start utterances shown in Fig. 6 are made up of the following predetermined utterances: (4) Occupant 90D makes an utterance saying "Listen to this," indicating that a sound source will be provided. (5) Occupant 90C (second user) makes an utterance saying "OK," responding affirmatively.

[0121] The detection unit 13 detects a sharing start behavior by detecting an utterance to provide a sound source and an utterance that responds positively to the utterance from the voice recognition result. Note that the utterance to provide a sound source and the utterance that responds positively to the utterance are not limited to the example of FIG. 6. That is, the detection unit 13 can detect multiple variations of the utterance to provide a sound source and the utterance that responds positively to the utterance. For example, the detection unit 13 refers to text indicating each of the multiple variations stored in the storage unit 16, and detects the utterance to provide a sound source and the utterance that responds positively to the utterance.

[0122] When the detection unit 13 detects both an utterance to provide a sound source and an utterance that responds affirmatively to the utterance, it determines that a sharing start utterance has been detected, and outputs the voice recognition result and the microphone identification information to the identification unit 14. On the other hand, when the detection unit 13 does not detect either of these utterances, it determines that a sharing start utterance has not been detected, and does not output the voice recognition result and the microphone identification information to the identification unit 14. For example, when the utterance by the occupant 90C is an utterance that responds negatively to the utterance "listen to this" by the occupant 90D, the detection unit 13 does not output the voice recognition result and the microphone identification information to the identification unit 14. Note that examples of utterances that respond negatively include, but are not limited to, "no," "sorry," and "not listening."

[0123] (Specific example of prescribed speech and behavior 3) Fig. 7 is a diagram showing yet another specific example of a sharing start speech / behavior performed in vehicle interior space 210. Note that, regarding the specific example shown in Fig. 7, the same parts as those in the specific examples shown in Fig. 5 and Fig. 6 have already been described, and therefore the description will not be repeated here.

[0124] The sharing start speech and behavior shown in Fig. 7 consists of a predetermined utterance, and the predetermined utterance includes an utterance expressing a desire to receive the sound source and an utterance responding affirmatively to the utterance. Specifically, the sharing start speech and behavior shown in Fig. 6 consists of the following predetermined utterances. (6) Occupant 90C utters "Mr. D, let me hear XX," expressing a desire to receive the sound source. (7) Occupant 90D (third user) utters "OK," responding affirmatively.

[0125] The detection unit 13 detects a sharing start behavior by detecting an utterance indicating a desire to receive the sound source and an utterance that responds positively to the utterance from the voice recognition result. Note that the utterance indicating a desire to receive the sound source and the utterance that responds positively to the utterance are not limited to the example of FIG. 7. That is, the detection unit 13 can detect multiple variations of the utterance indicating a desire to receive the sound source and the utterance that responds positively to the utterance. For example, the detection unit 13 refers to text indicating each of the multiple variations stored in the storage unit 16, and detects the utterance indicating a desire to receive the sound source and the utterance that responds positively to the utterance.

[0126] When the detection unit 13 detects both an utterance indicating a desire to enjoy the sound source and an utterance that responds affirmatively to the utterance, it determines that a sharing start utterance has been detected, and outputs the voice recognition result and the microphone identification information to the identification unit 14. On the other hand, when the detection unit 13 does not detect either of these utterances, it determines that a sharing start utterance has not been detected, and does not output the voice recognition result and the microphone identification information to the identification unit 14. For example, when the utterance by the occupant 90D is an utterance that responds negatively to the utterance by the occupant 90C of "Mr. D, let me hear XX", the detection unit 13 does not output the voice recognition result and the microphone identification information to the identification unit 14. Note that an utterance that responds negatively is, for example, "no", "sorry", etc., but is not limited thereto.

[0127] As described in specific examples 2 and 3, the occupants of the vehicle 200 can realize audio sharing among the occupants by having a general conversation that is carried out when multiple people share audio, and outputting audio emitted by the same sound source 2 from the directional speakers 3 provided in the seats where each occupant is present. As a result, the occupants of the vehicle 200 can realize audio sharing without performing input operations on a user interface, and therefore can realize audio sharing without feeling the hassle of the above-mentioned input operations.

[0128] (Modification) In the above embodiment, an example has been described in which the sound emitted by the sound source 2 is output from the directional speaker 3 via the control device 1, but the sound emitted by the sound source 2 may be output directly from the directional speaker 3 without going through the control device 1. For example, the directional speaker 3 may be directly connected to any of the sound sources 2 via Bluetooth and output the sound emitted by the sound source 2. This connection may be made without going through the control device 1, but it is preferable that either one of the sound source 2 or the directional speaker 3 transmits information indicating this connection, i.e., information indicating the paired sound source 2 and directional speaker 3, to the control device 1.

[0129] The output control unit 15 in this example instructs at least one of the sound source 2 identified by the identification unit 14, the directional speaker 3 that was connected to the sound source 2 before the identification, and the directional speaker 3 identified by the identification unit 14, so that the sound emitted by the sound source 2 identified by the identification unit 14 is also output from the directional speaker 3 identified by the identification unit 14.

[0130] In the above embodiment, an example was described in which the sharing start behavior is composed of only a predetermined action or only a predetermined utterance, but the sharing start behavior may be composed of a predetermined action and a predetermined utterance. For example, the sharing start behavior may be an utterance indicating that a sound source is to be provided or that a user wishes to receive the sound source, and an action that responds affirmatively to the utterance. In this example, the detection unit 13 outputs the video recognition result and camera identification information, and the voice recognition result and microphone identification information to the identification unit 14.

[0131] The identification unit 14 identifies the sound source 2 of interest from, for example, the voice recognition result and the microphone identification information. The identification unit 14 also identifies the directional speaker 3 to be controlled from, for example, the video recognition result and the camera identification information. The details of these identifications have been described in the first embodiment, and therefore will not be repeated here.

[0132] The sharing start behavior may be a motion or speech by one user. As an example, when the occupant 90D moves the sound source 2D closer to the occupant 90C, the detection unit 13 may detect only the motion as the sharing start behavior. In other words, the behavior for consent may be omitted.

[0133] Furthermore, the identification unit 14 may identify the directional speaker 3 to be controlled based on the content of the user's utterance. For example, when "everyone" is included in the speech recognition result indicating an utterance to the effect that the user wishes to enjoy a sound source, the identification unit 14 may identify all directional speakers 3 other than the directional speaker 3 set for the focused sound source 2 as the control target. In this example, the detection unit 13 may detect, for example, a speech behavior including an utterance "Everyone, listen to this" as a sharing start speech behavior.

[0134] Furthermore, the detection unit 13 may detect only the utterance "Everyone, listen to this" as the sharing start speech. In this example, the identification unit 14 may identify whether or not an occupant is sitting in each seat of the vehicle 200, and identify the directional speaker 3 provided in the seat where the occupant is sitting as the control target, among all the directional speakers 3 other than the directional speaker 3 set in the sound source 2 of interest. The identification unit 14 may identify whether or not an occupant is sitting, for example, based on measurement data of a weight sensor provided on the seat surface of the seat.

[0135] In addition, when the sound source 2 of interest is an information terminal, the output control unit 15 may measure in real time the radio wave strength of Bluetooth connecting the sound source 2 and the directional speaker 3 identified as the control target, and perform control according to the radio wave strength.

[0136] For example, the output control unit 15 may decrease (fade out) the volume of the sound output from the directional speaker 3 being the control target in response to a decrease in radio wave intensity. That is, the output control unit 15 decreases the volume of the sound output from the directional speaker 3 being the control target in response to an increase in the distance between the information terminal and the directional speaker 3. Then, when the distance between the information terminal and the directional speaker 3 becomes such that a connection by Bluetooth cannot be established, the connection by Bluetooth is released, and the sound emitted by the information terminal cannot be output from the directional speaker 3. Thus, with this configuration, the control device 1 can naturally end the sound output associated with the release of the connection between the information terminal and the directional speaker 3.

[0137] In addition, when the Bluetooth connection between the sound source 2 of interest and the directional speaker 3 specified as the control target is released, the output control unit 15 preferably stores the combination of the sound source 2 and the directional speaker 3 in the storage unit 16. Then, when the measured radio wave intensity of the information terminal that is the sound source 2 that has been released from the connection becomes equal to or greater than a predetermined intensity, the output control unit 15 may refer to the combination and reconnect the information terminal and the directional speaker 3. This resumes sharing of the sound emitted from the information terminal. In addition, after reconnecting the information terminal and the directional speaker 3, the output control unit 15 may increase (fade in) the volume of the sound output from the directional speaker 3 that is the control target in response to an increase in the radio wave intensity. With this configuration, the control device 1 can make the resumption of sound output accompanying the reconnection between the information terminal and the directional speaker 3 natural.

[0138] Moreover, instead of the multiple directional speakers 3, the audio output system 100 may include multiple speakers that do not have directionality.

[0139] Furthermore, the control device 1 may perform processes from the detection of a predetermined utterance to the identification of the sound source 2 of interest and the directional speaker 3 to be controlled by natural language processing. In this example, in order to realize the natural language processing, the control device 1 may use an estimation model that has been trained to estimate the sound source 2 of interest and the directional speaker 3 to be controlled when a specific word is included in the utterance. In this example, the components that acquire the voice recognition result, input it to the estimation model, and identify the sound source 2 and the directional speaker 3 based on the estimation result correspond to the detection unit 13 and the identification unit 14.

[0140] The above modifications can also be applied to the embodiments described below.

[0141] [Embodiment 2] Other embodiments of the present invention will be described below. For ease of explanation, the same reference numerals are given to members having the same functions as those described in the above embodiment, and the description thereof will not be repeated.

[0142] When the control device 1 according to the present embodiment detects a second predetermined speech or behavior by the user indicating an end, the control device 1 ends the output of the sound emitted by the identified sound source from each of the directional speakers 3 sharing the sound source. Hereinafter, the second predetermined speech or behavior will be referred to as a sharing end speech or behavior.

[0143] Specifically, the detection unit 13 detects the above-mentioned sharing end speech and behavior in addition to the predetermined speech and behavior described in the embodiment 1. The sharing end speech and behavior may be, for example, a predetermined action such as crossing both arms to make an "X" (hereinafter referred to as a second predetermined action), or a predetermined utterance including a phrase such as "the end" (hereinafter referred to as a second predetermined utterance).

[0144] In the former example, when detection unit 13 detects a second predetermined action from the video recognition result, it determines that it has detected a sharing end speech or behavior, and outputs the video recognition result and camera identification information to identification unit 14. In the latter example, when detection unit 13 detects a second predetermined utterance from the voice recognition result, it determines that it has detected a sharing end speech or behavior, and outputs microphone identification information to identification unit 14.

[0145] The identification unit 14 ends the output of the sound emitted by the sound source identified based on the sharing start speech and behavior from each of the directional speakers 3 identified based on the sharing start speech and behavior in response to the detected sharing end speech and behavior.

[0146] First, a case will be described in which the identification unit 14 acquires the recognition result of the image and the camera identification information as a result of the occupant performing the sharing end behavior consisting of the second predetermined action. The identification unit 14 identifies the camera 4 that captured the image based on the acquired camera identification information. Next, the identification unit 14 identifies the seat where the occupant who performed the sharing end behavior is sitting based on the identification result of the camera 4 and the recognition result of the image, thereby identifying the space in which the occupant who performed the sharing end behavior is present. Next, the identification unit 14 identifies the directional speaker 3 provided in the seat based on the identification result of the seat. Next, the identification unit 14 identifies the sound source 2 that is emitting the sound output by the directional speaker 3 based on the identification result of the directional speaker 3 and the sound source setting 161. Next, the identification unit 14 determines whether or not there is another directional speaker 3 that is outputting the sound emitted by the identified sound source 2 based on the identification result of the sound source 2 and the sound source setting 161.

[0147] When it is determined that the other directional speaker 3 exists, the identification unit 14 ends the output of the sound emitted by the sound source 2 identified based on the sharing start behavior from each of the directional speakers 3 identified based on the sharing start behavior. The directional speaker 3 is, in other words, a directional speaker 3 whose sharing control flag is 1. The identification unit 14 outputs an end instruction to the output control unit 15 to end the sound output for the directional speaker 3 whose sharing control flag is 1 in the sound source setting 161. The end instruction includes speaker information of the directional speaker 3 and pre-sharing sound source information in the record including the speaker information.

[0148] The identification unit 14 also updates the sound source setting 161. Specifically, the identification unit 14 updates the sound source information, the sharing control flag, and the sound source information before sharing for the record including the speaker information of the end instruction. More specifically, the identification unit 14 deletes the current sound source information and moves the sound source information before sharing to the "sound source" cell. As a result, the "sound source before sharing" cell becomes blank. The identification unit 14 also changes the sharing control flag to "0".

[0149] The output control unit 15 ends the current audio output for the directional speaker 3 indicated by the end instruction, and resets the sound source 2 for each of the identified directional speakers 3 to the sound source 2 that was set immediately before control by the output control unit 15 began. That is, the output control unit 15 starts audio output from the sound source 2 indicated by the end instruction. This allows the occupants of the vehicle 200 to relisten to the audio that they were listening to immediately before the sharing of the audio emitted by the sound source 2 of interest after the sharing ends.

[0150] As an example, the output control unit 15 may gradually lower the volume of the current audio output from the directional speaker 3 indicated by the end instruction, and when the volume falls below a predetermined volume, start audio output from the sound source 2 indicated by the end instruction, and gradually increase the volume until it reaches a second predetermined volume different from the above-mentioned predetermined volume. Note that the second volume is preferably a value greater than the predetermined volume. By configuring in this way, it is possible for the occupants of the vehicle 200 to re-listen in a natural way to the audio that they were listening to just before the audio emitted by the sound source 2 was shared.

[0151] Next, a case where the identification unit 14 acquires microphone identification information as a result of the occupant making a sharing end speech consisting of the second predetermined utterance will be described. The identification unit 14 identifies the microphone 5 that collected the sound based on the acquired microphone identification information. Next, the identification unit 14 identifies the seat where the occupant who made the sharing end speech is sitting based on the identification result of the microphone 5, thereby identifying the space in which the occupant who made the sharing end speech is present. Since each microphone 5 and each seat are in one-to-one correspondence, the identification unit 14 can identify the occupant who made the second predetermined utterance by identifying the microphone 5 that collected the second predetermined utterance. The subsequent processing is the same as that in the case where the recognition result of the image and the camera identification information are acquired as described above, and therefore the description will not be repeated here.

[0152] If it is determined that there is no other directional speaker 3 that is outputting the sound emitted by the identified sound source 2, the identification unit 14 does not output an end instruction to the output control unit 15. Note that a case where it is determined that there is no other directional speaker 3 is when, for example, an occupant present in the output destination space of a directional speaker 3 that is not outputting the sound emitted by the focused sound source 2 makes a sharing end statement.

[0153] (Flow of sharing termination process) Fig. 8 is a flowchart showing an example of the flow of the sharing end process executed by the control device 1. In the flowchart shown in Fig. 8, the same step numbers are assigned to steps that execute the same processes as the sharing process described in the first embodiment. Also, since the steps have already been described in the first embodiment, the description thereof will be omitted here.

[0154] In step S11, the detection unit 13 detects a sharing end behavior. The detection unit 13 determines whether or not a sharing end behavior has been performed based on the acquired video recognition result and voice recognition result. If it is determined that a sharing end behavior has been performed (YES in step S11), the detection unit 13 outputs the recognition result of the acquired video and the camera identification information or the microphone identification information to the identification unit 14, and the sharing process proceeds to step S12. If it is determined that a sharing end behavior has not been performed (NO in step S11), the sharing process returns to step S1.

[0155] In step S12, the identification unit 14 identifies the sound source 2 to which the directional speaker 3 sharing the sound source was connected immediately before the sharing. The identification unit 14 outputs an end instruction including information indicating the identified sound source 2 to the output control unit 15.

[0156] In step S13, the output control unit 15 outputs the sound emitted by the identified sound source 2 from each directional speaker 3. That is, the directional speaker 3 identified by the identification unit 14 as the control target stops outputting the sound emitted by the sound source 2 identified by the identification unit 14, and starts outputting the sound emitted by another sound source 2 that was set immediately before sharing.

[0157] (Modification) The process of the output control unit 15 when the sharing end command is executed is not limited to the process of outputting the sound emitted by the sound source 2 indicated by the pre-sharing sound source information. For example, the output control unit 15 may be configured to only perform the process of disconnecting the target sound source 2 from the directional speaker 3, that is, not to cause the directional speaker 3 to output sound after sharing ends.

[0158] [Embodiment 3] Further, another embodiment of the present invention will be described below. For convenience of explanation, the same reference numerals are given to members having the same functions as those described in the above embodiment, and the description thereof will not be repeated.

[0159] The control device 1 according to this embodiment controls audio output based on an eyelid opening value indicating how much the eyelids of each passenger are open.

[0160] As an example, the control device 1 shares the sound emitted by the sound source 2 identified by the identification unit 14 only with occupants for whom an eyelid opening value equal to or greater than a predetermined value has been detected. Specifically, the identification unit 14 identifies the eyelid opening value from the recognition result of the video captured by the camera 4, and compares the eyelid opening value with a predetermined value. Then, the identification unit 14 excludes from the control targets the directional speaker 3 that allows occupants whose eyelid opening value is less than the predetermined value to hear the sound.

[0161] For example, when the voice recognition result of the sharing start speech includes "everyone," the identification unit 14 identifies the eyelid opening value of each occupant of the vehicle 200 from the recognition result of the video. When there is an occupant whose eyelid opening value is less than a predetermined value, the identification unit 14 identifies the seat in which the occupant sits from the camera identification information, and identifies the directional speaker 3 provided in that seat. Then, the identification unit 14 excludes that directional speaker 3 from the control targets.

[0162] This allows the control device 1 to exclude occupants whose eyelid opening value is less than a predetermined value, in other words, occupants who are sleeping or about to sleep, from the sharing targets of the sound emitted from the sound source 2 identified by the identification unit 14. This makes it possible to reduce the possibility that the sound emitted from the sound source 2 will be heard by an occupant who does not wish to be the sharing target of the sound.

[0163] Note that the control using the eyelid opening value is not limited to this example. For example, the control device 1 may measure in real time the eyelid opening value of an occupant present in the output destination space of the directional speaker 3 identified as the control target, and control the audio output from the directional speaker 3.

[0164] For example, the identification unit 14 continues to acquire the recognition result of the video of the occupant of the vehicle 200 present in the output destination space of the directional speaker 3 identified as the control target, until the directional speaker 3 is no longer the control target. When the identification unit 14 identifies from the video recognition result that the eyelid opening value of the occupant is less than a predetermined value, the identification unit 14 reduces the volume of the directional speaker 3 that allows the occupant to hear the sound to a predetermined value. This enables the control device 1 not to disturb the sleep of the occupant.

[0165] [Software implementation example] The functions of the control device 1 (hereinafter referred to as the "device") can be realized by a program for causing a computer to function as the device, and a program for causing a computer to function as each control block of the device (in particular, the video recognition unit 11, the voice recognition unit 12, the detection unit 13, the identification unit 14, the output control unit 15, and the communication unit 17).

[0166] In this case, the device includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the program. The control device and storage device execute the program to realize each function described in each of the above embodiments.

[0167] The program may be non-transitory and may be recorded in one or more computer-readable recording media. The recording media may or may not be included in the device. In the latter case, the program may be provided to the device via any wired or wireless transmission medium.

[0168] In addition, some or all of the functions of each of the control blocks can be realized by a logic circuit. For example, an integrated circuit in which a logic circuit that functions as each of the control blocks is formed is also included in the scope of the present invention. In addition, the functions of each of the control blocks can be realized by, for example, a quantum computer.

[0169] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention. [Explanation of symbols]

[0170] 1 Control device 2, 2A~2N sound source 3, 3A~3N Directional speaker (audio output device) 4, 4A, 4B Camera (imaging device) 5, 5A~5D Microphone (audio input device) 13 Detection section 14 Specific section 15 Output control section S5 Detection Step S6 Specific Step S7 Specific Step S8 Output control step

Claims

1. A control device that outputs a sound generated by a sound source from a plurality of sound output devices, The sound source can be set for each of the audio output devices, A detection unit that detects a predetermined speech and behavior of a user present in an output destination space of the audio output device and a predetermined speech and behavior of a second user who is present in a space different from the output destination space and is different from the user; an identification unit that identifies a sound source of interest among the sound sources and the audio output device to be controlled in accordance with predetermined speech and behavior of the user and the second user detected by the detection unit; and an output control unit that causes the audio generated by the identified sound source to be output from each of the identified audio output devices.

2. The control device according to claim 1 , wherein the detection unit detects, as the predetermined behavior, a predetermined action of the user and the second user from an image captured by an imaging device.

3. The control device according to claim 2 , wherein the detection unit detects the predetermined action based on movements of body parts of the user and the second user included in the image.

4. A control device for outputting sound generated by a sound source from a plurality of sound output devices, comprising: The sound source can be set for each of the audio output devices, A detection unit that detects a predetermined speech or behavior of a user present in an output destination space of the audio output device; an identification unit that identifies a sound source of interest among the sound sources and the audio output device to be controlled in response to the detected predetermined speech or behavior; an output control unit that outputs the sound generated by the identified sound source from each of the identified sound output devices; The detection unit detects a predetermined movement of the user as the predetermined speech or behavior based on a movement of a body part of the user included in an image captured by an imaging device, the movement of the body part includes a movement of moving the position of an information terminal operating as a sound source along a predetermined path; The identification unit is configured to identify the information terminal as the sound source of interest.

5. the predetermined path is a path that allows the information terminal to approach from a side of a user present in an output destination space of an audio output device in which the information terminal is set as a sound source to a side of a user present in an output destination space of an audio output device in which the information terminal is not set as a sound source, The control device according to claim 4 , wherein the identification unit identifies, as the control target, an audio output device for which the information terminal is not set as a sound source.

6. The detection unit further detects a line of sight of a user to whom the information terminal is brought close, The identification unit is When the line of sight is directed toward the information terminal, the sound source of interest and the control target are identified; The control device according to claim 5 , wherein, when the line of sight is not directed toward the information terminal, the control device does not identify the sound source of interest and the control target.

7. The control device according to claim 1 , wherein the detection unit detects a predetermined utterance uttered by the user as the predetermined speech or behavior from a voice input to a voice input device.

8. A control device for outputting sound generated by a sound source from a plurality of sound output devices, comprising: The sound source can be set for each of the audio output devices, A detection unit that detects a predetermined speech or behavior of a user present in an output destination space of the audio output device; an identification unit that identifies a sound source of interest among the sound sources and the audio output device to be controlled in response to the detected predetermined speech or behavior; an output control unit that outputs the sound generated by the identified sound source from each of the identified sound output devices; The detection unit detects a predetermined utterance uttered by the user as the predetermined speech or behavior from a voice input to a voice input device, The predetermined utterance includes an instruction to provide a sound source, The detection unit further detects, after the predetermined utterance is uttered, a speech or behavior of a second user different from the user who uttered the predetermined utterance; The identification unit is If the speech or behavior is positive, a sound source of the sound being output to the space in which the user who uttered the predetermined utterance is present is identified as the sound source of interest, and at least a sound output device that outputs sound to the space in which the second user is present is identified as the control target; A control device that does not identify the sound source of interest and the control target when the behavior is negative.

9. The predetermined utterance includes a desire to receive the sound source, The detection unit further detects, after the predetermined utterance is uttered, a speech or behavior of a third user different from the user who uttered the predetermined utterance; The identification unit is If the speech or behavior is positive, a sound source of the sound being output to the space in which the third user is present is identified as the sound source of interest, and at least a sound output device that outputs sound to the space in which the user who uttered the predetermined utterance is present is identified as the control target; The control device according to claim 7 or 8, wherein, when the speech or behavior is negative, the control device does not identify the focused sound source and the controlled object.

10. The detection unit further detects a second predetermined speech or behavior of the user that is different from the predetermined speech or behavior and indicates an end; The control device according to claim 1 , wherein the output control unit terminates output of the audio emitted by the identified sound source from each of the identified audio output devices when the second specified speech or behavior is detected.

11. The control device according to claim 10 , wherein the output control unit, upon the termination, resets, for each of the identified audio output devices, a sound source that was set immediately before the control by the output control unit started.

12. Each of the audio output devices comprises: It has directivity in the direction of sound emission, The control device according to claim 1 , wherein the control device is provided so as to prevent a sound field in which a plurality of sounds output from different audio output devices arrive.

13. A method for controlling audio output by a control device that causes a sound generated by a sound source to be output from a plurality of audio output devices, comprising: The sound source can be set for each of the audio output devices, a detection step of detecting predetermined speech and behavior of a user present in an output destination space of the audio output device and a second user present in a space different from the output destination space and different from the user; a specifying step of specifying a sound source of interest among the sound sources and the sound output device to be controlled according to predetermined speech and behavior of the user and the second user detected in the detection step; and an output control step of causing each of the identified audio output devices to output the sound emitted by the identified sound source.

14. 2. A control program for causing a computer to function as the control device according to claim 1, the control program causing a computer to function as the detection unit, the identification unit, and the output control unit.

Images