Information processing method and control device

Abstract

To achieve both of gesture by a vehicle crewman and musical performance sounds regenerated in the related vehicle without causing discomfort to the crewman.SOLUTION: A control device determines regeneration start of musical performance sounds on the basis of a relative positional relationship between at least a part of a crewman and at least a part of a vehicle that are detected on the basis of a picked-up image in the vehicle when determining that a steering wheel has no effect on the vehicle's behavior.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing method and a control device. [Background technology]

[0002] Conventionally, there is known a technique for detecting a gesture by a user in a detection range, which is a predetermined space in a vehicle. For example, Patent Document 1 listed below discloses a gesture detection device and the like that aims to form an area (gesture detection area) for detecting a gesture by a vehicle occupant in an appropriate position. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2021-194982 Summary of the Invention [Problem to be solved by the invention]

[0004] The above-mentioned conventional techniques do not allow a gesture by a vehicle occupant and a musical performance sound to be reproduced inside the vehicle to be compatible without causing the occupant to feel uncomfortable.

[0005] In one aspect, the present invention has been made in consideration of the above circumstances, and its purpose is to provide an information processing method and control device that allows gestures by a vehicle occupant and musical sounds to be played inside the vehicle to be compatible without causing any discomfort to the occupant. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, one aspect of the present invention provides an information processing method that causes a processor to execute a process for playing musical sounds including at least one of the musical instrument sounds of a plurality of types of musical instruments and the cries of a plurality of types of animals, wherein the processor determines whether the steering wheel is in a state that does not affect the behavior of the vehicle, acquires an image captured by an imaging unit that continuously captures images of the interior of the vehicle in a three-dimensional space, detects at least a part of the body of an occupant of the vehicle based on the image, and, if it determines that the steering wheel is in a state that does not affect the behavior of the vehicle, determines to start playing the musical sounds based on the relative positional relationship between the detected at least a part of the body of the occupant and at least a part of the vehicle. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide an information processing method and a control device that can reconcile gestures by a vehicle occupant and musical sounds played inside the vehicle without causing the occupant to feel uncomfortable. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram showing a schematic configuration of a vehicle equipped with a control device according to an embodiment. [Figure 2] 2 is a diagram showing an example of the location of the control device in FIG. 1 in a vehicle interior. [Figure 3] An example is shown in which vehicle parts are associated with musical instruments for each seat position in the vehicle. [Figure 4] 2 is a flowchart showing an example of a processing procedure for information processing executed by the control device of FIG. 1; [Figure 5] 10 is a flowchart illustrating an example of a processing procedure for identifying a type of gesture and playing back a performance sound corresponding to the identified type of gesture. [Figure 6] An example is shown in which an occupant is making a "capping" gesture with their hands. [Figure 7]An example is shown in which an occupant is making a "rubbing" gesture with their hand. [Figure 8] 10 is a flowchart illustrating an example of a processing procedure for determining the start of reproduction of performance sounds based on the acceleration of a position change of a body part for a general user. [Figure 9] 10 is a flowchart illustrating an example of a processing procedure for identifying a "tapping" gesture made by a driver with his / her hands and playing back a musical sound corresponding to the "tapping" gesture. [Figure 10] 10 is a flowchart illustrating an example of a processing procedure for identifying a "rubbing" gesture made by a driver with his / her hand and playing back a musical sound corresponding to the "rubbing" gesture. [Figure 11] An example of a service provided to a passenger by the control device of FIG. 1 is shown below. DETAILED DESCRIPTION OF THE INVENTION

[0009] An embodiment according to one aspect of the present invention (hereinafter also referred to as "the present embodiment") will be described below with reference to the drawings. However, the present embodiment described below is merely an example of the present invention in all respects. Needless to say, various improvements and modifications can be made without departing from the scope of the present invention. In other words, when implementing the present invention, specific configurations according to the embodiment may be appropriately adopted. Note that, although data appearing in the present embodiment are described in natural language, more specifically, they are specified using pseudo-language, commands, parameters, machine language, etc. that can be recognized by a computer.

[0010] §1 Application Examples Conventionally, technologies for detecting gestures made by vehicle occupants have been known, but these technologies do not take into consideration the compatibility of gestures with musical performance, and are unable to reproduce sounds without causing discomfort to the occupant. Therefore, a control device (control device 10) according to this embodiment accurately detects at least a part of the body of an occupant Ue of a vehicle 1 and reproduces a musical performance sound PS based on the detection results. Therefore, the occupant Ue can perform music without feeling uncomfortable by performing gestures GT using at least that part of his or her body. In the following description, specific body parts of the occupant Ue, such as fingers, hands, arms, and legs, may be abbreviated as "BP (body part)." A BP is any body part suitable for the occupant Ue to execute control based on a gesture GT, and the BP includes at least one of the occupant's fingers, hands, arms, and legs. The control device 10 may detect the entire body of the occupant Ue, for example, detect the posture (of the entire body) of the occupant Ue, and play the performance sound PS in accordance with the detected posture (including changes in posture) of the occupant Ue. That is, the control device 10 accurately detects at least the BP (part of the body) of the occupant Ue of the vehicle 1, and plays the performance sound PS in accordance with the detection result (for example, changes in the position of the detected BP (particularly, the relative positional relationship RP with the part PV of the vehicle 1)).

[0011] The control device 10 determines the start of playback of the performance sounds PS, and the performance sounds PS include at least one of the musical instrument sounds MS of a plurality of types of musical instruments MI and the cries CR of a plurality of types of animals AM. When the control device 10 detects that the occupant Ue is making a gesture GT using the BP while the steering wheel is not affecting the behavior of the vehicle 1, the control device 10 determines the start of playback of the performance sounds PS. In particular, the control device 10 accurately detects the gesture GT of the occupant Ue based on the relative positional relationship RP between the BP of the occupant Ue and the part PV of the vehicle 1 (changes in this relative positional relationship RP), and determines the start of playback of the performance sounds PS based on the detection result. Therefore, the occupant Ue can perform music (i.e., playback of the performance sounds PS) using the gesture GT without feeling uncomfortable.

[0012] The control device 10 may associate at least one of a musical instrument MI and an animal AM with each of a plurality of parts PV of the vehicle 1. Then, for example, when the control device 10 detects that the occupant Ue is making a gesture GT with respect to a certain part PV(x) of the vehicle 1 using a BP, the control device 10 may execute the following process. That is, the control device 10 may play, as a performance sound PS, the musical instrument sound MS(x) (the cry CR(x)) of the musical instrument MI(x) (animal AM(x)) associated with the certain part PV(x). Furthermore, when the control device 10 identifies the type of gesture GT (musical performance action) that the occupant Ue is making with respect to the part PV using a BP, the control device 10 may play a performance sound PS(y) corresponding to the identified type of gesture GT (for example, gesture GT(y)). For example, the control device 10 may play, as the performance sound PS, an instrument sound MS(x, y) (a cry CR(x, y)) generated when a certain instrument MI(x) (animal AM(x)) is played with a certain type of performance motion (gesture GT(y)). In other words, when the control device 10 detects a gesture GT made by the occupant Ue using the BP toward a part PV of the vehicle 1, it may play at least one of the "instrument sound MS of the instrument MI" and the "cry CR of the animal AM" according to the part PV and gesture GT.

[0013] In this embodiment, when distinguishing between multiple occupants Ue of the vehicle 1, "(1)," "(2)," "(3)," ..., "(n)" are added after "Ue" to distinguish between them. Note that "n" is an integer equal to or greater than "1." When referring to multiple occupants Ue collectively without distinguishing between them, they are simply referred to as "occupant Ue." The same expression is used for BP, gesture GT, body part PV, musical instrument MI, animal AM, etc. Examples of body parts PV of the vehicle 1 include the steering wheel (steering wheel), door handles, seats, headrests, armrests, floor mats (floors), instrument panel (dashboard), glove box, shift lever, center console, etc. However, the body parts PV are not limited to the above-mentioned body parts. Furthermore, the body parts PV are, for example, parts inside the vehicle 1, but it is not necessary for the body parts PV to be parts inside the vehicle 1. In the following description, among the parts PV, "parts PV (e.g., steering wheel, shift lever, etc.) that can affect the behavior of vehicle 1 when the steering wheel is in a state that affects the behavior of vehicle 1" may be specifically referred to as driving parts DPV.

[0014] The control device 10 captures an image of the interior of the vehicle 1 (vehicle interior) using the imaging unit 110, and detects at least the BP of the occupant Ue of the vehicle 1 based on the captured image CI captured by the imaging unit 110. The control device 10 then determines the start of playback of the performance sound PS based on the position of the detected BP (including the posture of the occupant Ue), particularly based on the position of the BP relative to the part PV of the vehicle 1 (i.e., the relative positional relationship RP between the part PV and the BP). For example, the control device 10 plays the performance sound PS based on the position of the BP at each point in time (particularly the position of the BP relative to the part PV; i.e., the above-mentioned relative positional relationship RP), in other words, based on changes in the position of the BP over time. For example, when the control device 10 detects a gesture (gesture GT(4)) of the occupant Ue closing and opening their hand (BP(2)) (moving their hand into a fist and then an open hand), the control device 10 plays the musical instrument sound (musical instrument sound MS(1)) of a cymbal (musical instrument MI(1)). In particular, the control device 10 may play the instrument sound of a cymbal when it detects that the occupant Ue is closing and opening his / her hand while the distance between the part PV(1) of the vehicle 1 and the hand of the occupant Ue is a predetermined value and not changing (e.g., the distance is within a predetermined range). Furthermore, when the control device 10 detects a gesture (gesture GT(1)) in which the occupant Ue stomps his / her foot (BP(4)) on the floor (part PV(2)) of the vehicle 1, it may play the instrument sound (instrument sound MS(2)) of a drum (instrument MI(2)). For example, the control device 10 may play the instrument sound of a drum when it detects that the occupant Ue is stomping his / her foot on the floor or lifting his / her foot from the floor. The control device 10, which has been outlined above, will now be described in detail with reference to FIGS. 1 to 11. To facilitate understanding of the control device 10, an example will be described in which the performance sound PS is each of the instrument sounds MS of a plurality of types of musical instruments MI.

[0015] §2 Configuration example FIG. 1 is a block diagram showing a schematic configuration of a vehicle 1 equipped with a control device 10, and FIG. 2 is a diagram showing an example of the interior of a vehicle, showing an example of an installation location of the control device 10 shown in FIG. 1. As shown in FIG. 1, the control device 10 includes, as functional blocks, an imaging unit 110, a determination unit 120, a storage unit 130, a signal generation unit 140, and a transmission unit 150. In the control device 10, the functional blocks from the determination unit 120 to the transmission unit 150, other than the imaging unit 110, may be realized by a general-purpose or dedicated computer (information processing device), and may be realized, for example, by a CPU (processor), memory, etc. of such a computer. For example, the control device 10 loads a control program (not shown) stored in the storage unit 130 into a RAM. The control device 10 may then operate as a computer including the determination unit 120 to the transmission unit 150 as software modules by having the CPU interpret and execute instructions included in the control program loaded into the RAM to control each component. The imaging unit 110 and the functional blocks other than the imaging unit 110 (the above-mentioned computer) are connected, for example, by a CAN (Controller Area Network) or other in-vehicle LAN, and can transmit and receive information to and from each other. The external device in Fig. 1 may be, for example, a speaker or other audio output device for outputting voice, sound effects, etc. The control device 10 is provided in the vehicle 1, and in the example shown in Fig. 2, is provided in the interior of the vehicle 1.

[0016] The imaging unit 110 continuously captures images of the interior of the vehicle 1 as a three-dimensional space. For example, the angle of view of the imaging unit 110 is adjusted so that the imaging range covers the entire interior of the vehicle 1. If a single imaging unit 110 cannot cover the entire interior of the vehicle 1, multiple imaging units 110 may be provided, and the multiple imaging units 110 may be configured to cover the entire interior of the vehicle 1. The imaging unit 110 may be, for example, a stereo camera capable of capturing three-dimensional images or a so-called TOF (Time of Flight) camera. The imaging unit 110 may also be configured by combining multiple monocular cameras capable of capturing two-dimensional images. For example, as illustrated in FIG. 2, the imaging unit 110 is provided approximately in the center of the vehicle 1 in the vehicle width direction. In the illustrated example, the imaging unit 110 is provided approximately in the center of the vehicle width direction near the top of the front windshield.

[0017] The determination unit 120 includes a first determination unit 122 that determines whether the steering wheel (part PV(4)) of the vehicle 1 is in a state that does not affect the behavior of the vehicle 1, and a second determination unit 124 that determines the start of reproduction of the performance sound PS. The first determination unit 122 determines, for example, whether the vehicle 1 is in an entertainment mode in which the steered wheels are not turned even when the steering wheel is operated, and if the vehicle 1 is in the entertainment mode, determines that the steering wheel of the vehicle 1 is in a state that does not affect the behavior of the vehicle 1.

[0018] The second determination unit 124 includes a BP detection unit 1242, a positional relationship identification unit 1244, a musical performance action identification unit 1246, and a reproduction determination unit 1248. When the first determination unit 122 determines that the steering wheel of the vehicle 1 is in a state that does not affect the behavior of the vehicle 1, the second determination unit 124 may start executing a process that can reproduce the musical performance sound PS. When the first determination unit 122 determines that the steering wheel of the vehicle 1 is not in a state that does not affect the behavior of the vehicle 1, the second determination unit 124 does not start executing a process that can reproduce the musical performance sound PS. The second determination unit 124 may identify each of one or more occupants Ue of the vehicle 1, and may, for example, acquire identification information ID of each occupant Ue. The identification information ID of each occupant Ue may be associated with information that can identify the type of musical performance sound PS (musical instrument sound MS) to be reproduced for each occupant Ue, such as the musical instrument MI that each occupant Ue likes (e.g., is good at playing).

[0019] The BP detection unit 1242 detects at least the BP of the occupant Ue of the vehicle 1 based on the captured image CI (imaging data) captured by the imaging unit 110. For example, the BP detection unit 1242 detects at least the BP of each of the multiple occupants Ue from the captured image CI. The BP detection unit 1242 may detect one or more BPs for each occupant Ue. For example, the BP detection unit 1242 may detect the finger (BP(1)) and foot (BP(4)) of occupant Ue(1) from the captured image CI(Tx) at a certain time point (t=Tx), and may also detect the hand (BP(2)) of occupant Ue(2) from the captured image CI(Tx). In other words, the BP detection unit 1242 may detect at least one BP of each occupant Ue from the captured image CI. The BP detection unit 1242 may detect each BP of each occupant Ue at each time point based on the captured image CI at each time point.

[0020] The positional relationship specification unit 1244 specifies the position of each BP (each BP position) of each occupant Ue detected by the BP detection unit 1242, and saves (stores) the specified BP position of each occupant Ue as BP position information in the memory unit 130. For example, the positional relationship specification unit 1244 specifies the BP position of each occupant Ue at each time point, and saves the specified BP position of each occupant Ue at each time point as BP position information 132 in the memory unit 130.

[0021] In this embodiment, the positional relationship identification unit 1244 identifies a relative positional relationship RP between each BP of each occupant Ue detected by the BP detection unit 1242 and at least one of the multiple regions PV of the vehicle 1. For example, the positional relationship identification unit 1244 identifies the distance between each BP of each occupant Ue and each of the multiple regions PV as the relative positional relationship RP between each BP of each occupant Ue and each region PV. The positional relationship identification unit 1244 may identify the relative positional relationship RP between each BP of each occupant Ue and each region PV at each point in time.

[0022] The playing action identification unit 1246 determines whether each occupant Ue is performing a gesture GT (music action) using the BP based on the above-mentioned relative positional relationship RP identified by the positional relationship identification unit 1244. In particular, the playing action identification unit 1246 determines whether the occupant Ue is performing a gesture GT using the BP based on the relative positional relationship RP at each time point (change in the relative positional relationship RP). By using the relative positional relationship RP between the BP and the part PV (in particular, the relative positional relationship RP between the BP and the part PV at each time point), the playing action identification unit 1246 can precisely determine whether the occupant Ue is performing a gesture GT using the BP.

[0023] For example, the performance action identification unit 1246 identifies a change (change over time) in the relative positional relationship RP between each BP of each occupant Ue and each part PV from the "relative positional relationship RP between each BP of each occupant Ue and each part PV" at each time point. Then, with reference to the position change performance action correspondence information 134, the performance action identification unit 1246 determines whether the change in the relative positional relationship RP corresponds to a gesture GT predefined in the position change performance action correspondence information 134. As will be described in detail later, the position change performance action correspondence information 134 predefines one or more types of gestures GT, and each of the one or more types of gestures GT is associated with a predetermined "change in the relative positional relationship RP between the BP of the occupant Ue and each part PV of the vehicle 1." In the position change performance action correspondence information 134 according to this embodiment, each of the multiple types of gestures GT is predefined with a respective one of multiple types of "change in the relative positional relationship RP between the BP of the occupant Ue and each part PV of the vehicle 1." The performance action identification unit 1246 determines that the occupant Ue is performing the gesture GT using BP if the identified "change in the relative positional relationship RP" is a change corresponding to the gesture GT defined in the position change / performance action correspondence information 134. If the identified "change in the relative positional relationship RP" is not a change corresponding to the gesture GT defined in the position change / performance action correspondence information 134, the performance action identification unit 1246 determines that the occupant Ue is not performing the gesture GT using BP.

[0024] The performance action identification unit 1246 may further identify the type of gesture GT that the occupant Ue is making using the BP. The performance action identification unit 1246 can precisely identify the type of gesture GT that the occupant Ue is making using the BP by using the "relative positional relationship RP between the BP and the part PV of the occupant Ue (at each time point)." Specifically, the performance action identification unit 1246 identifies which of multiple types of gestures GT that the occupant Ue is making using the BP is predefined in the position change / performance action correspondence information 134. For example, the performance action identification unit 1246 identifies whether the identified "change in the relative positional relationship RP" matches (corresponds to) which of the changes associated with each gesture GT in the position change / performance action correspondence information 134.

[0025] The playing motion identification unit 1246 may further identify the mode of the gesture GT being made by the occupant Ue using the BP. For example, the playing motion identification unit 1246 may identify whether the gesture GT is a "gesture GT of a large motion" in which the change in the relative positional relationship RP is large, or a "gesture GT of a small motion" in which the change in the relative positional relationship RP is small. The playing motion identification unit 1246 may also identify whether the gesture GT is a "gesture GT of a fast motion" in which the speed (acceleration) of the change in the relative positional relationship RP is large, or a "gesture GT of a relaxed motion" in which the speed (acceleration) of the change is small. The playing motion identification unit 1246 can precisely identify the mode of the gesture GT being made by the occupant Ue using the BP by using the "relative positional relationship RP between the BP and the part PV of the occupant Ue (at each time point)."

[0026] The playing action identification unit 1246 may further identify which of the multiple parts PV of the vehicle 1 the gesture GT that the occupant Ue is making using the BP is directed to. As will be described in detail later, in the part-instrument correspondence information 136, each of the multiple parts PV of the vehicle 1 may be associated in advance with a plurality of types of musical instruments MI. Thus, after identifying the part PV that is the target of the gesture GT that the occupant Ue is making using the BP, the playing action identification unit 1246 may further identify the musical instrument MI associated with the identified part PV by referring to the part-instrument correspondence information 136.

[0027] When there is one part PV (musical instrument MI) that can be the target of the gesture GT performed by the occupant Ue using the BP, the playing action identification unit 1246 may determine that the target of the gesture GT is the relevant part PV (musical instrument MI). For example, when only one type of musical instrument MI is previously associated with (one or more) parts PV around the occupant Ue, the playing action identification unit 1246 may determine that the target of the gesture GT is the relevant one type of musical instrument MI (the part PV to which the relevant musical instrument MI is associated). When there are multiple parts PV (musical instrument MI) that can be the target of the gesture GT, for example, when multiple types of musical instruments MI are respectively associated with multiple parts PV around the occupant Ue, the playing action identification unit 1246 may make the following determination. That is, the playing action identification unit 1246 may identify the part PV that is closest to the relevant BP at a certain time as the target of the gesture GT. The "certain time" may be the time when execution of a process capable of reproducing the playing sound PS is started, or the time when the gesture GT is started. When there are multiple parts PV that can be targets of a gesture GT, the playing action identification unit 1246 may identify the part PV whose relative positional relationship RP (e.g., distance) with respect to the BP has changed the most in the gesture GT as the target of the gesture GT. When there are multiple parts PV that can be targets of a gesture GT, the playing action identification unit 1246 may identify the part PV that is the target of the gesture GT from among the multiple parts PV according to the type of the gesture GT. For example, when the gesture GT performed by the occupant Ue using the BP is a playing action of the type "hitting, plucking, and stepping," the playing action identification unit 1246 may identify the part PV associated with an instrument MI that is generally played with the playing action of "hitting, plucking, and stepping" as the target of the gesture GT. For example, when the gesture GT performed by the occupant Ue using the BP is a playing action of the type "rubbing," the playing action identification unit 1246 may identify the part PV associated with an instrument MI that is generally played with the playing action of "rubbing" as the target of the gesture GT.

[0028] The reproduction determination unit 1248 determines whether to start reproduction of the performance sound PS using the results of the determination by the first determination unit 122 and the results of the determination by the second determination unit 124. For example, it determines whether to reproduce the performance sound PS. Specifically, the reproduction determination unit 1248 first confirms whether the first determination unit 122 has determined that the steering wheel of the vehicle 1 is in a state that does not affect the behavior of the vehicle 1. If it is confirmed that the first determination unit 122 has determined that the steering wheel of the vehicle 1 is in a state that does not affect the behavior of the vehicle 1, the reproduction determination unit 1248 further executes the following process. That is, the reproduction determination unit 1248 determines whether the "relative positional relationship RP between the BP position of the occupant Ue and the part PV of the vehicle 1" identified by the positional relationship identification unit 1244 satisfies a predetermined condition. For example, the reproduction determination unit 1248 determines whether a change (change over time) in the "relative positional relationship RP" identified from the "relative positional relationship RP" at each time point satisfies a predetermined condition. Then, when it is confirmed that the relative positional relationship RP (change in the relative positional relationship RP) between the BP position of the occupant Ue and the part PV of the vehicle 1 satisfies a predetermined condition, the playback determination unit 1248 decides to start playing the performance sound PS, for example, decides to play the performance sound PS.

[0029] The reproduction determination unit 1248 may determine whether the relative positional relationship RP between the BP position of the occupant Ue and the part PV of the vehicle 1 (change in the relative positional relationship RP) satisfies a predetermined condition based on the determination result of the playing action identification unit 1246. For example, the reproduction determination unit 1248 checks whether the playing action identification unit 1246 has determined that the occupant Ue is performing a gesture GT (playing action) using the BP. Then, when it is confirmed that the playing action identification unit 1246 has determined that the occupant Ue is performing a gesture GT using the BP, the reproduction determination unit 1248 may determine that the relative positional relationship RP between the BP position of the occupant Ue and the part PV of the vehicle 1 satisfies a predetermined condition.

[0030] (About the performance sound according to the movement of BP) When the musical performance action specification unit 1246 specifies the type of gesture GT (type of musical performance action) that the occupant Ue is making using the BP, the reproduction determination unit 1248 may perform the following process: That is, the reproduction determination unit 1248 may determine to reproduce the musical performance sound PS that corresponds to the type of the specified gesture GT (musical performance action).

[0031] As described above, the second determination unit 124 (particularly, the playing action identification unit 1246) may identify the type of gesture GT that the occupant Ue is making using the BP from the "relative positional relationship RP between the BP of the occupant Ue and the part PV of the vehicle 1 (at each time point)" detected based on the captured image CI. That is, the second determination unit 124 may detect at least the movement of the BP of the occupant Ue (change in position (particularly, position relative to the part PV of the vehicle 1)), and may particularly identify the type of movement of the BP. Then, the second determination unit 124 (particularly, the playback determination unit 1248) may determine the start of playback of the playing sound PS corresponding to the type of gesture GT (playing action) identified by the playing action identification unit 1246. That is, the playing sound PS that the playback determination unit 1248 determines to play (start playback) may be changed according to the detected "movement of at least the BP of the occupant Ue." The second determination unit 124 may change the performance sound PS to be reproduced depending on the (type of) movement of the BP, that is, depending on the (type of) gesture GT that the occupant Ue is making using the BP. The control device 10 can reproduce the performance sound PS corresponding to the (type of) movement of the BP of the occupant Ue. In particular, the control device 10 can detect the (type of) movement of the BP with high accuracy by identifying the movement (change in position) of the BP of the occupant Ue in relation to the part PV of the vehicle 1, and reproduce the performance sound PS corresponding to the detected movement of the BP. The performance sound PS may be different, for example, depending on the contact form between the BP of the occupant Ue and the part PV of the vehicle 1.

[0032] (Sounds played according to vehicle parts) When the performance action identification unit 1246 identifies the part PV (instrument MI) that is the target of the gesture GT that the occupant Ue is making using the BP, the playback determination unit 1248 may decide to play the instrument sound MS of the identified instrument MI as the performance sound PS.

[0033] As described above, the reproduction determination unit 1248 determines the start of reproduction of the performance sound PS based on the relative positional relationship RP between at least the BP of the occupant Ue and at least a part of the body part PV of the vehicle 1. In particular, in this embodiment, each of the multiple body parts PV of the vehicle 1 is associated with one of multiple types of musical instruments MI or one of multiple types of animals AM. Therefore, the second determination unit 124 (particularly the reproduction determination unit 1248) may make the following determination based on the relative positional relationship RP between at least the BP of the occupant Ue detected by the BP detection unit 1242 and one of the multiple body parts PV of the vehicle 1. That is, the second determination unit 124 may determine the start of reproduction of the musical instrument sound MS(x) (sound CR(x)) of the musical instrument MI(x) (animal AM(x)) associated with one such body part PV (e.g., body part PV(x)). For example, the second determination unit 124 determines to reproduce, as the performance sound PS, the instrument sound MS of the instrument MI associated with the part PV identified as the target of the gesture GT being made by the occupant Ue using the BP. The control device 10 determines whether to start reproduction of the instrument sound MS (the sound CR) of the instrument MI (animal AM) associated with the part PV of the vehicle 1, particularly the "part PV of the vehicle 1" for which the relative positional relationship RP of the occupant Ue with the BP is determined. That is, the control device 10 determines whether the occupant Ue is performing a performance operation (gesture GT) on the instrument MI (animal AM) using the BP based on the relative positional relationship RP between the part PV with which the instrument MI is previously associated and the BP of the occupant Ue. Therefore, the control device 10 can accurately determine whether the occupant Ue is performing a gesture GT on the instrument MI (animal AM) using the BP.

[0034] When the first determination unit 122 determines that the steering wheel of the vehicle 1 is in a state that affects the behavior of the vehicle 1, the reproduction determination unit 1248 determines not to start reproduction of the performance sound PS, for example, does not reproduce the performance sound PS. Furthermore, when the first determination unit 122 determines that the relative positional relationship RP between the BP position of the occupant Ue and the part PV of the vehicle 1 does not satisfy a predetermined condition, the reproduction determination unit 1248 determines not to start reproduction of the performance sound PS, for example, does not reproduce the performance sound PS. For example, when the performance action identification unit 1246 determines that the occupant Ue is not performing a gesture GT using the BP, the reproduction determination unit 1248 determines not to start reproduction of the performance sound PS, for example, does not reproduce the performance sound PS.

[0035] The storage unit 130 is an example of a memory resource, and is configured, for example, with a hard disk drive, a solid state drive, etc. In this embodiment, the storage unit 130 stores various information such as BP position information 132, part-instrument correspondence information 136, and position change performance action correspondence information 134.

[0036] The BP position information 132 is information indicating the position of each BP (each BP position) of each occupant Ue, for example, information indicating each BP position of each occupant Ue at each time point. The BP position information 132 is saved (stored) in the memory unit 130 by the positional relationship specification unit 1244.

[0037] The position-change musical performance action correspondence information 134 is information that associates "(combinations of) relative positional relationships RP between the BP of the occupant Ue and the parts PV of the vehicle 1 (at each of a plurality of points in time)" with "gestures GT (music actions) performed by the occupant Ue using the BP." In other words, the position-change musical performance action correspondence information 134 is information that associates "changes in the relative positional relationship RP (changes over time)" with "gestures GT (music actions)." For example, in the position-change musical performance action correspondence information 134, multiple types of gestures GT are previously associated with multiple types of "changes in the relative positional relationship RP." Specifically, in the position-change musical performance action correspondence information 134, a certain "change in the relative positional relationship RP" may be associated with a gesture GT(1) (music action) of the "hitting, plucking, or stepping" type. Furthermore, another "change in the relative positional relationship RP" may be associated with a gesture GT(2) of the "rubbing" type. As explained above, the position change / music performance action correspondence information 134 may associate "(combinations of) relative positional relationships RP between the BP of the occupant Ue and the part PV of the vehicle 1 (at each of a plurality of points in time)" with "(each type of) gesture GT." For example, the position change / music performance action correspondence information 134 may associate "(conditions related to) the distance between the BP and the part PV at each of a plurality of consecutive points in time" with "each type of gesture GT."

[0038] The part-instrument correspondence information 136 is information that associates parts PV of the vehicle 1 with musical instruments MI (animal AM), for example, information that associates at least one part PV with at least one musical instrument MI (animal AM). In the part-instrument correspondence information 136 according to this embodiment, each of the multiple parts PV is previously associated with one of multiple types of musical instruments MI or one of multiple types of animals AM. The association between the part PV and the musical instrument MI may be performed by the occupant Ue, by a person other than the occupant Ue, or by a device such as the control device 10 or an external device. Furthermore, the association between the part PV and the musical instrument MI may be changed at a predetermined timing (for example, on a daily basis).

[0039] The body part instrument correspondence information 136 may associate one or more body part PVs of the vehicle 1 with one or more musical instruments MI for each seat position of the vehicle 1 (or each seat position of the occupant Ue). For example, in the body part instrument correspondence information 136, each body part PV may be associated with a musical instrument MI or an animal AM for each seat position of the vehicle 1. For example, in the body part instrument correspondence information 136, multiple body part PVs of the vehicle 1 may be grouped by seat position, and a musical instrument MI (animal AM) of a specific attribute (type) may be associated with the body part PV for each group.

[0040] FIG. 3 shows an example in which regions PV and musical instruments MI are associated with each seat position of the vehicle 1. In the example shown in FIG. 3, the vehicle 1 includes a driver's seat (seat ST(1)), a passenger seat (seat ST(2)), a first rear seat behind the driver's seat (seat ST(3)), and a second rear seat behind the passenger seat (seat ST(4)), and each seat ST is associated with a group GR. In the example shown in FIG. 3, the group GR(1) is associated with the seat ST(1), the group GR(2) is associated with the seat ST(2), the group GR(3) is associated with the seat ST(3), and the group GR(4) is associated with the seat ST(4). The region-musical instrument correspondence information 136 may associate one or more musical instruments MI with each of one or more region PV belonging to the group GR(1) (in other words, arranged in a position or area corresponding to the group GR(1)). The part instrument correspondence information 136 may associate one or more part PVs belonging to group GR(2) with one or more musical instruments MI, respectively. Similarly, the part instrument correspondence information 136 may associate one or more part PVs belonging to groups GR(3) and GR(4) with one or more musical instruments MI.

[0041] The part-instrument correspondence information 136 may associate a different type (attribute, type) of musical instrument MI with a part PV belonging to each group GR. For example, the part-instrument correspondence information 136 may associate a drum (musical instrument MI(2)) with a part PV(3-x) belonging to group GR(3), and a trumpet (musical instrument MI(3)) with a part PV(4-y) belonging to group GR(4). By using the part-instrument correspondence information 136 that associates a specific type of musical instrument MI with a part PV of the vehicle 1 for each group GR (seat ST), the control device 10 achieves the following effects. That is, the control device 10 enables, for example, multiple occupants Ue of the vehicle 1 to play musical instruments MI together without feeling any discomfort.

[0042] The body part instrument correspondence information 136 may associate a different type of instrument MI with a body part PV belonging to multiple groups GR (for example, when areas corresponding to multiple groups GR overlap, the body part PV arranged in the overlapping area). For example, in the example shown in FIG. 3, the area corresponding to group GR(3) overlaps with the area corresponding to group GR(4). Therefore, the body part instrument correspondence information 136 may associate a body part PV belonging to both group GR(3) and group GR(4) (for example, the armrest (body part PV(34)) between seat ST(3) and seat ST(4)) with an instrument MI as follows: That is, the body part instrument correspondence information 136 may associate a drum with the body part PV(34) for seat ST(3), and a trumpet with the body part PV(34) for seat ST(4). The body part / instrument correspondence information 136 may associate a drum with the body part PV(34) for the occupant Ue(3) sitting in the seat ST(3), and a trumpet with the body part PV(34) for the occupant Ue(4) sitting in the seat ST(4). By associating different types of musical instruments MI with each seat ST (occupant Ue sitting in the seat ST) even for the same body part PV in the body part / instrument correspondence information 136, the control device 10 can achieve the following effect. That is, the control device 10 can play instrument sounds MS of different types of musical instruments MI for each seat ST (occupant Ue sitting in the seat ST) even for gestures GT (musical performance actions) for the same body part PV. For example, the control device 10 may play a drum instrument sound (instrument sound MS(2)) in response to a gesture GT by the occupant Ue(3) sitting in the seat ST(3) with respect to the armrest (body part PV(34)) between the seat ST(3) and the seat ST(4). Furthermore, the control device 10 may play a trumpet instrument sound (instrument sound MS(3)) in response to a gesture GT made by the occupant Ue(4) sitting in the seat ST(4) with respect to the armrest.

[0043] The body part instrument correspondence information 136 may associate one or more body part PVs of the vehicle 1 with one or more musical instruments MI for each identification information ID of each occupant Ue of the vehicle 1. For example, the body part instrument correspondence information 136 may associate one or more body part PVs arranged around the seating position of the occupant Ue with one or more musical instruments MI for each identification information ID. For example, the body part instrument correspondence information 136 may associate a drum with a body part PV arranged around the seating position of an occupant Ue who likes to play the drums, and a trumpet with a body part PV arranged around the seating position of an occupant Ue who likes to play the trumpet. As described above, the control device 10 (e.g., the second determination unit 124) may acquire the identification information IDs of one or more occupants Ue of the vehicle 1. Then, the control device 10 may use the acquired identification information IDs and the above-described body part instrument correspondence information 136 to cause the occupant Ue corresponding to each identification information ID to play the musical instrument MI corresponding to each identification information ID.

[0044] When the reproduction determination unit 1248 determines to start reproduction of the performance sounds PS (to reproduce the performance sounds PS), the signal generation unit 140 generates a signal corresponding to the performance sounds PS to be reproduced. The transmission unit 150 transmits the signal generated by the signal generation unit 140 to an external device (an audio output device such as a speaker) to reproduce the performance sounds PS.

[0045] §3 Example of operation FIG. 4 is a flowchart showing an example of the information processing procedure executed by the control device 10 according to this embodiment. The procedure described below is an example of an information processing method PM in which a processor (e.g., the CPU of the control device 10) executes processing to reproduce the performance sound PS. However, the procedure described below is merely an example, and each step may be modified as much as possible. Furthermore, steps in the procedure described below may be omitted, replaced, or added as appropriate depending on the embodiment. Furthermore, the processing procedure described below is executed by, for example, the CPU of the control device 10, but for simplicity of explanation, the "CPU of the control device 10" will be abbreviated as "control device 10" below.

[0046] (Step S10) In step S10, the control device 10 operates as the first determination unit 122 and determines whether the steering wheel of the vehicle 1 is in a state that does not affect the behavior of the vehicle 1. If it is determined that the steering wheel of the vehicle 1 is in a state that does not affect the behavior of the vehicle 1 (Yes in step S10), the control device 10 proceeds to step S20. If it is determined that the steering wheel of the vehicle 1 is in a state that affects the behavior of the vehicle 1 (No in step S10), the control device 10 repeats the determination in step S10 until it determines that the steering wheel of the vehicle 1 is in a state that does not affect the behavior of the vehicle 1.

[0047] (Step S20) In step S20, the control device 10 operates as the second determination unit 124 and starts executing a process capable of reproducing the performance sound PS. When it is determined in step S10 that "the steering wheel of the vehicle 1 is in a state that does not affect the behavior of the vehicle 1" and the control device 10 receives a user operation instructing the execution of a process capable of reproducing the performance sound PS, the control device 10 may start executing the process capable of reproducing the performance sound PS. Such a user operation may be a predetermined gesture GT (gesture GT instructing the execution of a process capable of reproducing the performance sound PS) by the occupant Ue of the vehicle 1.

[0048] (Step S30) In step S30, the control device 10 operates as the BP detection unit 1242 and acquires captured images CI (captured data) captured by the imaging unit 110, which continuously captures images of the interior (car interior) of the vehicle 1 as a three-dimensional space. Then, the control device 10 detects at least the BP (part of the body) of each occupant Ue of the vehicle 1 from the captured images CI.

[0049] (Step S40) In step S40, the control device 10 operates as the reproduction determination unit 1248 and determines whether the relative positional relationship RP between the BP of the occupant Ue detected in step S30 and at least a portion of the region PV of the vehicle 1 satisfies a predetermined condition. If it is determined that the relative positional relationship RP between the BP of the occupant Ue and at least a portion of the region PV of the vehicle 1 satisfies the predetermined condition (Yes in step S40), the control device 10 proceeds to step S50. If it is determined that the relative positional relationship RP between the BP of the occupant Ue and at least a portion of the region PV of the vehicle 1 does not satisfy the predetermined condition (No in step S40), the control device 10 returns to the determination in step S10.

[0050] (Step S50) In step S50, the control device 10 operates as the transmitting unit 150 and transmits a signal corresponding to the performance sound PS to an external device (an audio output device such as a speaker) to play the performance sound PS. When the control device 10 identifies the type of gesture GT (musical performance action) being performed by the occupant Ue using the BP, it may play the performance sound PS corresponding to the identified type of musical performance action (gesture GT). When the control device 10 identifies the part PV (musical instrument MI) that is the target of the gesture GT being performed by the occupant Ue using the BP, it may play the musical instrument sound MS of the identified musical instrument MI as the performance sound PS.

[0051] <1. Identifying the type of gesture (musical action)> As described above, the control device 10 (the reproduction determination unit 1248) checks whether the relative positional relationship RP (a change in the relative positional relationship RP) between the BP of the occupant Ue and the part PV of the vehicle 1 (at each time point) satisfies a predetermined condition, and determines whether to start reproduction of the performance sound PS. For example, when the control device 10 determines that a certain "change in the relative positional relationship RP" corresponds to one of a plurality of predefined types of gestures GT, the control device 10 decides to start reproduction of the performance sound PS. In other words, when a certain "change in the relative positional relationship RP" is one of the changes pre-associated with each of a plurality of types of gestures GT in the position change / performance action correspondence information 134, the control device 10 decides to start reproduction of the performance sound PS. In particular, when the control device 10 identifies the type of gesture GT corresponding to a certain "change in the relative positional relationship RP," it reproduces the performance sound PS corresponding to the identified type of gesture GT. An example of the process executed by the control device 10 to "identify the type of gesture GT made by the occupant Ue using the BP and play back the performance sound PS corresponding to the identified type of gesture GT" will be described below with reference to FIGS. 5 to 7.

[0052] <1.1. Hitting, plucking, and stomping gestures (performance movements)> FIG. 5A is a flowchart showing an example of a processing procedure for identifying a "hit, plucking, stepping" gesture GT(1) as exemplified in FIG. 6 and reproducing a performance sound PS corresponding to the gesture GT(1).

[0053] FIG. 6 shows an example in which an occupant Ue (in the illustrated example, the driver (occupant Ue(1))) of a vehicle 1 is making a "smacking" gesture GT(1) using his / her hand (BP(2)). In FIG. 6, the BP position (t=T0) indicates the position of the hand of the occupant Ue(1) at a first time point (t=T0), and in particular indicates the position of the hand gripping the steering wheel (site PV(4)). That is, the BP position (t=T0) is the position of the driver's hand gripping the steering wheel (BP position), as well as the position of the steering wheel (site PV(4)), in particular the position of the gripped part of the steering wheel. The BP position (t=T1) indicates the position of the hand of the occupant Ue(1) at a second time point (t=T1) that is later than the first time point. The BP position (t=T2) indicates the position of the hand of the occupant Ue(1) at a third time point (t=T2) that is later than the second time point.

[0054] As illustrated in FIG. 6, in the “hit” gesture GT(1), the occupant Ue first increases the distance (relative distance) between the BP and the part PV of the vehicle 1 (the steering wheel in the illustrated example), for example, increasing the distance between them above a threshold value TH(1). In the illustrated example, the occupant Ue changes the position of the BP from the BP position (t=T0) to the BP position (t=T1), that is, moves the BP away from the part PV. Next, the occupant Ue decreases the “distance between the BP and the part PV” that was once increased, for example, decreasing the distance between them below a threshold value TH(2). In the illustrated example, the occupant Ue changes the position of the BP from the BP position (t=T1) to the BP position (t=T2), that is, moves the BP closer to the part PV (for example, causes the BP to collide with the part PV). In the illustrated example, the threshold value TH(1) is greater than the threshold value TH(2). By setting the threshold value TH(1) to be greater than the threshold value TH(2), it is possible to precisely determine whether the gesture GT(1) of "tapping" the part PV with the BP is being performed.

[0055] When detecting the "(change in) relative positional relationship RP between the BP and the part PV," the control device 10 determines that the occupant Ue is making a "hitting" gesture GT(1) using the BP, and plays a performance sound PS corresponding to the gesture GT(1). Specifically, the control device 10 executes the following steps S110 to S160 to determine the "hitting" gesture GT(1) and plays a performance sound PS corresponding to the determined gesture GT(1).

[0056] (Steps S110 to S130) Steps S110, S120, and S130 are similar to steps S10, S20, and S30, respectively, described with reference to FIG. 4, and therefore will not be described in detail.

[0057] (Step S140) In step S140, the control device 10 operates as the playing motion identification unit 1246 and performs the following determination regarding the relative distance (an example of the relative positional relationship RP) between the BP of the occupant Ue detected in step S130 and at least a part of the part PV of the vehicle 1. That is, the control device 10 determines whether the distance between the BP of the occupant Ue and the part PV of the vehicle 1 at a certain point in time (for example, a second point in time (t=T1) in FIG. 6) is greater than a first distance (for example, a threshold value TH(1) in FIG. 6). If the distance between them is greater than the first distance (Yes in step S140), the control device 10 determines that the occupant Ue has moved the BP away from the part PV, and proceeds to step S150. If the distance between them is equal to or less than the first distance (No in step S140), the control device 10 returns to the determination in step S110.

[0058] (Step S150) In step S150, the control device 10 operates as the playing action identification unit 1246 and performs the following determination regarding the relative distance between the BP of the occupant Ue and the location PV of the vehicle 1. That is, the control device 10 determines whether the distance between the BP of the occupant Ue and the location PV of the vehicle 1 at the next time point (for example, the third time point (t=T2) in FIG. 6) is smaller than a second distance (for example, the threshold value TH(2) in FIG. 6). If the distance between the two is smaller than the second distance (Yes in step S150), the control device 10 determines that the occupant Ue has brought the BP closer to (collided with) the location PV, identifies a "hitting" gesture GT(1) as exemplified in FIG. 6, and proceeds to step S160. If the distance between the two is equal to or greater than the second distance (No in step S150), the control device 10 returns to the determination in step S110.

[0059] (Step S160) In step S160, the control device 10 causes an external device (a sound output device such as a speaker) to reproduce the performance sound PS corresponding to the "hit" gesture GT(1) identified in step S150.

[0060] As described above, when the control device 10 determines that the occupant Ue is using the BP to make a "hitting" gesture GT(1) like a percussion instrument on the part PV that the BP was in contact with (e.g., gripping), the control device 10 determines to start playing the performance sound PS. For example, the control device 10 determines to play the performance sound PS corresponding to the "hitting" gesture GT(1). For example, when the control device 10 determines that the relative distance between the part PV of the vehicle 1 and the BP of the occupant Ue (e.g., at least a part of a hand or foot) increases and then falls below a predetermined value, the control device 10 determines that the occupant Ue is making the "hitting" gesture GT(1) using the BP. Then, the control device 10 determines whether to start playing the performance sound PS corresponding to the "hitting" gesture GT(1). The control device 10 may detect each finger of the occupant Ue as the BP of the occupant Ue and identify the "hitting" gesture GT(1) made by each finger. The control device 10 may assign different performance sounds PS (scales) to each finger. For example, the control device 10 may assign Do, Re, Mi, ... or chords (C, D, E, F, ...) to each finger from the thumb to the little finger. The control device 10 may change the playback method of the performance sounds PS depending on the magnitude, speed, acceleration, etc. of the change in the relative distance between the BP of the occupant Ue and the part PV of the vehicle 1. For example, when a gesture GT is made that significantly moves the position of the BP (e.g., moves it farther away from the part PV of the vehicle 1) or increases the speed of the change in the relative distance (i.e., taps quickly), the control device 10 may increase the volume of the playback sound PS. Furthermore, the control device 10 may change the condition (threshold TH) for determining whether a "tapping" gesture GT(1) is being performed depending on the BP (type of BP). For example, the threshold TH used to determine whether a finger (BP(1)) is being tapped may be smaller than the threshold TH used to determine whether a hand (BP(2)) is being tapped. The threshold value TH used for determining the hand may be smaller than the threshold value TH used for determining the foot (BP(4)). By changing the condition (threshold value TH) used for determining whether the gesture GT is being performed using the BP according to the type of BP, the control device 10 can determine whether the gesture GT is being performed using the BP appropriately according to the type of BP.Similarly, the control device 10 may change the condition (threshold value TH) used when determining whether the gesture GT is being made, depending on the part PV of the vehicle 1 (type of part PV).

[0061] <1.2. Rubbing gesture (musical action)> FIG. 5B is a flowchart showing an example of a processing procedure for identifying a "rub" gesture GT(2) as shown in FIG. 7 and playing back a performance sound PS corresponding to the gesture GT(2).

[0062] FIG. 7 shows an example in which an occupant Ue of a vehicle 1 (in the illustrated example, the driver (occupant Ue(1))) is making a "rubbing" gesture GT(2) using his / her hand (BP(2)). In FIG. 7, the BP position (t=T0) indicates the position of the hand of the occupant Ue(1) at a first time point (t=T0), and in particular indicates the position of the hand gripping the steering wheel (site PV(4)), and is similar to the BP position (t=T0) described using FIG. 6. The "hand position of the occupant Ue" may be the representative coordinate of the hand of the occupant Ue. The BP position (t=T1) indicates the position of the hand of the occupant Ue(1) at a second time point (t=T1) that is later than the first time point. The BP position (t=T2) indicates the position of the hand of the occupant Ue(1) at a third time point (t=T2) that is later than the second time point.

[0063] As illustrated in FIG. 7, in the "rub" gesture GT(2), the occupant Ue continues to change the position of the BP while keeping the distance between the BP and the part PV of the vehicle 1 (the steering wheel in the illustrated example) small (e.g., the distance in the axial direction of the steering wheel). That is, in the "rub" gesture GT(2), the occupant Ue moves the BP along the part PV, for example, while maintaining the distance between the BP and the part PV smaller than a threshold value TH(3) (not illustrated). In the illustrated example, the occupant Ue changes the position of the BP from the BP position (t=T0) to the BP position (t=T1), and further from the BP position (t=T1) to the BP position (t=T2), while maintaining the distance from the part PV smaller than the threshold value TH(3).

[0064] When detecting such "(change in) the relative positional relationship RP between the BP and the part PV," the control device 10 determines that the occupant Ue is making a "rubbing" gesture GT(2) using the BP, and plays a performance sound PS corresponding to the gesture GT(2). Specifically, the control device 10 executes the following steps S210 to S250 to determine the "rubbing" gesture GT(2) and plays a performance sound PS corresponding to the determined gesture GT(2).

[0065] (Steps S210 to S230) Steps S210, S220, and S230 are similar to steps S10, S20, and S30, respectively, described with reference to FIG. 4, and therefore details thereof will be omitted.

[0066] (Step S240) In step S240, the control device 10 operates as the playing motion identification unit 1246 and performs the following determination on the BP of the occupant Ue detected in step S230. That is, the control device 10 determines whether the distance between the BP of the occupant Ue and the part PV of the vehicle 1 is smaller than a third distance (for example, the above-mentioned threshold value TH(3)) and the position of the BP is continuously changing. "The distance between the BP of the occupant Ue and the part PV of the vehicle 1 is smaller than the third distance" may indicate, for example, a state in which "the BP and the part PV are in contact." If the distance between the BP and the part PV is smaller than the third distance and the position of the BP is continuously changing (Yes in step S240), the control device 10 determines that "the occupant Ue is moving the BP while keeping it in contact with the part PV (or along the part PV)." That is, the control device 10 determines that the occupant Ue is using the BP to make the "rubbing" gesture GT(2) as illustrated in Fig. 7 (identifies that the occupant Ue is using the BP to make the "rubbing" gesture GT(2)), and proceeds to step S250. If the distance between the BP of the occupant Ue and the part PV of the vehicle 1 is equal to or greater than the third distance, or if the position of the BP is not continuing to change (No in step S240), the control device 10 returns to the determination of step S210.

[0067] (Step S250) In step S250, the control device 10 causes an external device (a sound output device such as a speaker) to reproduce the performance sound PS corresponding to the "rub" gesture GT(2) identified in step S240.

[0068] As described above, when the control device 10 determines that the occupant Ue is using the BP to make a "rubbing" gesture GT(2) on the part PV that the BP was in contact with (e.g., gripping), the control device 10 determines to start playing the performance sound PS. For example, the control device 10 determines to play the performance sound PS corresponding to the "rubbing" gesture GT(2). For example, the control device 10 determines whether the distance between the part PV of the vehicle 1 and the BP (e.g., at least a part of a hand or foot) of the occupant Ue is equal to or less than a predetermined value (e.g., the two are in contact) and the position of the BP is continuously changing. The control device 10 may continue playing the performance sound PS corresponding to the "rubbing" gesture GT(2) while the distance between the part PV and the BP of the occupant Ue is equal to or less than the predetermined value and the position of the BP is continuously changing. The control device 10 may change the playback method of the performance sound PS depending on the magnitude, speed, acceleration, etc. of the change in the relative distance between the BP of the occupant Ue and the part PV of the vehicle 1. For example, if the change in the position of the BP is rapid (that is, if a quick "rubbing" gesture GT(2) is made), the control device 10 may raise the pitch of the played sound PS.

[0069] 2. Gesture (music movement) detection based on speed, acceleration, etc. of position change The control device 10 may determine to start reproducing the performance sound PS when the speed, acceleration, etc. of the change in the "relative positional relationship RP between the BP and the part PV" satisfy a predetermined condition. For example, the control device 10 may determine to start reproducing the performance sound PS for a general user (a passenger Ue other than the driver) when the speed (acceleration) of the change in the positional relationship between the BP and the part PV satisfies a predetermined condition. Specifically, the control device 10 may determine to start reproducing the performance sound PS for a general user when the distance between the BP and the part PV is equal to or less than a predetermined value and the speed (or acceleration) of the BP in the direction approaching the part PV, or the change in the speed (or acceleration), is greater than a predetermined value. An example of the process of the control device 10 for determining to start reproducing the performance sound PS for a general user based on the speed, acceleration, etc. of the change in the position of the BP (relative to the part PV) will be described below with reference to FIG. 8. FIG. 8 is a flowchart showing an example of the process of "determining to start reproducing the performance sound PS based on the acceleration of the change in the position of the BP" executed by the control device 10 for a general user.

[0070] (Steps S310 to S330) Steps S310, S320, and S330 are similar to steps S10, S20, and S30, respectively, described with reference to FIG. 4, and therefore will not be described in detail.

[0071] (Step S340) In step S340, the control device 10 operates as the playback determination unit 1248 and performs the following determination on the BP of the occupant Ue (e.g., a general user) detected in step S330. That is, the control device 10 determines whether the change in acceleration (acceleration per unit time) of the change in position of the BP of the occupant Ue (change in position relative to the part PV) is greater than a fourth reference value (threshold value TH(4)). The phrase "the change in acceleration of the change in the position of the BP is greater than the fourth reference value" may refer to, for example, "the BP suddenly started moving quickly" or "the BP suddenly stopped." If the change in acceleration of the change in the position of the BP is greater than the fourth reference value (Yes in step S340), the control device 10 proceeds to step S350. If the change in acceleration of the change in the position of the BP is equal to or less than the fourth reference value (No in step S340), the control device 10 returns to the determination in step S310.

[0072] (Step S350) In step S350, the control device 10 causes an external device (a sound output device such as a speaker) to reproduce the performance sound PS.

[0073] <3. Gestures (musical actions) by drivers> The control device 10 may identify the type of gesture GT performed by the driver (occupant Ue(1)) among the occupants Ue using the BP through the processing illustrated in FIGS. 9 and 10, and may play a performance sound PS corresponding to the identified type of gesture GT. In particular, the control device 10 may determine the start of playback of the performance sound PS based on the relative positional relationship RP between the driver's BP and a driving region DPV (e.g., the steering wheel (region PV(4)), the shift lever (region PV(5)), etc.) through the processing illustrated in FIGS. 9 and 10. FIGS. 9 and 10 each show an example of a processing procedure for determining whether to start playback of the performance sound PS based on the relative positional relationship RP between the steering wheel, one of the driving regions DPV, and the driver's BP (hands or fingers) (changes in the relative positional relationship RP).

[0074] <3.1. Driver's gestures of hitting, playing, and stepping (musical performance)> FIG. 9 is a flowchart showing an example of a processing procedure for identifying a gesture GT(1) of "hitting, flipping, stepping" performed by the driver using the BP, and for playing a performance sound PS corresponding to the gesture GT(1).

[0075] (Steps S410 to S430) Steps S410, S420, and S430 are similar to steps S10, S20, and S30, respectively, described using Fig. 4, and therefore will not be described in detail. For example, in step S430, the control device 10 (BP detection unit 1242) detects the hand (BP(2)) of the driver (occupant Ue(1)) from the captured image CI. The control device 10 may also detect the driver's finger (BP(1)) from the captured image CI.

[0076] (Step S440) In step S440, the control device 10 operates as the positional relationship identification unit 1244, and when the driver is gripping the steering wheel, detects the gripping position of the steering wheel and records it as a reference position. As described above, the control device 10 (positional relationship identification unit 1244) stores the position of each BP (each BP position) of each occupant Ue at each time point in the storage unit 130 as BP position information 132. Therefore, when the control device 10 determines that "the driver is gripping the steering wheel at a certain time point," it records the hand position (grip position) at that time point as a reference position in the BP position information 132. For example, the control device 10 records the BP position (t=T0) in FIG. 6 as the reference position.

[0077] (Step S450) In step S450, the control device 10 operates as the playing action identification unit 1246 and performs the following determination regarding the position of the driver's hands at the next time point (a time point after the time point at which it is determined that the steering wheel is being gripped; for example, the second time point (t=T1) in FIG. 6). That is, the control device 10 determines whether the relative distance between the position of the driver's hands at the next time point and the most recent reference position is greater than a fifth distance (for example, the threshold value TH(1) in FIG. 6). Explaining with reference to FIG. 6, the control device 10 determines whether the distance between the position of the driver's hands (BP position (t=T1)) at the second time point (t=T1) that is later than the first time point and the reference position (BP position (t=T0)) is greater than the fifth distance. If the distance between the two is greater than the fifth distance (Yes in step S450), the control device 10 proceeds to step S460. If the distance between them is equal to or less than the fifth distance (No in step S450), the control device 10 returns to the determination in step S410.

[0078] (Step S460) In step S460, the control device 10 performs the following determination on the position of the driver's hand at a later time point (a time point later than the time point at which it is determined that the distance between the driver's hand position and the reference position is greater than the fifth distance; for example, the third time point (t=T2) in FIG. 6). That is, the control device 10 determines whether the relative distance between the driver's hand position at the later time point and the most recent reference position is less than a sixth distance (for example, the threshold value TH(2) in FIG. 6). Explaining with reference to FIG. 6, the control device 10 determines whether the distance between the driver's hand position (BP position (t=T2)) at the third time point (t=T2) that is later than the second time point and the reference position (BP position (t=T0)) is less than the sixth distance. If the distance between the two is less than the sixth distance (Yes in step S460), the control device 10 determines that the driver has made a "clap" gesture GT(1) with their hands, and proceeds to step S470. That is, if the distance between them is shorter than the sixth distance, the control device 10 determines that the gesture made by the driver with his / her hand is the "tapping" gesture GT(1), and proceeds to step S470. If the distance between them is equal to or longer than the sixth distance (No in step S460), the control device 10 returns to the determination in step S410.

[0079] (Step S470) In step S470, the control device 10 causes an external device (a sound output device such as a speaker) to reproduce the performance sound PS corresponding to the "hit" gesture GT(1) identified in step S460.

[0080] <3.2. Rubbing gesture (musical performance) by the driver> FIG. 10 is a flowchart showing an example of a processing procedure for identifying a "rub" gesture GT(2) made by the driver using the BP and playing back a performance sound PS corresponding to the gesture GT(2).

[0081] (Steps S510 to S540) Steps S510, S520, S530, and S540 are similar to steps S410, S420, S430, and S440, respectively, described with reference to FIG. 9, and therefore details thereof will be omitted.

[0082] (Step S550) In step S550, the control device 10 operates as the playing action identification unit 1246 and performs the following determination regarding the position of the driver's hands at the next time point (a time point after the time point at which it is determined that the driver is gripping the steering wheel; for example, the second time point (t=T1) in FIG. 7). That is, the control device 10 determines whether the relative distance between the position of the driver's hands at the next time point and the most recent reference position is greater than a seventh distance (for example, a threshold value TH(3)). Explaining with reference to FIG. 7, the control device 10 determines whether the distance between the position of the driver's hands (BP position (t=T1)) at the second time point (t=T1) that is later than the first time point and the reference position (BP position (t=T0)) is greater than the seventh distance. If the distance between the two is greater than the seventh distance (Yes in step S550), the control device 10 determines that the driver has changed the position at which he or she grips the steering wheel, and proceeds to step S560. If the distance between them is equal to or less than the seventh distance (No in step S550), the control device 10 returns to the determination in step S510.

[0083] (Step S560) In step S560, the control device 10 performs the following determination on the position of the driver's hands at a later time point (a time point after the time point at which it is determined that the distance between the driver's hand position and the reference position is greater than the seventh distance; for example, the third time point (t=T2) in FIG. 7). That is, the control device 10 determines whether the relative distance between the driver's hand position at a later time point and the most recent reference position is smaller than the eighth distance (for example, the above-mentioned threshold value TH(2)) and whether the minimum value of the relative distance from the entire steering wheel is smaller than the eighth distance. For example, "the relative distance between the driver's hand position at a later time point and the most recent reference position is smaller than the eighth distance" may indicate a state such as "the driver has returned the grip position of the steering wheel." For example, "the minimum value of the relative distance between the driver's hand position at a later time point and the entire steering wheel is smaller than the eighth distance" may indicate a state such as "the driver has moved his / her hands while keeping them in contact with the steering wheel or along the steering wheel." Explaining this with reference to FIG. 7, the control device 10 determines whether the distance between the driver's hand position (BP position (t=T2)) at a third time point (t=T2) after the second time point and the reference position (BP position (t=T0)) is smaller than the eighth distance. The control device 10 also determines whether the distance between the driver's hand position (BP position (t=T2)) at the third time point and the entire steering wheel position is smaller than the eighth distance. If the relative distance between the driver's hand position and the most recent reference position is smaller than the eighth distance and the minimum value of the relative distance between the driver's hand position and the entire steering wheel is smaller than the eighth distance (Yes in step S560), the control device 10 determines that the "rubbing" gesture GT(2) is being performed. In other words, if the driver moves their hand while keeping it in contact with the steering wheel (or along the steering wheel), the control device 10 determines (specifies) that the driver is performing the "rubbing" gesture GT(2) on the steering wheel, and proceeds to step S570. If the relative distance between the driver's hand position and the nearest reference position is equal to or greater than the eighth distance, or if the minimum value of the relative distance between the driver's hand position and the entire steering wheel is equal to or greater than the eighth distance (No in step S560), the control device 10 returns to the judgment in step S510.

[0084] (Step S570) In step S570, the control device 10 causes an external device (a sound output device such as a speaker) to reproduce the performance sound PS corresponding to the "rub" gesture GT(2) identified in step S560.

[0085] As described above, for the driver of the occupant Ue, the control device 10 may determine whether to start playing the performance sound PS by performing the processing illustrated in FIGS. 9 and 10 on the relative positional relationship RP between the BP and the region PV (particularly, the driving region DPV) of the vehicle 1. That is, the control device 10 first identifies the position where the occupant Ue (driver)'s hands are gripping at least a part of the driving region DPV (the position of the occupant Ue's hands gripping the driving region DPV), in other words, the position of the part of the driving region DPV gripped by the occupant Ue. Then, when the control device 10 determines that the occupant Ue (driver)'s hands are not gripping the driving region DPV at a certain point in time, it records the position where the occupant Ue's hands last gripped the driving region DPV at a point in time prior to the certain point in time as a reference position. In the example of FIG. 6, when the control device 10 determines that the driver is not gripping the steering wheel at a second time point (t=T1), it records the position (BP position (t=T0)) of the hand (BP(2)) of the occupant Ue(1) at the first time point (t=T0) as a reference position in the BP position information 132. The control device 10 determines the start of playback of the performance sound PS based on the relative positional relationship RP between the recorded reference position and the hand of the occupant Ue (driver). In other words, the control device 10 determines the start of playback of the performance sound PS based on the relative positional relationship RP between the position (reference position) of the part of the driving region DPV that was gripped by the driver and the driver's hand.

[0086] The driver of the occupant Ue may be restricted from making a gesture GT (playing action) of releasing the hands from the driving region DPV (for example, the steering wheel (region PV(4)), the shift lever (region PV(5)), etc.). Therefore, the control device 10 uses a reference position to precisely determine whether the driver who is subject to such restrictions is making a gesture GT using the hands (to precisely identify the type of gesture GT being made using the hands). For example, the control device 10 can precisely determine whether the driver is making a gesture GT using the hands based on the relative positional relationship RP between the reference position (i.e., the position of the part of the driving region DPV that was being held by the driver) and the driver's hand at each point in time.

[0087] As illustrated in FIGS. 6 and 7, the driving region DPV is, for example, the steering wheel (region PV(4)). The relative positional relationship RP between the position of the portion of the steering wheel gripped by the driver (reference position) and the driver's hand (BP) may be, for example, the distance between the two in at least one of the direction of the steering wheel's rotation axis and the radial direction perpendicular to the rotation axis. Specifically, based on the distance between the reference position in the direction of the steering wheel's rotation axis and the driver's hand at each time, the control device 10 can precisely determine whether the driver is making a "smacking" gesture GT(1) with his / her hand. Furthermore, when the distance between the reference position in the direction of the steering wheel's rotation axis and the driver's hand at each time is equal to or less than a predetermined value and the distance between them changes in the radial direction, the control device 10 can precisely determine that the driver is making a "rubbing" gesture GT(2) with his / her hand.

[0088] <4. How to use the control device> FIG. 11 shows an example of a service provided by the control device 10 to the occupant Ue. The control device 10 may provide the occupant Ue with a "play with music" service. The control device 10 may identify the gestures GT being performed by each of the multiple occupants Ue in the interior (inside) of the vehicle 1 using the BP, and may play, for example, simultaneously, the performance sounds PS corresponding to each gesture GT. Through such processing by the control device 10, the multiple occupants Ue can enjoy a performance session inside the vehicle 1. The control device 10 may transmit the performance sounds PS being played in one vehicle 1 (vehicle 1(x)) to another vehicle 1 (vehicle 1(y)) via a network, thereby enabling the performance sounds PS to be shared among the multiple vehicles 1 (a session for everyone).

[0089] As described above, the correspondence between the parts PV of the vehicle 1 and the musical instruments MI in the part-musical instrument correspondence information 136 may be changed at a predetermined timing (e.g., daily). Using such a change, the control device 10 may provide the occupant Ue with a service such as a daily "sound source (musical instrument MI) search" as illustrated in FIG. 11A. For example, the control device 10 may change the correspondence between the parts PV of the vehicle 1 and the musical instruments MI so that, like a family treasure hunt, touching a location inside the vehicle (any part PV of the vehicle 1) produces a different musical sound PS each day. In the example illustrated in FIG. 11A, by previously associating a musical instrument MI with the back of the driver's seat (the part PV), a child searching for the musical instrument MI can play the musical instrument sound MS of the musical instrument MI by touching the back of the driver's seat.

[0090] The control device 10 may provide a service that supports the occupant Ue in exercising in the vehicle. For example, the control device 10 may play a sound in the vehicle that encourages exercise (for example, "Seat Driving Stretching" introduced by JAF (registered trademark) as illustrated in FIG. 11(B)) while the vehicle 1 is parked. The control device 10 may then play musical sounds PS in sync with the stretching movements (gestures GT) made by the occupant Ue.

[0091] [Features] As described above, the control device 10 according to this embodiment includes the first determination unit 122, the imaging unit 110, and the second determination unit 124. The first determination unit 122 determines whether the steering wheel is in a state that does not affect the behavior of the vehicle 1. The imaging unit 110 continuously captures images of the interior of the vehicle 1 as a three-dimensional space. The second determination unit 124 determines the start of playback of a musical performance sound PS, which includes at least one of musical instrument sounds MS of a plurality of types of musical instruments MI and cries CR of a plurality of types of animals AM. The second determination unit 124 detects at least the BP of the occupant Ue of the vehicle 1, i.e., detects at least a part of the body of the occupant Ue, based on the captured image CI captured by the imaging unit 110. Furthermore, when the first determination unit 122 determines that the steering wheel is in a state that does not affect the behavior of the vehicle 1, the second determination unit 124 makes the following determination. That is, the second determination unit 124 determines whether to start reproducing the performance sound PS based on the relative positional relationship RP between the detected BP of the occupant Ue and at least a part of the part PV of the vehicle 1.

[0092] The information processing method PM according to this embodiment is an information processing method that causes a processor (e.g., the CPU of the control device 10) to execute the process of reproducing the performance sound PS described above. The processor determines whether the steering wheel is in a state that does not affect the behavior of the vehicle 1 (step S10). The processor also acquires a captured image CI captured by the imaging unit 110 from the imaging unit 110, and detects at least the BP of the occupant Ue of the vehicle 1 based on the captured image CI (step S30). If the processor determines that the steering wheel is in a state that does not affect the behavior of the vehicle 1 (Yes in step S10), it determines whether to start reproducing the performance sound PS based on the relative positional relationship RP between the detected BP of the occupant Ue and at least a part of the body part PV of the vehicle 1 (step S40).

[0093] Therefore, the control device 10 (information processing method PM) can achieve both the gesture GT by the occupant Ue of the vehicle 1 and the performance sound PS to be played inside the vehicle 1 without causing the occupant Ue to feel uncomfortable. In particular, the control device 10 (information processing method PM) determines the start of playback of the performance sound PS based not on a simple gesture but on the relative positional relationship RP between the occupant Ue's BP and the part PV of the vehicle 1. Therefore, the occupant Ue can recognize that the part PV of the vehicle 1 has a function equivalent to the musical instrument MI (the part PV is associated with the musical instrument MI), and the occupant Ue can play music inside the vehicle 1 without feeling uncomfortable.

[0094] §4 Variations Although the embodiments of the present invention have been described above in detail, the above description is merely illustrative of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. For example, the following modifications are possible. Note that, in the following, the same reference numerals are used for components similar to those in the above embodiment, and descriptions of the same points as those in the above embodiment are omitted where appropriate. The following modifications can be combined as appropriate. [Explanation of symbols]

[0095] 1...vehicle, 10...control device, 110...imaging unit, 122...first determination unit, 124...Second Judgment Section, AM...Animal, BP...Body Parts, CI...image capture, CR...voice, DPV...driving part, GT...gesture (movement), MI...instrument, MS...instrument sound, PS...performance sound, PR...relative positional relationship, PV...vehicle part, Ue...occupant

Claims

1. An information processing method for causing a processor to execute a process of reproducing musical performance sounds including at least one of musical instrument sounds of a plurality of types of musical instruments and cries of a plurality of types of animals, the method comprising: The processor: Determine whether the steering wheel is in a state that does not affect the vehicle's behavior, acquiring, from an imaging unit that continuously images the interior of the vehicle as a three-dimensional space, images captured by the imaging unit; Detecting at least a part of a body of an occupant of the vehicle based on the captured image; when it is determined that the steering wheel is in a state that does not affect the behavior of the vehicle, determining whether to start reproducing the musical sound based on the detected relative positional relationship between at least a part of the body of the occupant and at least a part of the vehicle; Information processing methods.

2. The at least some portion of the vehicle includes a plurality of portions of the vehicle, each of the plurality of body parts is associated with one of the plurality of types of musical instruments or one of the plurality of types of animals; The processor: determining, based on the relative positional relationship between the detected at least part of the occupant's body and one of the plurality of body parts, whether to start playing a sound of one of the plurality of types of musical instruments or a sound of one of the plurality of types of animals associated with the one of the body parts; The information processing method according to claim 1 .

3. The processor: detecting a movement of at least a part of the body of the occupant based on the captured image captured by the imaging unit; changing the played sound in accordance with the detected movement; 3. The information processing method according to claim 1.

4. At least some of the parts of the vehicle include a driving part that can affect the behavior of the vehicle when the steering wheel is in a state that affects the behavior of the vehicle, The processor: Identifying a position where the driver's hand is holding at least a part of the driving part; When it is determined that the occupant's hands are not gripping the driving part at a certain time, the position where the occupant's hands last gripped the driving part at a time before the certain time is recorded as a reference position; determining a start of reproduction of the performance sound based on the relative positional relationship between the recorded reference position and the hand of the occupant; 3. The information processing method according to claim 1.

5. the driving portion includes the steering wheel, the relative positional relationship is a distance between the reference position and the hand of the occupant in a direction of a rotation axis of the steering wheel or in a radial direction perpendicular to the rotation axis; The information processing method according to claim 4.

6. a first determination unit that determines whether the steering wheel is in a state that does not affect the behavior of the vehicle; an imaging unit that continuously captures images of the interior of the vehicle as a three-dimensional space; a second determination unit that determines the start of playback of musical performance sounds, the musical performance sounds including at least one of the musical instrument sounds of a plurality of types of musical instruments and the cries of a plurality of types of animals; Equipped with The second determination unit detecting at least a part of a body of an occupant of the vehicle based on the captured image captured by the imaging unit; when the first determination unit determines that the steering wheel is in a state that does not affect the behavior of the vehicle, determining whether to start reproducing the performance sound based on a relative positional relationship between at least a part of the body of the occupant detected and at least a part of the vehicle; Control device.

Images