Control system and head-mounted display

Abstract

The invention provides a control system and a head-mounted display. A control system (12) is provided with: a head-mounted display (22) which is worn by a user who is in a vehicle (10) and which provides at least a display image to the user, the display image corresponding to an image acquired by an outside camera (16) which is provided in the vehicle (10) and which captures an image of the periphery of the vehicle (10), a detection unit (48), and a volume control unit (52); a detection unit (48) that detects a gesture made by a user on the basis of an image acquired by an in-vehicle camera (14) that is provided in a vehicle (10) and that captures an image in a vehicle compartment; the volume control unit (52) performs control for adjusting the volume (34) of a speaker provided in the head-mounted display (22) in accordance with the gesture detected by the detection unit (48). Accordingly, the head-mounted display can be effectively used in a vehicle compartment.

Description

Technical Field

[0001] The present invention relates to a control system and a head mounted display. Background Art

[0002] Japanese Patent Application No. 7243639 discloses an information processing device capable of wirelessly communicating between a headset and a smartphone. The headset includes a motion sensor capable of detecting user movements. The smartphone can operate based on the user movements detected by the motion sensor. The smartphone can also provide users with games, videos, and other content. Summary of the Invention

[0003] Japanese Patent Application No. 7243639 merely discloses operating a smartphone based on a user's motion detected by a motion sensor provided in a headset. Recently, there has been a demand for a technology that enables efficient use of a head-mounted display in a vehicle cabin.

[0004] The purpose of the present invention is to solve the above-mentioned technical problems.

[0005] The control system involved in the first embodiment of the present invention has a head-mounted display, a detection unit and a volume control unit, wherein the head-mounted display is worn by a user riding in a vehicle and provides at least a display image to the user, and the display image corresponds to an image acquired by an external camera installed in the vehicle and capturing the surroundings of the vehicle; the detection unit detects a gesture made by the user based on an image acquired by an internal camera installed in the vehicle and capturing the interior of the vehicle; and the volume control unit performs the following control to adjust the volume of a speaker equipped with the head-mounted display according to the gesture detected by the detection unit.

[0006] The head-mounted display involved in the second embodiment of the present invention is worn by a user riding in a vehicle, and has a display unit, a speaker, a detection unit and a volume control unit, wherein the display unit provides at least a display image to the user, and the display image corresponds to an image acquired by an external camera installed in the vehicle and capturing the surroundings of the vehicle; the speaker is used to provide sound to the user; the detection unit detects gestures made by the user based on images acquired by an internal camera installed in the vehicle and capturing the interior of the vehicle; and the volume control unit performs control to adjust the volume of the speaker based on the gesture detected by the detection unit.

[0007] According to the present invention, a head-mounted display can be effectively used in a vehicle cabin.

[0008] The above-mentioned objects, features and advantages will be easily understood from the following description of the embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 It is a schematic configuration diagram of a control system according to this embodiment.

[0010] Figure 2 This is a functional block diagram of a head-mounted display.

[0011] Figure 3 This is a flow chart of the volume adjustment process.

[0012] Figure 4A is a diagram showing a user performing a first gesture. Figure 4B FIG. 1 is a diagram showing a user performing a second gesture.

[0013] Figure 5 FIG. 1 is a diagram showing a user performing the third gesture.

[0014] Figure 6A This diagram shows a user performing a first gesture with both hands. Figure 6B This diagram shows a user performing the second gesture with both hands.

[0015] Figure 7 This diagram shows a user making the first gesture with one hand and lowering the other hand.

[0016] Figure 8 This diagram shows a user making a first gesture with one hand and indicating a direction with the other hand.

[0017] Figure 9 This diagram shows a user making the third gesture with one hand and indicating a direction with the other hand. DETAILED DESCRIPTION

[0018] The hope is that head-mounted displays will immerse users in content such as games and videos.

[0019] The user's eyes are covered by the display surface of the head-mounted display. Therefore, the user cannot see the scenery outside the vehicle while wearing the head-mounted display. When the vehicle is driving in this state, the user may experience motion sickness.

[0020] The embodiments described in this specification are less likely to impair the user's immersion in content such as games and videos. Furthermore, the embodiments described in this specification prevent motion sickness in users wearing head-mounted displays in vehicles.

[0021] [1 Structure of Control System 12] Figure 1This is a schematic diagram of the control system 12 according to this embodiment. The control system 12 adjusts the volume of the head-mounted display 22 based on gestures made by a passenger wearing the head-mounted display 22 among passengers in the vehicle 10. In this specification, a passenger wearing the head-mounted display 22 is referred to as a user U.

[0022] The control system 12 includes an in-vehicle camera 14, an out-vehicle camera 16, one or more in-vehicle microphones 18, an external microphone 20, and a head-mounted display 22. The in-vehicle camera 14 and the head-mounted display 22 are capable of wireless communication with each other. The out-vehicle camera 16 and the head-mounted display 22 are capable of wireless communication with each other. The multiple in-vehicle microphones 18 and the head-mounted display 22 are capable of wireless communication with each other. The external microphone 20 and the head-mounted display 22 are capable of wireless communication with each other. Furthermore, the head-mounted display 22 is capable of communicating with a server device 26 via a communication line 24 such as the Internet.

[0023] The interior camera 14 captures the interior of the vehicle 10. The interior camera 14 transmits an interior image signal representing the captured interior image to the head mounted display 22. In addition, a plurality of interior cameras 14 may be provided.

[0024] The exterior camera 16 captures images of the surroundings of the vehicle 10. The exterior camera 16 transmits an exterior image signal representing the acquired exterior image to the head mounted display 22. In addition, a plurality of exterior cameras 16 may be provided.

[0025] Multiple onboard microphones 18 are disposed within the interior of the vehicle 10. For example, one of the onboard microphones 18 is disposed near each seat in the vehicle 10. The onboard microphones 18 capture sounds around the seat in which they are disposed. Each onboard microphone 18 transmits an audio signal representing the captured sound to the head-mounted display 22.

[0026] The external microphone 20 is disposed outside the vehicle 10. The external microphone 20 captures sounds outside the vehicle and transmits an audio signal representing the captured sounds to the head-mounted display 22. A plurality of external microphones 20 may be provided.

[0027] Figure 2 2 is a functional block diagram of the head mounted display 22. In this specification, the head mounted display 22 is also referred to as HMD (Head Mounted Display) 22. The HMD 22 includes an operation unit 30, a display unit 32, an audio unit 34, a first communication unit 36, a second communication unit 38, a calculation unit 40, and a storage unit 42.

[0028] The operation unit 30 is a human-machine interface through which the user U can perform input operations. The operation unit 30 can be composed of a remote controller, a switch, etc. The operation unit 30 can output an operation signal corresponding to the operation performed by the user U to the calculation unit 40 .

[0029] The display unit 32 can be comprised of, for example, a display device. The display surface of the display device is positioned close to the eyes of the user U. The display unit 32 displays (projects) an image onto the display surface based on an image signal transmitted from the computing unit 40. The display unit 32 can provide the user U with images of digital content and images outside the vehicle. For example, the display unit 32 can display content related to augmented reality (AR), mixed reality (MR), and the like.

[0030] The sound unit 34 can be composed of, for example, an audio device. The speaker (including headphones or earphones) of the sound unit 34 is close to the ear 62 ( Figure 4A 、 Figure 4B The audio unit 34 outputs audio from the speaker based on the audio signal sent from the calculation unit 40. The audio unit 34 can provide the user U with audio in the digital content and can also provide the user U with sounds inside the vehicle.

[0031] The first communication unit 36 ​​can be formed, for example, of a communication module (integrated circuit module) having an antenna, etc. The first communication unit 36 ​​receives signals transmitted from the in-vehicle camera 14 , the out-vehicle camera 16 , the plurality of in-vehicle microphones 18 , and the external microphone 20 , and outputs the signals to the calculation unit 40 .

[0032] The second communication unit 38 can be formed of, for example, a communication module (integrated circuit module) having an antenna, etc. The second communication unit 38 receives the digital content signal transmitted from the server device 26 via the communication line 24 and outputs it to the calculation unit 40 .

[0033] The computing unit 40 can be comprised of, for example, a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). In other words, the computing unit 40 can be comprised of processing circuitry. At least a portion of the computing unit 40 can also be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array). At least a portion of the computing unit 40 can also be implemented by an electronic circuit including discrete components.

[0034] The computing unit 40 includes a first acquiring unit 44, a second acquiring unit 46, a detecting unit 48, a display control unit 50, and a volume control unit 52. The first acquiring unit 44, the second acquiring unit 46, the detecting unit 48, the display control unit 50, and the volume control unit 52 can be implemented by the computing unit 40 executing a program stored in the storage unit 42.

[0035] The first acquisition unit 44 can acquire the in-vehicle image signal output by the in-vehicle camera 14 via the first communication unit 36. Furthermore, the first acquisition unit 44 acquires the in-vehicle image signal output by the in-vehicle camera 16 via the first communication unit 36. Furthermore, the first acquisition unit 44 acquires the audio signal output by each in-vehicle microphone 18 via the first communication unit 36. Furthermore, the first acquisition unit 44 acquires the audio signal output by the external microphone 20 via the first communication unit 36. Meanwhile, the second acquisition unit 46 acquires the digital content signal output by the server device 26 via the second communication unit 38.

[0036] The detection unit 48 detects the gesture made by the user U based on the in-vehicle image signal acquired by the first acquisition unit 44. For example, the detection unit 48 detects the user U wearing the HMD 22 through image recognition. Furthermore, the detection unit 48 detects the hand 64 ( Figure 4A 、 Figure 4B etc.) and finger movements.

[0037] The display control unit 50 causes the display unit 32 to display an image corresponding to the vehicle exterior image signal acquired by the first acquisition unit 44. Furthermore, the display control unit 50 causes the display unit 32 to display an image corresponding to the digital content signal acquired by the second acquisition unit 46. The display control unit 50 can control the display unit 32 so that the digital content image and the vehicle exterior image are superimposed and displayed. The display control unit 50 can control the display unit 32 based on an operation signal output by the operation unit 30.

[0038] The volume control unit 52 causes the audio unit 34 to output sound corresponding to the sound signal acquired by the first acquisition unit 44. Furthermore, the volume control unit 52 causes the audio unit 34 to output sound corresponding to the digital content signal acquired by the second acquisition unit 46. The volume control unit 52 can control the audio unit 34 to simultaneously output the sound of the digital content, sounds inside the vehicle cabin, and sounds outside the vehicle. The volume control unit 52 can control the volume of the speaker of the audio unit 34 based on gestures detected by the detection unit 48. Furthermore, the volume control unit 52 can control the volume of the digital content based on an operation signal output by the operation unit 30.

[0039] The storage unit 42 is a computer-readable storage medium. It is composed of volatile memory (not shown) and non-volatile memory (not shown). Examples of volatile memory include RAM (Random Access Memory). Examples of non-volatile memory include ROM (Read Only Memory) and flash memory. Data, etc., are stored in the volatile memory. Programs, tables, and maps, etc., are stored in the non-volatile memory. At least a portion of the storage unit 42 may be provided in the aforementioned processor, integrated circuit, etc.

[0040] [2 Volume adjustment processing] Figure 3 This is a flow chart of the volume adjustment process. The volume adjustment process described below is performed by the computing unit 40 of the HMD 22 at predetermined intervals. The volume adjustment process described below is for adjusting the volume of sounds inside the vehicle cabin and the volume of sounds outside the vehicle, among the sounds output by the audio unit 34.

[0041] In step S1, the first acquisition unit 44 acquires an in-vehicle image signal from the in-vehicle camera 14 via the first communication unit 36. In step S2, the first acquisition unit 44 acquires audio signals from the multiple in-vehicle microphones 18 and the external microphone 20 via the first communication unit 36. In step S3, the detection unit 48 detects a gesture made by the user U based on the in-vehicle image signal acquired in step S1.

[0042] In step S4, the detection unit 48 determines whether a specific gesture has been detected. Examples of specific gestures that can be detected in this embodiment will be described later. If a specific gesture has been detected (step S4: Yes), the process transfers to step S5. On the other hand, if a specific gesture has not been detected (step S4: No), the process returns to step S1.

[0043] In step S5 , volume control unit 52 adjusts the volume of either the sound inside the vehicle or the sound outside the vehicle according to the gesture detected by detection unit 48 .

[0044] [3 Examples of Specific Gestures and Volume Adjustment] In the present embodiment, when the detection unit 48 detects a specific gesture made by the user U, the volume control unit 52 can increase or decrease the volume of the sound inside and outside the vehicle output by the audio unit 34 .

[0045] [3-1 Volume Adjustment Example 1] In the first volume adjustment example, the volume control unit 52 can increase or decrease the volume of the sounds acquired by the plurality of vehicle-mounted microphones 18 and the volume of the sounds acquired by the external microphone 20 , among the volumes output by the audio unit 34 .

[0046] like Figure 4A and Figure 4B As shown, the user U can make a gesture of placing the palm 60 around the ear 62 without blocking the ear 62 with the palm 60. For example, Figure 4A As shown in FIG. 1 , the user U can make a first gesture in which the palm 60 faces the front of the head and the palm 60 is located behind the ear 62. Figure 4B As shown, the user U can make a second gesture in which the palm 60 is directed toward the back of the head and in front of the ear 62. When the detection unit 48 detects the first gesture or the second gesture, the volume control unit 52 can increase the volume of the sounds acquired by the plurality of vehicle-mounted microphones 18 and the volume of the sounds acquired by the external microphone 20, among the volumes output by the audio unit 34.

[0047] like Figure 5 As shown, the user U can make a third gesture of covering the ear 62 with the hand 64. When the third gesture is detected by the detection unit 48, the volume control unit 52 can reduce the volume of the sound acquired by the plurality of vehicle-mounted microphones 18 and the volume of the sound acquired by the external microphone 20, among the volumes output by the audio unit 34.

[0048] [3-2 Second volume adjustment example] In the second volume adjustment example, the volume control unit 52 can selectively increase or decrease the volume of the sounds acquired by the plurality of vehicle-mounted microphones 18 and the volume of the sounds acquired by the external microphone 20 , among the volumes output by the audio unit 34 .

[0049] like Figure 6A As shown in FIG, the user U can make the first gesture with both hands 64a and 64b. Figure 6B As shown, the user U can make the second gesture with both hands 64a and 64b. Figure 6A The gesture shown or Figure 6B In the case of the gesture shown, the volume control unit 52 can selectively increase the volume of the sound acquired by the external microphone 20 among the volumes output by the audio unit 34 .

[0050] like Figure 7 As shown, the user U can make the first gesture or the second gesture with one hand 64a and put down the other hand 64b. Figure 7In the case of the gesture shown, the volume control unit 52 can selectively increase the volume of the sound picked up by the vehicle-mounted microphone 18 in the direction toward which the palm 60 of one hand 64 a is directed, among the volumes output by the audio unit 34 .

[0051] like Figure 8 As shown, the user U can make the first gesture or the second gesture with one hand 64a and indicate the direction with the other hand 64b. Figure 8 In the case of the gesture shown, the volume control unit 52 can selectively increase the volume of the sound picked up by the vehicle-mounted microphone 18 in the direction indicated by the other hand 64 b , among the volumes output by the audio unit 34 .

[0052] like Figure 9 As shown, the user U can make a third gesture with one hand 64a and indicate a direction with the other hand 64b. Figure 9 In the case of the gesture shown, the volume control unit 52 can selectively reduce the volume of the sound picked up by the vehicle-mounted microphone 18 in the direction indicated by the other hand 64 b , among the volumes output by the audio unit 34 .

[0053] [4 Other embodiments] The interior camera 14, exterior camera 16, in-vehicle microphone 18, and external microphone 20 can also communicate with the communication device of the vehicle 10. Furthermore, the HMD 22 can also communicate with the communication device of the vehicle 10. In other words, the various devices equipped in the vehicle 10 (the interior camera 14, exterior camera 16, in-vehicle microphone 18, and external microphone 20) and the HMD 22 can also communicate via the communication device of the vehicle 10.

[0054] [5 Effects] According to the present embodiment, the detection unit 48 detects the gesture of the user U based on the image acquired by the in-vehicle camera 14 that constantly captures the interior of the vehicle cabin. Therefore, the gesture of the user U can be stably detected.

[0055] When adjusting the volume of the head mounted display 22 using a remote controller or the like, there is a concern that the user U's sense of immersion may be impaired. With the above configuration, since the volume of the head mounted display 22 can be adjusted based on the user U's gestures, the user U's sense of immersion is less likely to be impaired.

[0056] According to the above configuration, the head mounted display 22 can display an image corresponding to an image acquired by the exterior camera 16 that captures the surroundings of the vehicle 10 , thereby preventing the user U from experiencing motion sickness.

[0057] As described above, according to the present embodiment, the head mounted display 22 can be effectively used in the vehicle cabin.

[0058] [6 Notes] Regarding the above-mentioned embodiment, the following additional notes are also disclosed.

[0059] (Note 1) The control system (12) of the present invention comprises a head-mounted display (22), a detection unit (48) and a volume control unit (52), wherein the head-mounted display (22) is worn by a user (U) riding in a vehicle (10) and provides at least a display image to the user, the display image corresponding to an image acquired by an external camera (16), wherein the external camera (16) is provided in the vehicle and photographs the surroundings of the vehicle; the detection unit (48) detects a gesture made by the user based on an image acquired by an internal camera (14), wherein the internal camera is provided in the vehicle and photographs the interior of the vehicle; and the volume control unit (52) performs control to adjust the volume of a speaker (34) based on the gesture detected by the detection unit, wherein the speaker is provided in the head-mounted display.

[0060] According to the above configuration, the detection unit detects a gesture made by the user based on an image acquired by the in-vehicle camera that constantly captures the interior of the vehicle cabin, and thus can stably detect a gesture made by the user.

[0061] According to the above configuration, since the volume of the head mounted display can be adjusted according to the user's gesture, the user's sense of immersion is less likely to be impaired.

[0062] According to the above configuration, the head mounted display displays an image corresponding to an image acquired by the exterior camera that captures the surroundings of the vehicle, thereby preventing the user from experiencing motion sickness.

[0063] As described above, according to the above configuration, the head mounted display can be effectively used in a vehicle cabin.

[0064] (Note 2) In the control system described in Supplement 1, the volume control unit may increase the volume when the detection unit detects the gesture of placing the palm (60) around the ear (62) instead of blocking the ear with the palm.

[0065] (Note 3) In the control system described in Supplement 1 or 2, the volume control unit may reduce the volume when the gesture of covering the ears with the hands (64) is detected by the detection unit.

[0066] (Note 4) In the control system described in any one of Notes 1 to 3, the volume control unit may selectively increase the volume of the sound acquired by the microphone (20) when the predetermined gesture is detected by the detection unit, wherein the predetermined gesture refers to a gesture in which the palm of the left hand (64a) is positioned around the left ear instead of blocking the left ear with the palm of the left hand and the palm of the right hand (64b) is positioned around the right ear instead of blocking the right ear with the palm of the right hand, and the microphone (20) is used to acquire sounds outside the vehicle.

[0067] (Note 5) In the control system described in any one of Notes 1 to 4, the volume control unit may selectively increase the volume of the sound from the direction indicated by the other hand (64b) when the predetermined gesture is detected by the detection unit, wherein the predetermined gesture refers to a gesture in which the palm of one hand (64a) is positioned around the ear instead of covering the ear with the one hand and indicating a direction with the other hand.

[0068] (Note 6) In the control system described in Supplementary Note 2, the volume of the sound coming from the direction toward which the palm is facing may be selectively increased.

[0069] According to the configurations of Supplementary Notes 2 to 6 , the user can adjust the volume by gestures instead of operating the device, and thus the user's sense of immersion is less likely to be impaired.

[0070] (Note 7) The head-mounted display of the present invention is worn by a user riding in a vehicle, and comprises a display unit (32), a speaker, a detection unit and a volume control unit, wherein the display unit (32) at least provides a display image to the user, and the display image corresponds to an image acquired by an external camera, wherein the external camera is provided in the vehicle and photographs the surroundings of the vehicle; the speaker is used to provide sound to the user; the detection unit detects a gesture made by the user based on an image acquired by an internal camera, wherein the internal camera is provided in the vehicle and photographs the interior of the vehicle; and the volume control unit performs control to adjust the volume of the speaker based on the gesture detected by the detection unit.

[0071] According to the above configuration, the detection unit detects a gesture made by the user based on an image acquired by the in-vehicle camera that constantly captures the interior of the vehicle cabin, and thus can stably detect the gesture of the user.

[0072] According to the above configuration, since the volume of the head mounted display can be adjusted according to the user's gesture, the user's sense of immersion is less likely to be impaired.

[0073] According to the above configuration, the head mounted display displays an image corresponding to an image acquired by the exterior camera that captures the surroundings of the vehicle, thereby preventing the user from experiencing motion sickness.

[0074] The present invention has been described in detail, but the present invention is not limited to the above-mentioned embodiments. These embodiments can be supplemented, replaced, changed, partially deleted, etc. in a manner that does not deviate from the scope of the present invention or in a manner that does not deviate from the scope of the present invention derived from the contents recorded in the technical solution and its equivalents. In addition, these embodiments can also be implemented in combination. For example, in the above-mentioned embodiment, the order of each action and the order of each processing are shown as an example, but are not limited to this. In addition, the same applies when numerical values ​​or mathematical formulas are used in the description of the above-mentioned embodiment.

Claims

1. A control system, characterized in that: It has a head-mounted display, a detection unit and a volume control unit, wherein: The head-mounted display is worn by a user riding in a vehicle and provides at least a display image to the user, the display image corresponding to an image acquired by an external camera, wherein the external camera is provided in the vehicle and captures the surroundings of the vehicle; The detection unit detects the gesture made by the user based on an image captured by an in-vehicle camera, wherein the in-vehicle camera is provided in the vehicle and captures the interior of the vehicle; The volume control unit performs control to adjust the volume of a speaker provided in the head-mounted display according to the gesture detected by the detection unit.

2. The control system according to claim 1, characterized in that: The volume control unit increases the volume when the gesture of placing the palm around the ear instead of covering the ear with the palm is detected by the detection unit.

3. The control system according to claim 1, characterized in that: The volume control unit reduces the volume when the gesture of covering the ears with hands is detected by the detection unit.

4. The control system according to claim 1, characterized in that: The volume control unit selectively increases the volume of the sound acquired by the microphone when the prescribed gesture is detected by the detection unit, wherein the prescribed gesture refers to a gesture in which the palm of the left hand is positioned around the left ear instead of blocking the left ear with the palm of the left hand and the palm of the right hand is positioned around the right ear instead of blocking the right ear with the palm of the right hand, and the microphone is used to acquire external sounds of the vehicle.

5. The control system according to claim 1, characterized in that: When the detection unit detects the prescribed gesture, the volume control unit selectively increases the volume of the sound coming from the direction indicated by the other hand, wherein the prescribed gesture refers to a gesture of placing the palm of one hand around the ear instead of blocking the ear with the one hand and indicating the direction with the other hand.

6. The control system according to claim 2, characterized in that: The volume of sound coming from the direction the palm is facing is selectively increased.

7. A head-mounted display worn by a user riding in a vehicle, characterized in that: It has a display unit, a speaker, a detection unit and a volume control unit, wherein: The display unit provides at least a display image to the user, the display image corresponding to an image acquired by an exterior camera, wherein the exterior camera is provided at the vehicle and captures a periphery of the vehicle; The speaker is used to provide sound to the user; The detection unit detects the gesture made by the user based on an image captured by an in-vehicle camera, wherein the in-vehicle camera is provided in the vehicle and captures the interior of the vehicle; The volume control section performs control to adjust the volume of the speaker according to the gesture detected by the detection section.

Images