Control system and head-mounted display

Abstract

To comply with a growing demand for technology that allows head-mounted displays to be used effectively inside a vehicle cabin.SOLUTION: A control system 12 is provided with: a head-mounted display 22 worn by a user boarding a vehicle 10 in which the head-mounted display 22 provides the user with at least display images based on images acquired by an exterior camera 16 equipped in the vehicle 10 and capturing surroundings of the vehicle 10; a detection unit 48 for detecting gestures made by the user based on images acquired by an interior camera 14 provided in the vehicle 10 and capturing the interior of a vehicle cabin; and a volume control unit 52 for executing control to adjust the volume of a speaker 34 equipped in the head-mounted display 22 based on the gestures detected by the detection unit 48.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a control system and a head-mounted display. [Background technology]

[0002] Patent Document 1 discloses an information processing device capable of wireless communication between headphones and a smartphone. In this information processing device, the headphones are equipped with a motion sensor capable of detecting a user's movements. The smartphone can be operated in response to the user's movements detected by the motion sensor. The smartphone can provide the user with games, videos, etc. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7243639 Summary of the Invention [Problem to be solved by the invention]

[0004] Patent Document 1 merely discloses that a smartphone is operated in response to a user's movements detected by a motion sensor provided in headphones. Recently, there has been a demand for technology that enables head-mounted displays to be used effectively inside a vehicle.

[0005] The present invention aims to solve the above-mentioned problems. [Means for solving the problem]

[0006] A control system according to a first aspect of the present disclosure includes a head-mounted display worn by a user in a vehicle, the head-mounted display providing at least the user with a display image corresponding to an image acquired by an exterior camera mounted on the vehicle and capturing an image of the area around the vehicle, a detection unit that detects a gesture by the user based on an image acquired by an interior camera mounted on the vehicle and capturing an image of the interior of the vehicle, and a volume control unit that executes control to adjust the volume of a speaker mounted on the head-mounted display based on the gesture detected by the detection unit.

[0007] A head-mounted display according to a second aspect of the present disclosure is a head-mounted display worn by a user riding in a vehicle, and includes a display unit that provides at least a display image to the user corresponding to an image acquired by an exterior camera that is provided in the vehicle and captures an image of the area around the vehicle, a speaker that provides sound to the user, a detection unit that detects a gesture by the user based on an image acquired by an interior camera that is provided in the vehicle and captures an image of the interior of the vehicle, and a volume control unit that executes control to adjust the volume of the speaker based on the gesture detected by the detection unit. [Effects of the Invention]

[0008] According to the present invention, a head-mounted display can be used satisfactorily inside a vehicle. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a schematic configuration diagram of a control system according to this embodiment. [Figure 2] FIG. 2 is a functional block diagram of the head-mounted display. [Figure 3] FIG. 3 is a flowchart of the volume adjustment process. [Figure 4]4A is a diagram illustrating a user making a first gesture, and FIG. 4B is a diagram illustrating a user making a second gesture. [Figure 5] FIG. 5 is a diagram illustrating a user making a third gesture. [Figure 6] 6A is a diagram showing a user making a first gesture with both hands, and FIG 6B is a diagram showing a user making a second gesture with both hands. [Figure 7] FIG. 7 is a diagram illustrating a user making a first gesture with one hand and lowering the other hand. [Figure 8] FIG. 8 is a diagram illustrating a user making a first gesture with one hand and pointing in a direction with the other hand. [Figure 9] FIG. 9 is a diagram illustrating a user making a third gesture with one hand and pointing in a direction with the other hand. DETAILED DESCRIPTION OF THE INVENTION

[0010] It is desirable for a head-mounted display to immerse the user in content such as games and videos.

[0011] The eyes of a user wearing a head-mounted display are covered by the display surface of the head-mounted display. Therefore, a user wearing a head-mounted display inside a vehicle cannot see the scenery outside the vehicle. If the vehicle is traveling in this state, the user may experience motion sickness.

[0012] The embodiments described herein make it difficult for a user to lose their sense of immersion in content such as games, videos, etc. Furthermore, the embodiments described herein reduce motion sickness in a user wearing a head-mounted display inside a vehicle.

[0013] [1 Configuration of Control System 12] 1 is a schematic configuration diagram of a control system 12 according to this embodiment. The control system 12 adjusts the volume of the head-mounted display 22 in response to a gesture made by an occupant wearing a head-mounted display 22 among occupants aboard the vehicle 10. In this specification, the occupant wearing the head-mounted display 22 is referred to as a user U.

[0014] The control system 12 includes an in-vehicle camera 14, an exterior camera 16, one or more in-vehicle microphones 18, an exterior microphone 20, and a head-mounted display 22. The in-vehicle camera 14 and the head-mounted display 22 are capable of wireless communication with each other. The exterior camera 16 and the head-mounted display 22 are capable of wireless communication with each other. The multiple in-vehicle microphones 18 and the head-mounted display 22 are capable of wireless communication with each other. The exterior microphones 20 and the head-mounted display 22 are capable of wireless communication with each other. The head-mounted display 22 is capable of communicating with a server device 26 via a communication line 24 such as the Internet.

[0015] The in-vehicle camera 14 captures an image of the interior of the vehicle 10. The in-vehicle camera 14 transmits an in-vehicle image signal indicating the captured in-vehicle image to the head-mounted display 22. Note that a plurality of in-vehicle cameras 14 may be provided.

[0016] The exterior camera 16 captures images of the surroundings of the vehicle 10. The exterior camera 16 transmits an exterior image signal indicating the captured exterior image to the head-mounted display 22. Note that a plurality of exterior cameras 16 may be provided.

[0017] Each of the multiple interior microphones 18 is disposed within the cabin of vehicle 10. For example, one of the interior microphones 18 is disposed around each seat in vehicle 10. Each interior microphone 18 captures sounds around the seat in which it is located. Each interior microphone 18 transmits an acoustic signal indicative of the captured sound to head-mounted display 22.

[0018] The exterior microphone 20 is disposed outside the vehicle 10. The exterior microphone 20 acquires sounds outside the vehicle. The exterior microphone 20 transmits an acoustic signal indicating the acquired sounds to the head-mounted display 22. Note that a plurality of exterior microphones 20 may be provided.

[0019] 2 is a functional block diagram of the head mounted display 22. In this specification, the head mounted display 22 is also referred to as an HMD (Head Mounted Display) 22. The HMD 22 includes an operation unit 30, a display unit 32, an audio unit 34, a first communication unit 36, a second communication unit 38, a calculation unit 40, and a storage unit 42.

[0020] The operation unit 30 is a human-machine interface that allows the user U to perform input operations. The operation unit 30 may be configured with a remote controller, a switch, etc. The operation unit 30 outputs an operation signal to the calculation unit 40 in accordance with the operation performed by the user U.

[0021] The display unit 32 may be configured, for example, by a display device. The display surface of the display device is located close to both eyes of the user U. The display unit 32 displays (projects) a display image on the display surface based on an image signal transmitted from the calculation unit 40. The display unit 32 may provide the user U with a display image of digital content, as well as a display image of the outside of the vehicle. For example, the display unit 32 may display content related to augmented reality, mixed reality, etc.

[0022] The acoustic unit 34 may be configured, for example, by an audio device. A speaker (including headphones and earphones) of the acoustic unit 34 is placed close to the ear 62 (FIGS. 4A, 4B, etc.) of the user U. The acoustic unit 34 outputs sound from the speaker based on an acoustic signal transmitted from the calculation unit 40. The acoustic unit 34 may provide the user U with sounds in the digital content and may also provide the user U with sounds inside the vehicle.

[0023] The first communication unit 36 ​​may be configured, for example, by a communication module (integrated circuit module) equipped with an antenna, etc. The first communication unit 36 ​​receives signals transmitted from the in-vehicle camera 14, the outside-vehicle camera 16, the multiple in-vehicle microphones 18, and the outside-vehicle microphone 20, and outputs the signals to the calculation unit 40.

[0024] The second communication unit 38 may be configured, for example, by a communication module (integrated circuit module) equipped with an antenna, etc. The second communication unit 38 receives a digital content signal transmitted from the server device 26 via the communication line 24, and outputs the signal to the calculation unit 40.

[0025] The calculation unit 40 may be configured by a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). That is, the calculation unit 40 may be configured by processing circuitry. At least a part of the calculation unit 40 may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array). At least a part of the calculation unit 40 may be realized by an electronic circuit including discrete devices.

[0026] The calculation unit 40 includes a first acquisition unit 44, a second acquisition unit 46, a detection unit 48, a display control unit 50, and a volume control unit 52. The first acquisition unit 44, the second acquisition unit 46, the detection unit 48, the display control unit 50, and the volume control unit 52 can be realized by the calculation unit 40 executing a program stored in the storage unit 42.

[0027] The first acquisition unit 44 acquires an interior image signal output by the interior camera 14 via the first communication unit 36. The first acquisition unit 44 also acquires an exterior image signal output by the exterior camera 16 via the first communication unit 36. The first acquisition unit 44 also acquires an acoustic signal output by each interior microphone 18 via the first communication unit 36. The first acquisition unit 44 also acquires an acoustic signal output by the exterior microphone 20 via the first communication unit 36. Meanwhile, the second acquisition unit 46 acquires a digital content signal output by the server device 26 via the second communication unit 38.

[0028] The detection unit 48 detects a gesture made by the user U based on the in-vehicle image signal acquired by the first acquisition unit 44. For example, the detection unit 48 detects the user U wearing the HMD 22 by image recognition. Furthermore, the detection unit 48 detects the position of the hand 64 (FIGS. 4A, 4B, etc.) of the user U and the movement of the fingers by gesture recognition.

[0029] The display control unit 50 causes the display unit 32 to display a display image corresponding to the outside-of-vehicle image signal acquired by the first acquisition unit 44. The display control unit 50 also causes the display unit 32 to display a display image corresponding to the digital content signal acquired by the second acquisition unit 46. The display control unit 50 can execute control to display the digital content display image and the outside-of-vehicle display image superimposed on each other on the display unit 32. The display control unit 50 can control the display unit 32 based on an operation signal output by the operation unit 30.

[0030] The volume control unit 52 causes the audio unit 34 to output sound corresponding to the audio signal acquired by the first acquisition unit 44. The volume control unit 52 also causes the audio unit 34 to output sound corresponding to the digital content signal acquired by the second acquisition unit 46. The volume control unit 52 can execute control to cause the audio unit 34 to simultaneously output the sound of the digital content, the sound inside the vehicle cabin, and the sound outside the vehicle. The volume control unit 52 can execute control to adjust the volume of the speaker of the audio unit 34 based on the gesture detected by the detection unit 48. The volume control unit 52 can also execute control to adjust the volume of the digital content based on the operation signal output by the operation unit 30.

[0031] The storage unit 42 is a computer-readable storage medium. The storage unit 42 is composed of a volatile memory (not shown) and a non-volatile memory (not shown). The volatile memory is, for example, a random access memory (RAM). The non-volatile memory is, for example, a read-only memory (ROM), a flash memory, etc. Data, etc. are stored in, for example, the volatile memory. Programs, tables, maps, etc. are stored in, for example, the non-volatile memory. At least a part of the storage unit 42 may be provided in the above-mentioned processor, integrated circuit, etc.

[0032] [2 Volume adjustment processing] 3 is a flowchart of the volume adjustment process. The volume adjustment process described below is performed at predetermined time intervals by the calculation unit 40 of the HMD 22. The volume adjustment process described below is a process for adjusting the volume of sounds inside the vehicle cabin and sounds outside the vehicle, among the sounds output by the audio unit 34.

[0033] In step S1, the first acquisition unit 44 acquires an in-vehicle image signal from the in-vehicle camera 14 via the first communication unit 36. In step S2, the first acquisition unit 44 acquires an acoustic signal from each of the multiple in-vehicle microphones 18 and the exterior microphone 20 via the first communication unit 36. In step S3, the detection unit 48 detects a gesture by the user U based on the in-vehicle image signal acquired in step S1.

[0034] In step S4, the detection unit 48 determines whether or not a specific gesture has been detected. Examples of specific gestures that can be detected in this embodiment will be described later. If a specific gesture has been detected (step S4: YES), the process proceeds to step S5. On the other hand, if a specific gesture has not been detected (step S4: NO), the process returns to step S1.

[0035] In step S5, the volume control unit 52 adjusts either the volume of the sound inside the vehicle or the volume of the sound outside the vehicle in accordance with the gesture detected by the detection unit 48. The speaker of the audio unit 34 outputs sound at a volume in accordance with the control of the volume control unit 52.

[0036] [3 Examples of specific gestures and volume adjustments] In this embodiment, when the detection unit 48 detects a specific gesture made by the user U, the volume control unit 52 can increase or decrease the volume of the sounds inside and outside the vehicle that are output from the audio unit 34.

[0037] [3-1 First volume adjustment example] In the first volume adjustment example, the volume control unit 52 increases or decreases the volume of the sounds picked up by the multiple in-vehicle microphones 18 and the volume of the sounds picked up by the exterior microphone 20 among the volumes output by the audio unit 34.

[0038] 4A and 4B, the user U may make a gesture of placing the palm 60 around the ear 62 without covering the ear 62 with the palm 60. For example, as shown in FIG. 4A, the user U may make a first gesture of facing the palm 60 forward of the head and placing the palm 60 behind the ear 62. As shown in FIG. 4B, the user U may make a second gesture of facing the palm 60 backward of the head and placing the palm 60 in front of the ear 62. When the detection unit 48 detects the first gesture or the second gesture, the volume control unit 52 increases the volume of the sounds acquired by the multiple in-vehicle microphones 18 and the volume of the sounds acquired by the exterior microphone 20, among the volumes output by the audio unit 34.

[0039] 5, the user U may make a third gesture of covering his / her ears 62 with his / her hands 64. When the third gesture is detected by the detection unit 48, the volume control unit 52 reduces the volume of the sounds picked up by the multiple in-vehicle microphones 18 and the volume of the sounds picked up by the exterior microphone 20, among the volumes output by the audio unit 34.

[0040] [3-2 Second volume adjustment example] In the second volume adjustment example, the volume control unit 52 selectively increases or decreases the volume of the sounds picked up by the multiple in-vehicle microphones 18 and the volume of the sounds picked up by the exterior microphone 20 among the volumes output by the audio unit 34.

[0041] As shown in Fig. 6A, the user U may perform a first gesture with both hands 64a, 64b. Furthermore, as shown in Fig. 6B, the user U may perform a second gesture with both hands 64a, 64b. When the gesture shown in Fig. 6A or the gesture shown in Fig. 6B is detected by the detection unit 48, the volume control unit 52 selectively increases the volume of the sound acquired by the exterior microphone 20 from among the volumes output by the audio unit 34.

[0042] 7, the user U may perform a first gesture or a second gesture with one hand 64a and lower the other hand 64b. When the gesture shown in Fig. 7 is detected by the detection unit 48, the volume control unit 52 selectively increases the volume of the sound picked up by the in-vehicle microphone 18 in the direction in which the palm 60 of the one hand 64a is facing, among the volumes output by the audio unit 34.

[0043] 8, the user U may make a first gesture or a second gesture with one hand 64a and point in a direction with the other hand 64b. When the gesture shown in Fig. 8 is detected by the detection unit 48, the volume control unit 52 selectively increases the volume of the sound picked up by the in-vehicle microphone 18 in the direction pointed by the other hand 64b, among the volumes output by the audio unit 34.

[0044] 9, the user U may make a third gesture with one hand 64a and point in a direction with the other hand 64b. When the gesture shown in Fig. 9 is detected by the detection unit 48, the volume control unit 52 selectively reduces the volume of the sound picked up by the in-vehicle microphone 18 in the direction pointed by the other hand 64b, among the volumes output by the audio unit 34.

[0045] [4 Other embodiments] The in-vehicle camera 14, the exterior camera 16, the in-vehicle microphone 18, and the exterior microphone 20 may communicate with a communication device of the vehicle 10. Furthermore, the HMD 22 may also communicate with the communication device of the vehicle 10. In other words, each device provided in the vehicle 10 (the in-vehicle camera 14, the exterior camera 16, the in-vehicle microphone 18, and the exterior microphone 20) and the HMD 22 may communicate via the communication device of the vehicle 10.

[0046] [5 Effects] According to this embodiment, the detection unit 48 detects gestures by the user U based on images acquired by the in-vehicle camera 14, which constantly captures the interior of the vehicle, and therefore can stably detect gestures by the user U.

[0047] When the volume of the head-mounted display 22 is adjusted using a device such as a remote controller, there is a risk that the sense of immersion of the user U may be impaired. According to the above configuration, the volume of the head-mounted display 22 can be adjusted by the gestures of the user U, so that the sense of immersion of the user U is less likely to be impaired.

[0048] According to the above configuration, the head-mounted display 22 displays a display image corresponding to an image acquired by the exterior camera 16 that captures the surroundings of the vehicle 10, thereby making it possible to suppress motion sickness in the user U.

[0049] As described above, according to this embodiment, the head-mounted display 22 can be used satisfactorily inside the vehicle.

[0050] [6 Notes] The following additional notes are further disclosed regarding the above embodiment.

[0051] (Appendix 1) The control system (12) of the present disclosure includes a head-mounted display (22) worn by a user (U) aboard a vehicle (10), the head-mounted display providing at least a display image to the user corresponding to an image acquired by an exterior camera (16) provided in the vehicle and capturing an image of the area around the vehicle, a detection unit (48) that detects a gesture by the user based on an image acquired by an interior camera (14) provided in the vehicle and capturing an image of the interior of the vehicle, and a volume control unit (52) that executes control to adjust the volume of a speaker (34) provided in the head-mounted display based on the gesture detected by the detection unit.

[0052] According to the above configuration, the detection unit detects a gesture made by the user based on an image acquired by an in-vehicle camera that constantly captures images of the interior of the vehicle, and therefore, can stably detect a gesture made by the user.

[0053] According to the above configuration, the volume of the head-mounted display can be adjusted by the user's gestures, which makes it less likely that the user's sense of immersion will be impaired.

[0054] According to the above configuration, the head-mounted display displays an image corresponding to an image acquired by an external camera that captures the surroundings of the vehicle, thereby making it possible to suppress motion sickness in the user.

[0055] As described above, according to the above configuration, the head-mounted display can be used satisfactorily inside the vehicle.

[0056] (Appendix 2) In the control system described in Appendix 1, when the detection unit detects the gesture of placing the palm (60) around the ear (62) without covering the ear (62) with the palm, the volume control unit may increase the volume.

[0057] (Appendix 3) In the control system according to Supplementary Note 1 or 2, when the detection unit detects the gesture of covering an ear with a hand (64), the volume control unit may decrease the volume.

[0058] (Appendix 4) In the control system described in any one of Supplementary Notes 1 to 3, when the detection unit detects the gesture of placing the palm of the left hand (64a) around the left ear without covering the left ear with the palm of the left hand, and placing the palm of the right hand (64b) around the right ear without covering the right ear with the palm of the right hand, the volume control unit may selectively increase the volume of the sound acquired by the microphone (20) that acquires sounds outside the vehicle.

[0059] (Appendix 5) In the control system described in any one of Supplementary Notes 1 to 4, when the detection unit detects the gesture of placing the palm of one hand (64a) around an ear without covering the ear with the hand and pointing in a direction with the other hand (64b), the volume control unit may selectively increase the volume of the sound from the direction pointed by the other hand.

[0060] (Appendix 6) In the control system described in Supplementary Note 2, the volume of the sound from the direction in which the palm is pointed may be selectively increased.

[0061] According to the configurations of Supplementary Notes 2 to 6, the user can adjust the volume by gestures instead of operating the device, which makes it less likely that the user's sense of immersion will be impaired.

[0062] (Appendix 7) The head-mounted display of the present disclosure is a head-mounted display worn by a user in a vehicle, and includes a display unit (32) that is provided in the vehicle and provides the user with at least a display image corresponding to an image acquired by an exterior camera that photographs the area around the vehicle, a speaker that provides sound to the user, a detection unit that detects gestures made by the user based on images acquired by an interior camera that is provided in the vehicle and photographs the interior of the vehicle, and a volume control unit that executes control to adjust the volume of the speaker based on the gesture detected by the detection unit.

[0063] According to the above configuration, the detection unit detects a gesture made by the user based on an image acquired by an in-vehicle camera that constantly captures images of the interior of the vehicle, and therefore, can stably detect a gesture made by the user.

[0064] According to the above configuration, the volume of the head-mounted display can be adjusted by the user's gestures, which makes it less likely that the user's sense of immersion will be impaired.

[0065] According to the above configuration, the head-mounted display displays an image corresponding to an image acquired by an external camera that captures the surroundings of the vehicle, thereby making it possible to suppress motion sickness in the user.

[0066] Although the present disclosure has been described in detail, the present disclosure is not limited to the individual embodiments described above. Various additions, substitutions, modifications, partial deletions, etc. are possible in these embodiments without departing from the gist of the present disclosure or the spirit of the present disclosure derived from the content of the claims and their equivalents. These embodiments can also be implemented in combination. For example, in the above-described embodiments, the order of each operation and the order of each process are shown as examples and are not limited to these. The same applies when numerical values ​​or mathematical expressions are used in the description of the above-described embodiments. [Explanation of symbols]

[0067] 10...Vehicle 12...Control system 14...In-car camera 16...Out-car camera 20...External microphone (microphone) 22...Head-mounted display (HMD) 34...Sound unit (speaker) 48...Detection unit 52...Volume control 60...Palm 62...Ear 64, 64a, 64b...Hands

Claims

1. a head-mounted display worn by a user in a vehicle, the head-mounted display being provided in the vehicle and providing at least a display image to the user corresponding to an image acquired by an exterior camera that captures an image of the periphery of the vehicle; a detection unit that detects a gesture made by the user based on an image acquired by an in-vehicle camera that is provided in the vehicle and captures an image of the interior of the vehicle; a volume control unit that executes control to adjust the volume of a speaker provided in the head-mounted display based on the gesture detected by the detection unit; A control system comprising:

2. 2. The control system of claim 1, The volume control unit increases the volume when the detection unit detects the gesture of placing the palm around the ear without covering the ear.

3. 2. The control system of claim 1, When the gesture of covering an ear with a hand is detected by the detector, the volume controller decreases the volume.

4. 2. The control system of claim 1, When the detection unit detects the gesture of placing the palm of the left hand around the left ear without covering the left ear with the palm of the left hand and placing the palm of the right hand around the right ear without covering the right ear with the palm of the right hand, the volume control unit selectively increases the volume of the sound captured by a microphone that captures sounds outside the vehicle.

5. 2. The control system of claim 1, A control system in which, when the palm of one hand is placed around an ear without covering the ear with the palm of the other hand and the gesture of pointing in a direction with the other hand is detected by the detection unit, the volume control unit selectively increases the volume of sound from the direction pointed by the other hand.

6. 3. The control system of claim 2, A control system that selectively increases the volume of sounds coming from the direction the palm is pointed.

7. A head-mounted display worn by a user in a vehicle, a display unit that is provided in the vehicle and that provides at least a display image to the user according to an image acquired by an exterior camera that captures an image of the surroundings of the vehicle; a speaker for providing sound to the user; a detection unit that detects a gesture made by the user based on an image acquired by an in-vehicle camera that is provided in the vehicle and captures an image of the interior of the vehicle; a volume control unit that executes control to adjust the volume of the speaker based on the gesture detected by the detection unit; A head-mounted display comprising:

Images