Method for controlling head-mounted information processing device
The head-mounted device addresses ambient sound interference by collecting and managing audio inputs to ensure clear audio output, adjusting sound emission methods to prioritize desired sounds over distracting noises, enhancing user experience.
Patent Information
- Application Number
- JP2023046346
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2023-03-23
- Publication Date
- 2024-11-07
- Estimated Expiration
- 2039-03-04
AI Technical Summary
Existing head-mounted information processing devices struggle to effectively manage ambient sounds that interfere with the user's ability to hear desired audio outputs, particularly when unnecessary ambient sounds obstruct the audio signals, making it difficult to hear important verbal calls or warnings.
The device includes an audio input section to collect ambient sound, a control section to determine interference, and a control unit to manage audio output based on volume levels and user or environmental conditions, adjusting sound emission methods such as changing frequency characteristics or sound image positioning to minimize interference.
Ensures that users can reliably hear desired audio despite unnecessary ambient sounds, preventing missed information and improving user experience by managing sound emission to avoid interference.
Smart Images

Figure 0007579378000001 
Figure 0007579378000002 
Figure 0007579378000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a method for controlling a head-mounted information processing device that is worn on the head and has a function for viewing and listening to images and sounds. [Background technology]
[0002] Patent document 1 shows a mobile terminal having a judgment unit that judges whether or not ambient sound should be output to earphones based on an audio signal input from at least one microphone, and an audio output control unit that outputs the audio signal input from the at least one microphone to the earphones when it is determined that ambient sound should be output to the earphones. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2015-211267 A Summary of the Invention [Problem to be solved by the invention]
[0004] In general, a head-mounted information processing device can display real space or virtual space (virtual objects) on a display unit worn by a user on the head. In recent years, such head-mounted information processing devices have been able to seamlessly fuse the real world and the virtual world in real time, allowing the user to experience as if a virtual object were actually present in the place. Conventionally, in a head-mounted information processing device that is used by wearing a display unit and an audio output unit such as headphones on the head, the audio output unit converts an audio signal output from inside the head-mounted information processing device into audio, emits the audio, and transmits the audio signal to the user.
[0005] Among the sound output units, air conduction headphones, which listen to air conduction sound transmitted by air vibrations, are worn so that they come into contact with the surface of the ear, and open-ear headphones in particular are sometimes worn without completely blocking the ears, which means that ambient sounds can pass around the headphones and enter the ears, picking up and hearing the ambient sounds.Bone conduction headphones, which listen to bone conduction sound transmitted by bone vibrations, are worn without blocking the ears at all, so ambient sounds can enter the ears and be heard directly.
[0006] Incidentally, there are ambient sounds that are necessary for the user (for example, calls from others, sirens of emergency vehicles, etc.) and those that are unnecessary for the user (the sounds of trains and cars running, conversations between others, loud and noisy sounds such as fireworks and thunder, the sounds of strong winds and heavy rain, etc.) In response to such ambient sounds, Patent Document 1 discloses a method in which, while earphones are worn on the ears, the ambient sounds collected by a microphone are judged as being highly necessary for the user, and the highly necessary ambient sounds are allowed to be heard through the earphones.
[0007] However, although Patent Document 1 describes listening to ambient sounds that are highly necessary for the user, it does not suggest any measures to be taken against ambient sounds that are unnecessary for the user, and there is a problem that the unnecessary ambient sounds prevent the user from hearing the audio signals output from the inside of the head-mounted information processing device. In particular, when the audio signals output from the inside of the head-mounted information processing device are expressed in words, there is a problem that the unnecessary ambient sounds make it difficult to hear the voices assisting the user, such as verbal calls and warnings to the user, and the voices are missed.
[0008] The present invention has been made in consideration of the above, and provides a head-mounted information processing device that enables a user to reliably hear desired sounds even when there are surrounding sounds that are unnecessary for the user. [Means for solving the problem]
[0009] Among the inventions disclosed in this application, a brief summary of representative inventions is as follows.
[0010] A head-mounted information processing device according to one embodiment is a device that has a function of viewing and listening to images or sounds and is worn on the head, and includes an audio input unit, an audio output unit, and a control unit that controls the operation of the head-mounted information processing device. The audio input unit is worn near the user's ears, and collects ambient sounds that are generated outside the head-mounted information processing device and enter the ears, and converts them into input audio signals. The audio output unit generates an output audio signal, converts the generated output audio signal into output audio, and emits the output audio toward the user. The control unit determines whether the ambient sound is in a state that prevents the output audio from being heard based on the volume level of the input audio signal from the audio input unit and the volume level of the output audio signal from the audio output unit, and controls the sound emission operation of the audio output unit based on the determined result.
[0011] Moreover, a head-mounted information processing device according to an embodiment includes a display unit, an audio output unit, a sensor device, a user state determination unit, and a control unit that controls the operation of the head-mounted information processing device. The display unit displays predetermined information including virtual space information or real space information to the user. The audio output unit generates an output audio signal, converts the generated output audio signal into an output audio, and emits the output audio toward the user. The sensor device detects the user's state or the state around the user. The user state determination unit determines whether the user's state or the state around the user is suitable for listening to the output audio based on the detection result of the sensor device. The control unit causes the audio output unit to emit sound when the state is suitable for listening to the output audio, and instructs the audio output unit to stop emitting sound when the state is not suitable for listening to the output audio, based on the determination result of the user state determination unit. Effect of the Invention
[0012] By using the head-mounted information processing device of the present invention, it becomes possible for the user to reliably hear the desired sound even in the presence of unwanted ambient sounds.
[0013] Furthermore, problems, configurations and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]
[0014] [Figure 1] 1 is a schematic diagram showing an example of an external configuration of a head-mounted information processing device according to a first embodiment of the present invention. [Diagram 2] 2 is a block diagram showing a schematic configuration example of the head-mounted information processing device of FIG. 1. [Diagram 3] 3 is a flowchart showing an example of detailed processing contents in a control unit in FIG. 2. [Figure 4] FIG. 2 is a block diagram showing an example of the configuration of a main part around a control unit related to voice input. [Diagram 5] 4 is a flowchart showing an example of detailed processing contents of a control process of the sound emission operation in FIG. 3. [Figure 6] FIG. 2 is a block diagram showing an example of the configuration of the main parts around a control unit related to audio output. [Figure 7] 5 is an explanatory diagram showing an example of an operation when a delay time is added when sound emission is resumed in FIG. 4. FIG. [Figure 8] 6 is a flowchart showing an example of detailed processing contents of the sound emission operation control processing in FIG. 3, which differs from FIG. 5. [Figure 9] 9 is a flowchart showing an example of detailed processing contents of the sound emission operation control processing in FIG. 3, which differs from FIG. 8. [Figure 10] 10 is a flow chart showing an example of detailed processing contents of a control unit in FIG. 2 in a head-mounted information processing device according to a second embodiment of the present invention. [Figure 11] FIG. 11 is a diagram illustrating an example of an inappropriate state for a user in FIG. [Figure 12] FIG. 11 is a block diagram showing a schematic configuration example of a head-mounted information processing device according to a third embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0015] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In all the drawings for explaining the embodiment, the same members are generally designated by the same reference numerals, and the repeated description will be omitted.
[0016] (Embodiment 1) Overview of head-mounted information processing device Fig. 1 is a schematic diagram showing an example of the external configuration of a head-mounted information processing device according to a first embodiment of the present invention. The head-mounted information processing device in Fig. 1 includes a head-mounted information processing device main body (head-mounted display) 100 mounted on the head of a user, a chest-worn wearable terminal 200 mounted on the chest of the user or a wristband-type wearable terminal 300 mounted on the arm of the user, and an input controller 400.
[0017] The head-mounted information processing device main body 100 includes a camera 111 and a sensor device that detects the state of the user or the state around the user. The sensor device includes a right eye gaze sensor 112, a left eye gaze sensor 113, an acceleration sensor 114, a gyro sensor 115, a geomagnetic sensor 116, a temperature and humidity sensor 117, a surrounding object detection sensor 118, a speech sound microphone 121, and first to third peripheral sound microphones 131 to 133. The head-mounted information processing device main body 100 captures an image in front of the user using the camera 111, and detects the user's gaze, the movement of the user's head, the temperature and humidity around the user, the presence or absence of objects around the user, and the like using the sensor device.
[0018] The head-mounted information processing device main body 100 also includes a display unit 119. The display unit 119 is installed in front of both eyes, and displays to the user predetermined information such as virtual space information and real space information captured by the camera 111. The vocal sound microphone 121 collects vocal sounds from the user and converts them into audio signals. The first to third peripheral sound microphones 131 to 133 constitute an audio input unit. The audio input unit is attached near the user's ears, collects ambient sounds generated outside the head-mounted information processing device and entering the ears, and converts them into input audio signals. The first peripheral sound microphone 131 is provided, for example, in the center of the head-mounted information processing device, and collects sounds uttered by others toward the user. The second and third peripheral sound microphones 132 and 133 are provided in contact with the left and right ears of the user, and collect ambient sounds entering the user's ears from the outside.
[0019] The head-mounted information processing device main body 100 also includes headphones 122 (122a, 122b) that are attached to the left and right ears of the user. The headphones 122a, 122b constitute an audio output unit, converting left and right output audio signals generated inside the head-mounted information processing device into left and right output sounds, respectively, and emitting the sounds toward the user. When the user listens to sound through the audio output unit, there are cases where the sound is heard as air-conducted sound that enters the ear and is transmitted by vibration of the air, and cases where the sound is heard as bone-conducted sound that is transmitted by vibration of the bone without passing through the ear. The headphones 122 may be of either an air-conducted type or a bone-conducted type.
[0020] The chest-worn wearable terminal 200 includes a heartbeat sensor 201, which is a sensor device, and detects the heart rate, which is the number of times the heart beats within a certain period of time. The wristband-type wearable terminal 300 includes a blood pressure sensor 301, which is a sensor device, and detects the user's blood pressure. The wristband-type wearable terminal 300 may also include a pulse sensor and detect the pulse rate, which is the number of times the artery beats. The input controller 400 allows the user to perform various input operations. The chest-worn wearable terminal 200, the wristband-type wearable terminal 300, and the input controller 400 transmit and receive information to and from the head-mounted information processing device main body 100 by short-range wireless communication. In this case, the information may be transmitted and received not only wirelessly but also by wire.
[0021] As described above, in the head-mounted information processing device worn closely by the user, the audio input units such as the second and third peripheral sound microphones 132, 133 of the head-mounted information processing device main body 100 can detect ambient sounds entering the ears. In addition, the various sensor devices of the head-mounted information processing device main body 100 and the wearable terminals 200, 300 can detect the user's state (e.g., mental and physical state and body movement) and the state around the user. In addition, based on the user's input operation via the input controller 400, the display unit 119 provided in front of the user's eyes in the head-mounted information processing device main body 100 can display predetermined information including real space information or virtual space information.
[0022] Fig. 2 is a block diagram showing a schematic configuration example of the head-mounted information processing device of Fig. 1. In Fig. 2, the head-mounted information processing device main body 100 includes the camera 111, various sensor devices 151 (112-118, 121, 131-133), display unit 119, and headphones 122 described in Fig. 1. In addition, the head-mounted information processing device main body 100 includes a user state determination unit 123, a vibration generation unit 124, a control unit 125, a memory 128, a soundproof ear cover detection unit 134, a virtual space information generation processing unit 141, and a short-distance wireless communication unit 142. These components are connected to each other via a bus 150.
[0023] The camera 111 is installed on the front of the head-mounted information processing device main body 100 and captures the scenery in front of the user. The captured image is displayed on the display unit 119 as an image in real space. The right eye gaze sensor 112 and the left eye gaze sensor 113 detect the gaze of the right eye and the left eye, respectively. The technology for detecting the user's gaze is generally known as eye tracking. For example, in eye tracking using corneal reflex, an infrared LED (Light Emitting Diode) is irradiated onto the face and an image is captured with an infrared camera, and the position on the cornea of the reflected light generated by the irradiation of the infrared LED (corneal reflex) is used as a reference point, and the gaze is detected based on the position of the pupil relative to the position of the corneal reflex.
[0024] The acceleration sensor 114 is a sensor that detects acceleration, which is a change in speed per second, and detects movement, vibration, impact, and the like. The gyro sensor 115 is a sensor that detects angular velocity in the rotation direction, and detects the vertical, horizontal, and diagonal posture states. The geomagnetic sensor 116 is a sensor that detects the magnetic force of the earth, and detects the direction in which the head-mounted information processing device main body 100 is facing. Therefore, by using the gyro sensor 115 or the geomagnetic sensor 116, and in some cases by using the acceleration sensor 114 in combination, it is possible to detect the movement of the head of the user wearing the head-mounted information processing device main body 100. In particular, by using a three-axis type geomagnetic sensor 116 that detects geomagnetism in the up-down direction in addition to the front-back and left-right directions, it is possible to detect the movement of the head with higher accuracy by detecting the geomagnetic change in response to the movement of the head.
[0025] The temperature and humidity sensor 117 is a sensor that detects the temperature and humidity around the user. The surrounding object detection sensor 118 is a sensor that detects the distance to an object, the direction and the relative speed of the object by emitting radio waves, light waves, ultrasonic waves, etc. and detecting the reflected waves from the object. By wearing the surrounding object detection sensor 118 on the user's head, it is possible to detect the distance to an object existing around the user with the user as a reference, the relative speed of the object, and the direction in which the object exists.
[0026] When an output audio signal generated inside the head-mounted information processing device is output as output audio via headphones 122 (audio output unit), user state determination unit 123 determines whether or not the state of the user or the state around the user is suitable for listening to the output audio, based on the detection results of various sensor devices 151. Although details will be described later with reference to FIG. 10 and subsequent figures, user state determination unit 123 determines that the state is not suitable for listening to the output audio in various predetermined cases, such as when the user is considered to be gazing at virtual space information or real space information on display unit 119.
[0027] The control unit 125 is composed of, for example, a CPU (Central Processing Unit) and the like, and controls each component and controls the operation of the entire head-mounted information processing device by executing a program 126, such as an OS (Operating System) and an operation control application, stored in a memory 128. Although details will be described later, the control unit 125 determines whether or not the ambient sound interferes with the hearing of the output sound based on the volume level of the input sound signal from the sound input unit (second and third peripheral sound microphones 132, 133) and the volume level of the output sound signal from the sound output unit (headphones 122, etc.), and controls the sound emission operation of the sound output unit based on the determination result.
[0028] The memory 128 is a flash memory, a RAM for work, etc. The memory 128 stores a program 126 such as an OS and an application for operation control used by the control unit 125. The memory 128 also stores various information data 127 such as data of output audio signals generated by the head-mounted information processing device main body 100, virtual space information generated by the virtual space information generation processing unit 141, and information from the wearable terminals 200 and 300. For example, data of output audio signals generated by the head-mounted information processing device main body 100 includes data of a voice assistant that responds to a user's request through a dialogue with the user by voice, such as calling to the user, guidance, conveying information, warning, etc.
[0029] The display unit 119 is configured with a liquid crystal panel or the like, and displays real space information and virtual space information by video, and also displays display contents such as presentation notification information and operation status to the user on a screen. For example, the display unit 119 may display information to notify the user when sound emission of an output audio signal is started, interrupted, or resumed. This allows the user to recognize that, for example, when sound emission is interrupted or resumed, the interruption or resume has been caused by a normal control operation and not by a malfunction.
[0030] The vibration generating unit 124 generates vibrations in response to an instruction from the control unit 125, and converts, for example, notification information for the user generated by the head-mounted information processing device main body 100 into vibrations. The vibration generating unit 124 generates vibrations while being attached closely to the user's head, thereby enabling the user to receive a notification with a higher degree of recognition. For example, when the imaging range of the camera 111 includes the ears, the soundproof ear cover detection unit 134 detects whether the user's ears are covered with hands or the like based on the image to prevent ambient sounds from entering the ears. It is also possible to detect whether ambient sounds are prevented from entering the ears based on the volume of ambient sounds entering the second and third ambient sound microphones 132 and 133.
[0031] The virtual space information generation processing unit 141 generates virtual space information that expresses a virtual space different from the real space with images and sounds. The short-distance wireless communication unit 142 is a communication interface that performs short-distance wireless communication with the chest-worn wearable terminal 200, the wristband-type wearable terminal 300, and the input controller 400 that are present within a range where short-distance wireless communication is possible. The short-distance wireless communication unit 142 receives detection information from sensors mounted on each of the wearable terminals 200 and 300, transmits control information for controlling a vibration generating unit mounted on each of the wearable terminals 200 and 300, transmits and receives input operation information to and from the input controller 400, and the like.
[0032] The short-distance wireless communication unit 142 is typically an electronic tag, but is not limited thereto, and may be anything that is capable of at least wireless communication when the head-mounted information processing device main body 100 is located near the chest-worn wearable terminal 200, the wristband-type wearable terminal 300, and the input controller 400. Examples of such a technology include Bluetooth (registered trademark), IrDA (Infrared Data Association), Zigbee (registered trademark), HomeRF (Home Radio Frequency, registered trademark), and wireless LAN (IEEE802.11a, IEEE802.11b, IEEE802.11g).
[0033] The chest-mounted wearable terminal 200 has a heart rate sensor 201, which is a sensor device, a vibration generating unit 202, and a short-range wireless communication unit 203. The heart rate sensor 201 is attached closely to the user's chest and detects the user's heart rate with high accuracy. The short-range wireless communication unit 203 transmits information on the detected heart rate to the head-mounted information processing device main body 100 by short-range wireless communication. The vibration generating unit 202 generates vibrations in response to a control input, and by being attached closely to the user's chest, the generated vibrations can be reliably transmitted to the user.
[0034] The wristband-type wearable terminal 300 has a blood pressure sensor 301, which is a sensor device, a vibration generating unit 302, and a short-range wireless communication unit 303. The blood pressure sensor 301 is wrapped around the user's arm and worn to accurately detect the user's blood pressure. The short-range wireless communication unit 303 transmits information about the detected blood pressure to the head-mounted information processing device main body 100 by short-range wireless communication. The vibration generating unit 302 generates vibrations in response to an input, and by wrapping the device around the user's arm and wearing it, the generated vibrations can be reliably transmitted to the user.
[0035] Here, the head-mounted information processing device main body 100 receives heart rate information from the heart rate sensor 201 and blood pressure information from the blood pressure sensor 301 via the short-range wireless communication unit 142. The user state determination unit 123 can determine whether the user's state (mental and physical state) is suitable for listening to the output sound based on the heart rate information and blood pressure information. In addition, notification information for the user transmitted from the head-mounted information processing device main body 100 is transmitted to the vibration generation units 202 and 302 of the wearable terminals 200 and 300 via the short-range wireless communication units 142, 203 and 303. The vibration generation units 202 and 302 can notify the user of the notification information by converting the notification information into vibration.
[0036] The vibration generating unit 124 of the head-mounted information processing device main body 100 and the vibration generating units 202, 302 of each wearable device 200, 300 may generate vibrations to notify the user when, for example, starting, interrupting, or resuming emission of an output audio signal. This allows the user to strongly recognize, for example, that sound emission has been interrupted or resumed. The head-mounted information processing device main body 100 may also notify the user by emitting a sound from the headphones 122 to notify the user of the interruption or resumption of sound emission.
[0037] The input controller 400 has an input operation unit 401 and a short-range wireless communication unit 402. The input operation unit 401 is an input means such as a keyboard or key buttons, and allows a user to set and input information that he or she wishes to input. The input operation unit 401 may also be an input means of a touch pad type such as a capacitance type. Information inputted by the input operation unit 401 is transmitted to the head-mounted information processing device main body 100 via the short-range wireless communication unit 402. Note that, although wireless communication is used here to improve usability, wired communication may of course also be used.
[0038] <<Details of the control unit>> FIG. 3 is a flow diagram showing an example of detailed processing contents in the control unit in FIG. 2. FIG. 4 is a block diagram showing an example of the configuration of the main parts around the control unit related to the voice input in FIG. 2. The control unit 125 in FIG. 2 repeatedly executes the flow shown in FIG. 3 at a predetermined control period. In FIG. 3, the control unit 125 detects an ambient sound using the voice input unit (the second and third ambient sound microphones 132, 133) (step S101). Next, the control unit 125 determines whether or not the audio output unit (the headphones 122, etc.) is emitting an output sound (step S102). If the output sound is being emitted in step S102, the control unit 125 determines whether or not the ambient sound is in a state where the output sound is prevented from being heard in terms of the volume level based on the volume level of the ambient sound detected in step S101 (step S103). In the specification, the state where the ambient sound is prevented from being heard in terms of the volume level is also called a volume disturbance state.
[0039] If a volume disturbance state occurs in step S103, the control unit 125 determines whether or not the problem cannot be dealt with by changing the sound emission method (step S104). Changing the sound emission method is, for example, a process of changing the frequency characteristics of an output sound signal, which will be described in detail later. If the problem cannot be dealt with by changing the sound emission method in step S104, the control unit 125 instructs the sound output unit to stop sound emission (step S105). Furthermore, in step S105, the control unit 125 notifies the user that sound emission will be stopped by displaying a notice on the display unit 119, using tactile vibrations from the vibration generating units 124, 202, 302, or using voice output from the sound output unit (headphones 122), and ends the process.
[0040] If no volume disturbance state occurs in step S103, the control unit 125 determines whether the audio output unit has changed the sound emission method and is emitting sound (step S106). If the sound emission method has been changed and sound is being emitted in step S106, the control unit 125 cancels the change in the sound emission method in step S107 (i.e., returns to the default sound emission method), and then continues sound emission by the audio output unit in step S108 and ends the process. On the other hand, if the sound emission method has not been changed and sound is being emitted in step S106 (i.e., if sound is being emitted with the default sound emission method), the control unit 125 continues sound emission by the audio output unit as it is and ends the process (step S108). Note that even if the problem can be dealt with by changing the sound emission method in step S104, the control unit 125 continues sound emission by the audio output unit as it is (step S108).
[0041] If sound for output is not being emitted in step S102, the control unit 125 determines whether sound emission by the sound output unit is paused or not (step S109). If sound emission is paused in step S109, the control unit 125 executes control processing of the sound emission operation in step S110, the details of which will be described later, and ends the processing. The control processing of the sound emission operation includes changing the sound emission method, etc. On the other hand, if sound emission is not paused in step S109, the control unit 125 ends the processing.
[0042] Fig. 4 shows an example of the configuration around the control unit 125 related to step S103 in Fig. 3 (i.e., determining whether or not a volume disturbance state is occurring). Fig. 4 shows the control unit 125 and an audio input unit 601. The audio input unit 601 includes second and third external sound microphones 132, 133 that collect ambient sounds entering the ears and convert them into an input audio signal (VI). On the other hand, the control unit 125 includes an ambient sound discrimination unit 610 and an audio input processing unit 611. The audio input processing unit 611 receives, for example, an input audio signal (VI) from the audio input unit 601 and detects its volume level (LI).
[0043] The ambient sound discrimination unit 610 includes a threshold generation unit 615, a threshold table 616, and a comparator 617. The threshold table 616 defines in advance a correspondence relationship between each range of the volume level (LOx) of the output audio signal (VO) from the audio output unit and a threshold level that indicates a relative allowable value of the ambient sound for each range. The threshold generation unit 615 receives the volume level (LOx) of the output audio signal (VO) that is recognized in advance within the device, and generates a threshold level (THx) according to the volume level (LOx) based on the threshold table 616.
[0044] The comparator 617 compares the volume level (LI) of the input audio signal (VI) from the audio input processing unit 611 with the threshold level (THx) from the threshold generation unit 615 to determine whether to instruct the audio output unit to suspend sound emission. In other words, the comparator 617 determines whether the ambient sound corresponding to the input audio signal (VI) interferes with the listening of the output audio corresponding to the output audio signal (VO), that is, whether a volume disturbance state occurs.
[0045] Specifically, when the volume level (LI) of the input audio signal (VI) is equal to or higher than the threshold level (THx), the comparator 617 determines that the ambient sound is preventing the output audio from being heard (a volume disturbance state is occurring), and issues an instruction to suspend sound emission (INT) to the audio output unit. Also, when the volume level (LI) of the input audio signal (VI) becomes less than the threshold level (THx) while the audio output unit is suspending sound emission (i.e., during the control process of the sound emission operation in step S110 of FIG. 3), the comparator 617 determines that the ambient sound is not preventing the output audio from being heard, and issues an instruction to resume sound emission (RES) to the audio output unit. In other words, the comparator 617 determines that the volume disturbance state is resolved, and issues an instruction to resume sound emission (RES) to the audio output unit.
[0046] Here, the volume level of the surrounding sound that is felt as an obstacle when the user listens to the output sound may vary from person to person. Therefore, the threshold generating unit 615 is capable of correcting the threshold level (THx) according to a user setting. Here, the threshold level (THx) is generated using the threshold table 616, but it is also possible to generate the threshold level (THx) using, for example, a predetermined arithmetic expression or the like.
[0047] In addition, here, a relative comparison between the volume level (LI) of the input audio signal (VI) and the volume level (LOx) of the output audio signal (VO) is used, but in some cases, an absolute comparison of the volume level (LI) of the input audio signal (VI) may be used, or a combination of relative and absolute comparisons may be used. For example, when the ambient sound is very loud, it is possible to determine that the ambient sound is preventing the output audio from being heard by an absolute comparison using only the volume level (LI) of the input audio signal (VI).
[0048] Furthermore, the ambient sound discrimination unit 610 and the voice input processing unit 611 may be provided in the user state discrimination unit 123 in Fig. 2 which discriminates whether or not the state of the user or the state around the user is suitable for listening to the output voice. That is, in Fig. 2, the control unit 125 and the user state discrimination unit 123 are separated mainly from the viewpoint of function for convenience, but the control unit 125 can include the function of the user state discrimination unit 123. Also, from the viewpoint of hardware, the user state discrimination unit 123 can be realized by program processing by a CPU, for example, and from this viewpoint as well, the control unit 125 can include the user state discrimination unit 123.
[0049] 《Sound emission control process[1]》 Fig. 5 is a flow diagram showing an example of detailed processing contents of the control processing of the sound emission operation in Fig. 3. Fig. 6 is a block diagram showing an example of the configuration of the main parts around the control unit related to the sound output in Fig. 2. Fig. 5 shows the processing contents of the control processing (step S110) of the sound emission operation performed while sound emission is interrupted in Fig. 3, and shows the processing contents of the control processing [1] as an example.
[0050] In Fig. 5, the control unit 125 determines whether or not the volume disturbance state (a state in which ambient sound prevents the output sound from being heard) as described in step S103 of Fig. 3 and Fig. 4 has been resolved (step S201). If the volume disturbance state has been resolved in step S201, the control unit 125 (specifically, the ambient sound discriminator 610) issues a resume instruction (RES) to instruct the audio output unit to resume sound output, as described in Fig. 4. In response to the resume instruction (RES), the audio output unit resumes sound output by going back to the point before it was interrupted (step S202).
[0051] Also, in step S202, the control unit 125 adds a delay time according to the volume level (LI) of the ambient sound (input audio signal (VI)) immediately before the volume disturbance state is eliminated when restarting sound emission, as will be described in detail later with reference to FIG. 7. Specifically, the control unit 125 issues a restart instruction (RES) to the audio output unit after a predetermined period has elapsed according to the volume level (LI) of the input audio signal (VI) immediately before the ambient sound becomes a state in which it does not disturb the listening of the output audio. Furthermore, in step S202, the control unit 125 notifies the user that sound emission will be restarted by displaying a notice on the display unit 119, using tactile vibrations from the vibration generating units 124, 202, 302, or using voice output from the audio output unit (headphones 122), and ends the process. In the subsequent control cycles, the sound emission operation continues in the flow of steps S101 → S102 → S103 → S106 → S108 in FIG. 3.
[0052] On the other hand, if the volume disturbance state still occurs in step S201, the control unit 125 analyzes the bias of the ambient sound (step S203) and determines whether or not there is a bias in the ambient sound (step S204). Specifically, for example, the audio input processing unit 611 in the control unit 125 in Fig. 4 determines whether or not there is a bias in the volume levels (LI) of the left and right input audio signals (VI) from the second and third peripheral sound microphones 132, 133 that is equal to or greater than a predetermined reference value.
[0053] If there is a bias in the ambient sound in step S204, the control unit 205 instructs the user to change the direction of the head using the display unit 119, and ends the process (step S205). When the user then changes the direction of the head, the bias in the ambient sound decreases accordingly. As a result, if the volume disturbance state is resolved when the process reaches step S201 in Fig. 4 via the flow in Fig. 3 in the next control cycle, sound emission is resumed in step S202.
[0054] On the other hand, even if the deviation of the ambient sound is minimized by changing the direction of the user's head, the volume disturbance state may not be resolved. In this case, the control unit 125 proceeds from step S204 to step S206, and in step S206, issues an instruction to the user to cover his / her ears using the display unit 119, and ends the process. After that, if the user, for example, really wants to listen to the output sound, covers his / her ears with his / her hands so as to include the second and third peripheral sound microphones 132, 133. As a result, usually, when the process reaches step S201 in FIG. 4 via the flow in FIG. 3 in the next control cycle, the volume disturbance state is resolved, and sound emission is resumed in step S202.
[0055] Here, the control unit 125 indirectly determines whether the user covers his / her ears based on the detection results of the second and third peripheral sound microphones 132, 133 by the peripheral sound discrimination unit 610 in FIG. 4, but may directly detect using the soundproof ear cover detection unit 134 described in FIG. 2. Here, the control unit 125 issues an instruction to cover the ears when there is no bias in the peripheral sound in step S204, but may issue an instruction to cover the ears regardless of whether there is bias in the peripheral sound (i.e., when the volume disturbance state is not resolved in step S201). Furthermore, when the user covers his / her ears, for example, the volume of the peripheral sound detected by the first peripheral sound microphone 131, etc. may be adjusted and superimposed on the output audio signal (VO). In this case, it is possible to prevent the peripheral sound from being completely inaudible.
[0056] Further, here, the control unit 125 issues an instruction to the user to change the direction of the head when there is a bias in the ambient sound, but in this case, the instruction may be issued including the direction of the head, etc. Specifically, the control unit 125 may, for example, determine the direction from which the ambient sound comes based on the detection results of the first to third peripheral sound microphones 131 to 133, calculate the direction of the head to move away from the direction of the sound, and guide the user to face in that direction in cooperation with a sensor device (for example, the gyro sensor 115 or the geomagnetic sensor 116).
[0057] Fig. 6 shows an example of the configuration around the control unit 125 related to step S202 (i.e., resumption of sound emission) in Fig. 5. Fig. 6 shows the control unit 125, headphones 122, and a voice dictionary 622. The control unit 125 includes a voice assistant processing unit 620 and a voice output processing unit 621. Here, the voice assistant processing unit 620 and the voice output processing unit 621, together with the headphones 122, constitute a voice output unit 602 that generates an output voice signal, converts the generated output voice signal into a voice for output, and emits the voice to the user.
[0058] The voice assistant processing unit 620 has a function of responding to a user's request through a voice-based dialogue with the user, and generates voice text data that is the source of the voice. The voice dictionary 622 includes waveform data representing each voice, and is stored, for example, in the memory 128 of FIG. 2. The voice output processing unit 621 generates an output voice signal (VO) by synthesizing the voice text data from the voice assistant processing unit 620 with the waveform data from the voice dictionary 622. The headphones 122 convert the output voice signal (VO) into a voice for output and emit it to the user.
[0059] Here, after instructing the audio output unit to suspend sound emission, when the ambient sound does not interfere with the audibility of the output sound in step S201 of Fig. 5, the control unit 125 issues a resume instruction (RES) to the audio output unit 602 in step S202. In response to the resume instruction (RES), the audio output unit 602 resumes sound emission by going back to the position before the interruption. Specifically, the audio output unit 602 resumes sound emission by going back to the beginning of the sentence that was interrupted in the middle, or by going back to the punctuation mark before the position that was interrupted in the middle or the previous phrase, or by going back a predetermined fixed number of characters or words.
[0060] In the process of generating voice text data, the voice assistant processing unit 620 recognizes sentence units, phrase units, punctuation positions, and the like, and stores the generated voice text data in a buffer (for example, the memory 128 in FIG. 2). The voice output processing unit 621 also sequentially stores the output voice signal (VO) generated corresponding to the voice text data in a buffer. Therefore, the voice output unit 602 can recognize the interruption point such as the sentence, phrase, or section between punctuation marks at the time of interruption based on the timing of receiving an interruption instruction (INT) from the control unit 125 (the ambient sound discrimination unit 610 in FIG. 4). After that, when the voice output unit 602 receives a resume instruction (RES) from the control unit 125 (the ambient sound discrimination unit 610 in FIG. 4), it can resume sound emission by going back to the point before the interruption based on the recognized interruption point.
[0061] As a specific example, assume that the audio output unit 602 emits the sentence "Tonight's weather forecast for the Tokyo area is cloudy, then fine." For example, when the audio output unit 602 receives an interrupt instruction (INT) while emitting the part "cloudy, then fine," and then receives a resume instruction (RES), it goes back to the beginning of the sentence and resumes emitting the sound from the part "tonight," or resumes emitting the sound from the part "cloudy, then fine."
[0062] In this way, the subject of control for interrupting or resuming sound emission is, for example, a voice from the voice assistant processing unit 620 that may cause problems if the user misses it, and sound such as music from a music player that does not cause any problems if the user misses it may be excluded from the control. In addition, the volume level (LOx) of the output audio signal (VO) is input to the audio output unit 602 in FIG. 6 in response to an instruction from the user via the input controller 400 (see FIG. 2). The audio output unit 602 emits sound at a volume corresponding to the volume level (LOx). The volume level (LOx) is also used by the ambient sound discrimination unit 610 shown in FIG. 4.
[0063] FIG. 7 is an explanatory diagram showing an example of operation when adding a delay time when restarting sound emission (step S202) in FIG. 4. FIG. 7 shows the relationship between the ambient sound noise level and the sound emission operation of the output audio signal. In FIG. 7, the control unit 125 starts emitting the output audio signal at time t1 when the ambient sound noise level 701 is sufficiently low, and accordingly, the sound emission operation state 702 becomes ON (operation execution). After that, when the ambient sound noise level 701 increases and becomes equal to or higher than the first ambient sound noise level 703 corresponding to the threshold level (THx) in FIG. 4 at time t2, the control unit 125 determines that the ambient sound is in a state where it interferes with the hearing of the output sound, and interrupts sound emission. Accordingly, the sound emission operation state 702 becomes OFF (operation stopped).
[0064] Thereafter, the ambient sound noise level 701 remains in a state of being equal to or higher than the first ambient sound noise level 703 but not exceeding a second ambient sound noise level 704 higher than the first ambient sound noise level 703, and then becomes lower than the first ambient sound noise level 703 at time t3. In response to this, the control unit 125 does not immediately issue a resume instruction (RES) to the audio output unit, but issues a resume instruction (RES) at time t4 after a predetermined period Td1 corresponding to the volume level of the input audio signal immediately before time t3 (i.e., immediately before the audio signal becomes unobstructed). Here, the volume level of the input audio signal becomes equal to or higher than the first ambient sound noise level 703 and lower than the second ambient sound noise level 704. The audio output unit resumes sound emission in response to the resume instruction (RES) at time t4, and the sound emission operation state 702 becomes ON (operation execution) accordingly.
[0065] Thereafter, the ambient sound noise level 701 becomes equal to or higher than the first ambient sound noise level 703 at time t5, and the sound emission operation state 702 is switched to OFF. The ambient sound noise level 701 remains equal to or higher than the second ambient sound noise level 704 and does not exceed a third ambient sound noise level 705 higher than the second ambient sound noise level 704 during the period from time t5 to time t6, and becomes lower than the first ambient sound noise level 703 at time t6. In response to this, the control unit 125 issues a resume instruction (RES) to the audio output unit after a predetermined period Td2 corresponding to the volume level of the input audio signal immediately before time t6 (i.e., equal to or higher than the second ambient sound noise level 704 and equal to or lower than the third ambient sound noise level 705). Here, the period Td2 is longer than the period Td1 because the corresponding volume level is higher than the volume level corresponding to the period Td1.
[0066] In the example of FIG. 7, the ambient sound noise level 701 rises again at time t7 before time t8 when the predetermined period Td2 has elapsed, and exceeds the first ambient sound noise level 703. For this reason, the resume instruction (RES) is not issued, and the audio output unit maintains the interrupted state of sound emission. After time t7, the ambient sound noise level 701 remains in a state equal to or higher than the third ambient sound noise level 705, and then becomes lower than the first ambient sound noise level 703 at time t9. In response to this, the control unit 125 issues a resume instruction (RES) to the audio output unit at time t10 after the elapse of a predetermined period Td3 (>Td2) corresponding to the volume level of the input audio signal immediately before time t9 (i.e., equal to or higher than the third ambient sound noise level 705). The audio output unit resumes sound emission in response to the resume instruction (RES) at time t10.
[0067] In general, a user may experience a period of hearing loss after a loud ambient sound. The higher the volume level of the ambient sound, the longer this period of hearing loss becomes. Therefore, by performing the control as shown in FIG. 7, even if the period of hearing loss changes according to the volume level of the ambient sound, the control unit 125 can reliably avoid the period of hearing loss and emit sound, and the user can hear the output sound in a good hearing state regardless of the volume level of the ambient sound. In the example of FIG. 7, the volume level of the ambient sound is divided into three stages, but it is also possible to divide it into finer stages or to use a method in which the predetermined period changes continuously according to the volume level. Furthermore, when the sound emission operation is resumed, control may be performed to slightly increase the volume level of the output sound signal.
[0068] 《Sound emission control process[2]》 Fig. 8 is a flow diagram showing an example of detailed processing contents different from Fig. 5 of the sound emission operation control processing in Fig. 3. Fig. 8 shows processing contents different from Fig. 5 of the sound emission operation control processing (step S110) performed while sound emission is interrupted in Fig. 3, and shows the processing contents of control processing [2] as an example. In Fig. 5, the control unit 125 takes measures against the ambient sound by having the user take various measures, but in Fig. 8, the control unit 125 changes the sound emission method (specifically, changes the frequency characteristics) to take measures against the ambient sound without the user's action.
[0069] In Fig. 8, the control unit 125 determines whether the volume disturbance state has been resolved, as in step S201 in Fig. 5, and if resolved, resumes sound emission, adds a delay time, and notifies the user, as in step S202 in Fig. 5. On the other hand, if the volume disturbance state still occurs in step S201, the control unit 125 analyzes the frequency characteristics of the input audio signal (VI) corresponding to the ambient sound, for example, using the audio input processing unit 611 in Fig. 4 (step S301).
[0070] Next, the control unit 125 determines whether or not the problem can be dealt with by changing the frequency characteristics of the output audio signal to be emitted based on the analysis result of the frequency characteristics in step S301 (step S302). That is, the control unit 125 determines whether or not the volume disturbance state cannot be resolved in terms of the volume level determined in step S201 of Fig. 9 or step S103 of Fig. 3, but whether or not the state in which the ambient sound interferes with the hearing of the output audio can be resolved by changing the sound emission method.
[0071] If it is determined in step S302 that the problem can be dealt with by changing the frequency characteristics, the control unit 125 changes the frequency characteristics of the output audio signal to be emitted (step S303), and then proceeds to step S202 to resume sound emission, add a delay time, notify the user, and end the process. In the control cycle thereafter, the sound emission operation continues with the frequency characteristics changed in the flow of steps S101 → S102 → S103 → S104 → S108 in FIG.
[0072] Furthermore, in the process of continuing the sound emission operation with the frequency characteristics changed, if the volume level of the ambient sound decreases and the volume disturbance state is resolved, the change in the frequency characteristics is released and the default frequency characteristics are restored in the flow of steps S103 → S106 → S107 in Fig. 3. Then, the sound emission operation continues in the flow of steps S101 → S102 → S103 → S106 → S108 in Fig. 3. On the other hand, if the control unit 125 determines in step S302 that the problem cannot be dealt with by changing the frequency characteristics (i.e., changing the sound emission method), it ends the process. As a result, the state in which sound emission is interrupted continues until the volume disturbance state is resolved.
[0073] Here, an example of a specific method for changing the frequency characteristics in steps S301 to S303 will be described. First, the control unit 125 (for example, the audio output processing unit 621 in FIG. 6) is provided with a plurality of frequency characteristics (for example, fundamental frequencies) applicable to the output audio signal (VO) in advance. The control unit 125 recognizes the frequency characteristics (for example, fundamental frequencies) of the input audio signal by analyzing the frequency characteristics of the input audio signal in step S301.
[0074] Then, the control unit 125 issues an instruction to the audio output unit 621 to select a frequency characteristic whose similarity to the frequency characteristic of the input audio signal is lower than a predetermined reference value from among a plurality of frequency characteristics applicable to the output audio signal (VO) (step S303). Specifically, the control unit 125 causes the audio output unit 621 to select, for example, a fundamental frequency of the output audio signal (VO) that is different from the fundamental frequency of the input audio signal by the reference value or more.
[0075] In this way, by changing the frequency characteristics of the output audio signal, the user can fully hear the output audio signal even when there is a relatively loud ambient sound. However, if there is no frequency characteristic whose similarity to the frequency characteristics of the input audio signal is lower than a reference value among the multiple frequency characteristics applicable to the output audio signal, it is determined in step S302 that changing the frequency characteristics is not possible, and sound emission is not resumed.
[0076] Note that the control unit 125 can selectively execute either the control process [2] shown in Fig. 8 or the control process [1] shown in Fig. 5 according to the settings, or execute the control process [2] shown in Fig. 8 and then the control process [1] shown in Fig. 5. Specifically, when the control unit 125 determines in step S302 of Fig. 8 that the problem cannot be solved by changing the frequency characteristics, it may execute the control process [1] of Fig. 5 to request the user to take a predetermined action.
[0077] 《Sound emission control process[3]》 Fig. 9 is a flow diagram showing an example of detailed processing contents of the control processing of the sound emission operation in Fig. 3, which differs from Fig. 8. Fig. 9 shows the processing contents of the control processing [3] which is a modification of the control processing [2] in Fig. 8. In Fig. 9, similar to the case of Fig. 8, the control unit 125 changes the sound emission method to take measures against the ambient sound without going through a user's action, but unlike the case of Fig. 8, the measures are taken by changing the sound image of the sound to be emitted for output.
[0078] In Fig. 9, the processes of steps S301 to S303 in Fig. 8 are replaced with processes of steps S401 to S403. In Fig. 9, if the volume disturbance state is not resolved in step S201, the control unit 125 analyzes the sound source position of the ambient sound (step S401). Specifically, the control unit 125 determines the sound source position of the ambient sound based on the volume levels and delay difference of the left and right input sound signals (VI) from the second and third peripheral sound microphones 132 and 133, for example, using the sound input processing unit 611 in Fig. 4.
[0079] Next, the control unit 125 determines whether or not the problem can be dealt with by changing the sound image of the output sound based on the analysis result of the sound source position of the ambient sound in step S401 (step S402). That is, the control unit 125 determines whether or not the state in which the ambient sound prevents the output sound from being heard can be resolved by changing the sound emission method. If it is determined in step S402 that the problem can be dealt with by changing the sound image of the output sound, the control unit 125 changes the sound image of the output sound (step S403), and then proceeds to step S202 to resume sound emission, add a delay time, notify the user, and end the process. On the other hand, if the control unit 125 determines in step S402 that the problem cannot be dealt with by changing the sound image of the output sound, end the process.
[0080] In step S403, the control unit 125 specifically instructs the audio output unit to generate left and right output audio signals so that the position of the sound image of the output audio is away from the position of the sound source of the ambient sound by a predetermined reference value or more. In response to this, for example, the audio output processing circuit 621 in Fig. 6 controls the sound images of the output audio emitted from the left and right headphones 122a, 122b by controlling the volume levels and delay difference of the left and right output audio signals. At this time, for example, when the sound source position of the ambient sound is diagonally forward to the left, the sound image position of the output audio is determined to be diagonally backward to the right, etc.
[0081] In this way, by changing the sound image of the output sound, the user can fully hear the output sound even when there is a relatively loud ambient sound. In addition, if the control unit 125 determines in step S402 that changing the sound image of the output sound is not possible, specifically, if it is not possible to create a sound image position that is farther away from the position of the ambient sound source than the reference value, the control unit 125 ends the process. As a result, the sound emission is suspended until the volume disturbance state is resolved.
[0082] The control unit 125 can selectively execute either the control process [3] shown in Fig. 9 or the control process [2] shown in Fig. 8 according to the settings, or execute the control process [3] shown in Fig. 9 before or after the control process [2] shown in Fig. 8 and before the control process [1] shown in Fig. 5. Specifically, when the control unit 125 determines that the problem cannot be solved by changing the sound emission method in both step S402 in Fig. 9 and step S302 in Fig. 8, it can execute the control process [1] in Fig. 5 to request the user to take a predetermined action.
[0083] Other Operations of the Control Unit As another example of operation, when the surrounding sound prevents the output sound from being heard, the control unit 125 may instruct the sound output unit to insert a fixed output sound signal representing a word that attracts the user's attention at the beginning of the generated output sound signal and emit the signal. Specifically, the control unit 125 issues such an instruction, for example, when notifying the user by voice in step S105 of FIG. 3, or when restarting sound emission after changing the sound emission method in FIG. 8 or FIG. 9, etc. In this way, by inserting a word that attracts the user's attention at the beginning of the output sound signal, the user can be clearly made aware of the fact that the surrounding sound increases and it becomes difficult to hear the output sound that is being emitted, and the selective attention of the cocktail party effect has the effect of making the emitted sound easier to hear. For example, a word that attracts the user's attention can be a user's name that is registered in the device in advance.
[0084] Furthermore, as another operation example, when the ambient sound is in a state where the output sound cannot be heard, the control unit 125 may perform a switching process so that the output sound signal from inside the head-mounted information processing device is displayed as text on the display unit 119 instead of emitting the output sound. At this time, the control unit 125 performs text display going back to the point before the interruption, as in the case of resuming sound emission. In this way, the head-mounted information processing device cannot convey predetermined information to the user through the sense of hearing, but can convey it through the sense of sight.
[0085] <<Major Effects of the First Embodiment>> As described above, by using the head-mounted information processing device of the first embodiment, it is typically possible for the user to reliably hear a desired sound even when there is an unnecessary ambient sound for the user. Also, it is possible for the user to conveniently hear a desired sound even when there is an unnecessary ambient sound for the user.
[0086] In more detail, when a loud ambient sound is generated and reaches the ears, making it impossible to hear the output audio signal from inside the head-mounted information processing device, the sound emission is interrupted to prevent the user from missing the output audio. When the ambient sound becomes small enough to no longer interfere with the hearing of the output audio, the sound emission is resumed from the point before it was interrupted, allowing the user to easily hear the output audio without missing anything. In particular, it becomes possible to ensure that the user can hear the audio, such as the output audio from a voice assistant, which may be problematic if it is missed, without missing anything.
[0087] Even if the hearing of the ear is impaired for a while due to loud ambient noise, the user can re-start emitting sound after waiting for the time required for the hearing to recover, so that the user can reliably and conveniently hear the desired sound. Furthermore, by changing the sound emitting method such as changing the frequency characteristics or sound image of the output sound as necessary, or by requesting the user to take measures such as covering their ears, the user can reliably and conveniently hear the desired sound even in a situation where ambient noise is still present. Furthermore, by notifying the user by display, sound, or vibration when sound emission is interrupted or resumed, the user's convenience is improved.
[0088] Although the sound emission operation is controlled in accordance with the surrounding sound here, the user may not want such control of the sound emission operation depending on the content of the head-mounted information processing device. Therefore, for example, the user may set whether or not to control the sound emission operation for each piece of content in the head-mounted information processing device main body 100. The head-mounted information processing device main body 100 may select whether or not to control the sound emission operation for each piece of content according to this user setting.
[0089] (Embodiment 2) <<Details of the control unit>> In the second embodiment, the control unit 125 controls the sound emission operation to reflect the user's state (e.g., mental and physical state) or the state around the user (e.g., the occurrence of a dangerous situation), unlike the state of the ambient sound described in the first embodiment. Schematically, in Fig. 2, the sensor device 151 in the head-mounted information processing device main body 100 and the sensor devices in each of the wearable terminals 200 and 300 detect the state of the user or the state around the user.
[0090] The user state determination unit 123 determines whether the state of the user or the state around the user is suitable for listening to the output sound based on the detection result of the sensor device. If the state is suitable for listening to the output sound based on the determination result of the user state determination unit 123, the control unit 125 causes the sound output unit (headphones 122, etc.) to emit sound, and if the state is not suitable for listening to the output sound, the control unit 125 instructs the sound output unit to stop emitting sound. Details of this will be described below.
[0091] Fig. 10 is a flow diagram showing an example of detailed processing contents of the control unit of Fig. 2 in the head-mounted information processing device according to the second embodiment of the present invention. Control unit 125 repeatedly executes the flow shown in Fig. 10 at a predetermined control period. In Fig. 10, control unit 125 uses user state determination unit 123 to detect the state of the user or the state of the user's surroundings with various sensor devices (step S501).
[0092] Next, the control unit 125 determines whether the audio output unit (headphones 122, etc.) is emitting output audio (step S502). If output audio is emitting audio in step S502, the control unit 125 uses the user state determination unit 123 to determine whether the state of the user or the state around the user is suitable for listening to the output audio, based on the detection result of step S501 (step S503). In the specification, a state in which the state of the user or the state around the user is not suitable for listening to the output audio is also referred to as an inappropriate user state.
[0093] If a user inappropriate state occurs in step S503, the control unit 125 instructs the audio output unit to suspend sound emission (step S504). Furthermore, in step S504, the control unit 125 notifies the user that sound emission will be suspended by displaying a notice on the display unit 119, or by using tactile vibrations from the vibration generating units 124, 202, 302, or by using voice output from the audio output unit (headphones 122), and ends the process. On the other hand, if a user inappropriate state does not occur in step S503, the control unit 125 continues sound emission by the audio output unit and ends the process (step S505).
[0094] If the output sound is not being emitted in step S502, the control unit 125 determines whether or not sound emission by the sound output unit is paused (step S506). If sound emission is not paused in step S506, the control unit 125 ends the process. On the other hand, if sound emission is paused in step S506, the control unit 125 determines whether or not the user's unsuitable state has been resolved (step S507).
[0095] When the user inappropriate state is resolved in step S507 (i.e., when the state becomes suitable for listening to the output sound), the control unit 125 issues a resume instruction (RES) to instruct the sound output unit to resume sound emission, as in the case of the first embodiment described in FIG. 4. In response to the resume instruction (RES), the sound output unit resumes sound emission from the point before the interruption (step S508). Also, in step S508, the control unit 125 notifies the user that sound emission will be resumed by displaying a notice on the display unit 119, using tactile vibrations from the vibration generating units 124, 202, and 302, or using voice output from the sound output unit (headphones 122), and ends the process. In the control cycle thereafter, the sound emission operation continues in the flow of steps S501→S502→S503→S505 in FIG. 10.
[0096] <<Details of User Unsuitable Condition>> Fig. 11 is a diagram for explaining an example of a user inappropriate state in Fig. 10. Fig. 11 shows specific examples of situations in which it is determined that a user inappropriate state has occurred in steps S503 and S507 in Fig. 10, and seven situations (1) to (7) are shown here. Situation (1) is a situation in which it is considered that the user is gazing at virtual reality (VR) space information or augmented reality (AR) space information on the display unit 119.
[0097] Specifically, for example, in step S501, the user state determination unit 123 detects the position of the user's gaze based on the right eye gaze sensor 112 and the left eye gaze sensor 113, and further obtains the display positions of the virtual space information and the real space information from the virtual space information generation processing unit 141 and the camera 11 in Fig. 2. Then, the user state determination unit 123 determines whether the time ratio during which the position of the user's gaze matches the display positions of the virtual space information and the real space information is equal to or greater than a predetermined reference value. If the time ratio during which they match is equal to or greater than a predetermined reference value, the user state determination unit 123 determines that the user is gazing at the virtual space information or the real space information, and in steps S503 and S507, determines that the state is not suitable for listening to the sound to be output (in other words, a user inappropriate state has occurred).
[0098] Situation (2) is a situation in which the user is deemed to be performing rapid eye movement. Specifically, the user state determination unit 123 determines, for example, whether the number of times the user's gaze changes within a predetermined time period is equal to or greater than a predetermined reference value based on the detection results of the gaze sensors 112 and 113 in step S501. Generally, when a human being moves his / her gaze from one viewpoint to another distant viewpoint, a rapid rotation of the eyeballs called a saccade occurs, and here, the occurrence or non-occurrence of this saccade is determined. If the number of times the user's gaze changes is equal to or greater than a predetermined reference value, the user state determination unit 123 determines that a saccade has occurred, and determines in steps S503 and S507 that the state is not suitable for listening to the output sound.
[0099] Situation (3) is a situation in which the user is deemed to be absent-minded. Specifically, the user state determination unit 123 determines, for example, whether the number of times the user's gaze changes within a predetermined time is equal to or less than a predetermined reference value based on the detection results of the gaze sensors 112, 113 in step S501. If the number of times the user's gaze changes is equal to or less than the predetermined reference value, the user is deemed to be absent-minded, feeling sleepy and not clearly conscious. If the user is absent-minded, the user state determination unit 123 determines in steps S503 and S507 that the state is not suitable for listening to the output sound.
[0100] Situation (4) is a situation where the user's head is moving significantly. Specifically, the user state determination unit 123 determines, for example, whether the user's head has moved at a predetermined reference speed or more and by a predetermined reference amount or more based on the detection results of the acceleration sensor 114, the gyro sensor (angular velocity sensor) 115, or the geomagnetic sensor 116 in step S501. When the user's head is moving significantly, for example, some abnormality has occurred in the external environment around the user, such as the user being in danger, and the user's visual or auditory attention may be directed thereto. When the user's head is moving significantly, the user state determination unit 123 determines in steps S503 and S507 that the state is not suitable for listening to the output sound so as not to disturb the user's concentration.
[0101] Situation (5) is a situation where the user's physical condition suddenly changes. Specifically, the user state determination unit 123 determines, for example, whether the rate of change over time (e.g., the rate of increase) of the user's heart rate or blood pressure is equal to or greater than a predetermined reference value based on the detection results of the heart rate sensor 201 and the blood pressure sensor 301 in step S501. If the heart rate or blood pressure increases suddenly, a sudden abnormality occurs in the user's mental and physical condition, and the user may not be able to pay attention to his or her hearing. If a change in the user's physical condition occurs, the user state determination unit 123 determines in steps S503 and S507 that the state is not suitable for listening to the sound to be output.
[0102] 1 and 2, the heart rate is detected by the heart rate sensor 201 of the chest-worn wearable device 200 placed in close contact with the chest where it can be detected most accurately, and the blood pressure is detected by the blood pressure sensor 301 of the wristband-type wearable device 300 placed in close contact with the arm where it can be detected easily. However, the heart rate and blood pressure are not limited to this, and may be acquired by communication from, for example, an AI (Artificial Intelligence) watch or the like. The acquired information is not limited to the heart rate and blood pressure, and may be any biological information that can detect the user's mental and physical condition.
[0103] Situation (6) is a situation in which the user is talking to another person. Specifically, in step S501, the user state determination unit 123 detects a voice by the first peripheral sound microphone 131 and the speech sound microphone 121, and detects eye movement by the gaze sensors 112 and 113. Based on the detected voice and eye movement, the user state determination unit 123 determines where the user's attention is and determines whether the user is facing another person or talking on the phone. At this time, the user state determination unit 123 may recognize the presence of another person based on the imaging result of the camera 11, or, if the head-mounted information processing device main body 100 is equipped with a telephone function, may recognize a telephone conversation based on that function. If the user is talking to another person, the user state determination unit 123 determines in steps S503 and S507 that the state is not suitable for listening to the output sound.
[0104] Situation (7) is a situation where an approaching object is present around the user. Specifically, in step S501, the user state determination unit 123 detects the distance and relative speed of an object such as a car, person, or animal that is present around the user based on the detection result of the surrounding object detection sensor 118, thereby detecting whether an approaching object is present within a certain range around the user.
[0105] When an approaching object is present within a certain range around the user, the user state determination unit 123 determines that the user's visual or auditory attention is directed to the object, and determines in steps S503 and S507 that the state is not suitable for listening to the output sound. Also, in steps S503 and S507, if the user state determination unit 123 determines that the state is not suitable for listening to the output sound based on the surrounding object detection sensor 118, the control unit 125 instructs the audio output unit to emit a warning sound indicating danger to the user.
[0106] The user state determination unit 123 may, for example, define the range for detecting an approaching object into two stages: a danger range around the user and a warning range outside the danger range. When an approaching object is in the warning range, the control unit 125 controls the output sound to be emitted from a direction opposite to the direction in which the approaching object is present, and notifies the user of the sound emission operation by display, sound, or vibration. In this case, when an approaching object is in the warning range, the user can hear the output sound with reduced disturbing noise due to the sound from the approaching object. Conversely, the control unit 125 may control the output sound to be emitted from the direction in which the approaching object is present. In this case, the user can easily recognize the direction of the approaching object.
[0107] In this manner, the user state determination unit 123 determines whether the state of the user or the state around the user is suitable for listening to the output sound (in other words, whether an inappropriate state for the user has occurred). Based on the result of this determination, if the state is suitable for listening to the output sound (if an inappropriate state for the user has not occurred), the control unit 125 causes the audio output unit to emit sound, and if the state is not suitable for listening to the output sound (if an inappropriate state for the user has occurred), the control unit 125 instructs the audio output unit to stop emitting sound.
[0108] As another example of an unsuitable state for the user, the user state determination unit 123 may determine whether an unsuitable state for the user has occurred using a temperature and humidity sensor 117 that detects the temperature and humidity around the user. Specifically, the user state determination unit 123 may determine that an unsuitable state for the user has occurred when the rate of change over time of the temperature and humidity is equal to or greater than a reference value. Also, the sensor device that detects the state around the user is not limited to the temperature and humidity sensor 117, and it is also possible to use an air pressure sensor that detects air pressure.
[0109] <<Major Effects of the Second Embodiment>> As described above, by using the head-mounted information processing device of the second embodiment, it is typically possible to allow the user to reliably hear the desired sound by reflecting the state of the user and the state of the user's surroundings. Also, it is possible to allow the user to conveniently hear the desired sound by reflecting the state of the user and the state of the user's surroundings.
[0110] More specifically, even if a sound emission operation is requested by the user, if the state of the user or the state around the user is not suitable for listening to the sound to be output, as shown in situations (1) to (7) in FIG. 11, the sound emission operation is controlled to be interrupted. This allows the user to reliably hear the sound to be output without missing anything. Furthermore, if the state of the user or the state around the user subsequently becomes suitable for listening to the sound to be output, the sound emission is resumed from the point before it was interrupted, as in the case of embodiment 1. This allows the user to reliably hear the sound to be output, and allows for easy listening.
[0111] Also, as in the case of the first embodiment, when the sound emission operation is interrupted or resumed, the user is notified of that fact, thereby improving usability for the user. Furthermore, with regard to situation (7), the sound emission operation is controlled according to the degree of the approaching object, making it easier for the user to hear the output sound. At this time, if the approaching object poses a danger to the user, the assistant sound other than attention calls and warnings is controlled not to be emitted, thereby ensuring the safety of the user and allowing the user to reliably hear the desired sound.
[0112] For example, as described in situation (3), in a situation where the user is deemed to be absent-minded, the head-mounted information processing device main body 100 may perform processing such as shining light into the user's eyes or displaying two or more blinking lights within the user's field of vision. This puts the user in an awake state in which the brain is awake and the user is clearly conscious, and the control unit 125 can detect this awake state via the user state determination unit 123 and resume the sound emission operation.
[0113] (Embodiment 3) Overview of head-mounted information processing device Fig. 12 is a block diagram showing a schematic configuration example of a head-mounted information processing device according to embodiment 3 of the present invention. Compared to the configuration example shown in Fig. 2, the head-mounted information processing device shown in Fig. 12 is a configuration example in which the virtual space information generation processing unit 141 in the head-mounted information processing device main body 100 of Fig. 2 is separated outside the head-mounted information processing device main body 100 and made into a separate device.
[0114] 12, a virtual space information generation server 500 generates virtual space information and the like, and transmits and receives the generated virtual space information and the like to and from a head-mounted information processing device main body 100 via an external network 600. The head-mounted information processing device main body 100 includes a transmitting / receiving antenna 1201 and a communication unit 1202, and transmits and receives virtual space information and the like from the virtual space information generation server 500.
[0115] The virtual space information generation server 500 includes a virtual space information generation processing unit 501, a memory 502, a control unit 503, a communication unit 504, and a transmission / reception antenna 505, which are connected to each other via a bus 506. The virtual space information generation processing unit 501 generates virtual space information that expresses a virtual space different from real space with images and sounds. The memory 502 is a flash memory or the like, and stores the virtual space information generated by the virtual space information generation processing unit 501 and various programs used by the control unit 503. The communication unit 504 is a communication interface that communicates with the head-mounted information processing device main body 100 via the transmission / reception antenna 505 and an external network 600.
[0116] As described above, the amount of information in the virtual space can be increased by generating virtual space information using a server device separate from the head-mounted information processing device main body 100 and having the head-mounted information processing device main body 100 acquire the virtual space information via communication. Also, the hardware and software resources of the head-mounted information processing device main body 100 can be reduced.
[0117] 2 and 12, the heart rate sensor 201 in the chest-worn wearable terminal 200, the blood pressure sensor 301 in the wristband-type wearable terminal 300, and the input operation unit 401 in the input controller 400 may be incorporated into the head-mounted information processing device main body 100. The heart rate sensor 201 can detect the heart rate when worn closely on the head, and the blood pressure sensor 301 can detect the blood pressure value in the head artery just below the scalp when worn closely on the head.
[0118] The input operation unit 401 may be installed in a position within the head-mounted information processing device main body 100 where the user can easily perform input operations. Alternatively, the user may vocalize a voice indicating an input operation, and the voice may be collected by the vocalization microphone 121 to capture the input operation information. Alternatively, an input operation screen may be displayed on the display unit 119, and the input operation information may be captured based on the position on the input operation screen to which the line of sight detected by each of the line-of-sight sensors 112 and 113 is directed, or a pointer may be displayed on the input operation screen and designated by a hand movement or the like to capture the input operation information. By using vocalization or display for input operations, it is possible to further improve usability.
[0119] The present invention is not limited to the above-described embodiment, but includes various modified examples. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and is not necessarily limited to an embodiment having all of the configurations described. It is also possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. It is also possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.
[0120] In addition, the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. In addition, the above-mentioned configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as the program, table, file, etc. that realizes each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.
[0121] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are connected to each other. [Explanation of symbols]
[0122] 100: Head-mounted information processing device main body (head-mounted display), 111: Camera, 112: Right eye gaze sensor, 113: Left eye gaze sensor, 114: Acceleration sensor, 115: Gyro sensor, 116: Geomagnetic sensor, 117: Temperature and humidity sensor, 118: Peripheral object detection sensor, 119: Display unit, 121: Speech microphone, 122: Headphones, 123: User state discrimination unit, 124, 202, 302: Vibration generation unit, 125, 503: Control unit, 128, 502: Memory, 131: First peripheral sound microphone, 132: Second peripheral sound microphone, 133: Third peripheral sound microphone, 134: Soundproof ear cover detection unit, 141: Virtual space information generation unit processing unit, 142, 203, 303, 402: short-range wireless communication unit, 151: sensor device, 200: chest-worn wearable terminal, 201: heart rate sensor, 300: wristband-type wearable terminal, 301: blood pressure sensor, 400: input controller, 401: input operation unit, 500: virtual space information generation server, 501: virtual space information generation processing unit, 600: external network, 601: voice input unit, 602: voice output unit 602, 610: ambient sound discrimination unit, 611: voice input processing unit, 615: threshold generation unit, 616: threshold table, 617: comparator, 620: voice assistant processing unit, 621: voice output processing unit, 622: voice dictionary.
Claims
1. A method for controlling a head-mounted information processing device that has a function of viewing and listening to images or sounds, is worn on a head, and includes an audio input unit, an audio output unit, and a control unit, comprising: a step of the audio input unit collecting ambient sound generated outside the head-mounted information processing device and entering the ears, and converting the collected sound into an input audio signal; a step in which the audio output unit generates an output audio signal, converts the generated output audio signal into an output audio, and outputs the output audio signal to a user; The control unit: determining whether or not the ambient sound is in a state where the output sound is prevented from being heard based on a volume level of the input sound signal from the sound input unit and a volume level of the output sound signal from the sound output unit; A step of controlling a sound output operation of the audio output unit based on a result of the determination; When the ambient sound does not interfere with the listening of the output sound, causing the sound output unit to emit sound; instructing the audio output unit to stop emitting sound when the ambient sound interferes with the listening of the output sound; generating a threshold level responsive to a volume level of the output audio signal; a step of determining whether to instruct the audio output unit to stop emitting sound by comparing a volume level of the input audio signal with the threshold level; A method for controlling a head-mounted information processing device having the same.
2. A method for controlling a head-mounted information processing device that has a function of viewing and listening to images or sounds, is worn on a head, and includes an audio input unit, an audio output unit, and a control unit, comprising: a step of the audio input unit collecting ambient sound generated outside the head-mounted information processing device and entering the ears, and converting the collected sound into an input audio signal; a step in which the audio output unit generates an output audio signal, converts the generated output audio signal into an output audio, and outputs the output audio signal to a user; The control unit: determining whether or not the ambient sound is in a state where the output sound is prevented from being heard based on a volume level of the input sound signal from the sound input unit and a volume level of the output sound signal from the sound output unit; A step of controlling a sound output operation of the audio output unit based on a result of the determination; When the ambient sound does not interfere with the listening of the output sound, causing the sound output unit to emit sound; instructing the audio output unit to stop emitting sound when the ambient sound interferes with the listening of the output sound; having After instructing the audio output unit to stop emitting sound, when the ambient sound becomes a state in which the output sound is not impeded from being heard, a step of the control unit instructing the audio output unit to resume sound output; a step of the audio output unit resuming sound output from a point before the sound output was interrupted in response to the instruction to resume; When the ambient sound becomes a state in which the output sound is not disturbed after the control unit instructs the audio output unit to suspend sound emission, the control unit instructs the audio output unit to resume sound emission after a predetermined period of time corresponding to a volume level of the input audio signal immediately before the ambient sound becomes a state in which the output sound is not disturbed has elapsed; A method for controlling a head-mounted information processing device having the same.
3. A method for controlling a head-mounted information processing device having a function of viewing and listening to images or sounds, worn on a head, and including an audio input unit, an audio output unit, a display unit, and a control unit, comprising: a step of the audio input unit collecting ambient sound generated outside the head-mounted information processing device and entering the ears, and converting the collected sound into an input audio signal; a step in which the audio output unit generates an output audio signal, converts the generated output audio signal into an output audio, and outputs the output audio signal to a user; The control unit: determining whether or not the ambient sound is in a state where the output sound is prevented from being heard based on a volume level of the input sound signal from the sound input unit and a volume level of the output sound signal from the sound output unit; A step of controlling a sound output operation of the audio output unit based on a result of the determination; When the ambient sound does not interfere with the listening of the output sound, causing the sound output unit to emit sound; instructing the audio output unit to stop emitting sound when the ambient sound interferes with the listening of the output sound; When the ambient sound interferes with the listening of the output sound, issuing an instruction to the user to cover his / her ears using the display unit; A method for controlling a head-mounted information processing device having the same.
4. A method for controlling a head-mounted information processing device having a function of viewing and listening to images or sounds, worn on a head, and including an audio input unit, an audio output unit, a display unit, and a control unit, comprising: a step of the audio input unit collecting ambient sound generated outside the head-mounted information processing device and entering the ears, and converting the collected sound into an input audio signal; a step in which the audio output unit generates an output audio signal, converts the generated output audio signal into an output audio, and outputs the output audio signal to a user; The control unit: determining whether or not the ambient sound is in a state where the output sound is prevented from being heard based on a volume level of the input sound signal from the sound input unit and a volume level of the output sound signal from the sound output unit; Controlling the sound output operation of the audio output unit based on the result of the determination; When the ambient sound does not interfere with the listening of the output sound, causing the sound output unit to emit sound; instructing the audio output unit to stop emitting sound when the ambient sound interferes with the listening of the output sound; having The audio input unit collects the left and right ambient sounds entering the left and right ears and converts them into the left and right input audio signals, respectively; The control unit: a step of determining whether or not there is a bias in the volume levels of the left and right input audio signals that is equal to or greater than a predetermined reference value after instructing the audio output unit to stop emitting sound; If there is a bias, instructing the user to change the direction of the head using the display unit; A method for controlling a head-mounted information processing device having the same.
5. A method for controlling a head-mounted information processing device that has a function of viewing and listening to images or sounds, is worn on a head, and includes an audio input unit, an audio output unit, and a control unit, comprising: a step of the audio input unit collecting ambient sounds generated outside the head-mounted information processing device and entering the ears, and converting the collected sounds into an input audio signal; a step in which the audio output unit generates an output audio signal, converts the generated output audio signal into an output audio, and outputs the output audio signal to a user; The control unit: determining whether or not the ambient sound is in a state where the output sound is prevented from being heard based on a volume level of the input sound signal from the sound input unit and a volume level of the output sound signal from the sound output unit; A step of controlling a sound output operation of the audio output unit based on a result of the determination; the audio output unit having a plurality of frequency characteristics applicable to the output audio signal; The control unit: analyzing a frequency characteristic of the input audio signal when the ambient sound interferes with the listening of the output audio; issuing an instruction to the audio output unit to select, from among the plurality of frequency characteristics applicable to the output audio signal, a frequency characteristic whose similarity to the frequency characteristic of the input audio signal is lower than a predetermined reference value; A method for controlling a head-mounted information processing device having the same.
6. A method for controlling a head-mounted information processing device that has a function of viewing and listening to images or sounds, is worn on a head, and includes an audio input unit, an audio output unit, and a control unit, comprising: a step of the audio input unit collecting ambient sound generated outside the head-mounted information processing device and entering the ears, and converting the collected sound into an input audio signal; a step in which the audio output unit generates an output audio signal, converts the generated output audio signal into an output audio, and outputs the output audio signal to a user; The control unit: determining whether or not the ambient sound is in a state where the output sound is prevented from being heard based on a volume level of the input sound signal from the sound input unit and a volume level of the output sound signal from the sound output unit; A step of controlling a sound output operation of the audio output unit based on a result of the determination; When the ambient sound is in a state where the output sound is prevented from being heard, instructing the audio output unit to insert a fixed output sound signal representing a word that attracts the user's attention at the beginning of the generated output sound signal and emit the fixed output sound signal; A method for controlling a head-mounted information processing device having the same.
7. A method for controlling a head-mounted information processing device that has a function of viewing and listening to images or sounds, is worn on a head, and includes an audio input unit, an audio output unit, and a control unit, comprising: a step of the audio input unit collecting ambient sound generated outside the head-mounted information processing device and entering the ears, and converting the collected sound into an input audio signal; a step in which the audio output unit generates an output audio signal, converts the generated output audio signal into an output audio, and outputs the output audio signal to a user; The control unit: determining whether or not the ambient sound is in a state where the output sound is prevented from being heard based on a volume level of the input sound signal from the sound input unit and a volume level of the output sound signal from the sound output unit; A step of controlling a sound output operation of the audio output unit based on a result of the determination; When the ambient sound does not interfere with the listening of the output sound, causing the sound output unit to emit sound; instructing the audio output unit to stop emitting sound when the ambient sound interferes with the listening of the output sound; The audio input unit collects the left and right ambient sounds entering the left and right ears and converts them into the left and right input audio signals, respectively; a step in which the audio output unit converts the generated left and right output audio signals into the left and right output audio signals, respectively, and outputs the sounds toward the user; The control unit: a step of instructing the audio output unit to stop emitting sound, and then determining a position of a sound source of the ambient sound based on the left and right input audio signals; instructing the audio output unit to generate the left and right output audio signals so that a position of a sound image of the output audio signal is away from a position of a sound source of the ambient sound by a predetermined reference value or more; A method for controlling a head-mounted information processing device having the same.
8. A method for controlling a head-mounted information processing device having a function of viewing and listening to images or sounds, worn on a head, and including an audio output unit, a sensor device, a user state determination unit, and a control unit, comprising: a step in which the audio output unit generates an output audio signal, converts the generated output audio signal into an output audio, and outputs the output audio signal to a user; The sensor device detects a state of the user; a step of determining by the user state determination unit whether or not the state of the user is suitable for listening to the output sound based on a detection result of the sensor device; The control unit: when the state is suitable for listening to the output sound based on a determination result of the user state determination unit, causing the sound output unit to emit sound; a step of instructing the audio output unit to stop emitting sound when the state is not suitable for listening to the output sound; A method for controlling a head-mounted information processing device having the same.
9. 9. The method for controlling a head-mounted information processing device according to claim 8, further comprising: a step of instructing the audio output unit to resume sound output when a state suitable for listening to the output sound is reached after the control unit instructs the audio output unit to suspend sound output; a step of the audio output unit resuming sound output from a point before sound output was interrupted in response to the instruction to resume; A method for controlling a head-mounted information processing device having the same.
10. 10. The method for controlling a head-mounted information processing device according to claim 9, further comprising: a step of the voice output unit resuming voice output from the beginning of the interrupted sentence; A method for controlling a head-mounted information processing device having the same.
11. A method for controlling a head-mounted information processing device having a display unit according to any one of claims 8 to 10, comprising: a step of displaying predetermined information including virtual space information or real space information to the user by the display unit; A step in which the sensor device, which is a gaze sensor, detects the gaze of the user; a step of determining, based on the detection result, that the user state determination unit is in a state that is not suitable for listening to the output sound when the user state determination unit determines that the user is gazing at the virtual space information or the real space information on the display unit, or that the user is performing rapid eye movement, or that the user is absentminded; A method for controlling a head-mounted information processing device having the same.
12. The method for controlling a head-mounted information processing device according to any one of claims 8 to 10, The sensor device, which is an angular velocity sensor or a geomagnetic sensor, detects a movement of the user's head; a step of determining, by the user state determination unit, that a state is not suitable for listening to the output sound when the user's head is significantly moving based on a detection result of the angular velocity sensor or the geomagnetic sensor; A method for controlling a head-mounted information processing device having the same.
13. The method for controlling a head-mounted information processing device according to any one of claims 8 to 10, The sensor device, which is a heart rate sensor or a blood pressure sensor for detecting blood pressure, detects the heart rate of the user; a step of determining, by the user state determination unit, that the user is in an unsuitable state for listening to the output sound when a rate of change in the user's heart rate or blood pressure over time is equal to or greater than a predetermined reference value based on a detection result of the heart rate sensor or the blood pressure sensor; A method for controlling a head-mounted information processing device having the same.
14. The method for controlling a head-mounted information processing device according to any one of claims 8 to 10, a step in which the sensor device, which is a speech microphone, collects speech from the user and converts it into a speech signal; The user state determination unit, determining whether the user is talking to another person based on the voice signal from the speech microphone; determining that the state is not suitable for listening to the output voice when the user is having a conversation with another person; A method for controlling a head-mounted information processing device having the same.
15. The method for controlling a head-mounted information processing device according to any one of claims 8 to 10, A step in which the sensor device, which is a surrounding object detection sensor, detects an object present in the vicinity of the user; a step of determining by the user state determination unit that a state is not suitable for listening to the output sound when an approaching object is present within a certain range around the user based on a detection result of the peripheral object detection sensor; a step of instructing the audio output unit to emit a warning sound indicating a danger to the user when the user state determination unit determines that the user state is not suitable for listening to the output sound based on a detection result of the peripheral object detection sensor; A method for controlling a head-mounted information processing device having the same.
16. A method for controlling a head-mounted information processing device according to any one of claims 1 to 15, A step in which the control unit controls a sound emission operation for the output voice signal from a voice assistant that responds to a request of the user through a voice dialogue with the user; A method for controlling a head-mounted information processing device having the same.
Citation Information
Patent Citations
Sound reproducing apparatus
JP2006295669A
Portable terminal and portable terminal system
JP2015211267A
Information processing program, information processing method and program
JP2017069687A
