Signal processing device, audio output device, and signal processing method

Abstract

To determine masked ambient sounds and content sounds and takes appropriate action.SOLUTION: A signal processing device includes a masking determination unit that determines a masking state between the content sound and the ambient sound using multiple channels of content sound data output from multiple audio output drivers arranged in the audio output device, ambient sound data obtained by multiple microphones arranged in an audio output device for collecting ambient sound, and information on the direction from which the ambient sound comes, and also includes an audio processing control unit that controls audio processing according to a result of the masking state determination by the masking determination unit.SELECTED DRAWING: Figure 29

Description

[Technical field]

[0001] The present technology relates to a signal processing device, an audio output device, and a signal processing method, and in particular to a technology suitable for the field of stereophonic sound reproduction equipped with a plurality of audio output drivers. [Background technology]

[0002] For example, technologies for stereophonic sound reproduction have been developed as 3D (three dimensions) audio, 360-degree audio, etc. Headphones used for stereophonic sound reproduction have multiple channel audio output drivers (driver units) in each of the left and right ear units, allowing the user to perceive content audio from various directions. Some such headphones also have multiple microphones for noise cancellation processing, collecting ambient sounds from various directions. One possible use case for such headphones, so-called multi-microphone, multi-driver headphones, is to listen to content such as 3D audio in an environment where there is ambient noise. The following Patent Document 1 discloses a technique for transmitting stereoscopic (3D) audio. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2021-152677 A Summary of the Invention [Problem to be solved by the invention]

[0004] Here, when considering two types of sound, content sound and ambient sound, consider the case where one sound is masked by the other. In other words, there are cases where the ambient sound is masked by the content sound and is not recognized by the user, and cases where the ambient sound masks part of the content sound components. For example, when performing noise cancellation processing on the ambient sound as noise, there are cases where the remaining components of the content sound are masked part of the content sound components.

[0005] It is believed that by determining the state of masking in this way, more efficient processing and processing that is desirable for the user can be realized. Therefore, the present technology proposes a technology that enables processing appropriate for the content sound and ambient sound situations to be performed. [Means for solving the problem]

[0006] The signal processing device according to the present technology includes a masking determination unit that determines a masking state of the content sound and the ambient sound using multiple channels of content sound data output from multiple audio output drivers arranged in an audio output device, ambient sound data obtained by multiple microphones arranged in the audio output device for collecting ambient sound, and information on the direction from which the ambient sound comes, and an audio processing control unit that performs control related to audio processing in accordance with the masking state determination result by the masking determination unit. Assume that a plurality of audio output drivers are provided in the housing of an audio output device such as headphones to play back stereophonic sound, and a plurality of microphones are arranged to collect ambient sound. In this case, a masking state is determined for content audio data and ambient audio data, and audio processing control is performed according to the masking state. [Brief description of the drawings]

[0007] [Figure 1] 1 is a block diagram of a headphone according to a first embodiment of the present technology. [Diagram 2] 2 is a schematic explanatory diagram of a microphone and an acoustic output driver in the headphones according to the embodiment. FIG. [Diagram 3] 3A and 3B are explanatory diagrams of sounds heard by a person wearing the headphones according to the embodiment. [Figure 4] FIG. 4 is an explanatory diagram of the relationship between the position of an acoustic output driver and noise cancellation characteristics. [Diagram 5] FIG. 1 is an explanatory diagram of the minimum audibility limit and simultaneous masking. [Figure 6] FIG. 13 is an explanatory diagram of masking when noise is added. [Figure 7] FIG. 13 is an explanatory diagram of masking when noise is added. [Figure 8] FIG. 11 is an explanatory diagram of masking when residual noise remains; [Figure 9] FIG. 11 is an explanatory diagram of masking when residual noise remains; [Figure 10] FIG. 1 is an explanatory diagram of spatial masking. [Figure 11] 11 is an explanatory diagram of a case where the arrival directions of noise and content sound are the same. FIG. [Figure 12] 11 is an explanatory diagram of masking when noise and content sound come from the same direction. FIG. [Figure 13] 11 is an explanatory diagram of a state in which noise cancellation processing is performed when the arrival directions of noise and content sound coincide with each other. FIG. [Figure 14] 11 is an explanatory diagram of masking in a state where noise cancellation processing is added when the arrival directions of noise and content sound coincide with each other. FIG. [Figure 15] 11 is an explanatory diagram of masking when noise and content sound come from the same direction. FIG. [Figure 16] 11 is an explanatory diagram of a case where noise and content sound arrive from different directions. FIG. [Figure 17] 11 is an explanatory diagram of a state in which noise cancellation processing is applied when noise and content sound arrive from different directions. FIG. [Figure 18] 11A and 11B are explanatory diagrams of masking with noise cancellation processing added when noise and content sound arrive from different directions. [Figure 19] 11 is an explanatory diagram of a case where the arrival direction of noise is included in the arrival direction of content sound. FIG. [Figure 20] FIG. 11 is an explanatory diagram of different masking states for each direction of arrival. [Figure 21] 11 is an explanatory diagram of a case where noise cancellation processing is performed in a masking state that differs for each direction of arrival. FIG. [Figure 22] FIG. 11 is an explanatory diagram of a state in which noise cancellation processing has been performed. [Figure 23] 11 is an explanatory diagram of a case where noise comes from a direction other than the direction from which the content sound comes. FIG. [Figure 24] FIG. 11 is an explanatory diagram of different masking states for each direction of arrival. [Diagram 25] 11 is an explanatory diagram of a case where noise cancellation processing is performed in a masking state that differs for each direction of arrival. FIG. [Figure 26] FIG. 11 is an explanatory diagram of a state in which noise cancellation processing has been performed. [Figure 27] 13 is a flowchart of a process example of a determination unit according to the embodiment. [Figure 28] 13 is a flowchart of a process example of a determination unit according to the embodiment. [Figure 29] FIG. 11 is a block diagram of a headphone according to a second embodiment. [Diagram 30] 11A and 11B are explanatory diagrams of examples of messages displayed on the host device to notify users of ambient sound; [Diagram 31] 11A and 11B are explanatory diagrams of examples of the display of ambient sound and arrival direction notification on the host device. [Diagram 32] 11A and 11B are explanatory diagrams of examples of the display of ambient sound and arrival direction notification on the host device. [Diagram 33] FIG. 13 is an explanatory diagram of a display example including a masking state in the host device. [Diagram 34] FIG. 13 is an explanatory diagram of a display example including a masking state in the host device. [Diagram 35] FIG. 13 is an explanatory diagram of a display example including a masking state in the host device. [Diagram 36] 13 is a flowchart of a process example of a determination unit according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] The embodiments will be described below in the following order. <1. First embodiment> <2. Second embodiment> <3. Summary and examples of modifications>

[0009] <1. First embodiment> In the embodiment of the present disclosure, a headphone equipped with multiple microphones and multiple drivers is given as an example of an audio output device, and a case where content including stereophonic data such as 3D audio is listened to using the headphone will be described as an example. Note that the audio output driver equipped in the headphone will hereinafter also be referred to simply as a "driver."

[0010] First, in the first embodiment, a process for dealing with an increase in the amount of content audio data transmitted in order to perform stereophonic sound reproduction will be described.

[0011] In the case of 3D sound playback, the number of sound sources has increased dramatically compared to content that plays conventional 2-channel stereo sound, and the transmission bit rate of the content sound data has ballooned.

[0012] On the other hand, in addition to the content sound, there is also ambient sound (noise) that is emitted from the surroundings and reaches the eardrum. In this case, by utilizing a phenomenon called the masking effect, it is possible to reduce the transmission bit rate of the content sound data, improve the signal-to-noise ratio (S / N) of the content sound data, improve the sense of separation of sounds, improve the noise cancellation (hereafter, noise cancellation will be abbreviated as "NC") effect, and even extend the battery life of headphones by reducing power consumption.

[0013] There is a phenomenon called the masking effect, where a certain sound is disturbed by another sound and becomes inaudible. When there are two sound sources, the masking effect can occur, where one signal blocks the other. The side that masks the other is called the "masker" and the side that is masked is called the "maskie."

[0014] In this embodiment, this masking effect is utilized to analyze, when stereophonic content is viewed in a noisy environment, which signal will be dominant (masker) based on the noise, the stereophonic content sound data, and the direction from which the noise is coming, and to what extent it will degrade or drown out the other signal (masker).

[0015] If the analysis result indicates that noise is a masker, the quantization bits of the content sound data for stereophonic reproduction are reduced, thereby achieving a reduction in the transmission bit rate.

[0016] Furthermore, when listening to content sound in a noisy environment, the NC function is turned on. In this case, the NC effect and the quality of the content sound after NC processing, such as the S / N ratio, depend on the direction from which the noise comes. Therefore, by determining which driver in the headphones should be used to cancel noise, or which driver does not require noise canceling, and changing the settings, a more comfortable S / N ratio for the reproduced sound, sound separation, and improved NC effect can be achieved.

[0017] FIG. 1 shows an example of the configuration of headphones 1 according to the first embodiment. The headphones 1 receive content sound data CT as stereophonic data from the host device 100, and output a reproduced sound based on the content sound data CT. Note that while the host device 100 is assumed to be a device separate from the headphones 1, it may also be a device section provided inside the headphones 1. For example, it may be an audio streaming control section in the headphones 1. In other words, the host device 100 in the embodiment may be in any form as long as it is a device or circuit section that serves as a source of content audio data CT to be played back by the headphones 1, and it does not matter whether it is integrated with or separate from the headphones 1.

[0018] The headphones 1 are provided with N drivers 2 (2A, 2B,...2(N)) that output, for example, N-channel stereophonic sound. The drivers 2A, 2B,...2(N) are disposed in different positions corresponding to the respective channels within the left and right housings of the headphones 1. For example, as shown in Fig. 2, drivers 2A, 2B...2(N) are provided in a housing 10 so as to emit sound toward a user's ear 200. As a result, the channels of the content sound that reach an eardrum 201 come from different directions.

[0019] For the sake of simplicity, the housing 10 on one side of the left and right ears of the user will be illustrated and described, but the other housing is similar. N channel (N pieces) drivers are provided in each of the left and right housings 10. Fig. 1 also shows a configuration corresponding to one ear 200, but the configuration on the other side is similar.

[0020] As shown in Fig. 1 and Fig. 2, M microphones 3 (3A, 3B, ... 3(M)) are provided at different positions facing the outside of the housing 10. This allows M channels of ambient sound to be collected. The microphones 3 are arranged at appropriate positions for performing NC processing, for example, by the FF (feedforward) method.

[0021] It should be noted that FIG. 2 is merely a schematic diagram, and the driver 2 and the microphone 3 are not necessarily aligned in the cross-sectional direction of the housing 10 as shown in the figure. For example, the multiple drivers 2 are arranged at various locations on the inner surface side of the housing 10, and the multiple microphones 3 are arranged at various locations on the outer surface side of the housing 10.

[0022] The host device 100 for the headphones 1 is a device that is a source of content sound data CT, and is envisioned to be, for example, a smartphone, an HMD (head mounted display), a game device, a tablet, a personal computer, or any other type of device.

[0023] The host device 100, for example, has a display unit to display the content video, and transmits the content audio data CT to the headphones 1. This allows the user to view the content in the form of video and audio. In this case, the content audio data CT is N-channel stereophonic data to be played back by the headphones 1. In other words, it is assumed that the content audio data CT has been subjected to signal processing on the host device 100 side according to the number of channels and the position of the driver 2 of the headphones 1.

[0024] The headphones 1 that receive the content sound data CT are provided with the functions of a determination unit 4, an ambient sound type determination unit 5, an NC signal generation unit 6, and an output signal generation unit 7, for example, by one or more microprocessors.

[0025] The sound collected by the microphone 3 is converted into ambient sound data S1 as digital data, for example, at the output stage of the microphone 3, and is supplied to the determination unit 4, the ambient sound type determination unit 5, and the NC signal generation unit 6. The conversion into digital data may be performed at the input stage of a microprocessor constituting each of these units. For example, an A / D conversion terminal is provided in the microprocessor, so that the determination unit 4, the ambient sound type determination unit 5, and the NC signal generation unit 6 acquire the ambient sound data S1 as digital data.

[0026] The determination unit 4 has a function of acquiring the ambient sound data S1 from the microphone 3 and the content sound data CT to perform determination and control. Specifically, it has functions as a masking determination unit 4a and an audio processing control unit 4b.

[0027] The masking determination unit 4a performs a process of determining the masking state of the content sound and the ambient sound using N-channel content sound data CT output from the N-channel driver 2, M-channel ambient sound data S1 obtained by the microphone 3, and information on the direction from which the ambient sound is coming. For this reason, the masking determination unit 4a determines the direction from which the ambient sound (noise) comes from based on the M-channel ambient sound data S1. Moreover, the masking determination unit 4a calculates the frequency characteristics of the M-channel ambient sound data S1. Moreover, the masking determination unit 4a calculates the frequency characteristics of the N-channel content sound data CT. The masking determination unit 4a determines the masking state regarding the ambient sound and the content sound based on this information, as will be described in detail later. In the case of stereophonic content as in this embodiment, the sound data of each channel of the content sound data CT is sound data output from a different driver 2. In other words, the channel number corresponds to the arrangement position of each driver 2. Therefore, the channel information of the content sound data CT corresponds to information on the arrival direction of the content sound with respect to the user. Therefore, the level of each channel in the multi-channel content sound data CT can be used to determine the level of each arrival direction of the content sound. In other words, the content sound data CT itself includes information on the direction from which the content sound comes with respect to the user.

[0028] The sound processing control unit 4b controls sound processing according to the result of the masking state determination by the masking determination unit 4a. For example, the acoustic processing control unit 4b outputs a control signal to the NC signal generating unit 6 according to the masking state, and controls the NC operation. The control of the NC operation includes turning on / off the NC process, and selecting the driver 2 that outputs the NC signal. For example, it determines which driver 2 should output the NC signal S2 to maximize the NC effect in response to the incoming noise, and sends a command to the NC signal generating unit 6.

[0029] Furthermore, the acoustic processing control section 4b performs processing to transmit notification information SS to the host device 100, which is an external device, in accordance with the result of the masking state determination, for example. In the first embodiment, the notification information SS includes quantization bit information required for the content sound data CT. The quantization bit information includes, for example, information on the channel and band for reducing the number of quantization bits in the content sound data CT.

[0030] The ambient sound type determination unit 5 performs a process of determining the type of sound from the ambient sound data obtained by the microphone 3. Note that the type determination does not necessarily involve a specific determination of the sound, and may be a determination of whether or not the sound is to be treated as noise. The ambient sound type determination unit 5 is mainly required for the processing of the second embodiment, and may not be provided for the processing of the first embodiment. However, in the first embodiment, for example, processing may be performed in which the NC processing is turned on / off depending on the type of sound in the ambient sound data S1.

[0031] The NC signal generating unit 6 has a function of generating an NC signal S2 for canceling noise, which is the ambient sound data obtained by the microphone 3. For example, the NC signal S2 is generated by processing according to the FF-NC algorithm.

[0032] The output signal generating unit 7 has a function of generating a signal to be output from the driver 2. Basically, it generates a signal to drive the driver 2 of each channel based on the data of each channel of the content sound data CT. Note that it may also include equalizer processing etc. for the content sound data CT. Based on the input NC signal S2, the output signal generating unit 7 generates a signal to drive the driver 2 of a specified channel. Note that the channel of the driver 2 that outputs the NC signal may be specified by the acoustic processing control unit 4b as described above.

[0033] The masking determination process performed by the determination unit 4 will now be described. There are various types of masking effects. There are "simultaneous masking (frequency masking)" which blocks adjacent frequency sounds at the same time, and "temporal masking" which blocks sounds immediately before and after. In this disclosure, we mainly utilize the above-mentioned "simultaneous masking" and "spatial masking" that occurs due to differences in the direction from which sounds arrive. "Spatial masking" is when the masker and the maskee are coming from the same direction as the listener, and the masking effect is strongest when the directions are different. Figure 3 shows a schematic diagram of a situation in which the ambient noise AN and the content sound AC are coming from different directions.

[0034] In the first embodiment, the ambient sound is considered as noise AN. Considering noise AN and content sound AC output from driver 2 of headphone 1, the following cases can be considered from the perspective of the masker / masquee relationship and the direction from which each sound comes.

[0035] Noise AN masks the content sound AC (sound equivalent to all or part of the quantized bits) due to the simultaneous masking effect. · Content sound AC masks noise AN

[0036] Noise AN and content sound AC come from the same direction (masking effect: large) Noise AN and content sound AC come from different directions (masking effect: small)

[0037] Combining these cases will result in improved S / N ratios, better sound separation, improved NC effects, and reduced bit rates when listening to surround sound content.

[0038] Here, we will explain the following three points regarding the phenomena and effects assumed. Driver position and NC effect - Threshold and simultaneous masking Spatial Masking

[0039] First, we will discuss the driver position and the NC effect. Figure 4 shows the results of an experiment to show the driver selection and NC performance in a multi-driver headphone. In the experiment, a multi-driver headphone equipped with four drivers, No. 1 to No. 4, was used.

[0040] The horizontal axis of Figure 4 is frequency and the vertical axis is sound pressure level, with the downward direction on the vertical axis representing quieter conditions, or in other words, a higher NC effect. The measurement result C1 is the sound pressure at the eardrum when no headphones are worn. Measurement result C2 is the sound pressure at the eardrum when the headphones are simply worn (NC processing off). Measurement result C3 shows the case where an NC signal is output using driver No. 1, measurement result C4 shows the case where an NC signal is output using driver No. 2, measurement result C5 shows the case where an NC signal is output using driver No. 3, and measurement result C6 shows the case where an NC signal is output using driver No. 4.

[0041] The measurement results showed that C6 has a high NC effect in the low range, but the NC effect decreases above 4kHz. On the other hand, the measurement result C5 shows that the NC effect is particularly high in the 1 to 6 kHz range, although the low range is not as good as the measurement result C6. The measurement results C3 and C4 also have different characteristics from the other measurement results.

[0042] As shown above, from the measurement results C3, C4, C5, and C6 when NC processing was performed with different drivers, it was found that with regard to noise cancellation of multi-driver headphones, there are bands where the NC effect is high and low depending on the driver that outputs the NC signal. This is due to the placement of each driver and the characteristics up to the eardrum. In other words, this feature means that by selecting and combining the drivers that output the NC signal, you can choose which frequency band to focus on for noise cancellation processing, which can be said to be the advantage of noise canceling in multi-driver headphones.

[0043] Next, we will explain the threshold of hearing and simultaneous masking (frequency direction). The minimum audibility level indicates the maximum sound pressure level that humans can hear in each frequency band, and sounds below the minimum audibility level cannot be heard. Coincidence masking is a phenomenon in which, when a certain frequency component (F1) reaches the eardrum and is heard, a nearby frequency component (F2) is masked by the frequency component (F1) and cannot be heard unless the frequency component (F2) is relatively large compared to the frequency component (F1).

[0044] Considering these points, quantization error can be tolerated in the portion that corresponds to the masking accuracy and the portion below the minimum audible limit, and the transmission bit rate can be reduced.

[0045] This will be explained using Figure 5. The horizontal axis is frequency and the vertical axis is amplitude. In the figure, the minimum audible limit of 40 is indicated by a dashed line. The figure also shows frequency components 20, 21, 22, and 23 of sounds that occur at the same time. Masking levels 30, 31, 32, and 33 that are masked at the same time by the frequency components 20, 21, 22, and 23 are shown by dashed lines. Sounds lower than the masking level 30 are masked by the sound of frequency component 20. Masking level 30 spreads out like an umbrella to other frequencies, with the frequency of frequency component 20 as its peak. In other words, sounds close to frequency component 20 are easily masked even if they are relatively loud, and the further away the frequency is, the less likely it is to be masked. The masking level 31 of the frequency component 21, the masking level 32 of the frequency component 22, and the masking level 33 of the frequency component 23 also have a similar tendency.

[0046] In the example of FIG. 5, the level of frequency component 22 is below the minimum audible limit. The frequency component 21 has a masking level 30 lower than that of the frequency component 20 , that is, it is masked by the frequency component 20 . The frequency component 23 is masked by the frequency component 20 in a region 23M. The frequency components 20 are below the threshold of audibility in the region 20M. In other words, the black areas 20M, 23M and the entire frequency components 21, 22 in the figure are audio components that are masked or below the minimum audible limit, and it can be thought that highly accurate information is not required for these parts.

[0047] In this way, by determining the masking state of the generated sound, it is possible to determine the area where quantization accuracy is not required. Therefore, when the determination unit 4 transmits information on the channel and band for reducing the number of quantization bits in the content sound data CT based on the determination of the masking state to the host device 100, the host device 100 controls the quantization process of the content sound data CT, thereby reducing the transmission bit rate.

[0048] Below, various cases will be shown, similar to FIG. Fig. 6 shows a state when noise is added to Fig. 5. For example, assume that single-frequency noise 24 is added to an environment in which content sound having frequency components 20, 21, 22, and 23 as shown in Fig. 5 is being listened to. In the case of FIG. 5, the noise 24 exceeds the level at which it is simultaneously masked by the frequency component 20, and therefore deteriorates the content sound. In such a case, NC processing is performed with the aim of canceling the noise 24 to a level below that at which it is masked at the same time.

[0049] Fig. 7, like Fig. 6, shows a case where single-frequency noise 24 is added to an environment where a content sound having frequency components 20, 21, 22, and 23 as shown in Fig. 5 is being listened to. However, in the case of Fig. 7, the noise 24 is originally below the masking level 30 caused by the frequency component 20 of the content sound. In such a case, there is no need to perform NC processing.

[0050] The example of FIG. 8 is an example in which the residual noise 24 that remains after the NC process exceeds a masking level of 30 at which the noise 24 is simultaneously masked by the content sound. The noise 24, in this case the residual noise from the NC process, deteriorates the accuracy of the lower bits of the content sound, that is, the portion indicated by the area below the masking level 34 by the noise 24 in the frequency components 20, 21, 22, and 23. The quantization error in this region is buried in residual noise 24 caused by the NC process, and is therefore subject to a reduction in the transmission bit rate.

[0051] The example in Fig. 9 is a case where the residual noise 24 of the NC process is higher than the content sound level. The parts of the frequency components 20, 21, 22, and 23 that are below the masking level 34 caused by the noise 24 are buried in the noise 24. In other words, the allowable range of the quantization error that is the target of the bit rate reduction is wider than in the case of Fig. 8.

[0052] Next, spatial masking will be described. Figure 10 shows the results of the spatial masking experiment. From the user's (listener's) point of view, the front direction is defined as 0 degrees. Figure 10 shows the results of an experiment in which the masker was positioned at 0 degrees and moved in 30 degree increments. As shown by curve 41, it can be seen that the volume at which a musky becomes inaudible (masked) tends to vary depending on the angle. At the 90 degree position, compared to 0 degrees (where the masker and maskee directions are the same), it was found that masking was not possible unless the masker was approximately 6.4 dB louder, indicating that masking was difficult when there was an angle difference.

[0053] In this embodiment, the three phenomena and effects of the driver position and NC effect, the minimum audibility limit and simultaneous masking, and spatial masking are utilized. The first to fourth cases will be explained.

[0054] In each of Figs. 11 to 26, the content sound AC, noise AN, residual noise AN (NC), and NC sound ANC are each indicated by arrows. Each arrow indicates the direction from which a sound reaches the eardrum 201, and the thickness of the arrow indicates the loudness of the sound.

[0055] 11 to 26, in order to prioritize ease of viewing, the reference numerals for the drivers 2 (2A, 2B,...2(N)) and microphones 3 (3A, 3B,...3(M)) are omitted. It should be understood that the configurations of the drivers 2 and microphones 3 are the same as those in Figs. 2 and 3.

[0056] [Case 1: When the direction of arrival of noise and content sound is the same] FIG. 11 shows an example in which the noise AN and the content sound AC come from the same direction, and the noise AN acts as a masker. For the purpose of explaining the basic concept of the embodiment, the content sound is assumed to be one channel, the noise is assumed to be unidirectional, and the magnitude relationship is assumed to be the same across all bands.

[0057] FIG. 12 shows that noise AN masks content sound AC. By performing NC processing to prevent noise AN from masking the content sound AC, the content sound AC can be heard more clearly. Basically, it is effective to perform NC processing using the driver 2 located in the same direction as the noise arrival direction for NC effects. Therefore, the NC sound ANC is output by selecting a driver according to the direction from which the noise arrives. Figure 13 shows that the NC sound ANC is output from the driver 2 in the direction from which the noise arrives.

[0058] However, it is not possible to completely cancel the noise AN that reaches the eardrum when the NC sound reaches the eardrum. Fig. 14 shows that there is residual noise AN(NC). Note that Fig. 14 shows that the noise AN is cancelled from the position of the driver 2, but this is for convenience of illustration, and the noise that is actually cancelled is the position of the eardrum 201. The same is true for the illustration of residual noise AN(NC) in other figures. Regarding the residual noise AN(NC), the following cases (A) and (B) are assumed.

[0059] (A) When the residual noise AN(NC) is greater than the minimum audible level and the quantization noise of the content signal, the residual noise will bury the sound of the lower quantized bits of the content sound AC due to the simultaneous masking effect. Therefore, the quantization bits of the content sound data CT are reduced.

[0060] (B) When the residual noise AN (NC) disappears to the point where it is imperceptible, or when it is masked by the content sound AC, the allocated bits are maximized in the transmission of the content sound data CT. However, bit reduction is performed taking into account masking due to the frequency characteristics of the content sound data CT itself.

[0061] Next, a case will be described in which the noise AN and the content sound AC come from the same direction and the content sound AC acts as a masker. 15 shows that the noise AN is masked by the content sound AC. When the level of the content sound AC is high and the noise AN is masked in this way, the noise AN is not perceived by the listener, so there is no need to perform NC processing. In transmitting the content sound data CT, the allocated bits are maximized to achieve a level of detail that can be recognized by the user.

[0062] [Case 2: When noise and content sound come from different directions] FIG. 16 shows a case where noise AN and content sound AC arrive from different directions. As in the first case, the content sound is one channel, the noise is unidirectional, and the loudness relationship is the same across all bands.

[0063] First, consider the case where the noise AN level is large. Due to the spatial masking effect, the content sound AC and the noise AN are each easy to hear, and the noise AN stands out. Alternatively, if the difference in loudness is large enough, the noise AN completely masks the content sound AC.

[0064] Therefore, the content signal is made easier to hear by outputting the NC sound ANC using a driver 2 that is suited to the direction and characteristics of the noise AN. For example, as shown in Fig. 17, it is usually assumed that the NC sound ANC is output using a driver 2 that is located in the same direction as the direction from which the noise AN arrives.

[0065] However, due to the difference in the arrival direction, the residual noise AN(NC) shown in Fig. 18 tends to be easier to hear. In other words, due to the effect of spatial masking, the masking effect tends to be smaller. As a result, the degradation of the content sound AC due to the bit reduction of the content sound data CT is easily noticeable. Therefore, the determination is made by adding spatial masking to the simultaneous masking described in Figures 5 to 9. For example, it can be considered that the masking levels (30, 31, 32, 33, 34) shown in Figures 5 to 9 are shifted up or down depending on the angle difference in the arrival direction.

[0066] [Case 3: When the direction of arrival of noise (multiple) sounds is included in the direction of arrival of content sounds (multiple) sounds] An example of the third case is shown in Fig. 19. As the content sound AC, the content sounds AC1, AC2, and AC3 are output from the respective drivers 2. Also, as the noise AN, the noises AN1 and AN2 arrive. The noise AN1 and the content sound AC1 arrive from the same direction, and the noise AN2 and the content sound AC2 arrive from the same direction.

[0067] When the level of the noise AN is large, the NC process is executed in the same manner as in the first case. Also, the transmission bits of the content sound data CT are determined according to the residual noise AN (NC) resulting from the NC process.

[0068] When multiple noises AN and content sounds AC exist, the relationship between the masker and the maskee may differ for each direction, as shown in FIG. In the example of Fig. 20, the noise AN1 masks the content sound AC1, while the content sound AC2 masks the noise AN2. Since such a case is anticipated, it is determined whether the noise AN or the content sound AC is the masker for each arrival direction.

[0069] If the level of the noise AN is greater than the content sound AC, the NC process is performed on the noise AN. For example, when the noise AN1 masks the content sound AC1, the state in which the NC sound ANC is output using the driver 2 in the same direction is shown in FIG. Moreover, the residual noise AN(NC) resulting from the NC processing is shown in Fig. 22. Regarding the residual noise AN(NC), the processing described as (A) or (B) in the first case may be carried out.

[0070] On the other hand, when the content sound AC is so high in level that it masks the noise AN, there is no need to perform NC processing on the noise AN. For example, FIGS. 21 and 22 show that noise AN2 is masked by content sound AC2.

[0071] 19, noise AN from one arrival direction may mask content sound AC from another arrival direction, for example, when the level of noise AN1 is so high that it masks content sound AC2. NC processing is performed on the noise AN1. Fig. 21 shows the state in which the NC sound ANC is output from the driver 2 in the same direction as the noise AN1, and Fig. 22 shows the residual noise AN(NC).

[0072] At this time, the transmission bit allocation for each of the content sounds AC1, AC2, and AC3 is determined depending on how much the remaining noise AN(NC) interferes with the content sounds AC1, AC2, and AC3.

[0073] [Case 4: When the direction of arrival of the noise(s) is different from the direction of arrival of the content sound(s)] An example of the fourth case is shown in Fig. 23. As the content sound AC, the content sounds AC1 and AC2 are output from each driver 2. Also, as the noise AN, the noises AN1, AN2, and AN3 arrive. The noise AN1 and the content sound AC1 arrive from the same direction, and the noise AN2 and the content sound AC2 arrive from the same direction. The noise AN3 arrives from a direction other than the content sound AC.

[0074] In this case as well, as in the third case, NC processing is performed as necessary, and the transmission bits of the content sound data CT are determined in accordance with the masking effect that occurs. In the example of Fig. 23, the noise AN1 masks the content sound AC1 as shown in Fig. 24. Also, the content sound AC2 masks the noise AN2. Therefore, NC processing is performed on noises AN1 and AN3. Figure 25 shows how NC sound ANC1 is output from the driver 2 in the same direction as noise AN1, and NC sound ANC3 is output from the driver 2 in the same direction as noise AN3. Figure 26 shows the residual noises AN1 (NC) and AN3 (NC).

[0075] At this time, the transmission bit allocation for each of the content sounds AC1, AC2, and AC3 is determined depending on how much the remaining noises AN1(NC) and AN3(NC) interfere with the content sounds AC1, AC2, and AC3.

[0076] 1 (masking determination unit 4a and audio processing control unit 4b) controls the NC processing in the NC signal generation unit 6 based on the characteristics and direction of noise AN, and the relationship between the characteristics of content sound AC and the position of the driver 2, as in the first to fourth cases described above. That is, it controls whether or not to perform NC processing for each characteristic of noise AN. The determination unit 4 also determines how many quantization bits are sufficient to be allocated to the transmission of content sound data CT based on the minimum audibility limit and masking effect, and performs processing to transmit a request to the host device 100.

[0077] An example of the process performed by the determination unit 4 is shown in FIG. 27 and FIG. In addition, in Figures 27 and 28, "CN1" and "CN2" indicate connections in the flowchart.

[0078] During the period when the headphones 1 receive the content sound data CT and output the content sound AC from the driver 2, the determination unit 4 repeats the processes in Fig. 27 and Fig. 28. Step S101 is a determination as to whether the repeat loop has ended. For example, the processes in Fig. 27 and Fig. 28 are ended by turning off the power, changing the operation mode, etc.

[0079] During the execution of the loop process, the decision unit 4 analyzes the frequency characteristics and the direction of arrival of the ambient sound obtained by the microphone 3, that is, the noise AN, in step S102. In addition, in step S103, the determination unit 4 analyzes the frequency characteristics of the content sound data CT. Note that the determination unit 4 can determine the direction from which the content sound AC comes from for the user, that is, which component of the sound is output from which driver 2, based on the channel number of the content sound data CT.

[0080] In step S110, it is determined whether the process loop from step S111 to step S118 is to be continued or ended. The processes from step S111 to step S118 are performed for each direction of arrival. For example, the processes from step S111 to step S118 are performed for each of the first direction to the Nth direction in accordance with the number of channels of the driver 2. When these processes are completed for all directions of arrival, the loop ends.

[0081] In step S111, the determination unit 4 compares the noise AN from a certain direction with the minimum audible limit. The level of the noise AN also takes into account the amount of noise blocked by the housing of the headphones 1. If (noise AN)<(minimum audibility limit), that is, if all frequency components constituting the noise AN are below the minimum audibility limit, the decision unit 4 proceeds to step S115 and sets that NC processing is not required for the noise AN. Then, in step S116, the decision unit 4 sets the inaudible flag to ON for the noise from the arrival direction currently being processed.

[0082] If (noise AN)<(minimum audibility limit) is not true, the decision unit 4 proceeds to step S112 and decides whether or not content sound AC is present from the arrival direction. If the content sound AC does not exist, the determination unit 4 proceeds to step S117, where it sets the NC processing to ON, and also sets the driver 2 to output the NC sound ANC. Then, in step S118, the decision unit 4 sets the inaudible flag for the noise from the arrival direction currently being processed to OFF.

[0083] If it is determined in step S112 that the content sound AC is present from the arrival direction, the determination unit 4 compares in step S113 the level of the noise AN with the level of the content sound AC output based on the content sound data CT. If it is determined that the level of the noise AN is higher than the content sound AC, the determination unit 4 performs the processes of steps S117 and S118 described above.

[0084] If it is determined that the level of the noise AN is equal to or lower than the content sound AC, the determination unit 4 determines in step S114 whether the content sound AC masks the noise by simultaneous masking. If the content sound AC does not mask the noise AN, the determination unit 4 performs the processes of steps S117 and S118 described above. If the content sound AC masks the noise AN, the decision unit 4 performs the processes of steps S115 and S116 described above.

[0085] By performing the above process for each direction of arrival, the NC processing setting is determined for each direction of arrival. That is, the inaudible flag is set to OFF for directions that require NC processing, and the inaudible flag is set to ON for directions that do not require NC processing. Then, the decision unit 4 ends the loop in step S110 and proceeds to step S120 in FIG.

[0086] In step S120, it is determined whether to continue or end the loop of the processes from step S121 to step S125. The processes from step S121 to step S125 are also performed for each direction of arrival.

[0087] In step S121, the decision unit 4 checks the inaudible flag for one arrival direction that is the processing target. If the inaudible flag is on, that is, if the flag is set to not require NC processing, the process returns to step S120 for that arrival direction and moves to processing for the next arrival direction.

[0088] If the inaudible flag is off in step S121, the decision unit 4 proceeds to step S122 and estimates the frequency characteristics and level of the residual noise AN(NC) of the NC process. Then, in step S123, the residual noise AN(NC) is compared with the minimum audible limit.

[0089] If (residual noise AN(NC))<(minimum audibility limit), that is, if all of the frequency components that make up the residual noise AN(NC) are below the minimum audibility limit, the judgment unit 4 proceeds to step S125 and sets the inaudible flag for the residual noise AN(NC) to on.

[0090] If (residual noise AN(NC))<(minimum audibility limit) is not true, the decision unit 4 proceeds to step S124 and decides whether or not the content sound AC masks the residual noise AN(NC) by simultaneous masking. If masking is to be performed, the decision unit 4 proceeds to step S125 and sets the inaudible flag for the unerased noise AN(NC) to ON. If it is determined that the residual noise AN(NC) is not masked, the inaudible flag is kept OFF and the process returns to step S120.

[0091] By performing the above processes from step S121 to step S125 for each direction of arrival, the residual noise AN(NC) is estimated for each direction, and if the residual noise AN(NC) is below the minimum audible limit or is masked by the content sound AC, the inaudible flag for that direction of arrival is changed to ON.

[0092] After completing the above loop processing, the decision unit 4 proceeds from step S120 to step S130. In step S130, the decision unit checks whether the inaudible flag is on for all directions of arrival.

[0093] If the inaudible flag is on in all directions of arrival, the decision unit 4 determines the required quantization bits for all channels of the content sound data CT in step S137. In this case, since high-precision content sound data CT is required, the maximum allocation is requested as the number of quantization bits.

[0094] If it is confirmed in step S130 that there is an arrival direction with the inaudible flag off, the decision unit 4 proceeds to step S131, and performs processing in step S132 for each arrival direction with the inaudible flag off. In step S132, the decision unit 4 calculates the spatial masking effect that the direction has on other directions. This allows the spatial masking effect on other directions to be found for one or more arrival directions with the inaudible flag off.

[0095] In step S133, the decision unit 4 decides whether to continue or end the loop for each direction of arrival. That is, the process from step S134 to step S136 is performed for each direction of arrival. In step S134, the decision unit 4 decides whether or not there is any spatial masking that will have an effect on the arrival direction being processed.

[0096] If the direction of arrival being processed is not affected by spatial masking, the determination unit 4 proceeds to step S135 to determine the required quantization bits of the content sound data CT for that direction. In this case, since high-precision content sound data CT is required, the maximum allocation is requested as the number of quantization bits.

[0097] If the direction of arrival being processed is affected by spatial masking, the determination unit 4 proceeds to step S136 to determine the required quantization bits of the content sound data CT for that direction. In this case, since there is an area where highly accurate information is not required due to masking, a reduction in the number of quantization bits is requested.

[0098] In the loop starting from step S133, the number of quantization bits is set for each direction of arrival in either step S135 or step S136.

[0099] The number of quantization bits is set for each direction in step S135, S136, or step S137. The number of quantization bits is set for each channel by associating each direction to be the target of the loop processing with each channel. In step S140, the decision unit 4 transmits the notification information SS to the host device. In this case, the notification information SS includes information on the required number of quantization bits for each channel.

[0100] After transmitting the notification information SS in step S140, the decision unit 4 returns to step S101 in Fig. 27. Then, the above processes are repeated.

[0101] As described above, in the first embodiment, the determination unit 4 performs the processes in Figs. 27 and 28, thereby reducing the transmission bits of the content audio data according to the situation, thereby realizing a reduction in the transmission bit rate. In addition, by determining which driver 2 in the headphones 1 should be used to cancel noise AN, or which driver 2 does not require NC processing, and successively changing the settings, it is possible to achieve a more comfortable S / N ratio for the content sound AC, improvement in sound separation, and NC effect.

[0102] <2. Second embodiment> In the second embodiment, a process for making the user recognize the surrounding sound will be described. In the first embodiment, the surrounding sound is regarded as noise AN and necessary processes are performed, but in the second embodiment, necessary processes are performed on the surrounding sound that the user wants to recognize.

[0103] Examples of such ambient sounds include the following: -Sounds of cars approaching from behind or the sides -Sounds approaching your room (the user wearing headphones 1) (footsteps, etc.) Announcements (public transportation and public facilities, etc.) Alerts, sirens (emergency vehicle sounds, emergency earthquake alerts, etc.) - A voice calling you (the user wearing headphone 1)

[0104] Fig. 29 shows a configuration example of the headphones 1. However, the same parts as in Fig. 1 are given the same reference numerals to avoid duplicated explanations. What is different from Fig. 1 is that an ambient sound signal processor 8 is provided. The ambient sound signal processing unit 8 performs processing on the ambient sound data S1 obtained by the microphone 3 under the control of the acoustic processing control unit 4b of the determination unit 4. For example, the ambient sound data S1 is subjected to noise reduction processing, voice enhancement processing, etc., and the processed sound data S3 is output. Alternatively, the ambient sound signal processing unit 8 may perform processing to generate sound data S3 such as a beep sound or an announcement sound.

[0105] The sound data S3 that has been signal-processed or generated by the ambient sound signal processing unit 8 is supplied to the output signal generating unit 7. The output signal generating unit 7 generates an output signal to the driver 2 in accordance with a specified channel based on the sound data S3, together with the content sound data CT and the NC sound data S2.

[0106] The ambient sound type determination unit 5 performs a process of determining the type of sound from the ambient sound data obtained by the microphone 3. For example, it determines the specific type of sound, such as the sound of an approaching car, footsteps, or announcements for trains, etc. Note that the type determination does not necessarily involve determining the specific sound, and may instead be a determination of whether or not the sound is to be treated as noise.

[0107] The determination unit 4 receives the ambient sound data S1 and the type information of the ambient sound data S1, and performs processing in a masking determination unit 4a and an audio processing control unit 4b. The masking determination unit 4a determines the masking state in the relationship between the ambient sound data S1 and the content sound data CT, in the same manner as in the first embodiment. In this case, it is also determined whether or not the ambient sound required by the type information is masked by the content sound AC.

[0108] The sound processing control unit 4b controls sound processing according to the result of the masking state determination by the masking determination unit 4a. For example, the acoustic processing control unit 4b outputs a control signal to the NC signal generating unit 6 according to the masking state to control the NC operation. The control of the NC operation includes turning on / off the NC process and selecting the driver 2 that outputs the NC signal.

[0109] In addition, the sound processing control unit 4b performs control so that a sound that makes the driver 2 aware of the surrounding sound is output according to the type of sound in the surrounding sound data S1 and the result of the masking state determination. In this case, the acoustic processing control section 4b selects a channel of the driver 2 that outputs a sound that makes the user aware of the ambient sound, depending on the direction from which the ambient sound comes.

[0110] In addition, the acoustic processing control unit 4b controls the ambient sound signal processing unit 8 so that a sound based on the ambient sound obtained by the microphone 3, i.e., a sound obtained by signal processing the ambient sound data S1, is output from the driver 2 as a sound that causes the user to recognize the ambient sound. Alternatively, the acoustic processing control unit 4b controls the ambient sound signal processing unit 8 so that a generated sound that allows the driver 2 to recognize the ambient sound is output.

[0111] Furthermore, the acoustic processing control section 4b performs processing to transmit notification information SS to the host device 100, which is an external device, in accordance with the result of the masking state determination, for example. In the second embodiment, the acoustic processing control unit 4b transmits, as the notification information SS, information used for displaying the ambient sound. The information used for displaying the ambient sound may include some or all of information on the direction from which the ambient sound comes, information on the type of the ambient sound, and information on the masking state determination result.

[0112] In the second embodiment, the following process is carried out. Among the ambient sounds, there are sounds that the user should be aware of, as in the above example, so the ambient sound type determination unit 5 determines the type of the sound.

[0113] The determination unit 4 (masking determination unit 4a) determines whether the ambient sound data S1 is a sound that should be recognized by the user based on the type information, and if it is a sound that should be recognized, determines whether the sound is masked by the content sound AC. It also determines whether the sound is canceled by the NC process.

[0114] When necessary ambient sounds are masked or canceled as noise, the user cannot perceive the sounds as they are, so the determination unit 4 (sound processing control unit 4b) performs processing to make the user perceive the ambient sounds.

[0115] As a process for making the user aware of surrounding sounds, there is a process for outputting sounds. For example, the determination unit 4 causes the ambient sound signal processing unit 8 to perform processing on the ambient sound data S1 to make it easier to hear, such as noise reduction processing or voice enhancement processing, and causes the sound based on the processed sound data S3 to be output from the driver 2. In this case, it is considered that the determination unit 4 instructs the driver 2 to output based on the direction from which the ambient sound comes. This allows the user, when listening to the content sound AC, to hear the ambient sound itself from the actual direction from which the ambient sound comes.

[0116] As a method for outputting sound, instead of the ambient sound itself, a sound indicating an alert such as a beep or a message sound may be output. For example, the determination unit 4 causes the ambient sound signal processing unit 8 to execute a sound data generation process and causes the driver 2 to output a sound based on the generated sound data S3, such as a beep or a message sound.

[0117] This allows the user to recognize that some necessary ambient sound is occurring when listening to the content sound AC. In this case, the determination unit 4 may instruct the driver 2 to output based on the direction from which the ambient sound comes. This allows the user to recognize that the necessary ambient sound is coming from the direction from which the beep sound or the like comes. A beep is suitable in cases where only notification of the presence of ambient sound is required.

[0118] In addition, for example, the driver 2 that outputs the ambient sound itself, the beep sound, and the message sound may be selected as a driver of a channel that is not affected by spatial masking by the content sound AC. This can improve the recognition of the necessary ambient sound.

[0119] In addition, by outputting the ambient sound itself, the user can recognize what kind of sound it is, but with beeps and message sounds, the type of sound may not be recognized. Therefore, the message content may be changed according to the type and urgency of the sound, or the sound quality and volume of the beep may be changed. This can increase the alertness and allow the user to recognize the type of ambient sound. Examples of the message sound may include specific content such as "A car is approaching from behind" or "Someone is approaching the room."

[0120] It is also possible to combine this with GPS location information, change the criteria based on the size of the road you are walking (or running) on, or the number of cars passing by, and issue a notification based on the result.

[0121] In order to recognize the surrounding sounds, instead of or in addition to the notification by sound output from the headphones 1 as described above, a notification may be given using a display on the host device 100. That is, the determination unit 4 conveys the determination result to the host device 100, such as a smartphone or an HMD, so that the host device 100 notifies the user.

[0122] When a user is watching game content, video content, or the like, the user is gazing at the screen of the host device, so it is preferable to notify the user of ambient sounds by displaying a message on the screen, for example.

[0123] For example, FIG. 30 shows an example in which a message 61 is displayed on a screen 60 of the host device 100. In FIG. The display position of such a message 61 may be changed depending on the direction from which the surrounding sound is coming. For example, the display position of the message 61 saying "A car is approaching" may be controlled so that it is displayed at the bottom of the screen as shown in FIG. 30 if the car is approaching from behind, at the top or center of the screen if the car is approaching from the front, on the left side of the screen if the car is approaching from the left, and on the right side of the screen if the car is approaching from the right.

[0124] There is also a method of making an effect image 62 appear on a screen 60, as shown in Figures 31 and 32. In this case, it is also possible to make the effect image appear according to the direction from which the surrounding sound comes. Figure 31 shows the case where sound comes from the left, and Figure 32 shows the case where sound comes from the front. The size of the effect image 62 may represent the volume of the surrounding sound.

[0125] These displayed notifications allow users to respond to ambient sounds while watching videos or playing games such as virtual reality (VR). For example, when footsteps approaching one's room are masked and cannot be heard, this process is performed, allowing the user to take appropriate action, such as pausing the game, based on the notification.

[0126] In addition to the above-mentioned visual notification, the host device 100 or the headphones 1 may notify the user by vibration or the like. The visual notification and the above-mentioned sound notification in the headphones 1 may be used in combination.

[0127] 33, 34, and 35 show examples of screen displays that notify the user of more detailed ambient sound conditions. In each drawing, an image of a space centered on the user's head is displayed as spatial coordinates 50. Using this spatial coordinates 50 as a reference, a content sound image 51, ambient sound type images 55 and 57, and effect images 56 and 58 are displayed at positions according to the direction from which the sound is coming.

[0128] The content sound image 51 is an image showing the type and masking range of the content sound AC. In the case of Fig. 33A, the sound of a musical instrument such as a violin is localized to the left of the user, and the range in which other sounds are masked by the sound of the musical instrument is shown in a circular shape.

[0129] The ambient sound type images 55 and 57 are images showing the type of ambient sound, such as an image of a car, an image of footprints showing footsteps, etc. The effect images 56 and 58 show ambient sounds. The amount of masked ambient sound is indicated by the size of the ambient sound type images 55, 57 and the effect images 56, 58. The display positions of the ambient sound type images 55, 57 and the effect images 56, 58 indicate the direction from which the ambient sound comes.

[0130] Also displayed on the screen 60 is a setting section 53. The setting section 53 is an operation section for the user to arbitrarily set the notification function to on / off. For example, the setting section 53 is provided with setting fields for "ambient sound extraction ON / OFF," "automobile ON / OFF," and "footsteps ON / OFF."

[0131] 33A shows a state in which the ambient sound extraction is turned off, and a content sound image 51 is displayed. 33B shows a state in which the ambient sound extraction is turned on and the type of ambient sound is set to "car." When the car is turned on, a slide bar 54 is displayed. The user can set the extraction level using the slide bar 54.

[0132] When a car sound is detected, an ambient sound type image 55 and an effect image 56 are displayed according to the direction from which the car sound is coming and the amount of masking, as shown in Fig. 33B. In this case, it is shown that the car sound is coming from the same direction as the content sound and is being masked to a certain extent.

[0133] 34A shows a case where the content sound has moved. For example, when the localization of the content sound data CT is changed as the original content, the user may change the localization of the musical instrument by an operation. Alternatively, the determination unit 4 may request the host device 100 to automatically change the localization of the musical instrument. For example, the determination unit 4 requests the host device 100 to change the localization of the content sound in response to the fact that the sound of an automobile is masked. This causes the host device 100 to change the channel of the content sound data CT and change the localization. 34A, the amount of masking due to the spatial masking effect is reduced, making it easier for the user to hear the car sound. The size of the ambient sound type image 55 and the effect image 56 is reduced by the amount of masking.

[0134] 35A shows a case where footsteps are set as the extraction target in setting unit 53. In this case as well, by turning on footsteps, slide bar 59 is displayed, and the user can use slide bar 59 to set the footstep extraction level. 35B shows a case where footsteps coming from the right side are detected. In response to this, an ambient sound type image 57 for the footsteps and an effect image 58 are displayed on the right side. This example shows a state in which the footsteps are masked to some extent by the content sound.

[0135] By displaying the images shown in Figs. 33 to 35, the user can recognize the surrounding sounds and their masking state in more detail.

[0136] The above assumes 3D audio content, but in games, for example, the position of the sound source moves automatically. This is tracked in real time, the masking effect is analyzed, and NC processing and the transmission bit rate of the content sound data CT are appropriately set. The combination of the HMD as the host device 100 and the headphones 1 has a high affinity because the directions of the user's vision and hearing match. When the user turns his or her head, the position of the headphones 1 (microphone 3) naturally changes at the same time, so the movement of the surrounding sound source as seen by the user can be handled in real time by the change in the signal from the microphone 3.

[0137] Although Figs. 33 to 35 show images in which stereophonic content is present, displays such as those shown in Figs. 33 to 35 may also be made in the case of non-stereophonic content or in a silent state in which there is no content.

[0138] So far, an example of notifying the user of necessary ambient sounds by voice or display has been described as the second embodiment. An example of processing by the determination unit 4 that realizes such an operation is shown in FIG.

[0139] During the period when the headphones 1 receive the content sound data CT and output the content sound AC from the driver 2, the determination unit 4 repeats the process of Fig. 36. Step S201 is a step for determining whether the repeat loop has ended. For example, the process of Fig. 36 ends when the power is turned off or the operation mode is changed.

[0140] During the loop period, in step S202, the determination unit 4 analyzes the frequency characteristics and the direction of arrival of the ambient sound, i.e., noise AN, obtained by the microphone 3. Also, based on the type information from the ambient sound type determination unit 5, the type of sound is determined.

[0141] In step S203, the determination unit 4 analyzes the frequency characteristics of the content sound data CT. The arrival direction of the content sound AC can be determined by the channel number of the content sound data CT.

[0142] In step S204, the determination unit 4 determines whether or not there is a sound that the user should recognize among the ambient sounds. If the result of the ambient sound type determination indicates that there is a sound such as a car sound, footsteps, an announcement, or an alert, it is determined that the sound that should be recognized is present, and the frequency characteristics and the direction of arrival of the sound are determined.

[0143] In step S205, the determination unit 4 determines the masking state and whether or not NC processing is to be performed. As in the first embodiment, the determination unit 4 determines the masking state based on the relationship between the ambient sound and the content sound, and then determines whether the NC process is on or off accordingly.

[0144] The determination unit 4 also determines whether the NC process is on or off depending on whether or not there is a sound that the user should recognize in the surrounding sounds. For example, when the ambient sound does not include a sound that the user should recognize, the determination unit 4 determines that the normal NC process is to be performed on the ambient sound. If the ambient sound includes a sound that the user should recognize, the NC processing is executed for at least the frequency components other than the relevant sound, and it is determined that the NC processing is not executed for the sound that the user should recognize. In this case, it may be determined that the normal NC processing is executed. For example, when generating a beep sound or a message sound, the NC processing for the ambient sound may always be executed normally. When the ambient sound is only the sound that the user should be aware of (or when the sound that the user should be aware of is dominant), it is determined that the NC process is not to be performed. Note that, in this case, it may also be determined that the normal NC process is to be performed as described above.

[0145] In step S206, the judgment unit 4 controls the NC signal generation unit 6 based on the judgment result in step S205. For example, it instructs the NC processing to be turned on / off. In some cases, it may control the NC processing to be disabled only for a specific frequency component. The judgment unit 4 also instructs the channel of the driver 2 that outputs the NC sound.

[0146] In step S207, the determination unit 4 branches the process depending on whether or not it is time to notify the user by displaying the host device 100. For example, if the ambient sounds include sounds that the user should be aware of and notification by display is set to ON, the process proceeds to step S220 and transmits notification information SS to the host device 100. This notification information SS includes information on the type of ambient sound, the direction of arrival, and the level of masking, making it possible to display the types of sounds described in Figs. 30 to 35 on the host device 100.

[0147] In step S208, the determination unit 4 branches the process depending on whether or not a notification by voice is to be performed. For example, if the ambient sound includes a sound that the user should recognize, the determination unit 4 proceeds to step S210 and branches the process depending on what type of notification is to be turned on.

[0148] When the ambient sound itself is to be output, the determination unit 4 proceeds to step S211, and instructs the ambient sound signal processing unit 8 to execute noise reduction processing, voice enhancement processing, etc. on the ambient sound data S1. It also instructs the channel of the driver 2 to output according to the arrival direction.

[0149] If the notification sound is to be generated, the determination unit 4 proceeds to step S212 and instructs the ambient sound signal processing unit 8 to generate a beep sound or a message sound. It also instructs the channel of the driver 2 to output according to the arrival direction.

[0150] For example, by repeating the above-described process of FIG. 36, the operation of notifying the surrounding sound by voice or display as described above is realized.

[0151] <3. Summary and examples of modifications> According to the above first and second embodiments, the following effects can be obtained.

[0152] The signal processing device of the embodiment is realized as a processor having the functions of a determination unit 4 including a masking determination unit 4a and an audio processing control unit 4b. The audio output device of the embodiment is realized as headphones 1 including such a determination unit 4. In these, the masking determination unit 4a determines the masking state of the content sound and the ambient sound using multiple channels of content sound data CT output from multiple drivers 2 arranged in the headphones 1, ambient sound data S1 obtained by multiple microphones 3 arranged in the headphones 1 and collecting ambient sound, and information on the direction from which the ambient sound is coming. The sound processing control unit 4b controls sound processing according to the result of the masking state determination by the masking determination unit 4a. This makes it possible to execute appropriate control of acoustic processing in cases such as when content sound AC is masked by ambient sound (noise AN) or when necessary ambient sound is masked by content sound, etc. In particular, by determining the masking state taking into account the level and direction of the ambient sound, as well as the channel of the content sound (information on which output position it is output from) and the level of each channel, it is possible to more appropriately determine the masking situation in so-called multi-microphone, multi-driver headphones. Therefore, when playing 3D audio and other stereophonic content, it is possible to control appropriate sound processing according to highly accurate masking situation judgment. For example, it is possible to enable the user to recognize necessary ambient sounds, enable comfortable listening to content sounds through appropriate NC processing, and reduce the load on system processing.

[0153] In the second embodiment, an example has been described in which control is performed so that a sound that makes the driver 2 aware of the ambient sound is output depending on the type of sound in the ambient sound data S1 and the determination result of the masking state (see FIG. 36). For example, for a user enjoying stereophonic content, not all ambient sounds are noise. For example, there are sounds that are necessary for safety or daily life. Therefore, if it is determined that the ambient sound is necessary depending on the type of the ambient sound, the ambient sound is output from the driver 2 and the user is made aware of it. This allows the user to enjoy stereophonic sound while being able to hear the ambient sounds appropriately.

[0154] In the second embodiment, an example has been given in which the driver 2 that outputs a sound that allows the driver to recognize an ambient sound is determined depending on the direction from which the ambient sound comes. By determining the driver 2 (i.e., the channel) in this way, the user can recognize the sound coming from the direction corresponding to the determined channel. Therefore, the user can hear the ambient sound itself, and the notification sound or message sound in place of the ambient sound, and can also recognize the direction from which the actual ambient sound is coming. The direction from which the ambient sound comes can be tracked in real time according to the user's own actions and movements by constantly analyzing the ambient sound data S1 collected by the microphone 3 of the headphones 1.

[0155] In the second embodiment, an example was given in which control is performed so that a sound obtained by signal processing of ambient sound data S1 obtained by microphone 3 is output from driver 2 as a sound that causes the driver to recognize the ambient sound (step S211 in FIG. 36). This prevents necessary ambient sounds from being inaudible due to masking, and enables the user to recognize actual ambient sounds even when listening to stereophonic sound through the headphones 1.

[0156] In the second embodiment, an example has been described in which control is performed so that a generated sound that allows the driver 2 to recognize an ambient sound is output (step S212 in FIG. 36). For example, sounds that indicate some kind of caution, warning, or notification, such as beeps or voice messages, are generated and output. This allows the user to recognize the surrounding situation (a situation in which necessary ambient sounds are occurring) even if necessary ambient sounds cannot be heard due to masking by the content sounds or noise cancellation processing.

[0157] In the second embodiment, an example is given in which the judgment unit 4 (acoustic processing control unit 4b) performs processing to transmit notification information SS used for displaying the ambient sound to the host device 100 depending on the type of sound in the ambient sound data S1 and the judgment result of the masking state (step S220 in Figure 36). For example, when an ambient sound that is determined to be necessary for the user is detected, information used for displaying the ambient sound to make the user aware of the ambient sound is transmitted to the host device 100. This causes the host device 100 to execute a display for making the user aware of the ambient sound as described with reference to Figures 30 to 35. When viewing stereophonic content accompanied by images, the user is also gazing at the screen, so notifying the user of the necessary ambient sound by display is also effective.

[0158] In the second embodiment, an example has been given in which the notification information SS used for the display to make the user aware of the ambient sound includes information on the direction from which the ambient sound comes. This allows an external device such as the host device 100 to perform display according to the direction from which the ambient sound comes (see FIGS. 30 to 35).

[0159] In the second embodiment, an example has been given in which the notification information SS used for the display to make the user aware of the ambient sound includes information on the type of the ambient sound. This allows an external device such as the host device 100 to perform display according to the type of surrounding sound, for example, the sound of a car, footsteps, etc. (see FIGS. 31 to 35).

[0160] In the second embodiment, an example has been given in which the notification information SS used for displaying the surrounding sound includes information on the determination result of the masking state. This makes it possible for an external device such as the host device 100 to perform a display showing a situation in which the ambient sound is being masked by the content sound (see FIGS. 33 to 35).

[0161] In the first and second embodiments, an example has been described in which the acoustic processing control section 4b controls the noise cancellation processing for the ambient sound depending on the result of the masking state determination. By performing NC processing on the ambient sound, the ambient sound can be reduced or eliminated for a user enjoying stereophonic content. However, NC processing does not need to be performed on ambient sound that is originally masked. Therefore, the efficiency of NC processing can be improved by controlling the NC processing according to the masking determination result. In other words, NC processing can be performed on ambient sound that is not masked by the content sound due to its frequency component or direction of arrival, and NC processing can be turned off for ambient sound that is masked. Furthermore, the sound processing control section 4b may also use the type of sound of the ambient sound data obtained by the microphone 3 to control the NC processing. For example, when it is determined that the ambient sound is necessary for the user, the NC processing can be turned off so that the user can hear the sound.

[0162] In the first and second embodiments, an example has been described in which the acoustic processing control unit 4b performs control to determine the driver 2 (i.e., the channel) that outputs NC sound for the ambient sound, depending on the result of the masking state determination and information on the direction from which the ambient sound comes. When performing NC processing on ambient sound, the NC effect can be made more effective by determining the driver 2 that outputs NC sound ANC according to the direction from which the ambient sound comes.

[0163] In the first embodiment, an example has been described in which the acoustic processing control section 4b performs processing for transmitting quantization bit information required for the content audio data CT to the host device 100 in accordance with the result of the masking state determination. Regarding stereophonic sound reproduction, the transmission bit rate is increasing drastically due to multi-viewpoint and free viewpoint, and reducing the transmission bit rate of content audio data is one of the important issues. Since the information of the sound components to be masked here does not need to be transmitted in the first place, the number of quantization bits can be reduced. Therefore, quantization bit information required for the content audio data is transmitted to the host device 100 according to the masking state determination result. This allows the host device 100 to reduce the amount of content audio data, resulting in a reduction in the transmission bit rate of the content audio data, an improvement in the S / N ratio of the content signal, an improvement in the sense of separation of sounds, an improvement in the NC effect, and even a longer battery life of the headphones 1 due to reduced power consumption.

[0164] In the first embodiment, the quantization bit information transmitted to the host device 100 includes information on the channel and band for reducing the number of quantization bits in the content audio data. This allows the host device 100 to reduce the number of quantization bits in a specified band in a specified channel.

[0165] The determination unit 4 (sound processing control unit 4b) can also selectively perform power-off control of the driver 2. For example, the control is to turn off the power supply to the driver 2 of a channel that does not output NC sound and does not output content sound. By constantly monitoring the driver 2 that is not being used temporarily in this way and turning off the power supply, power consumption is reduced, which contributes to extending the battery life of the headphones 1.

[0166] Furthermore, in the embodiment, the headphones 1 have been described as an example of an audio output device, but the technology of the present disclosure can also be applied to audio output devices such as various types of earphones, such as inner-ear type and canal type.

[0167] It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0168] The present technology can also be configured as follows. (1) a masking determination unit that determines a masking state between the content sound and the ambient sound using multiple channels of content sound data output from multiple audio output drivers arranged in the audio output device, ambient sound data obtained by multiple microphones arranged in the audio output device for collecting ambient sound, and information on the direction from which the ambient sound comes; a sound processing control unit that controls sound processing in accordance with a result of the masking state determination by the masking determination unit. Signal processing device. (2) The sound processing control unit controls the sound output driver to output a sound that allows the user to recognize the ambient sound, depending on the type of sound of the ambient sound data obtained by the microphone and the result of the determination of the masking state. A signal processing device according to (1) above. (3) The acoustic processing control unit determines, from among the plurality of acoustic output drivers, an acoustic output driver that outputs a sound that allows the user to recognize the ambient sound, depending on the direction from which the ambient sound comes. A signal processing device according to (2) above. (4) The acoustic processing control unit performs control so that a sound obtained by signal processing of an ambient sound obtained by the microphone is output from the acoustic output driver as a sound for recognizing the ambient sound. A signal processing device according to (2) or (3) above. (5) The sound processing control unit performs control so that a generated sound that causes the user to recognize an ambient sound is output from the sound output driver. A signal processing device according to (2) or (3) above. (6) The acoustic processing control unit performs a process of transmitting information used for displaying the ambient sound to an external device according to the type of sound of the ambient sound data obtained by the microphone and the result of the determination of the masking state. A signal processing device according to any one of (1) to (5) above. (7) The acoustic processing control unit performs processing for transmitting information used for displaying the surrounding sound to the external device, the information including information on the direction from which the surrounding sound comes. A signal processing device according to (6) above. (8) The sound processing control unit performs processing for transmitting information used for displaying the surrounding sound to the external device, the information including information on the type of the surrounding sound. A signal processing device according to (6) or (7) above. (9) The sound processing control unit performs processing for transmitting information used for displaying the surrounding sound to the external device, the information including information on the determination result of the masking state. A signal processing device according to any one of (6) to (8) above. (10) The acoustic processing control unit controls a noise cancellation process for the ambient sound depending on a result of the masking state determination. A signal processing device according to any one of (1) to (9) above. (11) The acoustic processing control unit performs control to determine an acoustic output driver that outputs a noise cancellation signal for the ambient sound, among the plurality of acoustic output drivers, in accordance with a result of the masking state determination and information on the direction from which the ambient sound comes. A signal processing device according to any one of (1) to (10) above. (12) The audio processing control unit performs a process of transmitting quantization bit information required for the content audio data to an external device according to a result of the masking state determination. A signal processing device according to any one of (1) to (11) above. (13) The quantization bit information to be transmitted to the external device includes information on a channel and a band for reducing the number of quantization bits in the content audio data. The signal processing device according to (12) above. (14) A plurality of acoustic output drivers; A plurality of microphones for collecting ambient sounds; a masking determination unit that determines a masking state between the content sound and the ambient sound using the content sound data of multiple channels output from the audio output driver, the ambient sound data obtained by the microphone, and information on the direction from which the ambient sound comes; a sound processing control unit that controls sound processing in accordance with a result of the masking state determination by the masking determination unit. Audio output device. (15) a masking determination process for determining a masking state between the content sound and the ambient sound using multiple channels of content sound data output from multiple audio output drivers arranged in the audio output device, ambient sound data obtained by multiple microphones arranged in the audio output device for collecting ambient sound, and information on the direction from which the ambient sound comes; a sound processing control process for controlling sound processing in accordance with a result of the masking state determination process; The signal processing method is executed by a signal processing device. [Explanation of symbols]

[0169] 1. Headphones 2,2A···2(N) Sound Output Driver (Driver) 3,3A···3(M) Microphone 4 Judgment section 4a Masking judgment section 4b Sound processing control section 5. Ambient sound type determination section 6 NC signal generation section 7 Output signal generator 8 Ambient sound signal processing section 100 Host Device 200 ears 201 Eardrum AC content sound AN Noise AN(NC) Residual noise ANC NC sound S1 Ambient sound data S2 NC sound data S3 Sound Data SS notification information CT content sound data

Claims

1. a masking determination unit that determines a masking state between the content sound and the ambient sound using multiple channels of content sound data output from multiple audio output drivers arranged in the audio output device, ambient sound data obtained by multiple microphones arranged in the audio output device for collecting ambient sound, and information on the direction from which the ambient sound comes; a sound processing control unit that controls sound processing in accordance with a result of the masking state determination by the masking determination unit; Signal processing device.

2. The sound processing control unit controls the sound output driver to output a sound that allows the user to recognize the ambient sound, depending on the type of sound of the ambient sound data obtained by the microphone and the result of the determination of the masking state. The signal processing device according to claim 1 .

3. The acoustic processing control unit determines, from among the plurality of acoustic output drivers, an acoustic output driver that outputs a sound that allows the user to recognize the ambient sound, depending on the direction from which the ambient sound comes. The signal processing device according to claim 2 .

4. The acoustic processing control unit performs control so that a sound obtained by signal processing of an ambient sound obtained by the microphone is output from the acoustic output driver as a sound for recognizing the ambient sound. The signal processing device according to claim 2 .

5. The sound processing control unit performs control so that a generated sound that causes the user to recognize an ambient sound is output from the sound output driver. The signal processing device according to claim 2 .

6. The acoustic processing control unit performs a process of transmitting information used for displaying the ambient sound to an external device according to the type of sound of the ambient sound data obtained by the microphone and the result of the determination of the masking state. The signal processing device according to claim 1 .

7. The acoustic processing control unit performs processing for transmitting information used for displaying the surrounding sound to the external device, the information including information on the direction from which the surrounding sound comes. The signal processing device according to claim 6.

8. The sound processing control unit performs processing for transmitting information used for displaying the surrounding sound to the external device, the information including information on the type of the surrounding sound. The signal processing device according to claim 6.

9. The sound processing control unit performs processing for transmitting information used for displaying the surrounding sound to the external device, the information including information on the determination result of the masking state. The signal processing device according to claim 6.

10. The acoustic processing control unit controls a noise cancellation process for the ambient sound depending on a result of the masking state determination. The signal processing device according to claim 1 .

11. The acoustic processing control unit performs control to determine an acoustic output driver that outputs a noise cancellation signal for the ambient sound, among the plurality of acoustic output drivers, in accordance with a result of the masking state determination and information on the direction from which the ambient sound comes. The signal processing device according to claim 1 .

12. The audio processing control unit performs a process of transmitting quantization bit information required for the content audio data to an external device according to a result of the masking state determination. The signal processing device according to claim 1 .

13. The quantization bit information to be transmitted to the external device includes information on a channel and a band for reducing the number of quantization bits in the content audio data. The signal processing device according to claim 12.

14. A plurality of acoustic output drivers; A plurality of microphones for collecting ambient sounds; a masking determination unit that determines a masking state between the content sound and the ambient sound using the content sound data of multiple channels output from the audio output driver, the ambient sound data obtained by the microphone, and information on the direction from which the ambient sound comes; a sound processing control unit that controls sound processing in accordance with a result of the masking state determination by the masking determination unit; Audio output device.

15. a masking determination process for determining a masking state between the content sound and the ambient sound using multiple channels of content sound data output from multiple audio output drivers arranged in the audio output device, ambient sound data obtained by multiple microphones arranged in the audio output device for collecting ambient sound, and information on the direction from which the ambient sound comes; a sound processing control process for controlling sound processing in accordance with a result of the masking state determination process; The signal processing method is executed by a signal processing device.

Images