SIGNAL PROCESSING DEVICE, ACOUSTIC OUTPUT DEVICE AND SIGNAL PROCESSING METHOD

Abstract

This signal processing device comprises a masking evaluation unit for evaluating the masking state of a content tone and an ambient tone using content tone data, which has multiple channels and is output by multiple acoustic output drivers arranged in an acoustic output device, ambient tone data received by multiple microphones arranged on the acoustic output device and capturing ambient tones, and information relating to the arrival directions of the ambient tones. The signal processing device also comprises an acoustic processing control unit for performing control of the acoustic processing according to the result of the masking state evaluation by the masking evaluation unit.

Description

Signal processing device, sound output device, and signal processing method

[0001] The present technology relates to a signal processing device, an audio output device, and a signal processing method, and in particular to a technology suitable for the field of stereophonic sound reproduction using a plurality of audio output drivers.

[0002] For example, technologies for stereophonic sound reproduction have been developed, such as 3D (three dimensions) audio and 360-degree audio. Headphones used for stereophonic sound reproduction have multi-channel audio output drivers (driver units) in each of the left and right ear units, allowing the user to perceive content audio from various directions. Some such headphones also have multiple microphones for noise cancellation processing, collecting ambient sounds from various directions. Such so-called multi-microphone / multi-driver headphones are expected to be used in use cases where content such as 3D audio is listened to in an environment where ambient sounds are generated. Patent Document 1 below discloses technology related to the transmission of stereophonic (3D) audio.

[0003] Japanese Patent Application Laid-Open No. 2021-152677

[0004] Here, when considering two types of sound, content sound and ambient sound, consider the case where one sound is masked by the other. That is, there are cases where the ambient sound is masked by the content sound and is not recognized by the user, and cases where the ambient sound masks part of the content sound components. For example, when performing noise cancellation processing on the ambient sound as noise, there are cases where the remaining component masks part of the content sound components.

[0005] It is believed that determining the state of masking in this way will enable more efficient processing and processing that is desirable for the user. Therefore, this technology proposes a technology that enables processing appropriate to the situation of content sound and ambient sound.

[0006] The present technology provides a signal processing device that includes a masking determination unit that determines a masking state between content sound and ambient sound using multiple channels of content sound data output from multiple audio output drivers arranged in an audio output device, ambient sound data obtained by multiple microphones arranged in the audio output device that collect ambient sound, and information on the direction from which the ambient sound is coming, and an audio processing control unit that controls audio processing in accordance with the masking state determination result by the masking determination unit. Assume that the audio output device, such as headphones, has multiple audio output drivers in its housing to play stereophonic sound and multiple microphones arranged to collect ambient sound. In this case, the signal processing device determines the masking state for the content sound data and the ambient sound data, and controls audio processing in accordance with the masking state.

[0007] 1 is a block diagram of headphones according to a first embodiment of the present technology; FIG. 2 is a schematic explanatory diagram of a microphone and an acoustic output driver in the headphones according to the embodiment; FIG. 3 is an explanatory diagram of sound heard by a wearer of the headphones according to the embodiment; FIG. 4 is an explanatory diagram of the position of the acoustic output driver and noise cancellation characteristics; FIG. 5 is an explanatory diagram of the minimum audible limit and simultaneous masking; FIG. 6 is an explanatory diagram of masking in a state where noise is added; FIG. 7 is an explanatory diagram of masking in a state where noise is added; FIG. 8 is an explanatory diagram of masking when there is residual noise; FIG. 9 is an explanatory diagram of masking when there is residual noise; FIG. 10 is an explanatory diagram of spatial masking; FIG. 11 is an explanatory diagram of a case where the arrival directions of noise and content sound coincide; FIG. 12 is an explanatory diagram of masking when the arrival directions of noise and content sound coincide; FIG. 13 is an explanatory diagram of a state where noise cancellation processing is applied when the arrival directions of noise and content sound coincide; FIG. 14 is an explanatory diagram of masking when noise and content sound coincide; 1 is an explanatory diagram of a state in which noise cancellation processing is applied when the directions from which noise and content sound arrive are different. FIG. 1 is an explanatory diagram of masking in a state in which noise cancellation processing is applied when the directions from which noise and content sound arrive are different. FIG. 2 is an explanatory diagram of a case in which the direction from which noise arrives is included in the direction from which content sound arrives. FIG. 3 is an explanatory diagram of a case in which a masking state that differs for each direction of arrival is applied. FIG. 4 is an explanatory diagram of a case in which noise cancellation processing is applied in a masking state that differs for each direction of arrival. FIG. 5 is an explanatory diagram of a state in which noise cancellation processing has been performed. FIG. 6 is an explanatory diagram of a case in which the direction from which noise arrives is other than the direction from which content sound arrives. FIG. 7 is an explanatory diagram of a masking state that differs for each direction of arrival is applied. FIG. 8 is an explanatory diagram of a case in which noise cancellation processing is applied in a masking state that differs for each direction of arrival. FIG. 9 is an explanatory diagram of a state in which noise cancellation processing has been performed. FIG. 10 is a flowchart of an example of processing by a determination unit of an embodiment. FIG. 11 is a flowchart of an example of processing by a determination unit of an embodiment. FIG. 12 is a block diagram of headphones of a second embodiment. FIG. 13 is an explanatory diagram of an example of a message display for ambient sound notification in a host device. FIG. 14 is an explanatory diagram of an example of a display for ambient sound and arrival direction notification in a host device.Fig. 10 is an explanatory diagram of a display example of ambient sound and arrival direction notification in the host device; Fig. 11 is an explanatory diagram of a display example including a masking state in the host device; Fig. 12 is an explanatory diagram of a display example including a masking state in the host device; Fig. 13 is an explanatory diagram of a display example including a masking state in the host device; Fig. 14 is a flowchart of a processing example of a determination unit of an embodiment.

[0008] The embodiments will be described below in the following order: <1. First embodiment> <2. Second embodiment> <3. Summary and variations>

[0009] 1. First Embodiment In the embodiment of the present disclosure, a headphone equipped with multiple microphones and multiple drivers is used as an example of an audio output device, and a case where content including stereophonic audio data such as 3D audio is listened to using the headphone is described as an example. Note that the audio output driver equipped in the headphone is also referred to simply as a "driver" hereinafter.

[0010] First, in the first embodiment, a process for dealing with an increase in the amount of content sound data transmitted in order to perform stereophonic sound reproduction will be described.

[0011] In the case of stereophonic sound reproduction, the number of sound sources has increased dramatically compared to content that reproduces conventional two-channel stereo sound, and the transmission bit rate of the content sound data has actually increased.

[0012] On the other hand, in addition to the content sound, there is also ambient sound (noise) that is emitted from the surroundings and reaches the eardrum. In this case, by utilizing a phenomenon known as the masking effect, it is possible to reduce the transmission bit rate of the content sound data, improve the signal-to-noise ratio (S / N) of the content sound data, improve the sense of sound separation, improve the noise cancellation (hereafter referred to as "NC") effect, and even extend the battery life of headphones by reducing power consumption.

[0013] There is a phenomenon called the masking effect, in which a sound is obstructed by another sound and becomes inaudible. When there are two sound sources, one signal can sometimes mask the other, creating a masking effect. The signal that masks the other is called the "masker," and the signal that is masked is called the "maskie."

[0014] In this embodiment, this masking effect is utilized to analyze which signal will be dominant (masker) when stereophonic content is viewed in a noisy environment, based on the noise, the stereophonic content sound data, and the direction from which the noise is coming, and to what extent the other signal (maskey) will be degraded or drowned out.

[0015] If the noise becomes a masker according to the analysis results, the transmission bit rate is reduced by reducing the quantization bits of the content sound data for stereophonic reproduction.

[0016] Furthermore, when listening to content sound in a noisy environment, the NC function is turned on. In this case, the NC effect and the quality of the content sound, such as the S / N ratio after NC processing, depend on the direction from which the noise is coming. Therefore, by determining which driver in the headphones should be used to cancel noise, or which driver does not require noise cancellation, and changing the settings, a more comfortable S / N ratio for the reproduced sound, sound separation, and improved NC effect can be achieved.

[0017] 1 shows an example configuration of headphones 1 according to a first embodiment. Content audio data CT as stereophonic data is transmitted to the headphones 1 from a host device 100, and the headphones 1 output playback sound based on the content audio data CT. Note that while the host device 100 is assumed to be a device separate from the headphones 1, it may also be a device section provided inside the headphones 1. For example, it may be an audio streaming control section within the headphones 1. In other words, the host device 100 in this embodiment may be in any form as long as it is a device or circuit section that serves as the source of the content audio data CT to be played back by the headphones 1, and it may be integrated with or separate from the headphones 1.

[0018] The headphones 1 are provided with N drivers 2 (2A, 2B, ... 2(N)) that output, for example, N channels of stereophonic sound. The drivers 2A, 2B, ... 2(N) are arranged in different positions corresponding to the respective channels within the left and right housings of the headphones 1. For example, as schematically shown in FIG. 2 , the drivers 2A, 2B, ... 2(N) are arranged within the housing 10 so as to emit sound toward the user's ear 200. As a result, the channels of content sound that reach the eardrum 201 arrive from different directions.

[0019] For ease of explanation, the housing 10 on one side of the user's left or right ear will be illustrated and described, but the other housing is similar. N channel (N pieces) drivers are provided in each of the left and right housings 10. While Fig. 1 also shows a configuration corresponding to one ear 200, the configuration on the other side is similar.

[0020] 1 and 2, M microphones 3 (3A, 3B, ... 3(M)) are provided at different positions facing the outside of the housing 10. This allows M channels of ambient sound to be collected. The microphones 3 are arranged at appropriate positions for performing NC processing using, for example, the FF (feedforward) method.

[0021] 2 is merely a schematic diagram, and the drivers 2 and microphones 3 are not necessarily aligned in the cross-sectional direction of the housing 10 as shown in the figure. For example, multiple drivers 2 may be arranged at various locations on the inner surface of the housing 10, and multiple microphones 3 may be arranged at various locations on the outer surface of the housing 10.

[0022] The host device 100 for the headphones 1 is a device that serves as a source of content sound data CT, and may be, for example, a smartphone, an HMD (head-mounted display), a game device, a tablet, a personal computer, or any other type of device.

[0023] The host device 100, for example, has a display unit to display content video, and transmits content audio data CT to the headphones 1. This allows the user to listen to content in the form of video and audio. In this case, the content audio data CT is N-channel stereophonic audio data played back by the headphones 1. In other words, it is assumed that the content audio data CT has undergone signal processing on the host device 100 side in accordance with the number and position of channels of the driver 2 of the headphones 1.

[0024] The headphones 1 that receive the content sound data CT are provided with the functions of a determination unit 4, an ambient sound type determination unit 5, an NC signal generation unit 6, and an output signal generation unit 7, each of which is implemented by one or more microprocessors, for example.

[0025] The sound collected by the microphone 3 is converted into ambient sound data S1 as digital data at the output stage of the microphone 3, for example, and supplied to the determination unit 4, ambient sound type determination unit 5, and NC signal generation unit 6. Note that the conversion into digital data may be performed at the input stage of the microprocessor that constitutes each of these units. For example, by providing the microprocessor with an A / D conversion terminal, the determination unit 4, ambient sound type determination unit 5, and NC signal generation unit 6 acquire the ambient sound data S1 as digital data.

[0026] The determination unit 4 has a function of obtaining the ambient sound data S1 and the content sound data CT from the microphone 3 and performing determination and control. Specifically, it has the functions of a masking determination unit 4a and an acoustic processing control unit 4b.

[0027] The masking determination unit 4a performs processing to determine the masking state of the content sound and the ambient sound using the N-channel content sound data CT output from the N-channel driver 2, the M-channel ambient sound data S1 obtained by the microphone 3, and information on the arrival direction of the ambient sound. Therefore, the masking determination unit 4a determines the arrival direction of the ambient sound (noise) from the M-channel ambient sound data S1. The masking determination unit 4a also calculates the frequency characteristics of the M-channel ambient sound data S1. The masking determination unit 4a also calculates the frequency characteristics of the N-channel content sound data CT. The masking determination unit 4a determines the masking state of the ambient sound and the content sound based on this information. Details will be described later. In the case of stereophonic content as in this embodiment, the sound data for each channel of the content sound data CT is sound data output from a different driver 2. In other words, the channel number corresponds to the location of each driver 2. Therefore, the channel information of the content sound data CT corresponds to information on the arrival direction of the content sound relative to the user. Therefore, the level of each direction from which the content sound comes can be determined from the level of each channel in the multi-channel content sound data CT. In other words, the content sound data CT itself includes information on the direction from which the content sound comes relative to the user.

[0028] The acoustic processing control unit 4b controls acoustic processing in accordance with the masking state determination result by the masking determination unit 4a. For example, the acoustic processing control unit 4b outputs a control signal to the NC signal generation unit 6 in accordance with the masking state to control NC operation. Control of NC operation includes turning NC processing on / off and selecting the driver 2 that outputs the NC signal. For example, in response to incoming noise, the acoustic processing control unit 4b determines which driver 2 should output the NC signal S2 to maximize the NC effect, and sends a command to the NC signal generation unit 6.

[0029] Furthermore, the acoustic processing control unit 4b performs processing to transmit notification information SS to the external device, i.e., the host device 100, in accordance with, for example, the masking state determination result. In the first embodiment, the notification information SS includes quantization bit information required for the content sound data CT. The quantization bit information includes, for example, information on the channel and band for reducing the number of quantization bits in the content sound data CT.

[0030] The ambient sound type determination unit 5 performs processing to determine the type of sound for the ambient sound data obtained by the microphone 3. Note that the type determination does not necessarily involve determining the specific sound, but may instead determine whether or not the sound is to be treated as noise. Note that the ambient sound type determination unit 5 is mainly required for the processing in the second embodiment, and therefore may not be provided for the processing in the first embodiment. However, in the first embodiment, for example, processing may be performed to turn NC processing on / off depending on the type of sound in the ambient sound data S1.

[0031] The NC signal generating unit 6 has a function of generating an NC signal S2 for canceling out the ambient sound data obtained by the microphone 3 as noise. For example, the NC signal S2 is generated by processing in accordance with an FF-NC algorithm.

[0032] The output signal generation unit 7 has a function of generating a signal to be output from the driver 2. Basically, it generates a signal to drive the driver 2 of each channel based on the data of each channel of the content sound data CT. Note that this may also include equalizer processing of the content sound data CT. The output signal generation unit 7 also generates a signal to drive the driver 2 of a specified channel based on the input NC signal S2. Note that the channel of the driver 2 that outputs the NC signal may be specified by the acoustic processing control unit 4b as described above.

[0033] The masking determination process performed by the determination unit 4 will be described below. There are various types of masking effects, including "simultaneous masking (frequency masking)," which blocks adjacent frequency sounds at the same time, and "temporal masking," which blocks sounds immediately before and after them. This disclosure mainly utilizes the above-mentioned "simultaneous masking" and "spatial masking," which occurs due to differences in the direction of sound arrival. "Spatial masking" is when the masker and maskee arrive from the same direction as the listener, resulting in the strongest masking effect, and when the directions of arrival are different, the masking effect is weaker. Note that FIG. 3 schematically shows a state in which the ambient noise AN and content sound AC arrive from different directions.

[0034] In the first embodiment, the ambient sound is considered to be noise AN. When noise AN and content sound AC output from the driver 2 of the headphones 1 are considered, the following cases can be considered from the perspective of the masker / maskie relationship and the direction from which each sound comes.

[0035] The noise AN masks the content sound AC (sound equivalent to all or part of the quantized bits) due to the simultaneous masking effect. The content sound AC masks the noise AN.

[0036] The noise AN and the content sound AC come from the same direction (masking effect: large). The noise AN and the content sound AC come from different directions (masking effect: small).

[0037] Combining these cases will improve the S / N ratio when listening to stereophonic content, improve the sense of separation of sounds, improve the NC effect, and reduce the bit rate.

[0038] Here, we will explain the following three points regarding the phenomena and effects assumed: Driver position and NC effect Minimum audibility limit and simultaneous masking Spatial masking

[0039] First, we will discuss the driver position and NC effect. Figure 4 shows the experimental results to demonstrate the driver selection and NC performance in multi-driver headphones. The experiment used multi-driver headphones equipped with four drivers, numbered No. 1 to No. 4.

[0040] The horizontal axis of Figure 4 is frequency, and the vertical axis is sound pressure level, with the downward direction of the vertical axis representing quieter conditions, i.e., a higher NC effect. Measurement result C1 is the sound pressure at the eardrum position when headphones are not being worn. Measurement result C2 is the sound pressure at the eardrum position when headphones are only being worn (NC processing off). Measurement result C3 shows the case where an NC signal is output using driver No. 1, measurement result C4 shows the case where an NC signal is output using driver No. 2, measurement result C5 shows the case where an NC signal is output using driver No. 3, and measurement result C6 shows the case where an NC signal is output using driver No. 4.

[0041] Measurement result C6 has a high NC effect in the low range, but the NC effect decreases above 4 kHz. On the other hand, measurement result C5 has a particularly high NC effect in the 1 to 6 kHz range, although not as high as measurement result C6. Measurement results C3 and C4 also have characteristics that differ from the other measurement results.

[0042] As shown in the measurement results C3, C4, C5, and C6, where NC processing was performed with different drivers, it can be seen that with regard to noise cancellation in multi-driver headphones, there are frequency bands where the NC effect is high and low depending on the driver that outputs the NC signal. This is due to the placement of each driver and the characteristics up to the eardrum. In other words, by selecting and combining the drivers that output the NC signal, it is possible to select which frequency bands to focus noise cancellation processing on, which can be said to be an advantage of noise canceling in multi-driver headphones.

[0043] Next, we will explain the minimum audibility limit and simultaneous masking (frequency direction). The minimum audibility limit indicates the sound pressure level at the limit that humans can hear in each frequency band, and sounds lower than the minimum audibility limit cannot be heard. Simultaneous masking is a phenomenon in which, when a certain frequency component (F1) reaches the eardrum and is heard, a nearby frequency component (F2) is masked by the frequency component (F1) and cannot be heard unless it is somewhat larger than the frequency component (F1).

[0044] Considering these points, quantization errors can be tolerated in the parts that correspond to the masking accuracy and parts that are below the minimum audible limit, and the transmission bit rate can be reduced.

[0045] This will be explained using Figure 5. The horizontal axis represents frequency, and the vertical axis represents amplitude. The figure shows the minimum audible limit 40 with a dashed line. The figure also shows frequency components 20, 21, 22, and 23 of sounds occurring simultaneously. The masking levels 30, 31, 32, and 33, which are masked by each frequency component 20, 21, 22, and 23 simultaneously, are shown with dashed lines. Sounds lower than the masking level 30 are masked by the sound of frequency component 20. The masking level 30 spreads out like an umbrella, with the frequency of frequency component 20 as its peak, to other frequencies. In other words, sounds close to frequency component 20 are easily masked, even if they are relatively loud, and the further away the frequency is, the less likely they are to be masked. The masking level 31 of frequency component 21, the masking level 32 of frequency component 22, and the masking level 33 of frequency component 23 also exhibit a similar tendency.

[0046] In the example of Figure 5, the level of frequency component 22 is below the minimum audible limit. Frequency component 21 is below the masking level 30 of frequency component 20, meaning it is masked by frequency component 20. Frequency component 23 is masked by frequency component 20 in region 23M. Frequency component 20 is below the minimum audible limit in region 20M. In other words, the blacked-out regions 20M and 23M and the entire frequency components 21 and 22 in the figure are audio components that are masked or below the minimum audible limit, and it can be considered that highly accurate information is not required for these portions.

[0047] In this way, by determining the masking state of the generated sound, it is possible to determine the area where quantization accuracy is not required. Therefore, when the determination unit 4 transmits information on the channel and band for reducing the number of quantization bits in the content sound data CT to the host device 100 based on the determination of the masking state, the host device 100 controls the quantization process of the content sound data CT, thereby reducing the transmission bit rate.

[0048] Below, various cases will be shown, similar to those in Fig. 5. Fig. 6 shows the state when noise is added to Fig. 5. For example, assume that single-frequency noise 24 is added to an environment in which content sound having frequency components 20, 21, 22, and 23 as shown in Fig. 5 is being listened to. In the case of Fig. 6, the noise 24 exceeds the level at which it is simultaneously masked by frequency component 20, and therefore degrades the content sound. In such a case, NC processing is performed with the aim of canceling the noise 24 to a level below that at which it is simultaneously masked.

[0049] Like Fig. 6, Fig. 7 also shows a case where single-frequency noise 24 is added to an environment where content sound having frequency components 20, 21, 22, and 23 as shown in Fig. 5 is being listened to. However, in the case of Fig. 7, the noise 24 is originally below the masking level 30 caused by the frequency component 20 of the content sound. In such a case, there is no need to perform NC processing.

[0050] 8 shows an example in which the residual noise 24 from the NC processing exceeds the masking level 30 at which the content sound is simultaneously masked. The noise 24, in this case the residual noise from the NC processing, deteriorates the accuracy of the lower bits of the content sound. That is, this is the portion shown by the area below the masking level 34 from the noise 24 in frequency components 20, 21, 22, and 23. The quantization error in this area is buried in the residual noise 24 from the NC processing, and is therefore subject to a reduction in the transmission bit rate.

[0051] The example in Fig. 9 shows a case where residual noise 24 from NC processing is greater than the level of the content sound. The portions of frequency components 20, 21, 22, and 23 that are below masking level 34 caused by noise 24 are buried in noise 24. In other words, the allowable range of quantization error that is subject to bit rate reduction is wider than in the case of Fig. 8.

[0052] Next, spatial masking will be described. FIG. 10 shows the results of a spatial masking experiment. From the user's (listener's) perspective, the front direction is defined as 0 degrees. FIG. 10 shows the results of an experiment in which the masker was positioned at 0 degrees and the maskee was moved in 30-degree increments. As shown by curve 41, it can be seen that the volume at which the maskee becomes inaudible (is masked) tends to vary depending on the angle. At the 90-degree position, compared to 0 degrees (where the masker and maskee are in the same direction), it can be seen that masking is not possible unless the masker is about 6.4 dB louder, and the results show that masking is difficult when there is an angle difference.

[0053] In this embodiment, the three phenomena and effects of the driver position and NC effect, the minimum audible limit and simultaneous masking, and spatial masking are utilized. Cases 1 to 4 will be described.

[0054] 11 to 26, the content sound AC, noise AN, residual noise AN (NC), and NC sound ANC are each indicated by an arrow. Each arrow indicates the direction from which the sound reaches the eardrum 201, and the thickness of the arrow indicates the loudness of the sound.

[0055] 11 to 26, in order to prioritize readability, the reference numerals for the drivers 2 (2A, 2B, ... 2(N)) and microphones 3 (3A, 3B, ... 3(M)) are omitted. It should be understood that the configurations of the drivers 2 and microphones 3 are the same as those in FIGS. 2 and 3.

[0056] [Case 1: When the Arrival Directions of Noise and Content Sound Match] Fig. 11 shows an example where the arrival directions of noise AN and content sound AC match, with noise AN acting as a masker. For the purpose of explaining the basic concept of the embodiment, it is assumed that the content sound is one channel, the noise is unidirectional, and the magnitude relationship is the same across all bands.

[0057] FIG. 12 shows that noise AN masks content sound AC. By performing NC processing to counter the masking of content sound AC by noise AN, the content sound AC can be heard more clearly. Basically, performing NC processing using a driver 2 located in the same direction as the noise arrival direction is effective for the NC effect. Therefore, the NC sound ANC is output by selecting a driver according to the noise arrival direction. FIG. 13 shows that NC sound ANC is output from a driver 2 in the noise arrival direction.

[0058] However, it is not possible to completely cancel the noise AN that reaches the eardrum due to the NC sound. Figure 14 shows that there is residual noise AN (NC). Note that in Figure 14, it is shown as if the noise AN is canceled from the position of the driver 2, but this is for convenience of illustration, and the noise that is actually canceled is at the position of the eardrum 201. The same applies to the illustration of residual noise AN (NC) in other figures. The following cases (A) and (B) are assumed regarding residual noise AN (NC).

[0059] (A) When the residual noise AN(NC) is greater than the minimum audible level and the quantization noise of the content signal, the residual noise will bury the sound of the lower quantized bits of the content sound AC due to a simultaneous masking effect. Therefore, the quantization bits of the content sound data CT are reduced.

[0060] (B) When the residual noise AN (NC) is reduced to the point where it is imperceptible, or when it is masked by the content sound AC, the allocated bits are maximized for the transmission of the content sound data CT, but bit reduction is performed taking into account masking due to the frequency characteristics of the content sound data CT itself.

[0061] Next, we will discuss the case where the noise AN and content sound AC are coming from the same direction and the content sound AC acts as a masker. Figure 15 shows that the noise AN is masked by the content sound AC. When the level of the content sound AC is high and masks the noise AN, the noise AN is not perceived by the listener, so NC processing is not necessary. By maximizing the allocated bits in the transmission of the content sound data CT, we can achieve a level of detail that is perceptible to the user.

[0062] [Case 2: When noise and content sound arrive from different directions] Fig. 16 shows a case where noise AN and content sound AC arrive from different directions. As in the first case, the content sound is one channel, the noise is also unidirectional, and the magnitude relationship is the same across all bands.

[0063] First, consider the case where the noise AN level is high. Due to the spatial masking effect, the content sound AC and the noise AN are each easily audible, making the noise AN stand out. Alternatively, if the difference in volume is large enough, the noise AN completely masks the content sound AC.

[0064] Therefore, the content signal can be made easier to hear by outputting the NC sound ANC using a driver 2 that is suited to the direction and characteristics of the noise AN. For example, as shown in Figure 17, it is usually assumed that the NC sound ANC is output using a driver 2 that is positioned in the same direction as the direction of arrival of the noise AN.

[0065] However, due to differences in the arrival direction, the residual noise AN (NC) shown in FIG. 18 tends to be more audible. In other words, this is because the masking effect tends to be small due to the effect of spatial masking. As a result, the degradation of the content sound AC due to bit reduction of the content sound data CT is easily noticeable. Therefore, spatial masking is added to the simultaneous masking described in FIGS. 5 to 9 for the determination. For example, it is possible to think of the masking levels (30, 31, 32, 33, 34) shown in FIGS. 5 to 9 as shifting up or down depending on the angular difference in the arrival direction.

[0066] [Case 3: When the direction of arrival of noise(s) is included in the direction of arrival of content sound(s)] Fig. 19 shows an example of the third case. As content sound AC, content sounds AC1, AC2, and AC3 are output from each driver 2. Also, noises AN1 and AN2 arrive as noise AN. The directions of arrival of noise AN1 and content sound AC1 match, and the directions of arrival of noise AN2 and content sound AC2 match.

[0067] When the noise AN level is high, the NC process is executed in the same manner as in the first case, and the transmission bits of the content sound data CT are determined according to the residual noise AN (NC) remaining after the NC process.

[0068] When multiple noises AN and content sounds AC are present, the relationship between the masker and the maskee may differ for each direction, as shown in Fig. 20. In the example of Fig. 20, noise AN1 masks content sound AC1. On the other hand, content sound AC2 masks noise AN2. Since such cases are anticipated, it is determined whether the noise AN or the content sound AC is the masker for each direction of arrival.

[0069] If the level of noise AN is higher than the content sound AC, NC processing is performed on noise AN. For example, when noise AN1 masks content sound AC1, a state in which NC sound ANC is output using driver 2 in the same direction is shown in Figure 21. The residual noise AN(NC) resulting from this NC processing is shown in Figure 22. For the residual noise AN(NC), the processing described as (A) or (B) in the first case can be performed.

[0070] On the other hand, if the level of the content sound AC is so high that it masks the noise AN, there is no need to perform NC processing on the noise AN. For example, Figures 21 and 22 show that noise AN2 is masked by content sound AC2.

[0071] 19, noise AN from one direction may mask content sound AC from another direction. For example, this may occur when noise AN1 is so loud that it masks content sound AC2. NC processing is performed on noise AN1. Fig. 21 shows the NC sound ANC being output from driver 2 in the same direction as noise AN1, and Fig. 22 shows the residual noise AN(NC).

[0072] At this time, the transmission bit allocation for each of the content sounds AC1, AC2, and AC3 is determined depending on how much the remaining noise AN(NC) interferes with the content sounds AC1, AC2, and AC3.

[0073] [Fourth Case: When the Direction from Which Noises (Multiple) Come is Other than the Direction from Which Content Sounds (Multiple) Come] An example of the fourth case is shown in Figure 23. As content sounds AC, content sounds AC1 and AC2 are output from each driver 2. Furthermore, noises AN1, AN2, and AN3 arrive as noise AN. The directions from which noise AN1 and content sound AC1 come are the same, and the directions from which noise AN2 and content sound AC2 come are the same. Noise AN3 is noise that comes from a direction other than the direction from which content sounds AC come.

[0074] In this case, as in the third case, NC processing is performed as necessary, and the transmission bits of the content sound data CT are determined according to the masking effect that occurs. In the example of Fig. 23, noise AN1 masks content sound AC1 as shown in Fig. 24. Also, content sound AC2 masks noise AN2. Therefore, NC processing is performed for noises AN1 and AN3. Fig. 25 shows the state in which NC sound ANC1 is output from the driver 2 in the same direction as noise AN1, and NC sound ANC3 is output from the driver 2 in the same direction as noise AN3. Fig. 26 shows the residual noises AN1 (NC) and AN3 (NC).

[0075] At this time, the transmission bit allocation for each of the content sounds AC1, AC2, and AC3 is determined depending on how much the remaining noises AN1(NC) and AN3(NC) interfere with the content sounds AC1, AC2, and AC3.

[0076] 1 (masking determination unit 4a and acoustic processing control unit 4b) controls the NC processing in the NC signal generation unit 6 based on the relationship between the characteristics and direction of noise AN, and the characteristics of content sound AC and the position of the driver 2, as in the first to fourth cases described above. In other words, it controls whether or not to perform NC processing for each characteristic of noise AN. The determination unit 4 also determines how many quantization bits are sufficient to allocate for the transmission of content sound data CT based on the minimum audibility limit and masking effect, and performs processing to transmit a request to the host device 100.

[0077] An example of the processing performed by the determination unit 4 is shown in Figures 27 and 28. In Figures 27 and 28, "CN1" and "CN2" indicate connections in the flowcharts.

[0078] While the headphones 1 receive the content sound data CT and output the content sound AC from the driver 2, the determination unit 4 repeats the processes in Figures 27 and 28. Step S101 is a step for determining whether the repeat loop has ended. The processes in Figures 27 and 28 are ended, for example, by power-off, a change in the operating mode, etc.

[0079] During execution of the loop processing, in step S102, the determination unit 4 analyzes the frequency characteristics and direction of arrival of the ambient sound, i.e., noise AN, obtained by the microphone 3. In addition, in step S103, the determination unit 4 analyzes the frequency characteristics of the content sound data CT. Note that the determination unit 4 can determine the direction from which the content sound AC is coming from for the user, i.e., which sound component is output from which driver 2, based on the channel number of the content sound data CT.

[0080] In step S110, it is determined whether to continue or end the processing loop from step S111 to step S118. The processing from step S111 to step S118 is performed for each direction of arrival. For example, the processing from step S111 to step S118 is performed for each of the first to Nth directions in accordance with the number of channels of the driver 2. When this processing has been completed for all directions of arrival, the loop ends.

[0081] In step S111, the determination unit 4 compares noise AN from a certain direction with the minimum audible limit. The level of noise AN also takes into account the amount of noise blocked by the housing of the headphones 1. If (noise AN) < (minimum audible limit), that is, if all of the frequency components that make up noise AN are below the minimum audible limit, the determination unit 4 proceeds to step S115 and determines that NC processing is not necessary for that noise AN. Then, in step S116, the determination unit 4 sets the inaudibility flag to ON for the noise from the direction of arrival that is currently being processed.

[0082] If (noise AN)<(minimum audibility limit) is not true, the determination unit 4 proceeds to step S112 to determine whether or not content sound AC is present from the direction of arrival. If content sound AC is not present, the determination unit 4 proceeds to step S117 to turn on NC processing. The determination unit 4 also sets the driver 2 to output NC sound ANC. Then, in step S118, the determination unit 4 turns off the inaudible flag for the noise from the direction of arrival that is currently being processed.

[0083] If it is determined in step S112 that content sound AC is present from the direction of arrival, the determination unit 4 compares the level of noise AN with the level of content sound AC output based on content sound data CT in step S113. If it is determined that the level of noise AN is higher than the content sound AC, the determination unit 4 performs the processes of steps S117 and S118 described above.

[0084] If it is determined that the level of the noise AN is equal to or lower than the content sound AC, the determination unit 4 determines in step S114 whether the content sound AC masks the noise by simultaneous masking. If the content sound AC does not mask the noise AN, the determination unit 4 performs the processes of steps S117 and S118 described above. If the content sound AC masks the noise AN, the determination unit 4 performs the processes of steps S115 and S116 described above.

[0085] By performing the above process for each direction of arrival, the NC processing setting is determined for each direction of arrival. That is, the inaudible flag is set to OFF for directions that require NC processing, and the inaudible flag is set to ON for directions that do not require NC processing. Then, the determination unit 4 ends the loop in step S110 and proceeds to step S120 in FIG. 28.

[0086] In step S120, it is determined whether to continue or terminate the loop of processing from step S121 to step S125. The processing from step S121 to step S125 is also performed for each direction of arrival.

[0087] In step S121, the determination unit 4 checks the inaudible flag for one direction of arrival that is the processing target. If the inaudible flag is set to ON, i.e., NC processing is not required, the process returns to step S120 for that direction of arrival and moves on to processing the next direction of arrival.

[0088] If the inaudible flag is OFF in step S121, the decision unit 4 proceeds to step S122 to estimate the frequency characteristics and level of the residual noise AN(NC) of the NC processing, and then compares the residual noise AN(NC) with the minimum audible limit in step S123.

[0089] If (residual noise AN(NC))<(minimum audible limit), that is, if all of the frequency components constituting the residual noise AN(NC) are below the minimum audible limit, the determination unit 4 proceeds to step S125 and sets the inaudible flag for the residual noise AN(NC) to ON.

[0090] If (residual noise AN(NC))<(minimum audibility limit) is not true, the determination unit 4 proceeds to step S124 to determine whether the content sound AC masks the residual noise AN(NC) by simultaneous masking. If masking is determined, the determination unit 4 proceeds to step S125 to set the inaudible flag for the residual noise AN(NC) to ON. If it is determined that the residual noise AN(NC) is not masked, the inaudible flag remains OFF and the process returns to step S120.

[0091] By performing the above processing from step S121 to step S125 for each direction of arrival, the residual noise AN(NC) is estimated for each direction, and if the residual noise AN(NC) is below the minimum audible limit or is masked by the content sound AC, the inaudible flag for that direction of arrival is changed to ON.

[0092] After completing the above loop processing, the decision unit 4 proceeds from step S120 to step S130. In step S130, the decision unit 4 checks whether the inaudible flag is on for all directions of arrival.

[0093] If the inaudible flag is on for all directions of arrival, the determination unit 4 determines the required quantization bits for all channels of the content sound data CT in step S137. In this case, since high-precision content sound data CT is required, the maximum allocation of the number of quantization bits is required.

[0094] If it is confirmed in step S130 that there is an arrival direction for which the inaudible flag is off, the determination unit 4 proceeds to step S131, and performs the process of step S132 for each arrival direction for which the inaudible flag is off. In step S132, the determination unit 4 calculates the spatial masking effect that the direction has on other directions. This allows the spatial masking effect that the direction has on other directions to be determined for one or more arrival directions for which the inaudible flag is off.

[0095] In step S133, the determination unit 4 determines whether to continue or end the loop for each direction of arrival. That is, the processes from step S134 to step S136 are performed for each direction of arrival. In step S134, the determination unit 4 determines whether or not there is spatial masking that affects the direction of arrival being processed.

[0096] If the direction of arrival being processed is not affected by spatial masking, the determination unit 4 proceeds to step S135 to determine the required quantization bits for the content sound data CT for that direction. In this case, since high-precision content sound data CT is required, the maximum allocation of the number of quantization bits is required.

[0097] If the direction of arrival being processed is affected by spatial masking, the determination unit 4 proceeds to step S136 to determine the required quantization bits for the content sound data CT for that direction. In this case, a reduction in the number of quantization bits is requested because there is an area where high-precision information is not required due to masking.

[0098] In the loop starting from step S133, the number of quantization bits is set for each direction of arrival in either step S135 or step S136.

[0099] In step S135, S136, or step S137, the number of quantization bits is set for each direction. By associating each direction to be the target of the loop processing with each channel, the number of quantization bits for each channel is set. In step S140, the determination unit 4 transmits notification information SS to the host device. In this case, the notification information SS includes information on the number of quantization bits required for each channel.

[0100] After transmitting the notification information SS in step S140, the determination unit 4 returns to step S101 in Fig. 27. Then, the above processing is repeated.

[0101] As described above, in the first embodiment, the determination unit 4 performs the processes shown in Figures 27 and 28, thereby reducing the transmission bits of content sound data depending on the situation and realizing a reduction in the transmission bit rate. Also, by determining which driver 2 in the headphones 1 should be used to cancel noise AN, or which driver 2 does not require NC processing, and successively changing the settings, it is possible to achieve a more comfortable S / N ratio of the content sound AC, improvements in sound separation, and improvements in the NC effect.

[0102] 2. Second Embodiment In the second embodiment, a process for making a user aware of ambient sounds will be mainly described. In the first embodiment described above, ambient sounds are treated as noise AN and necessary processes are performed, but in the second embodiment, necessary processes are performed on ambient sounds that the user wants to be aware of.

[0103] Specific examples of such ambient sounds include the following: - Sounds such as cars approaching from behind or the side - Sounds (footsteps, etc.) approaching the room of the user (the user wearing the headphones 1) - Announcements (public transportation announcements and announcements from various public facilities, etc.) - Alerts and sirens (sounds of emergency vehicles, emergency earthquake alerts, etc.) - Voices calling the user (the user wearing the headphones 1).

[0104] FIG. 29 shows an example of the configuration of headphones 1. However, the same parts as in FIG. 1 are assigned the same reference numerals to avoid redundant description. What differs from FIG. 1 is the provision of an ambient sound signal processing unit 8. The ambient sound signal processing unit 8 processes ambient sound data S1 obtained by the microphone 3 under the control of the acoustic processing control unit 4b of the determination unit 4. For example, the ambient sound data S1 is subjected to noise reduction processing, voice enhancement processing, etc., and the processed sound data S3 is output. Alternatively, the ambient sound signal processing unit 8 may perform processing to generate sound data S3 such as beeps or voice announcements.

[0105] The sound data S3 that has been signal processed or generated by the ambient sound signal processing unit 8 is supplied to the output signal generation unit 7. The output signal generation unit 7 generates an output signal to the driver 2 in accordance with a specified channel based on the sound data S3, together with the content sound data CT and the NC sound data S2.

[0106] The ambient sound type determination unit 5 performs processing to determine the type of sound from the ambient sound data obtained by the microphone 3. For example, it determines the specific type of sound, such as the sound of an approaching car, footsteps, or the sound of an announcement from a train or the like. Note that the type determination does not necessarily involve determining the specific sound, but may instead determine whether or not the sound is to be treated as noise.

[0107] The determination unit 4 receives the ambient sound data S1 and type information of the ambient sound data S1, and performs processing by the masking determination unit 4a and the sound processing control unit 4b. As in the first embodiment, the masking determination unit 4a determines the masking state in the relationship between the ambient sound data S1 and the content sound data CT. In this case, it also determines whether the ambient sound required by the type information is masked by the content sound AC.

[0108] The acoustic processing control unit 4b controls acoustic processing in accordance with the masking state determination result by the masking determination unit 4a. For example, the acoustic processing control unit 4b outputs a control signal to the NC signal generation unit 6 in accordance with the masking state to control NC operation. Control of NC operation includes turning NC processing on / off and selecting the driver 2 that outputs the NC signal.

[0109] In addition, the acoustic processing control unit 4b controls the driver 2 to output a sound that makes the driver aware of the ambient sound, depending on the type of sound in the ambient sound data S1 and the result of the masking state determination.In this case, the acoustic processing control unit 4b selects a channel for the driver 2 that outputs a sound that makes the driver aware of the ambient sound, depending on the direction from which the ambient sound is coming.

[0110] Furthermore, the acoustic processing control unit 4b controls the ambient sound signal processing unit 8 so that a sound based on the ambient sound obtained by the microphone 3, i.e., a sound obtained by signal processing the ambient sound data S1, is output from the driver 2 as a sound that causes the driver 2 to recognize the ambient sound. Alternatively, the acoustic processing control unit 4b controls the ambient sound signal processing unit 8 so that a generated sound that causes the driver 2 to recognize the ambient sound is output from the driver 2.

[0111] Furthermore, the acoustic processing control unit 4b performs processing to transmit notification information SS to the host device 100, which is an external device, in accordance with, for example, the masking state determination result. In the second embodiment, the acoustic processing control unit 4b transmits, as the notification information SS, information used for displaying the ambient sound to make the user aware of the ambient sound. The information used for displaying the ambient sound to make the user aware of the ambient sound may include some or all of information on the direction from which the ambient sound is coming, information on the type of the ambient sound, and information on the masking state determination result.

[0112] In the second embodiment, the following process is performed: Among the ambient sounds, there are sounds that the user should be aware of, as in the example above. Therefore, the ambient sound type determination unit 5 determines the type of the sound.

[0113] The determination unit 4 (masking determination unit 4a) determines whether the ambient sound data S1 is a sound that should be recognized by the user based on the type information, and if it is a sound that should be recognized, determines whether that sound is masked by the content sound AC. It also determines whether the sound will be canceled by the NC process.

[0114] If necessary ambient sounds are masked or canceled as noise, the user will not be able to perceive them as they are, so the determination unit 4 (sound processing control unit 4b) performs processing to make the user perceive the ambient sounds.

[0115] One process for making the user aware of ambient sounds is to output sound. For example, the determination unit 4 causes the ambient sound signal processing unit 8 to perform processes on the ambient sound data S1 to make it easier to hear, such as noise reduction and voice enhancement, and then causes the driver 2 to output sound based on the processed sound data S3. In this case, the determination unit 4 may instruct the driver 2 to output sound based on the direction from which the ambient sound is coming. This allows the user to hear the ambient sound itself from the actual direction from which the ambient sound is coming when listening to the content sound AC.

[0116] As a method for outputting sound, instead of the ambient sound itself, a sound indicating an alert such as a beep or a message sound may be output. For example, the determination unit 4 causes the ambient sound signal processing unit 8 to execute a sound data generation process and cause the driver 2 to output a sound based on the generated sound data S3, such as a beep or a message sound.

[0117] This allows the user to recognize that some necessary ambient sound is occurring when listening to the content sound AC. In this case, too, the determination unit 4 may instruct the driver 2 to output based on the direction from which the ambient sound is coming. This allows the user to recognize that necessary ambient sound is coming from the direction from which the beep sound or the like is coming. The beep sound is suitable in cases where it is only necessary to be notified of the presence of ambient sound.

[0118] It is also possible to consider selecting a driver 2 for a channel that is not affected by spatial masking by the content sound AC as the driver 2 that outputs the ambient sound itself, beeps, and message sounds. This can improve the recognizability of the necessary ambient sounds.

[0119] Furthermore, while the user can recognize the type of sound by outputting the ambient sound itself, the type of sound may not be recognizable with a beep or a message sound. Therefore, the message content may be changed or the sound quality and volume of the beep may be changed depending on the type and urgency of the sound. This can increase the alertness and allow the user to recognize the type of ambient sound. Examples of message sounds may include specific content such as "A car is approaching from behind" or "Someone is approaching the room."

[0120] It is also possible to combine this with GPS location information, change the criteria based on the size of the road you are walking (or running) on, or the number of cars passing by, and issue a notification based on the result.

[0121] To recognize the ambient sound, instead of or in addition to the notification by sound output from the headphones 1 as described above, notification may be made using a display on the host device 100. That is, the determination unit 4 transmits the determination result to the host device 100, such as a smartphone or HMD, so that the host device 100 notifies the user.

[0122] When a user is watching game content, video content, or the like, the user is gazing at the screen of the host device, so it is preferable to notify the user of ambient sounds by displaying a message on the screen, for example.

[0123] 30 shows an example in which a message 61 is displayed on a screen 60 of the host device 100. The display position of such a message 61 may be changed depending on the direction from which the ambient sound is coming. For example, the display position of the message 61 saying "A car is approaching" may be controlled so that it is displayed at the bottom of the screen as shown in FIG. 30 if a car is approaching from behind, at the top or center of the screen if a car is approaching from the front, on the left side of the screen if a car is approaching from the left, or on the right side of the screen if a car is approaching from the right.

[0124] There is also a technique of displaying an effect image 62 on a screen 60, as shown in Figures 31 and 32. In this case, too, it is possible to display the effect image 62 according to the direction from which the ambient sound is coming. Figure 31 shows the case where sound is coming from the left, and Figure 32 shows the case where sound is coming from the front. The size of the effect image 62 may be used to represent the volume of the ambient sound.

[0125] These displayed notifications allow users to respond to ambient sounds while watching videos or playing games such as virtual reality (VR). For example, if footsteps approaching one's room are masked and cannot be heard, this processing can be performed, allowing the user playing the game to take appropriate action, such as pausing the game, based on the notification.

[0126] In addition to the above-described visual notification, the host device 100 or the headphones 1 may notify the user by vibration or the like. The visual notification and the above-described sound notification in the headphones 1 may be used in combination.

[0127] 33, 34, and 35 show examples of screen displays that notify the user of more detailed ambient sound conditions. Each display shows an image of a space centered around the user's head as spatial coordinates 50. Using this spatial coordinates 50 as a reference, a content sound image 51, ambient sound type images 55 and 57, and effect images 56 and 58 are displayed at positions according to the direction from which the sound is coming.

[0128] The content sound image 51 is an image showing the type and masking range of the content sound AC. In the case of Fig. 33A, the sound of an instrument such as a violin is localized to the left of the user, and the range in which other sounds are masked by the sound of the instrument is shown in a circular shape.

[0129] The ambient sound type images 55, 57 are images that indicate the type of ambient sound, such as an image of a car or an image of footprints indicating footsteps. The effect images 56, 58 indicate ambient sounds. The size of the ambient sound type images 55, 57 and the effect images 56, 58 indicates the amount of masking in the ambient sound. The display position of the ambient sound type images 55, 57 and the effect images 56, 58 also indicates the direction from which the ambient sound is coming.

[0130] The screen 60 also displays a setting section 53. The setting section 53 is an operation section that allows the user to arbitrarily set the notification function to on or off. For example, the setting section 53 has setting fields for "ambient sound extraction ON / OFF," "car ON / OFF," and "footsteps ON / OFF."

[0131] Fig. 33A shows a state in which ambient sound extraction is turned off. In this case, a content sound image 51 is displayed. Fig. 33B shows a state in which ambient sound extraction is turned on and automobile is selected as the type of ambient sound. When automobile is selected, a slide bar 54 is displayed. The user can set the extraction level using the slide bar 54.

[0132] When a car sound is detected, an ambient sound type image 55 and an effect image 56 are displayed according to the direction from which the car sound is coming and the amount of masking, as shown in Fig. 33B. In this case, it is shown that the car sound is coming from the same direction as the content sound and is being masked to a certain extent.

[0133] FIG. 34A shows a case where the content sound has moved. For example, when the localization of the content sound data CT is changed based on the original content, a user may change the localization of the musical instrument through operation. Alternatively, the determination unit 4 may request the host device 100 to automatically change the localization of the musical instrument. For example, the determination unit 4 requests the host device 100 to change the localization of the content sound in response to the masking of the sound of a car. This causes the host device 100 to change the channel of the content sound data CT and change the localization. When the localization is changed as shown in FIG. 34A, the amount of masking due to the spatial masking effect is reduced, making it easier for the user to hear the sound of the car. As the amount of masking is reduced, the size of the ambient sound type image 55 and the effect image 56 is reduced as shown in FIG. 34B.

[0134] Fig. 35A shows the case where footsteps are set as the extraction target in setting unit 53. In this case, too, turning footsteps on displays slide bar 59, allowing the user to set the footstep extraction level using slide bar 59. Fig. 35B shows the case where footsteps arriving from the right are detected. In response, an ambient footstep sound type image 57 and an effect image 58 are displayed on the right. This example shows a state in which the footsteps are masked to some extent by the content sound.

[0135] By displaying the images shown in FIGS. 33 to 35, the user can recognize the ambient sounds and their masking states in more detail.

[0136] The above assumes stereophonic content, but in games, for example, the position of the sound source moves automatically. This is followed in real time, the masking effect is analyzed, and NC processing and the transmission bit rate of the content sound data CT are appropriately set. A combination of an HMD as the host device 100 and headphones 1 has high compatibility because the directions of the user's vision and hearing are aligned. When the user turns their head, the position of the headphones 1 (microphone 3) naturally changes at the same time, so the movement of the ambient sound source as seen by the user can be accommodated in real time by changes in the signal from the microphone 3.

[0137] Although FIGS. 33 to 35 show images in which stereophonic content is present, the displays shown in FIGS. 33 to 35 may also be produced for content that is not stereophonic, or for a silent state in which there is no content.

[0138] Up to this point, an example of notifying the user of necessary ambient sounds by voice or display has been described as the second embodiment. An example of processing by the determination unit 4 that realizes such an operation is shown in FIG.

[0139] While the headphones 1 receive the content sound data CT and output the content sound AC from the driver 2, the determination unit 4 repeats the process of Fig. 36. Step S201 is a step for determining whether the repeat loop has ended. The process of Fig. 36 ends, for example, when the power is turned off or the operating mode is changed.

[0140] During the loop period, in step S202, the determination unit 4 analyzes the frequency characteristics and direction of arrival of the ambient sound, i.e., noise AN, obtained by the microphone 3. The determination unit 4 also determines the type of sound based on type information from the ambient sound type determination unit 5.

[0141] In step S203, the determination unit 4 analyzes the frequency characteristics of the content sound data CT. The direction from which the content sound AC comes can be determined by the channel number of the content sound data CT.

[0142] In step S204, the determination unit 4 determines whether or not there is any ambient sound that the user should be aware of. If the result of the ambient sound type determination indicates that there is a sound such as the sound of a car, footsteps, an announcement, or an alert, the determination unit 4 determines that the sound that should be aware is present and determines its frequency characteristics and direction of arrival.

[0143] In step S205, the determination unit 4 determines the masking state and whether or not NC processing is being performed. As in the first embodiment, the determination unit 4 determines the masking state based on the relationship between the ambient sound and the content sound. Then, it determines whether or not NC processing is to be performed accordingly.

[0144] The determination unit 4 also determines whether to turn on or off NC processing depending on whether or not the ambient sound includes a sound that the user should recognize. For example, if the ambient sound does not include a sound that the user should recognize, the determination unit 4 determines that normal NC processing should be performed on the ambient sound. If the ambient sound includes a sound that the user should recognize, the determination unit 4 determines that NC processing should be performed on at least frequency components other than the sound in question, and that NC processing should not be performed on the sound that the user should recognize. In this case, it may be determined that normal NC processing should be performed. For example, when a beep or a message sound is generated, NC processing for the ambient sound may always be performed normally. If the ambient sound includes only sounds that the user should recognize (or if the sound that the user should recognize is dominant), it is determined that NC processing should not be performed. In this case, it may also be determined that normal NC processing should be performed, as described above.

[0145] In step S206, the determination unit 4 controls the NC signal generation unit 6 based on the determination result of step S205. For example, the determination unit 4 instructs the NC processing to be turned on or off. In some cases, the determination unit 4 may perform control such that the NC processing is disabled for only specific frequency components. The determination unit 4 also instructs the channel of the driver 2 that outputs the NC sound.

[0146] In step S207, the determination unit 4 branches the process depending on whether or not a notification to the user is to be made by a display on the host device 100. For example, if the ambient sounds include sounds that the user should be aware of and notification by display is set to on, the process proceeds to step S220, where notification information SS is transmitted to the host device 100. This notification information SS includes information on the type of ambient sound, the direction of arrival, and the level of masking, which enables the host device 100 to display the information as described in Figs. 30 to 35.

[0147] In step S208, the determination unit 4 branches the process depending on whether or not a sound notification is to be performed. For example, if the ambient sound includes a sound that the user should recognize, the determination unit 4 proceeds to step S210 and branches the process depending on what type of notification is to be turned on.

[0148] If the ambient sound itself is to be output, the determination unit 4 proceeds to step S211 and instructs the ambient sound signal processing unit 8 to perform noise reduction processing, voice enhancement processing, etc. on the ambient sound data S1. The determination unit 4 also instructs the channel of the driver 2 to output the ambient sound according to the direction of arrival.

[0149] If a notification sound is to be generated, the determination unit 4 proceeds to step S212 and instructs the ambient sound signal processing unit 8 to generate a beep or a message sound. It also instructs the channel of the driver 2 to output according to the direction of arrival.

[0150] For example, by repeating the process of FIG. 36, the operation of notifying the surrounding sound by voice or display as described above is realized.

[0151] 3. Summary and Modifications According to the first and second embodiments described above, the following effects can be obtained.

[0152] The signal processing device of the embodiment is realized as a processor or the like having the functions of a determination unit 4 including a masking determination unit 4a and an acoustic processing control unit 4b. The acoustic output device of the embodiment is realized as headphones 1 including such a determination unit 4. In these, the masking determination unit 4a determines the masking state between the content sound and the ambient sound using multi-channel content sound data CT output from multiple drivers 2 arranged in the headphones 1, ambient sound data S1 obtained by multiple microphones 3 arranged in the headphones 1 that collect ambient sound, and information on the direction from which the ambient sound is coming. The acoustic processing control unit 4b controls acoustic processing according to the masking state determination result by the masking determination unit 4a. This allows appropriate control of acoustic processing to be performed in cases such as when the content sound AC is masked by ambient sound (noise AN) or when necessary ambient sound is masked by the content sound. In particular, by determining the masking state taking into account the level and direction of arrival of the ambient sound, the channel of the content sound (information on which output position the content sound is output from), and the level of each channel, the masking situation in so-called multi-microphone / multi-driver headphones can be more appropriately determined. Therefore, when playing stereophonic content such as 3D audio, it is possible to control appropriate sound processing in accordance with highly accurate masking status determination, which makes it possible, for example, to enable the user to recognize necessary ambient sounds, enable comfortable listening to content sounds through appropriate NC processing, and reduce the load on system processing.

[0153] In the second embodiment, an example was described in which control is performed to output sounds from the driver 2 that make the user aware of the ambient sounds, depending on the type of sound in the ambient sound data S1 and the result of the masking state determination (see FIG. 36). For example, to a user enjoying stereophonic content, not all ambient sounds are noise. For example, there are sounds that are necessary for safety or daily life. Therefore, if it is determined that the ambient sounds are necessary based on their type, the ambient sounds are output from the driver 2 and the user is made aware of them. This allows the user to enjoy stereophonic sound in a state where the ambient sounds can also be heard appropriately.

[0154] In the second embodiment, an example was given in which the driver 2 that outputs the sound that makes the user aware of the ambient sound is determined according to the direction from which the ambient sound is coming. By determining the driver 2 (i.e., determining the channel) in this way, the user is able to recognize that the sound is coming from the direction corresponding to the determined channel. Therefore, the user can hear the ambient sound itself, a notification sound or a message voice that replaces the ambient sound, and also recognize the actual direction from which the ambient sound is coming. The direction from which the ambient sound is coming can be tracked in real time according to the user's own actions and movements by constantly analyzing the ambient sound data S1 collected by the microphone 3 of the headphones 1.

[0155] In the second embodiment, an example was given in which control is performed such that sound resulting from signal processing of ambient sound data S1 obtained by microphone 3 is output from driver 2 as sound for recognizing ambient sound (step S211 in FIG. 36 ). This prevents necessary ambient sounds from being inaudible due to masking, and allows the user to recognize actual ambient sounds even when listening to stereophonic sound through headphones 1.

[0156] In the second embodiment, an example was also described in which a generated sound for making the user aware of the ambient sound is output from the driver 2 (step S212 in FIG. 36 ). For example, a sound such as a beep or a voice message, which indicates some kind of caution, warning, or announcement, is generated and output. This allows the user to recognize the surrounding situation (a situation in which necessary ambient sound is occurring) even if necessary ambient sound cannot be heard due to masking by the content sound or noise cancellation processing.

[0157] In the second embodiment, an example was given in which the determination unit 4 (acoustic processing control unit 4b) performs processing to transmit notification information SS to be used for a display that makes the user aware of the ambient sound to the host device 100, depending on the type of sound in the ambient sound data S1 and the determination result of the masking state (step S220 in FIG. 36 ). For example, when ambient sound that is determined to be necessary for the user is detected, information to be used for a display that makes the user aware of the ambient sound is transmitted to the host device 100. This causes the host device 100 to execute a display that makes the user aware of the ambient sound, as described with reference to FIGS. 30 to 35 . When viewing stereophonic content accompanied by images, the user is also gazing at the screen, so notifying the user of necessary ambient sounds by display is also effective.

[0158] In the second embodiment, an example was given in which the notification information SS used for displaying the ambient sound includes information about the direction from which the ambient sound is coming, which allows an external device such as the host device 100 to perform a display according to the direction from which the ambient sound is coming (see FIGS. 30 to 35 ).

[0159] In the second embodiment, an example was given in which the notification information SS used for displaying the ambient sound includes information on the type of ambient sound, which allows an external device such as the host device 100 to display information according to the type of ambient sound, such as the sound of a car or footsteps (see FIGS. 31 to 35).

[0160] In the second embodiment, an example was given in which the notification information SS used for displaying the ambient sound includes information on the masking state determination result, which allows the external device such as the host device 100 to display information indicating a state in which the ambient sound is being masked by the content sound (see FIGS. 33 to 35 ).

[0161] In the first and second embodiments, the acoustic processing control unit 4b controls noise cancellation processing for ambient sounds according to the masking state determination result. By performing NC processing on ambient sounds, the ambient sounds can be reduced or eliminated for a user enjoying stereophonic content. However, NC processing does not need to be performed on ambient sounds that are originally masked. Therefore, controlling NC processing according to the masking determination result can improve the efficiency of NC processing. In other words, NC processing can be performed on ambient sounds that are not masked by content sounds due to their frequency components or direction of arrival, and NC processing can be turned off for masked ambient sounds. Furthermore, the acoustic processing control unit 4b can also control NC processing using the sound type of the ambient sound data obtained by the microphone 3. For example, if the acoustic processing control unit 4b determines that the ambient sounds are necessary for the user, NC processing can be turned off to allow the user to hear the ambient sounds.

[0162] In the first and second embodiments, an example was described in which the acoustic processing control unit 4b performs control to determine the driver 2 (i.e., channel) that outputs NC sound for the ambient sound, depending on the masking state determination result and information on the direction from which the ambient sound is coming. When performing NC processing for the ambient sound, the NC effect can be made more effective by determining the driver 2 that outputs NC sound ANC according to the direction from which the ambient sound is coming.

[0163] In the first embodiment, an example was described in which the acoustic processing control unit 4b performs processing to transmit quantization bit information required for the content audio data CT to the host device 100 in accordance with the masking state determination result. With regard to stereophonic reproduction, multi-viewpoint and free viewpoint technologies are facing significant increases in transmission bit rates, making reducing the transmission bit rate of the content audio data one of the important issues. Since information on the masked audio components does not need to be transmitted in the first place, the number of quantization bits can be reduced. Therefore, the quantization bit information required for the content audio data is transmitted to the host device 100 in accordance with the masking state determination result. This allows the host device 100 to reduce the amount of content audio data, resulting in a reduction in the transmission bit rate of the content audio data, an improvement in the S / N ratio of the content signal, an improvement in sound separation, an improvement in the NC effect, and even a longer battery life for the headphones 1 due to reduced power consumption.

[0164] In the first embodiment, the quantization bit information transmitted to the host device 100 includes information on the channel and band for which the number of quantization bits is to be reduced in the content audio data. This allows the host device 100 to reduce the number of quantization bits in a specified band in a specified channel.

[0165] The determination unit 4 (acoustic processing control unit 4b) can also selectively control the power-off of the driver 2. For example, the control is to turn off the power supply to the driver 2 of a channel that does not output NC sound or content sound. By constantly monitoring the driver 2 that is not temporarily used in this way and turning off the power supply, power consumption can be reduced, contributing to extending the battery life of the headphones 1.

[0166] Furthermore, in the embodiment, the headphones 1 have been described as an example of an audio output device, but the technology of the present disclosure can also be applied to audio output devices such as various types of earphones, such as inner-ear type and canal type.

[0167] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0168] The present technology may also be configured as follows. (1) A signal processing device comprising: a masking determination unit that determines a masking state between content sound and ambient sound using content sound data of multiple channels output from multiple audio output drivers arranged in an audio output device, ambient sound data obtained by multiple microphones arranged in the audio output device that collect ambient sound, and information on the direction from which the ambient sound arrives; and an audio processing control unit that controls audio processing in accordance with the masking state determination result by the masking determination unit. (2) The signal processing device described in (1) above, in which the audio processing control unit controls output of a sound that makes the user perceive the ambient sound from the audio output drivers in accordance with the type of sound of the ambient sound data obtained by the microphones and the masking state determination result. (3) The signal processing device described in (2) above, in which the audio processing control unit determines an audio output driver that outputs a sound that makes the user perceive the ambient sound, among the multiple audio output drivers, in accordance with the direction from which the ambient sound arrives. (4) The signal processing device according to (2) or (3) above, wherein the acoustic processing control unit controls so that a sound obtained by signal processing of the ambient sound obtained by the microphone is output from the acoustic output driver as a sound that causes the user to recognize the ambient sound. (5) The signal processing device according to (2) or (3) above, wherein the acoustic processing control unit controls so that a generated sound that causes the user to recognize the ambient sound is output from the acoustic output driver. (6) The signal processing device according to any of (1) to (5) above, wherein the acoustic processing control unit performs processing to transmit to an external device information used for a display that causes the user to recognize the ambient sound, depending on the sound type of the ambient sound data obtained by the microphone and the result of determining a masking state. (7) The signal processing device according to (6) above, wherein the acoustic processing control unit performs processing to transmit to the external device information used for a display that causes the user to recognize the ambient sound, including information on the direction from which the ambient sound is coming. (8) The signal processing device according to (6) or (7) above, wherein the acoustic processing control unit performs processing to transmit to the external device information used for a display that causes the user to recognize the ambient sound, including information on the type of the ambient sound.(9) The signal processing device according to any one of (6) to (8), wherein the acoustic processing control unit performs processing to include information on the masking state determination result in information used for displaying the ambient sound to be recognized, and transmits the information to the external device. (10) The signal processing device according to any one of (1) to (9), wherein the acoustic processing control unit controls noise cancellation processing for the ambient sound according to the masking state determination result. (11) The signal processing device according to any one of (1) to (10), wherein the acoustic processing control unit controls determining, from among the plurality of acoustic output drivers, an acoustic output driver that outputs a noise cancellation signal for the ambient sound according to the masking state determination result and information on the direction from which the ambient sound is coming. (12) The signal processing device according to any one of (1) to (11), wherein the acoustic processing control unit performs processing to transmit quantization bit information required for content audio data to the external device according to the masking state determination result. (13) The signal processing device according to (12), wherein the quantization bit information to be transmitted to the external device includes information on a channel and a band for reducing the number of quantization bits in the content audio data. (14) A sound output device comprising: a plurality of sound output drivers; a plurality of microphones that collect ambient sound; a masking determination unit that determines a masking state between the content sound and the ambient sound using multiple channels of content sound data output from the sound output drivers, ambient sound data acquired by the microphones, and information on the direction from which the ambient sound arrives; and a sound processing control unit that performs control related to sound processing in accordance with a result of the determination of the masking state by the masking determination unit. (15) A signal processing method in which a signal processing device performs the following: a masking determination process that determines a masking state between the content sound and the ambient sound using multiple channels of content sound data output from multiple sound output drivers arranged in the sound output device, ambient sound data acquired by multiple microphones that are arranged in the sound output device and collect ambient sound, and information on the direction from which the ambient sound arrives; and a sound processing control process that performs control related to sound processing in accordance with a result of the determination of the masking state by the masking determination process.

[0169] REFERENCE SIGNS LIST 1 headphones 2, 2A...2(N) acoustic output driver (driver) 3, 3A...3(M) microphone 4 determination unit 4a masking determination unit 4b acoustic processing control unit 5 ambient sound type determination unit 6 NC signal generation unit 7 output signal generation unit 8 ambient sound signal processing unit 100 host device 200 ear 201 eardrum AC content sound AN noise AN(NC) residual noise ANC NC sound S1 ambient sound data S2 NC sound data S3 sound data SS notification information CT content sound data

Claims

1. A signal processing device comprising: a masking determination unit that determines the masking state between content sound and ambient sound using multiple channels of content sound data output from multiple audio output drivers arranged in an audio output device, ambient sound data obtained by multiple microphones arranged in the audio output device that collect ambient sound, and information on the direction from which the ambient sound is coming; and an audio processing control unit that controls audio processing in accordance with the masking state determination result made by the masking determination unit.

2. The signal processing device according to claim 1, wherein the acoustic processing control unit controls the output of a sound from the acoustic output driver that makes the user aware of the ambient sound, depending on the type of sound in the ambient sound data obtained by the microphone and the result of the masking state determination.

3. The signal processing device according to claim 2, wherein the acoustic processing control unit determines, from among the plurality of acoustic output drivers, an acoustic output driver that outputs a sound that makes the user perceive the ambient sound, depending on the direction from which the ambient sound is coming.

4. The signal processing device according to claim 2, wherein the acoustic processing control unit controls the acoustic output driver to output a signal-processed sound obtained by the microphone as a sound that is recognized as an ambient sound.

5. The signal processing device according to claim 2, wherein the acoustic processing control unit controls the acoustic output driver to output a generated sound that makes the user aware of ambient sounds.

6. The signal processing device according to claim 1, wherein the acoustic processing control unit performs processing to transmit to an external device information used for displaying the ambient sound to make the user aware of the ambient sound, depending on the type of sound in the ambient sound data obtained by the microphone and the result of the masking state determination.

7. The signal processing device according to claim 6, wherein the acoustic processing control unit performs processing to include information on the direction from which the ambient sound is coming in information used for displaying the ambient sound and transmit the information to the external device.

8. The signal processing device according to claim 6, wherein the acoustic processing control unit performs processing to include information on the type of ambient sound in information used for displaying the ambient sound to make the user aware of the ambient sound, and transmits this information to the external device.

9. The signal processing device according to claim 6, wherein the acoustic processing control unit performs processing to transmit to the external device information used for displaying the ambient sound to be recognized, including information on the masking state determination result.

10. The signal processing device according to claim 1, wherein the acoustic processing control unit controls noise cancellation processing for ambient sounds according to the result of the masking state determination.

11. The signal processing device according to claim 1, wherein the acoustic processing control unit controls the selection of an acoustic output driver from among the plurality of acoustic output drivers to output a noise cancellation signal for the ambient sound, depending on the masking state determination result and information on the direction from which the ambient sound is coming.

12. The signal processing device according to claim 1, wherein the acoustic processing control section performs processing to transmit quantization bit information required for content audio data to an external device in accordance with the result of the masking state determination.

13. The signal processing device according to claim 12, wherein the quantization bit information transmitted to the external device includes information on a channel and a band for reducing the number of quantization bits in the content audio data.

14. An audio output device comprising: a plurality of audio output drivers; a plurality of microphones for collecting ambient sounds; a masking determination unit for determining a masking state between content sounds and ambient sounds using multiple channels of content sound data output from the audio output drivers, ambient sound data obtained by the microphones, and information on the direction from which the ambient sounds are coming; and an audio processing control unit for controlling audio processing in accordance with the masking state determination result by the masking determination unit.

15. A signal processing method in which a signal processing device executes the following steps: a masking determination process that determines the masking state between content sound and ambient sound using multiple channels of content sound data output from multiple audio output drivers arranged in an audio output device, ambient sound data obtained by multiple microphones arranged in the audio output device that collect ambient sound, and information on the direction from which the ambient sound is coming; and an audio processing control process that controls audio processing in accordance with the masking state determination result obtained by the masking determination process.

Images