Acoustic processing device, acoustic processing method, acoustic processing program and acoustic processing system
The sound processing device enhances stereoscopic sound by measuring viewer and speaker positions, correcting sound output to simulate an ideal speaker setup, overcoming limitations of limited speaker arrangements for a more immersive audio experience.
Patent Information
- Application Number
- KR1020267036686
- Authority / Receiving Office
- KR · KR
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-08-06
- Filing Date
- 2022-03-23
- Publication Date
- 2026-11-09
AI Technical Summary
Existing technologies struggle to accurately recreate stereoscopic sound in home environments without the ideal speaker arrangement, as they fail to account for the viewer's position, environmental factors, and speaker placement, leading to a suboptimal sense of presence.
A sound processing device that measures the viewer's location, speaker arrangement, and room shape to correct sound output, simulating an ideal speaker setup using virtual speakers, even with limited physical speakers, by employing methods like AoA, AoD, and SLAM to enhance spatial awareness.
Enables realistic three-dimensional sound reproduction without the need for multiple speakers, providing a more immersive audio experience by simulating the recommended speaker arrangement based on measured environmental data.
Smart Images

Figure P1020267036686_ABST
Abstract
Description
Technology Field
[0001] The present disclosure relates to a sound processing device, a sound processing method, a sound processing program, and a sound processing system for performing sound field processing during content playback. Background Technology
[0002] In movies or audio content, so-called stereoscopic sound (3D audio) is sometimes employed to enhance the sense of presence during playback by emitting sound from above or behind the viewer's head.
[0003] Ideally, to realize stereoscopic sound, multiple speakers should be placed to surround the viewer, but it is practically difficult to install multiple speakers in a typical home. As a solution to this problem, a technology is known that enables pseudo-stereoscopic sound to be realized even without an ideal speaker arrangement by installing a microphone at the viewing position and performing signal processing based on the collected sound (e.g., Patent Document 1). Additionally, a technology is known that synthesizes waveforms output from multiple speakers to make them appear as if they were emitted from a single pseudo-virtual speaker (e.g., Patent Document 2). Prior art literature
[0004] Japanese Patent Publication No. 6737959 and U.S. Patent No. 9749769 Specification The problem to be solved
[0005] However, in stereoscopic audio, to further enhance the viewer's sense of presence, it is required to understand the spatial configuration, such as the viewer's position, the environment of the playback device, and the distance to the ceiling or walls. In other words, to realize stereoscopic audio, it is desirable to make corrections by comprehensively utilizing information such as the viewer's location within the space, the number and placement of speakers, and reflections from walls and ceilings.
[0006] Therefore, the present disclosure proposes a sound processing device, a sound processing method, a sound processing program, and a sound processing system that can make content feel more realistic in a sound field. means of solving the problem
[0007] To solve the above problem, an audio processing device of one form according to the present disclosure comprises: an acquisition unit for acquiring a recommended environment defined for each content, including an ideal arrangement of speakers in a space where the content is played; a measurement unit for measuring the location of a viewer located in the space, the number and arrangement of speakers, and the shape of the space; and a correction unit for correcting the sound of the content, which is observed at the location of the viewer and is emitted from speakers located in the space, into a sound emitted from virtual speakers ideally arranged in the recommended environment, based on information measured by the measurement unit. Brief explanation of the drawing
[0008] Figure 1 is a diagram illustrating an overview of acoustic processing regarding an embodiment. Figure 2 is a drawing (1) for explaining the speaker placement regarding the recommended environment. Figure 3 is a drawing (2) for explaining the speaker placement regarding the recommended environment. Figure 4 is a drawing (3) for explaining the speaker placement regarding the recommended environment. Figure 5 is a drawing (1) for explaining acoustic processing regarding an embodiment. FIG. 6 is a drawing (2) for explaining acoustic processing regarding an embodiment. FIG. 7 is a drawing (3) for explaining acoustic processing regarding an embodiment. FIG. 8 is a drawing (4) for explaining acoustic processing regarding an embodiment. FIG. 9 is a drawing illustrating an example of the configuration of an acoustic processing device according to an embodiment. FIG. 10 is a drawing illustrating an example of a speaker information storage unit regarding an embodiment. FIG. 11 is a drawing illustrating an example of a measurement result memory unit regarding an embodiment. FIG. 12 is a drawing (1) for explaining the measurement process regarding an embodiment. FIG. 13 is a drawing (2) for explaining the measurement processing regarding an embodiment. FIG. 14 is a drawing illustrating an example of the configuration of a speaker according to an embodiment. FIG. 15 is a flowchart (1) showing the flow of processing regarding an embodiment. FIG. 16 is a flowchart (2) showing the flow of processing regarding an embodiment. FIG. 17 is a flowchart (3) showing the flow of processing regarding an embodiment. FIG. 18 is a hardware configuration diagram showing an example of a computer that realizes the function of an acoustic processing device. Specific details for implementing the invention
[0009] Hereinafter, embodiments will be described in detail based on the drawings. In addition, in each of the following embodiments, identical parts are assigned the same number to omit redundant descriptions.
[0010] The present disclosure is described in the order of the items shown below.
[0011] 1. Embodiment
[0012] 1-1. Overview of Acoustic Processing Regarding Embodiments
[0013] 1-2. Configuration of the acoustic processing device according to the embodiment
[0014] 1-3. Configuration of the speaker regarding the embodiment
[0015] 1-4. Procedure of processing regarding the embodiment
[0016] 1-5. Variations regarding embodiments
[0017] 2. Other embodiments
[0018] 3. Effects of the acoustic processing device according to the present disclosure
[0019] 4. Hardware Configuration
[0020] (1. Embodiment)
[0021] (1-1. Overview of acoustic processing regarding the embodiment)
[0022] Using FIG. 1, an example of acoustic processing according to an embodiment of the present disclosure is described. FIG. 1 is a diagram illustrating an overview of acoustic processing according to an embodiment. Specifically, FIG. 1 shows components of an acoustic processing system (1) that performs acoustic processing according to an embodiment.
[0023] As illustrated in FIG. 1, the sound processing system (1) includes a sound processing device (100), a speaker (200A), a speaker (200B), a speaker (200C), and a speaker (200D). The sound processing system (1) outputs a voice signal to a user (50) who is a viewer, or corrects the output voice signal.
[0024] The sound processing device (100) is an example of an information processing device that performs sound processing according to the present disclosure. Specifically, the sound processing device (100) controls the voice signals output by the speaker (200A), the speaker (200B), the speaker (200C), and the speaker (200D). For example, the sound processing device (100) controls the playback of content such as movies or music and controls the output of the voice included in the content from the speaker (200A), etc. In addition, if the content includes video, the sound processing device (100) may control the output of the video from the display (300). In addition, although details will be described later, the sound processing device (100) is equipped with various sensors, etc. for measuring the location of the user (50) or the speaker (200A), etc.
[0025] Speaker (200A), Speaker (200B), Speaker (200C), and Speaker (200D) are voice output devices that output voice signals. In the following description, if there is no need to distinguish between Speaker (200A), Speaker (200B), Speaker (200C), and Speaker (200D), they are collectively referred to as "Speaker (200)." Speaker (200) is wirelessly connected to the sound processing device (100) to receive voice signals or receive control regarding the measurement processing described later.
[0026] Additionally, each device in FIG. 1 conceptually represents a function in the sound processing system (1) and may take various forms depending on the embodiment. For example, the sound processing device (100) may be composed of two or more devices, each with a different function described later. Also, the speakers (200) included in the sound processing system (1) do not necessarily have to be four.
[0027] As described above, in the example illustrated in FIG. 1, the sound processing system (1) is a wireless audio speaker system realized by a combination of a sound processing device (100), which is a control unit that performs voice signal processing, and a speaker (200) that is wirelessly connected to the sound processing device (100). The sound processing system (1) provides the user (50) with so-called stereoscopic sound (3D audio) that enhances the sense of presence during content playback by emitting sound from above the viewer's head or behind their back.
[0028] However, content containing stereoscopic sound includes audio signals that assume the placement of not only flat-direction so-called surround speakers but also height-direction so-called height speakers (hereinafter collectively referred to as "ceiling speakers"). To properly reproduce such content, it is required that flat speakers and ceiling speakers be correctly positioned relative to the viewer's location. Correct placement refers to the recommended speaker placement defined, for example, in stereoscopic sound technical standards. According to these standards, realizing stereoscopic sound requires placing multiple speakers to surround the viewer, but it is practically difficult to install a large number of speakers in an ordinary home.
[0029] Therefore, in order to reproduce a sound field close to that of a standard arrangement even if it is not, there exists a technology that installs a microphone at the viewing position during initial setup and performs signal processing based on the sound collected there. According to this technology, sound field correction is performed so that the sound is heard from the correct arrangement according to the standard. Furthermore, according to this technology, if ceiling speakers cannot be installed, the sound is reflected off the ceiling to serve as a substitute for ceiling speakers, or the sound is corrected so that the viewer artificially perceives the sound of ceiling speakers by using signal processing technology (referred to as a virtualizer, etc.). However, to correct more accurately, it is desirable to accurately measure the positions of the viewer and speakers, identify the shape and characteristics of the room, and make corrections by comprehensively using such information, including cases where the room's space is limited.
[0030] In this regard, the sound processing system (1) according to the embodiment obtains a recommended environment defined for each content, including the ideal placement of speakers in the space where the content is played, and measures the location of a viewer located in the space, the number and placement of speakers, and the shape of the space. In addition, based on the measured information, the sound processing system (1) corrects the sound of the content emitted from speakers located in the space, which is the sound observed at the location of the viewer, into the sound emitted from virtual speakers ideally placed in the recommended environment.
[0031] In this way, the sound processing system (1) measures the location of the viewer in the real space and the placement of speakers, and based on this information, corrects the real sound to be closer to the sound emitted from the virtual speakers installed in the recommended environment. With this configuration, the user (50) can experience realistic three-dimensional sound without having to place multiple speakers as prescribed in the recommended environment. Furthermore, according to this method, the user (50) can realize three-dimensional sound without any burden, without having to go through the trouble of installing a microphone at the viewing position and performing initial settings.
[0032] The configuration and overview of the sound processing system (1) were shown above using FIG. 1. Next, the sound processing according to the present disclosure will be specifically described using FIG. 2 and below.
[0033] FIG. 2 is a diagram (1) for explaining speaker placement regarding a recommended environment. FIG. 2 shows an example of speaker placement recommended when watching 3D audio content in which voice regarding stereoscopic sound is recorded. Specifically, FIG. 2 shows an example of a recommended environment defined by Dolby Atmos (registered trademark (Dolby Atmos)).
[0034] In the example of FIG. 2, center speakers (10A) are positioned in front of the user (50), left front speakers (10B) in front of the left, right front speakers (10C) in front of the right, left surround speakers (10D) in rear of the left, and right front speakers (10E) in rear of the right. Additionally, above the user's (50) head, that is, as ceiling speakers, left top front speakers (10F) in front of the upper left, right top front speakers (10G) in front of the upper right, left top rear speakers (10H) in rear of the upper left, and right top rear speakers (10I) in rear of the upper right are positioned. Also, although not shown in FIG. 2, in a recommended environment, a low-frequency subwoofer may be added. In the arrangement of the example of FIG. 2, there are five speakers in the horizontal direction, a subwoofer, and four speakers on the ceiling, so it is also referred to as a "5.1.4" channel environment. In addition, as a recommended environment, environments such as "7.1.4" or "5.1.2" may also be used.
[0035] The sound processing device (100) obtains information such as the number or arrangement of speakers as shown in FIG. 2, and distance from the user (50) (viewing location), as information regarding the recommended environment for content playback. For example, the sound processing device (100) may obtain the recommended environment from metadata included in the content during content playback, or the recommended environment may be installed in advance by the manager of the sound processing device (100) or the user (50). In addition, below, when there is no need to distinguish each speaker that has realized the ideal arrangement in the recommended environment as shown in FIG. 2, they are collectively referred to as "virtual speakers (10)."
[0036] As shown in FIG. 2, in the recommended environment, the number of flat speakers (speakers installed at approximately the same height as the user (50)) and ceiling speakers, the distance or angle from the user (50), and the angle or distance between virtual speakers (10) are specified.
[0037] Next, using FIG. 3, the planar arrangement of the virtual speaker (10) regarding the ceiling speaker is described. FIG. 3 is a drawing (2) for explaining the speaker arrangement regarding the recommended environment.
[0038] For example, as illustrated in FIG. 3, in a recommended environment, the left top front speaker (10F) and the right top front speaker (10G) are specified to be installed at an angle of approximately 45 degrees each directly in front of the user (50). Additionally, the left top rear speaker (10H) and the right top rear speaker (10I) are specified to be installed at an angle of approximately 135 degrees each directly in front of the user (50).
[0039] Next, using FIG. 4, the installation height of the virtual speaker (10) regarding the ceiling speaker is described. FIG. 4 is a drawing (3) for explaining the speaker arrangement regarding the recommended environment. FIG. 4 shows a cross-sectional view corresponding to the arrangement shown in FIG. 3.
[0040] For example, as illustrated in FIG. 4, in a recommended environment, the left top front speaker (10F) (and the right top front speaker (10G), which is not illustrated) is specified to be installed at an angle of approximately 45 degrees upward at the direct front of the user (50). Additionally, the left top rear speaker (10H) (and the right top rear speaker (10I), which is not illustrated) is specified to be installed at an angle of approximately 135 degrees backward at the direct front of the user (50). Furthermore, with the user (50) as the center point, it is recommended that the left top front speaker (10F) and the left top rear speaker (10H) be installed at an angle of approximately 90 degrees. In addition, the recommended environment shown in FIGS. 2 to 4 is an example, and there are various recommended environments for each content, such as the number or placement of speakers, the installation distance to the user (50), etc., for example, the specifications of the stereoscopic sound or the regulations of the content production company.
[0041] As described above, the sound processing device (100) according to the embodiment corrects the sound output from the speaker (200) installed in reality as if the virtual speaker (10) were placed in the recommended environment, in a playback environment different from the recommended environment. First, prior to the correction processing, the sound processing device (100) obtains a recommended environment showing the arrangement of the virtual speaker (10) as illustrated in FIGS. 2 to 4. Then, the sound processing device (100) corrects the sound output from the speaker (200) installed in the actual space based on the recommended environment. This processing will be explained using FIGS. 5 and below.
[0042] FIG. 5 is a drawing (1) for explaining sound processing regarding an embodiment. As shown in FIG. 5, in the space where the user (50) is located, speakers (200A), speakers (200B), speakers (200C), and speakers (200D) are installed in a different arrangement than the recommended environment.
[0043] In the recommended environment, the number and arrangement of virtual speakers (10), and the distance from each virtual speaker (10) to the user (50) are specified, so in order to perform correction processing, it is necessary to determine the arrangement of speakers (200) and the location of the user (50). Therefore, the sound processing device (100) measures the arrangement of speakers (200) and the location of the user (50).
[0044] For example, the sound processing device (100) measures the location of each speaker (200) by utilizing a wireless transmission and reception function (specifically, a wireless module and an antenna) provided by the speaker (200). Although details will be described later, the sound processing device (100) may employ a method (Angle of Arrival (AoA)) to estimate the direction of the transmitting side (speaker (200)) by receiving a signal transmitted by each speaker (200) at a plurality of antennas and detecting the phase difference of the signal. Alternatively, the sound processing device (100) may use a method (Angle of Departure (AoD)) to estimate the angle (i.e., the arrangement as viewed from the sound processing device (100)) from the phase difference received by each speaker (200) by transmitting a signal while switching between the plurality of antennas provided by the sound processing device (100).
[0045] Additionally, when the sound processing device (100) measures the location of the user (50), it may use a wireless communication device, such as a smartphone, owned by the user (50). For example, the sound processing device (100) may transmit voice from a smartphone through a dedicated application, etc., receive such voice at the sound processing device (100) and speaker (200), and measure the location of the user (50) based on the arrival time. Alternatively, the sound processing device (100) may measure the location of the smartphone using the method described above, such as AoA, and estimate the measured location of the smartphone as the location of the user (50). Furthermore, the sound processing device (100) may detect a smartphone existing in space using radio waves such as Bluetooth, or it may receive a registration of the smartphone used by the user (50) in advance.
[0046] Alternatively, the sound processing device (100) may measure the location of the user (50) or each speaker (200) using a depth sensor such as a Time of Flight (ToF) sensor or an image sensor equipped with an AI chip that has completed prior training to recognize a human face.
[0047] Continuing, the acoustic processing device (100) measures the spatial shape. For example, the acoustic processing device (100) measures the spatial shape by transmitting a measurement signal from the speaker (200). This point is explained using FIG. 6. FIG. 6 is a drawing (2) for explaining acoustic processing regarding an embodiment.
[0048] As illustrated in FIG. 6, the speaker (200) has a horizontal unit (251) that outputs sound in a horizontal direction toward the user (50), and a ceiling unit (252) that outputs sound toward the ceiling. That is, the speaker (200) according to the embodiment is configured to emit separate sounds in two directions. By reflecting the sound emitted from the ceiling unit (252) from the ceiling (20), the speaker (200) can make the user (50) feel as if the sound is being emitted from a virtual speaker (260) that serves as a substitute for the ceiling speaker.
[0049] Additionally, the speaker (200) may measure the shape of the space using a measurement signal output from a ceiling unit (252). This method is referred to as FMCW (Frequency Modulated Continuous Wave), etc. This method outputs a sound whose frequency changes linearly over time from the speaker (200), detects a reflected wave from a microphone equipped with the speaker (200), and calculates the distance to the ceiling from the frequency difference (beat frequency).
[0050] Specifically, when the acoustic processing device (100) requests the measurement of the spatial shape, the speaker (200) transmits a measurement signal toward the ceiling (20). Then, the speaker (200) measures the distance to the ceiling by observing the reflected sound of the measurement signal with a microphone provided. Since the acoustic processing device (100) is aware of the number and arrangement of the speakers (200), it can obtain information regarding the spatial shape where the speakers (200) are installed by obtaining the ceiling height information transmitted from the speakers (200).
[0051] Additionally, the sound processing device (100) may acquire map information of the space where the user (50) is located using a technology such as SLAM (Simultaneous Localization and Mapping) using a depth sensor or an image sensor, and estimate the spatial shape from this information.
[0052] Additionally, the spatial shape may include information indicating the characteristics of the space. For example, depending on the material of the walls or ceiling of the space, the sound pressure or sound quality of the reflected sound may change. For example, the sound processing device (100) may receive input of information regarding the material of the room manually by the user (50), or it may estimate the material of the room by investigating a measurement signal in the space.
[0053] As described above, the sound processing device (100) can obtain the number or arrangement of speakers (200) located in the space, the location of the user (50), the shape of the space, etc., through measurement processing. Based on this information, the sound processing device (100) performs sound field correction processing. This point is explained using FIG. 7. FIG. 7 is a drawing (3) for explaining sound processing regarding an embodiment.
[0054] As described above, the recommended environment for playing 3D audio content is defined, but in the embodiment, the user (50) is in a situation where only four speakers, such as speakers (200A), speakers (200B), speakers (200C), and speakers (200D), can be placed. However, even if the ideal arrangement as shown in the drawing cannot be realized, if the user (50) can feel that the sound is ringing with the recommended speaker arrangement through voice signal correction processing, it can be said that the playback of 3D audio content with a sense of realism can be realized. The sound processing device (100) performs such sound processing using four speakers (200) installed in the real space.
[0055] This point is explained using FIG. 8. FIG. 8 is a drawing (4) for explaining acoustic processing regarding an embodiment.
[0056] In the example of FIG. 8, a situation is shown in which a new virtual speaker (260E) is created using three sound sources: a speaker (200A), a speaker (200B), and a virtual speaker (260B) utilizing ceiling reflection. Specifically, the sound processing device (100) synthesizes sound based on the positional relationship of the speakers (200) or reflected sound sources that can be placed in reality, and generates a wavefront of a monopole sound source at the location of the virtual speaker (260E). Such wavefront synthesis can be realized, for example, by the method described in the aforementioned Patent Document 2. Specifically, the sound processing device (100) can form a synthesized sound field based on the recommended environment by synthesizing four reflected sound sources created by four speakers (200) and the ceiling unit (252) of the speakers (200) by using the method of “Synthesis Monopoles (Monopole Synthesis)” described in Patent Document 2.
[0057] As illustrated in FIGS. 1 to 8, the sound processing device (100) obtains a recommended environment defined for each content, including an ideal arrangement of speakers in a space where the content is played. Additionally, the sound processing device (100) measures the location of a viewer in the space, the number and arrangement of speakers, and the shape of the space. Then, based on the measured information, the sound processing device (100) corrects the sound of the content, which is observed at the location of the user (50) and is emitted from a speaker (200) in the space, into a sound emitted from a virtual speaker (10) ideally arranged in the recommended environment.
[0058] Accordingly, the user (50) can feel as though they are watching sound output from a virtual speaker (10) placed in the recommended environment shown in FIG. 2, even if the speaker placement is different from the recommended environment shown in FIG. 7. That is, the sound processing device (100) can make the 3D audio content feel as real as the recommended environment, even if the speaker placement is different from the recommended environment.
[0059] In addition, according to the sound processing of the embodiment, a virtual speaker (260E) can be formed at a distance from the user (50) when viewed from a distance greater than the actually installed speaker (200) or reflected sound source. Because of this, the sound processing device (100) can form a virtual speaker (260E) at a location where it cannot be installed due to room size constraints, thereby playing sound at a distance recommended by content such as movies, or making the sound field space feel larger.
[0060] (1-2. Configuration of the acoustic processing device according to the embodiment)
[0061] Next, the configuration of the sound processing device (100) will be described. FIG. 9 is a drawing illustrating an example of the configuration of the sound processing device (100) according to an embodiment.
[0062] As illustrated in FIG. 9, the sound processing device (100) has a communication unit (110), a memory unit (120), a control unit (130), and a sensor (140). Additionally, the sound processing device (100) may have an input unit (e.g., a touch display or button, etc.) for receiving various operations from an administrator or user (50) managing the sound processing device (100), and a display unit (e.g., a liquid crystal display, etc.) for displaying various information.
[0063] The communication unit (110) is realized, for example, by a NIC (Network Interface Card) or a network interface controller. The communication unit (110) is connected to a network N via a wired or wireless connection and transmits and receives information to a speaker (200), etc., through the network N. The network N is realized using a wireless communication standard or method such as, for example, Bluetooth (registered trademark), the Internet, Wi-Fi (registered trademark), UWB (Ultra Wide Band), LPWA (Low Power Wide Area).
[0064] The sensor (140) is a functional part for detecting various information. The sensor (140) includes, for example, a ToF sensor (141), an image sensor (142), or a microphone (143).
[0065] The ToF sensor (141) is a depth sensor that measures the distance to an object located in space.
[0066] The image sensor (142) is a pixel sensor that records a space captured by a camera or the like as pixel information (still image or moving image). Additionally, the image sensor (142) may include a pre-trained AI chip for image recognition of a human face or the shape of a speaker. In this case, the image sensor (142) can detect a user (50) or a speaker (200) by image recognition while capturing a space with a camera.
[0067] The microphone (143) is a voice sensor that picks up voice output by the speaker (200) or voice spoken by the user (50).
[0068] Additionally, the sensor (140) may include a touch sensor that detects when a user comes into contact with the sound processing device (100), or a sensor that detects the current location of the sound processing device (100). For example, the sensor (140) may receive radio waves transmitted from a GPS (Global Positioning System) satellite and detect location information (e.g., latitude and longitude) indicating the current location of the sound processing device (100) based on the received radio waves.
[0069] Additionally, the sensor (140) may include a radio wave sensor that detects radio waves emitted by a smartphone or speaker (200), or an electromagnetic wave sensor that detects electromagnetic waves (antenna). Additionally, the sensor (140) may detect the environment in which the sound processing device (100) is placed. Specifically, the sensor (140) may include an illuminance sensor that detects the illuminance around the sound processing device (100), or a humidity sensor that detects the humidity around the sound processing device (100).
[0070] Additionally, the sensor (140) does not necessarily have to be provided inside the sound processing device (100). For example, the sensor (140) may be installed outside the sound processing device (100) as long as it is possible to transmit the sensed information to the sound processing device (100) using communication or the like.
[0071] The memory unit (120) is realized by a semiconductor memory device such as RAM (Random Access Memory) or Flash Memory, for example, or by a storage device such as a hard disk or an optical disk. The memory unit (120) has a speaker information memory unit (121) and a measurement result memory unit (122). Hereinafter, each memory unit will be described in order using FIGS. 10 and FIGS. 11.
[0072] FIG. 10 is a drawing illustrating an example of a speaker information storage unit (121) according to an embodiment. As shown in FIG. 10, the speaker information storage unit (121) has items such as "speaker ID" and "acoustic characteristics". In addition, in FIG. 10 and FIG. 11, information stored in the storage unit (120) is conceptually represented as "A01", but in reality, each of the information described below is stored in the storage unit (120).
[0073] "Speaker ID" is identification information for identifying a speaker. "Acoustic characteristics" represent the acoustic characteristics of each speaker. For example, acoustic characteristics may include information such as voice output values, frequency characteristics, the number or orientation of units, the efficiency of units, or the speed of response (time from voice signal input to output). The acoustic processing device (100) may acquire information regarding acoustic characteristics from a speaker manufacturer, etc. via network N, or may acquire acoustic characteristics by using methods such as outputting a measurement signal from a speaker and measuring it with a microphone provided by the acoustic processing device (100).
[0074] Next, the measurement result memory unit (122) will be described. FIG. 11 is a drawing illustrating an example of a measurement result memory unit according to an embodiment.
[0075] In the example illustrated in FIG. 11, the measurement result memory unit (122) has items such as "measurement result ID," "user location information," and "speaker placement information." "Measurement result ID" represents identification information that identifies the measurement result. The measurement result ID may include the time of measurement or location information indicating the location of the space where the measurement was taken.
[0076] "User location information" indicates the measured location of the user. "Speaker placement information" indicates the placement or number of the measured speakers. Additionally, user location information and speaker placement information may be stored in any format. For example, user location information and speaker placement information may be stored as objects placed in space based on SLAM. Additionally, user location information and speaker placement information may be stored as coordinate information or distance information centered on the location of the sound processing device (100). That is, user location information and speaker placement information may be in any format as long as the sound processing device (100) can identify the location of the user (50) or the speaker (200) in space.
[0077] Returning to Fig. 9, the explanation continues. The control unit (130) is realized by executing a program stored inside the sound processing device (100) (e.g., a sound processing program according to the present disclosure) using a RAM (Random Access Memory), etc., as a work area, by means of, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), etc. Additionally, the control unit (130) is a controller and may be realized by an integrated circuit such as, for example, an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).
[0078] As illustrated in FIG. 9, the control unit (130) has an acquisition unit (131), a measurement unit (132), and a correction unit (133).
[0079] The acquisition unit (131) acquires various information. For example, the acquisition unit (131) acquires a recommended environment defined for each content, including the ideal placement of speakers in the space where the content is played.
[0080] When the acquisition unit (131) acquires content such as movies or 3D audio via network N, it may acquire the recommended environment specified in the content from the metadata included in the content. Additionally, the acquisition unit (131) may acquire a recommended environment suitable for each content by receiving input from the user (50).
[0081] The measuring unit (132) measures the location of the user (50) located in the space, the number and arrangement of the speakers (200), and the shape of the space.
[0082] For example, the measuring unit (132) measures the number and arrangement of speakers located in the space by using radio waves transmitted or received by a plurality of speakers located in the space to measure the relative positions of the sound processing device (100) and the plurality of speakers (200).
[0083] This point is explained using FIGS. 12 and FIGS. 13. FIGS. 12 is a drawing (1) for explaining the measurement process regarding an embodiment.
[0084] In the example illustrated in FIG. 12, a situation is shown in which a receiver (70) having multiple antennas receives radio waves transmitted by a transmitter (60). For example, the transmitter (60) is an acoustic processing device (100), and the receiver (70) is a speaker (200). The acoustic processing device (100) transmits radio waves from an antenna (61) and can estimate the relative angle θ between the receiving side and the transmitting side by detecting the phase difference of the signals received by the multiple antennas (71), antenna (72), and antenna (73) equipped by the speaker (200). The acoustic processing device (100) measures the position of the speaker (200) based on the estimated angle θ. This method is referred to as AoA, etc.
[0085] Next, another example is described using FIG. 13. FIG. 13 is a drawing (2) for explaining measurement processing regarding an embodiment.
[0086] In the example illustrated in FIG. 13, a situation is shown in which a receiver (70) receives radio waves transmitted by a transmitter (60) from a plurality of antennas. For example, the transmitter (60) is an acoustic processing device (100), and the receiver (70) is a speaker (200). The acoustic processing device (100) transmits a signal while switching between a plurality of antennas, such as antenna (65), antenna (66), and antenna (67), and estimates the relative angle θ between the receiving side and the transmitting side from the phase difference when each speaker (200) receives radio waves from antenna (75). The acoustic processing device (100) measures the position of the speaker (200) based on the estimated angle θ. This method is referred to as AoD, etc.
[0087] The processing illustrated in FIGS. 12 and 13 is an example of measurement, and the measuring unit (132) may use other methods. For example, the measuring unit (132) may use a ToF sensor (141) that detects objects located in the space to measure at least one of the location of the user (50) located in the space, the number and arrangement of speakers (200), and the shape of the space.
[0088] Additionally, the measuring unit (132) may measure the location of the user (50) or speaker (200) located in the space by using an image sensor (142) provided by the sound processing device (100) to image-recognize the user (50) or speaker (200).
[0089] Additionally, the measuring unit (132) may measure the location of the user (50) or speaker (200) located in the space by using an image sensor provided by an external device to image-recognize the user (50) or speaker (200). For example, the measuring unit (132) may use an image sensor provided by the speaker (200) or display (300), or a USB camera connected to the display (300). Specifically, the measuring unit (132) acquires an image captured by the speaker (200) or display (300) and measures the location of the user (50) or speaker (200) by identifying and tracking the user (50) or speaker (200) through image analysis. Additionally, the measuring unit (132) may measure the shape of the space where the user (50) is located, or acoustic characteristics of the space based on the material of the walls or ceiling, etc., based on this image recognition. Additionally, when image analysis is performed by a speaker (200) or a display (300), the speaker (200) or the display (300) may convert the location or spatial shape of the user (50) obtained by the analysis into abstract data (metadata) and transmit the converted data to a sound processing device (100) via a video / audio connection cable such as HDMI (registered trademark) or a wireless system such as Wi-Fi.
[0090] Additionally, the measuring unit (132) may measure the location of the user (50) in the space by using radio waves transmitted or received by the smartphone carried by the user (50). That is, the measuring unit (132) measures the location of the user (50) using the smartphone by estimating the location of the smartphone using the AoA or AoD method described above. In addition, if there are multiple viewers in the same space in addition to the user (50), the measuring unit (132) can measure the location of all viewers by performing sequential measurements on all viewers. In addition, the measuring unit (132) may measure the location of the user (50), etc. by outputting a measurement signal (audible sound or ultrasonic) from a device held by each user (50) or other viewers and detecting it with a microphone (143).
[0091] Additionally, the measuring unit (132) measures the distance to the ceiling of the space based on the reflected sound of the sound emitted from the ceiling unit (252) equipped with the speaker (200) located in the space as the spatial shape of the space. For example, the measuring unit (132) controls the speaker (200) to output a measurement signal as shown in FIG. 6, and measures the distance to the ceiling based on the time until the speaker (200) receives the measurement signal emitted by the speaker (200).
[0092] Additionally, the measuring unit (132) may generate map information based on an image captured by an external device such as an image sensor (142) or a smartphone or speaker (200), and based on the generated map information, may measure at least one of the position of the sound processing device (100), the position of the user (50), the number and arrangement of speakers (200), and the spatial shape. That is, the measuring unit (132) may create spatial shape data where speakers (200) are arranged by using SLAM technology, and measure the arrangement of the user (50) or speakers (200) located in that space.
[0093] Additionally, the measuring unit (132) may continuously measure the location of the user (50) in the space, the number and arrangement of speakers, and the shape of the space. For example, the measuring unit (132) continuously measures the location of the user (50) at the timing when the content is stopped, or at the timing at regular intervals after power is supplied to the sound processing device (100). In this case, the correction unit (133) uses the information continuously measured by the measuring unit (132) to correct the sound of the content emitted from the speakers (200) in the space. By doing so, the measuring unit (132) can continuously measure and identify the change even if, for example, the arrangement of the speakers (200) is changed by the user (50) who cleaned the room, and thus can perform appropriate sound correction without the user (50) being aware of it.
[0094] The correction unit (133), based on information measured by the measurement unit (132), corrects the voice of the content emitted from the speaker (200) located in the space, which is the voice observed at the location of the user (50), into the voice emitted from the virtual speaker (10) ideally placed in the recommended environment.
[0095] For example, the correction unit (133) corrects the voice of the speaker (200) into the voice emitted from the virtual speaker (10) by using a method of forming a virtual speaker by synthesizing voice waveforms emitted from a plurality of speakers (200) as described using FIGS. 7 and 8.
[0096] Additionally, the correction unit (133) may receive input from the user (50) and reflect this information in the correction. For example, the correction unit (133) provides information measured by the measurement unit (132) to the smartphone used by the user (50). Then, the correction unit (133) receives a change in information on the smartphone application from the user (50) who sees the information displayed on the smartphone application. For example, the correction unit (133) corrects the voice of the content based on at least one of the user's (50) location in the space, the number and arrangement of speakers (200), and the shape of the space, which are corrected by the user (50) on the smartphone. By doing so, the correction unit (133) can perform correction based on location information finely adjusted by the user (50) who has grasped the actual situation, and thus can perform correction based on the recommended environment more accurately.
[0097] Additionally, the correction unit (133) may also correct the audio of the content corrected by the correction unit (133) based on corrections made by the user (50). For example, after watching the audio of the content corrected by the correction unit (133), the user (50) may want to change the frequency being emphasized or adjust the arrival time (delay) of the audio output from the speaker (200). The correction unit (133) receives this information and corrects the audio to match the user's (50) request. By doing so, the correction unit (133) can form a sound field that the user (50) prefers.
[0098] Additionally, the correction unit (133) may correct the voice of the content based on the behavior pattern of the user (50) learned based on the information measured by the measurement unit (132) or the placement pattern of the speaker (200).
[0099] For example, the correction unit (133) acquires location information of the user (50) or the speaker (200) that the measurement unit (132) has continuously tracked. Additionally, the correction unit (133) acquires correction information of the sound field adjusted by the user (50). Furthermore, by learning such history with artificial intelligence (AI), the correction unit (133) can provide the optimal sound field that the user (50) desires.
[0100] Additionally, the correction unit (133) may make various suggestions to the user (50) through a smartphone application, etc. by combining the constant monitoring of the audio of the content being played with the microphone (143) and the continuous learning processing of the AI. For example, the correction unit (133) may suggest to the user (50) that the orientation of the speaker (200) be rotated slightly or the installation position be changed slightly so that it is closer to the sound field that the user (50) is expected to like. In addition, the correction unit (133) may predict the location where the user (50) is expected to be next based on the tracking history of the user's (50) location and perform sound field correction tailored to the predicted location. By doing so, the correction unit (133) can perform appropriate correction tailored to the location after the user (50) moves immediately after the user (50) moves.
[0101] In addition, the sound processing performed by the control unit (130) is realized by being implemented, for example, by a manufacturer producing a sound processing device (100) or a speaker (200), but there may also be a form in which it is embedded in a software module provided for content and then implemented in a sound processing device (100) or a speaker (200) for use.
[0102] (1-3. Configuration of the speaker regarding the embodiment)
[0103] Next, the configuration of the speaker (200) will be described. FIG. 14 is a drawing illustrating an example of the configuration of the speaker (200) according to an embodiment.
[0104] As illustrated in FIG. 14, the speaker (200) has a communication unit (210), a memory unit (220), and a control unit (230).
[0105] The communication unit (210) is realized, for example, by a NIC or a network interface controller. The communication unit (210) is connected to a network N via a wired or wireless connection and transmits and receives information to and from an audio processing device (100), etc., through the network N.
[0106] The memory unit (220) is realized by a semiconductor memory device such as RAM or flash memory, or by a storage device such as a hard disk or optical disk. The memory unit (220) stores the measurement results, for example, when the spatial shape is measured by the control of the sound processing device (100) or when the location of the user (50) is measured.
[0107] The control unit (230) is realized by executing a program stored inside the speaker (200) using RAM or the like as a work area, for example, by a CPU, MPU, GPU, etc. Additionally, the control unit (230) is a controller and may be realized by an integrated circuit such as an ASIC or FPGA, for example.
[0108] As illustrated in FIG. 14, the control unit (230) has an input unit (231), an output control unit (232), and a transmission unit (233).
[0109] The input unit (231) receives inputs such as voice signals corrected by the sound processing device (100) or control signals by the sound processing device (100).
[0110] The output control unit (232) controls the process of outputting voice signals, etc., from the output unit (250). For example, the output control unit (232) controls the output of a voice signal corrected by the sound processing device (100) from the output unit (250). In addition, the output control unit (232) controls the output of a measurement signal from the output unit (250) according to the control by the sound processing device (100).
[0111] The transmitting unit (233) transmits various information. For example, when the transmitting unit (233) is controlled to perform measurement processing from the sound processing device (100), it transmits the measurement result to the sound processing device (100).
[0112] The sensor (240) is a functional part for detecting various information. The sensor (240) includes, for example, a microphone (241).
[0113] The microphone (241) detects sound. For example, the microphone (241) detects the reflected sound of a measurement signal output from the output unit (250).
[0114] Additionally, the speaker (200) may be equipped with various sensors other than those shown in FIG. 14. For example, the speaker (200) may be equipped with a ToF sensor or an image sensor for detecting a user (50) or another speaker (200).
[0115] The output unit (250) outputs a voice signal according to the control of the output control unit (232). That is, the output unit (250) is a speaker unit that emits voice. The output unit (250) includes a horizontal unit (251) and a ceiling unit (252). In addition, the speaker (200) may be equipped with more units in addition to the horizontal unit (251) and the ceiling unit (252).
[0116] (1-4. Procedure of processing regarding the embodiment)
[0117] Next, using FIGS. 15 to 17, the procedure of processing regarding the embodiment will be explained. First, using FIG. 15, the procedure of the entire acoustic processing regarding the embodiment will be explained. FIG. 15 is a flowchart (1) showing the flow of processing regarding the embodiment.
[0118] As illustrated in FIG. 15, the sound processing device (100) determines whether it has received a measurement operation, for example, from a user (50) (step S101). If it has not received a measurement operation (step S101; "No"), the sound processing device (100) waits until it receives a measurement operation.
[0119] Meanwhile, when a measurement operation is received (Step S101; "Yes"), the sound processing device (100) measures the placement of the speaker (200) installed in the space (Step S102). After that, the sound processing device (100) measures the location of the user (50) (Step S103).
[0120] Next, the sound processing device (100) determines whether the user (50) has acquired the content to be played (step S104). If the content has not been acquired, the sound processing device (100) waits until the content is acquired (step S104; "No").
[0121] Meanwhile, if content is acquired (Step S104; "Yes"), the sound processing device (100) acquires a recommended environment corresponding to the content (Step S105). The sound processing device (100) starts playing the content (Step S106).
[0122] At this time, the sound processing device (100) corrects the audio signal of the played content as if it were being played in the recommended environment of the content (step S107).
[0123] After that, the sound processing device (100) determines whether the playback of the content has ended, for example, according to the operation of the user (50) (step S108). If the playback of the content has not ended (step S108; "No"), the sound processing device (100) continues the playback of the content.
[0124] Meanwhile, when the playback of the content ends (Step S108; "Yes"), the sound processing device (100) determines whether a predetermined time has elapsed (Step S109). If the predetermined time has not elapsed (Step S109; "No"), the sound processing device (100) waits until the predetermined time has elapsed.
[0125] Meanwhile, when a predetermined time has elapsed (step S109; "Yes"), the sound processing device (100) measures the placement of the speaker (200) again (step S102). That is, the sound processing device (100) tracks the position of the speaker (200) or the user (50) at predetermined intervals, so that when content is played next, it can perform correction based on appropriate position information.
[0126] Next, using FIG. 16, the procedure for the measurement processing regarding the speaker (200) will be explained. FIG. 16 is a flowchart (2) showing the flow of processing regarding an embodiment.
[0127] As illustrated in FIG. 16, when measuring the position or number of speakers (200) in step S102, the sound processing device (100) transmits a command for position measurement to each speaker (200) (step S201). A command is, for example, a control signal indicating the intention to start a measurement.
[0128] Additionally, the sound processing device (100) measures the arrangement of each speaker (200) (step S202). This processing may be performed by the sound processing device (100) itself using a ToF sensor (141), or by the speaker (200) or the smartphone using an image sensor equipped with an image sensor such as that of the speaker (200) or the smartphone owned by the user (50).
[0129] Next, the sound processing device (100) measures the distance from each speaker (200) to the ceiling (step S203). The distance to the ceiling may be obtained by executing a measurement method on the speaker (200) that utilizes the reflection of a measurement signal emitted by the speaker (200), or the sound processing device (100) itself may execute it using a ToF sensor (141), etc.
[0130] After that, the sound processing device (100) obtains measurement results from each speaker (200) (step S204). Then, the sound processing device (100) stores the measurement results in the measurement result memory unit (122) (step S205).
[0131] Next, using FIG. 17, the procedure for the measurement processing regarding the user (50) will be explained. FIG. 17 is a flowchart (3) showing the flow of processing regarding an embodiment.
[0132] As illustrated in FIG. 17, when the location of the user (50) is measured in step S103, the sound processing device (100) connects to the terminal device used by the user (50) (it may be a smartphone or a wearable device such as a smart watch or smart glasses worn by the user (50)) (step S301).
[0133] Next, the sound processing device (100) measures the location of the terminal device using any of the methods described above (step S302). This processing may be performed on the terminal device using an image sensor equipped in the terminal device, or the sound processing device (100) itself may perform it using a ToF sensor (141), etc.
[0134] After that, the sound processing device (100) obtains a measurement result from the terminal device (step S303). Then, the sound processing device (100) stores the measurement result in the measurement result memory unit (122) (step S304).
[0135] (1-5. Variations regarding embodiments)
[0136] In each of the above embodiments, the sound processing system (1) is shown as an example including a sound processing device (100) and four speakers (200). However, the sound processing system (1) may have a different configuration.
[0137] For example, the sound processing system (1) may be configured by combining multiple speakers having different functions or sound characteristics, provided that it can be connected to the sound processing device (100) via communication. That is, the sound processing system (1) may include existing speakers owned by the user (50) or speakers from other companies that are different from the speaker (200). In this case, the sound processing device (100) may emit sound measurement signals, etc., as described above to acquire the sound characteristics of these speakers.
[0138] Additionally, the speaker (200) does not necessarily have to be configured to have a horizontal unit (251) and a ceiling unit (252). If the speaker (200) is not equipped with a ceiling unit (252), the sound processing device (100) may measure spatial shape, such as the distance from the speaker (200) to the ceiling, by using a ToF sensor (141) or an image sensor (142), etc., instead of the speaker (200). Additionally, instead of the sound processing device (100), a display (300), etc. equipped with a camera may measure spatial shape, such as the distance from the speaker (200) to the ceiling.
[0139] Additionally, the sound processing system (1) may include shoulder-mounted speakers, headphones with an open structure that can hear external sounds, or bone conduction headphones with a structure that does not block the ears. In this case, the sound processing device (100) may measure the head-related transfer function (HRTF) of the user (50) as a characteristic embedded in the output device mounted on the user (50). In this case, the sound processing device (100) treats the output device mounted on the user (50) as a single speaker and synthesizes the waveform with the voice output from another speaker.
[0140] That is, the sound processing device (100) acquires the head transfer function of the user (50) and corrects the sound of a speaker placed near the user (50) based on the head transfer function of the user (50). By doing so, the sound processing device (100) can generate a sound field by combining a nearby speaker with clear sound field localization and another speaker placed in that space, thereby allowing the user (50) to experience a higher sense of realism.
[0141] (2. Other embodiments)
[0142] The processing for each of the above-described embodiments may be carried out in various other forms in addition to each of the above-described embodiments.
[0143] In addition, among the processes described in each of the above embodiments, all or part of the processes described as being performed automatically may be performed manually, or all or part of the processes described as being performed manually may be performed automatically by a known method. Furthermore, information including processing procedures, specific names, and various data or parameters shown in the above documents or drawings may be changed at will, except where specifically noted. For example, the various information shown in each drawing is not limited to the information depicted.
[0144] Furthermore, each component of each device described is functionally conceptual and is not required to be physically configured as described. That is, the specific form of distribution and integration of each device is not limited to what is described, and all or part thereof may be functionally or physically distributed and integrated into any unit according to various loads or usage conditions. For example, the measurement unit (132) and the correction unit (133) may be integrated.
[0145] In addition, each of the embodiments and variations described above can be appropriately combined to the extent that the processing content does not contradict each other.
[0146] Furthermore, the effects described in this specification are merely examples and are not limited thereto, and other effects may also be present.
[0147] (3. Effects of the acoustic processing device according to the present disclosure)
[0148] As described above, the sound processing device (in an embodiment, the sound processing device (100)) according to the present disclosure comprises an acquisition unit (in an embodiment, the acquisition unit (131)), a measurement unit (in an embodiment, the measurement unit (132)), and a correction unit (in an embodiment, the correction unit (133)). The acquisition unit acquires a recommended environment defined for each content, including the ideal arrangement of speakers in a space where the content is played. The measurement unit measures the location of a viewer (in an embodiment, the user (50)), the number and arrangement of speakers (in an embodiment, the speakers (200)), and the shape of the space. Based on the information measured by the measurement unit, the correction unit corrects the sound of the content, which is observed at the viewer's location and is emitted from speakers in the space, into the sound emitted from virtual speakers (in an embodiment, virtual speakers (10)) that are ideally arranged in the recommended environment.
[0149] In this way, the sound processing device according to the present disclosure can send sound to a viewer as if it were placed in a recommended environment by measuring the user's position and correcting the sound, even if physical speakers are not placed in a recommended environment for viewing 3D audio content, etc. By doing so, the sound processing device can make the content feel like it is in a more realistic sound field.
[0150] In addition, the measuring unit measures the number and arrangement of speakers located in the space by using radio waves transmitted or received by a plurality of speakers located in the space to measure the relative positions of the sound processing device and the plurality of speakers.
[0151] In this way, the acoustic processing device can measure the position of the speaker at high speed and accurately by measuring the position based on the radio waves between the acoustic processing device and the speaker.
[0152] In addition, the measuring unit uses a depth sensor that detects objects located in the space to measure at least one of the location of a viewer located in the space, the number and arrangement of speakers, and the shape of the space.
[0153] In this way, the acoustic processing device can accurately determine the distance to the speaker or the spatial shape by using a depth sensor, thereby enabling accurate measurement or correction processing.
[0154] In addition, the measuring unit measures the location of a viewer or speaker located in a space by using an image sensor equipped with an audio processing device or an external device (in an embodiment, a speaker (200), a display (300), a smartphone, etc.) to image-recognize the viewer or speaker.
[0155] In this way, the sound processing device can accurately measure the position of the speaker, etc., even in situations where measurement is difficult with other sensors, by using a camera (image sensor) equipped with a television or speaker to perform the measurement.
[0156] In addition, the measuring unit measures the location of a viewer in a space by using radio waves transmitted or received by a terminal device carried by the viewer (such as a smartphone or wearable device in an embodiment).
[0157] In this way, the sound processing device can accurately measure the location of a viewer even when the viewer cannot be identified by an image sensor or the like by determining the location using a terminal device.
[0158] In addition, the measuring unit measures the distance to the ceiling of the space based on the reflected sound of the sound emitted from the sound emitting unit (in the embodiment, the ceiling unit (252)) equipped with a speaker located in the space, as the spatial shape of the space.
[0159] In this way, the acoustic processing device can rapidly measure the spatial shape without undergoing complex processing such as image recognition by measuring the spatial shape using reflected sound output from a speaker.
[0160] In addition, the measurement unit continuously measures the location of viewers in the space, the number and arrangement of speakers, and the shape of the space. The correction unit uses the information continuously measured by the measurement unit to correct the audio of the content emitted from the speakers in the space.
[0161] In this way, the sound processing device can perform optimal correction tailored to the condition by tracking the position of the viewer or speaker, for example, even if the speaker is moved or the user moves due to some circumstances.
[0162] In addition, the acquisition unit acquires the recommended environment specified in the content from the metadata included in the content.
[0163] In this way, the sound processing device can perform correction processing based on the recommended environment required for each content by acquiring a recommended environment tailored to the content.
[0164] Additionally, the acquisition unit acquires the viewer's head transfer function. The correction unit corrects the sound from speakers placed near the viewer based on the viewer's head transfer function.
[0165] In this way, the sound processing device can provide the viewer with a more realistic sound field experience by performing corrections that incorporate open-type headphones, etc., as part of the system.
[0166] In addition, the measuring unit generates map information based on an image captured by an image sensor equipped with an acoustic processing device or an external device, and based on the generated map information, measures at least one of the acoustic processing device's self-position, the viewer's position, the number and arrangement of speakers, and the spatial shape.
[0167] In this way, the acoustic processing device can perform acoustic correction that includes obstacles, such as the location of pillars or walls in space, by performing measurements using map information.
[0168] In addition, the correction unit provides information measured by the measurement unit to a terminal device used by a viewer, and corrects the voice of the content based on at least one of the location of the viewer located in the space corrected by the viewer on the terminal device, the number and arrangement of speakers, and the shape of the space.
[0169] In this way, the sound processing device can perform more accurate correction by providing the measured situation through an application of a terminal device, etc., and receiving more fine position corrections, etc., from the viewer.
[0170] In addition, the correction unit corrects the audio of the content corrected by the correction unit, based on the correction made by the viewer.
[0171] In this way, the sound processing device can correct the sound to better suit the user's preferences, such as frequency emphasis points or delay conditions, by receiving requests from the viewer regarding the corrected sound.
[0172] In addition, the correction unit corrects the audio of the content based on the viewer's behavioral patterns or speaker placement patterns learned based on information measured by the measurement unit.
[0173] In this way, the sound processing device can perform sound field correction tailored to the situation at that location by learning the situation in which the viewer or speaker is moved, such as optimizing the voice to the location where the viewer is likely to be, or correcting the voice by estimating the position of the speaker after it has been moved.
[0174] (4. Hardware Configuration)
[0175] Information devices such as the sound processing device (100) according to each embodiment described above are realized by a computer (1000) with a configuration as shown in FIG. 18, for example. Below, the sound processing device (100) according to the present disclosure will be described by way of example. FIG. 18 is a hardware configuration diagram showing an example of a computer (1000) that realizes the function of the sound processing device (100). The computer (1000) has a CPU (1100), RAM (1200), ROM (Read Only Memory) (1300), HDD (Hard Disk Drive) (1400), a communication interface (1500), and an input / output interface (1600). Each part of the computer (1000) is connected by a bus (1050).
[0176] The CPU (1100) operates based on a program stored in the ROM (1300) or HDD (1400) and controls each part. For example, the CPU (1100) deploys a program stored in the ROM (1300) or HDD (1400) to the RAM (1200) and executes processing corresponding to various programs.
[0177] The ROM (1300) stores boot programs such as the BIOS (Basic Input Output System) that are executed by the CPU (1100) when the computer (1000) starts, or programs that depend on the hardware of the computer (1000).
[0178] The HDD (1400) is a computer-readable recording medium that non-temporarily records a program executed by the CPU (1100) and data used by such program. Specifically, the HDD (1400) is a recording medium that records an audio processing program according to the present disclosure, which is an example of program data (1450).
[0179] The communication interface (1500) is an interface for the computer (1000) to connect with an external network (1550) (e.g., the Internet). For example, the CPU (1100) receives data from another device or transmits data generated by the CPU (1100) to another device through the communication interface (1500).
[0180] The input / output interface (1600) is an interface for connecting an input / output device (1650) and a computer (1000). For example, the CPU (1100) receives data from an input device, such as a keyboard or a mouse, through the input / output interface (1600). Additionally, the CPU (1100) transmits data to an output device, such as a display, a speaker, or a printer, through the input / output interface (1600). Furthermore, the input / output interface (1600) may function as a media interface for reading a program, etc., recorded on a predetermined recording medium (media). Media is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or PD (Phase change rewritable Disk), an optical-magnetic recording medium such as a MO (Magneto-Optical Disk), a tape medium, a magnetic recording medium, or a semiconductor memory.
[0181] For example, when a computer (1000) functions as an audio processing device (100) in accordance with an embodiment, the CPU (1100) of the computer (1000) realizes functions such as a control unit (130) by executing an audio processing program loaded on RAM (1200). Additionally, an audio processing program in accordance with the present disclosure or data within a memory unit (120) is stored in the HDD (1400). Furthermore, the CPU (1100) reads and executes program data (1450) from the HDD (1400), but as another example, these programs may be acquired from another device through an external network (1550).
[0182] In addition, the present technology may also take the following configuration.
[0183] (1) A acquiring unit that acquires a recommended environment defined for each content, including an ideal placement of speakers in the space where the content is played, and
[0184] A measuring unit that measures the location of a viewer located in the above space, the number and arrangement of speakers, and the shape of the space, and
[0185] A correction unit that, based on information measured by the above-mentioned measuring unit, corrects the sound of the content emitted from a speaker located in the space, which is the sound observed at the viewer's location, into the sound emitted from a virtual speaker ideally positioned in the recommended environment.
[0186] An acoustic processing device equipped with
[0187] (2) The above measuring unit is,
[0188] A sound processing device described in (1) that measures the number and arrangement of speakers located in the space by using radio waves transmitted or received by a plurality of speakers located in the space and measuring the relative positions of the sound processing device and the plurality of speakers.
[0189] (3) The above measuring unit is,
[0190] A sound processing device described in (1) or (2) that uses a depth sensor to detect objects located in the space to measure at least one of the location of a viewer located in the space, the number and arrangement of speakers, and the shape of the space.
[0191] (4) The above measuring unit is,
[0192] A sound processing device described in any one of (1) to (3) that measures the location of a viewer or speaker located in the space by using an image sensor provided by the sound processing device or external device to image-recognize the viewer or speaker.
[0193] (5) The above measuring unit is,
[0194] A sound processing device described in any one of (1) to (4) that measures the location of a viewer located in the space by using radio waves transmitted or received by a terminal device carried by the viewer.
[0195] (6) The above measuring unit is,
[0196] A sound processing device described in any one of (1) to (5), which measures the distance to the ceiling of the space based on the reflected sound of a sound emitted from a sound irradiation unit equipped with a speaker located in the space, as the spatial shape of the space.
[0197] (7) The above measuring unit is,
[0198] Continuously measuring the location of viewers in the above space, the number and arrangement of speakers, and the shape of the space, and
[0199] The above correction unit is,
[0200] A sound processing device described in any one of (1) to (6) that uses information continuously measured by the above measuring unit to correct the sound of the content emitted from a speaker located in the space.
[0201] (8) The above acquisition part is,
[0202] A sound processing device described in any one of (1) to (7) above, which obtains a recommended environment specified in the content from metadata included in the content above.
[0203] (9) The above acquisition part is,
[0204] Obtain the head transfer function of the above viewer, and
[0205] The above correction unit is,
[0206] A sound processing device described in any one of (1) to (8) that corrects the sound of the speaker placed near the viewer based on the viewer's head transfer function.
[0207] (10) The above measuring unit is,
[0208] A sound processing device described in any one of (1) to (9), which generates map information based on an image captured by an image sensor or external device equipped with the sound processing device, and measures at least one of the self-position of the sound processing device, the position of the viewer, the number and arrangement of speakers, and the spatial shape based on the generated map information.
[0209] (11) The above correction unit is,
[0210] An audio processing device described in any one of (1) to (10), which provides information measured by the above measuring unit to a terminal device used by the viewer, and corrects the voice of the content based on at least one of the location of the viewer located in the space corrected by the viewer on the terminal device, the number and arrangement of speakers, and the shape of the space.
[0211] (12) The above correction unit is,
[0212] A sound processing device described in any one of (1) to (11) that corrects the voice of the content corrected by the correction unit, based on the correction performed by the viewer.
[0213] (13) The above correction unit is,
[0214] A sound processing device described in any one of (1) to (12) that corrects the sound of the content based on the behavioral pattern of the viewer learned based on the information measured by the measurement unit or the arrangement pattern of the speaker.
[0215] (14) The computer,
[0216] Obtaining a recommended environment defined for each content, including the ideal placement of speakers in the space where the content is played, and
[0217] Measure the location of viewers in the above space, the number and arrangement of speakers, and the shape of the space, and
[0218] Based on the above measured information, the sound of the content emitted from a speaker located in the space, which is the sound observed at the viewer's location, is corrected to sound emitted from a virtual speaker ideally placed in the recommended environment.
[0219] A sound processing method including the following.
[0220] (15) computer,
[0221] A acquiring unit that acquires a recommended environment defined for each content, including the ideal placement of speakers in the space where the content is played, and
[0222] A measuring unit that measures the location of a viewer located in the above space, the number and arrangement of speakers, and the shape of the space, and
[0223] A correction unit that, based on information measured by the above-mentioned measuring unit, corrects the sound of the content emitted from a speaker located in the space, which is the sound observed at the viewer's location, into the sound emitted from a virtual speaker ideally positioned in the recommended environment.
[0224] A sound processing program to function as.
[0225] (16) A sound processing system including a sound processing device and a speaker, and
[0226] The above sound processing device is,
[0227] A acquiring unit that acquires a recommended environment defined for each content, including the ideal placement of speakers in the space where the content is played, and
[0228] A measuring unit for measuring the location of a viewer located in the above space, the number and arrangement of the above speakers, and the shape of the space, and
[0229] Based on information measured by the above-mentioned measuring unit, the correction unit corrects the sound of the content, which is observed at the location of the viewer and is emitted from a speaker located in the space, into the sound emitted from a virtual speaker ideally positioned in the recommended environment.
[0230] The above speaker is,
[0231] A voice irradiation unit that irradiates a voice signal toward a predetermined location in the above space, and
[0232] The apparatus is equipped with an observation unit that observes the reflected sound of a voice signal investigated by the voice investigation unit described above, and
[0233] The above measuring unit is,
[0234] Measuring the spatial shape based on the time from when a voice signal is irradiated by the voice irradiation unit until the reflected sound is observed by the observation unit,
[0235] Sound processing system. Explanation of the symbols
[0236] 1: Acoustic processing system 10: Virtual Speaker 50: User 100: Acoustic processing unit 110: Communications Department 120: Memory Department 121: Speaker information memory 122: Measurement result memory unit 130: Control unit 131: Acquisition Department 132: Measurement section 133: Correction section 140: Sensor 200: Speaker
Claims
Claim 1 An audio processing device comprising: an acquisition unit for acquiring a recommended environment defined for each content, including an ideal arrangement of speakers in a space where content is played; a measurement unit for measuring the location of a viewer located in the space, the number and arrangement of speakers, and the shape of the space; and a correction unit for correcting the sound of the content, which is observed at the location of the viewer and is emitted from speakers located in the space, into a sound emitted from virtual speakers ideally arranged in the recommended environment, based on information measured by the measurement unit.
