Acoustic processing method, acoustic processing system, and acoustic processing program

Abstract

To provide an acoustic processing device that heightens a sense of realism given to acoustics listeners.SOLUTION: An acoustic processing device 1 comprises an acquisition unit 41, a storage unit 3, a processing unit 42, and a provision unit 43. The acquisition unit 41 acquires place information concerning a place where the sound field of an acoustic content 32 is reproduced. The storage unit 3 stores acoustic information 34 related to acoustics in a location corresponding to the place information. The processing unit 42 converts the acoustic characteristic of the acoustic content 32 into an acoustic characteristic corresponding to the location on the basis of the place information. The provision unit 43 provides the acoustic content 32 and the acoustic information 34 having been converted into the acoustic characteristic corresponding to the location. The acquisition unit 41 acquires the altered acoustic information 34. The storage unit 3 stores the acoustic content 32 and the acoustic information 34 associating with each other.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to an audio processing device, Acoustic processing system, acoustic processing method, and acoustic processing program Regarding. [Background technology]

[0002] There is a sound field collection and generation device that drives speakers with drive signals generated using wave field synthesis from sound field signals collected by multiple microphones, virtually reproducing the sound field at the sound collection location (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-171111 Summary of the Invention [Problem to be solved by the invention]

[0004] However, there are cases where simply reproducing a sound field is not enough to give a listener a sufficient sense of realism. Therefore, the present disclosure proposes a sound processing device that can enhance the sense of realism given to a listener. [Means for solving the problem]

[0005] The sound processing device according to the present disclosure includes an acquisition unit, a storage unit, and process The acquisition unit acquires location information of a location where sound fields of audio content are to be reproduced. The storage unit stores a location information corresponding to the location information. Sound with reverberant or reverberant characteristics The acoustic characteristics relating to the sound are stored. process The department, based on the location information, 、 The audio content , including reverberation or reflection characteristics The providing unit converts the acoustic characteristics into acoustic characteristics according to the location. The providing unit provides the acoustic content converted into the acoustic characteristics according to the location and the acoustic characteristics. The acquisition unit acquires the changed acoustic characteristics. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 is an explanatory diagram illustrating an overview of sound processing performed by a sound processing device according to the present disclosure. [Figure 2] 1 is a block diagram illustrating an example of a configuration of a sound processing device according to the present disclosure. [Figure 3] FIG. 1 is an explanatory diagram of a method for creating a VR spherical image according to the present disclosure. [Figure 4] FIG. 10 is an explanatory diagram illustrating an example of acoustic information according to the present disclosure. [Figure 5] 1 is an explanatory diagram of a method for measuring HRTFs according to the present disclosure. [Figure 6] 1 is an explanatory diagram of a method for measuring HRTFs according to the present disclosure. [Figure 7] FIG. 10 is an explanatory diagram illustrating an example of the operation of the sound processing device according to the present disclosure. [Figure 8] FIG. 10 is an explanatory diagram illustrating an example of the operation of the sound processing device according to the present disclosure. [Figure 9] FIG. 10 is an explanatory diagram illustrating an example of the operation of the sound processing device according to the present disclosure. [Figure 10] FIG. 10 is an explanatory diagram illustrating an example of the operation of the sound processing device according to the present disclosure. [Figure 11] 10 is a flowchart illustrating an example of processing executed by a control unit of the sound processing device according to the present disclosure. [Figure 12] 10 is a flowchart illustrating an example of processing executed by a control unit of the sound processing device according to the present disclosure. [Figure 13] 10 is a flowchart illustrating an example of processing executed by a control unit of the sound processing device according to the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0007] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and redundant description will be omitted.

[0008] (1. Overview of Acoustic Processing) 1 is an explanatory diagram showing an overview of sound processing by a sound processing device 1 according to the present disclosure. The sound processing device 1 is a device that reproduces a sound field in a state where the sound field of the recording location is reproduced by converting the acoustic characteristics of audio content such as movies, live music performances, music promotional videos, television programs, and songs into the acoustic characteristics of the recording location.

[0009] Here, the sound processing device 1 may not be able to give a sufficient sense of realism to the listener of the audio content simply by reproducing the sound field of the recording location. Specifically, when the measurement location where the acoustic characteristics of the audio content are measured and the playback location of the audio content match, the sound processing device 1 can give the listener a sense of realism as if they were actually there, but when this is not the case, the sense of realism is reduced by half.

[0010] For example, even if the sound processing device 1 provides a user watching a movie at home with movie sound content converted into sound characteristics that reproduce the reverberation characteristics and echo characteristics of a movie theater, it is difficult to give the user the feeling that they are in a movie theater because the user is at home.

[0011] Therefore, the sound processing device 1 uses technology such as virtual reality (hereinafter referred to as VR) to reproduce the measurement location where the acoustic information of the sound content was measured, and then reproduces the sound field of the sound content that reproduces the acoustic characteristics of the measurement location.

[0012] 1, the sound processing device 1 stores in advance a movie video content D1, a movie audio content D2, movie theater audio information D3, and a VR spherical image D4 of the movie theater (step S01). The movie theater audio information D3 includes various parameters related to the acoustic characteristics of the movie theater.

[0013] Then, when the sound processing device 1 receives a request for movie content from a user terminal 11 such as a smartphone carried by a user U, it transmits and provides the movie video content D1 and the movie audio content D2 to the user terminal 11 of the user U.

[0014] At this time, the sound processing device 1 converts the acoustic characteristics of the movie sound content D2 into the acoustic characteristics of the movie theater based on the movie theater sound information D3, and further provides the converted acoustic characteristics to the user terminal 11 together with the VR spherical image D4 inside the movie theater (step S02).

[0015] The VR spherical image D4 of the movie theater includes an image of a screen installed in the movie theater and an image of the surrounding environment of the screen, including audience seats, the walls and ceiling of the movie theater, etc. The sound processing device 1 adds information to the movie video content D1 indicating that the position of the screen in the VR spherical image D4 in the movie theater is to be the display position of the movie video content D1, and provides the added information to the user terminal 11.

[0016] As a result, the sound processing device 1 can, for example, display a VR spherical image D4 of a movie theater on the head-mounted display 12 worn by the user U, and display the movie video content D1 on a screen within the VR spherical image D4 (step S03).

[0017] At the same time, the sound processing device 1 can reproduce the sound field of the sound content D2 of the video content D1, the acoustic characteristics of which have been converted to those of a movie theater, by, for example, the earphones 13 worn by the user U (step S04).

[0018] In this way, the sound processing device 1 can allow the user U to listen to the sound content D2 while allowing the user U to see not only the video content D1 of the movie projected on the screen, but also the surrounding environment of the screen, such as the movie theater seats, walls, ceiling, etc.

[0019] This allows the sound processing device 1 to provide a user U who is watching movie video content at home with a sense of realism as if they were watching the movie in a movie theater. Here, the VR spherical image D4 of the inside of the movie theater is displayed on the head-mounted display 12, but the sound processing device 1 may display a 3DCG (Dimensional Computer Graphics) image that reproduces the inside of the movie theater instead of the VR spherical image D4 of the inside of the movie theater. Note that the operation of the sound processing device 1 described with reference to FIG. 1 is just one example. Other operation examples of the sound processing device 1 will be described later with reference to FIGS. 7 to 10.

[0020] (2. Configuration of sound processing device) Next, an example of the configuration of the sound processing device 1 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the configuration of the sound processing device 1 according to the present disclosure. As shown in Fig. 2, the sound processing device 1 includes a communication unit 2, a storage unit 3, and a control unit 4.

[0021] The communication unit 2 is realized by, for example, a network interface card (NIC), etc. The communication unit 2 is connected to the user terminal 11, the audio information creation device 100, and the creator terminal 101 via a communication network N such as the Internet, either wired or wirelessly, so as to be able to communicate information with them.

[0022] The audio information creation device 100 is a device that creates audio information 34, which will be described later. The creator terminal 101 is a terminal device used by a creator that creates audio content 32 to be provided to a user U by the audio processing device 1.

[0023] The storage unit 3 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 3 stores video content 31, audio content 32, VR video information 33, audio information 34, etc.

[0024] The video content 31 is a plurality of pieces of video data such as movies, live music performances, promotional videos of songs, and television programs, and is content data provided to the user terminal 11 by the sound processing device 1.

[0025] The audio content 32 is a plurality of pieces of audio data such as movies, live music, promotional videos for songs, television programs, and songs, and is content data provided to the user terminal 11 by the audio processing device 1.

[0026] The VR video information 33 includes a plurality of VR spherical images captured at various locations. Here, an example of a method for creating a VR spherical image will be described with reference to Fig. 3. Fig. 3 is an explanatory diagram of the method for creating a VR spherical image according to the present disclosure.

[0027] As shown in FIG. 3, when creating a VR spherical image, a 360° camera 102 is installed at each location where audio content 32 is played back, and the 360° camera 102 captures images in all directions, front, back, up, down, left, and right, of the location, thereby capturing the VR spherical image Vr.

[0028] As a result, for example, by installing a 360° camera 102 in a movie theater and taking images, it is possible to create a VR spherical video Vr that includes images of the movie theater screen shown in FIG. 1 and the surrounding environment of the screen, such as the theater seats, walls, ceiling, etc.

[0029] Note that the example shown in FIG. 1 is a movie theater with one screen, but this disclosure also allows for the creation of a VR spherical image Vr of a movie theater with four screens: in front, on both sides, and at the bottom.

[0030] In this case, the sound processing device 1 displays the video content 31 on one of the four screens at the front, and displays the surrounding environment of the movie theater on the other three screens. This also allows the sound processing device 1 to improve the sense of realism given to the user.

[0031] Returning to Fig. 2, the acoustic information 34 will be described. The acoustic information 34 includes a plurality of pieces of information related to the acoustics at each location where the acoustic content 32 is reproduced in the sound field. An example of the acoustic information 34 will now be described with reference to Fig. 4. Fig. 4 is an explanatory diagram showing an example of the acoustic information 34 according to the present disclosure.

[0032] 4, the acoustic information 34 is provided for each user to whom the video content 31 and the audio content 32 are provided by the audio processing device 1. The acoustic information 34 is information in which a user ID, a user's head-related transfer function (hereinafter referred to as HRTF: Head Related Transfer Function), a location, a VR video, and audio parameters are associated with each other.

[0033] The user ID is identification information for identifying each user. The HRTF is function information unique to each user that mathematically represents how sound reaches the user's ears from the sound source. Here, we will explain how to measure the HRTF with reference to Figures 5 and 6.

[0034] 5 and 6 are explanatory diagrams of a method for measuring HRTFs according to the present disclosure. For example, when measuring the HRTFs contained in the movie theater acoustic information D3 described with reference to Fig. 1, as shown in Fig. 5, a user U is asked to wear an ear microphone 14 in a seat in a movie theater Mt, and a test signal Ts is output from a speaker SP in the movie theater.

[0035] Then, the acoustic information creation device 100 acquires an audio signal SL collected by an ear microphone 14 attached to the left ear of the user U, and an audio signal SR collected by an ear microphone 14 attached to the right ear of the user U.

[0036] The acoustic information creation device 100 then derives the HRTF of the user U based on the time difference, the difference in signal level (intensity), the difference in reverberation, etc. between the two acquired audio signals SL and SR. In this way, the acoustic information creation device 100 can derive an accurate HRTF of the user U by actually measuring the test signal Ts heard by the user U.

[0037] The HRTF differs depending on the location (environment) where the user U listens to the test signal Ts. For this reason, for example, if there are multiple videos of different locations that the user wants to view while listening to the audio content 32, the user must visit each location to measure and derive the HRTF, which is a burden on the user.

[0038] Therefore, the acoustic information creation device 100 can derive the HRTFs of the user U at multiple locations while reducing the burden on the user U. For example, as shown in Fig. 6, the audio signal collected by the ear microphone 14 attached to the ear of the user U has a period in the beginning where the sound wave characteristics depend on the user U, and thereafter a period where the sound wave characteristics depend on the location.

[0039] For this reason, the acoustic information creation device 100, for example, has the user U come to one location, collects the test signal Ts with the ear microphone 14, and acquires the audio signal waveform of the time portion that depends on the user U. After that, the acoustic information creation device 100 places a dummy doll DM equipped with the ear microphone 14 in multiple locations desired by the user U, and acquires the audio signal waveform of the time portion whose sound wave characteristics depend on the location.

[0040] Then, the acoustic information creation device 100 synthesizes the audio signal waveform of the time portion dependent on the user U with the audio signal waveform of the time portion dependent on the location obtained using a dummy doll at multiple locations, and derives the HRTF of the user U at each location based on the synthesized signal.

[0041] As a result, the acoustic information creating device 100 can derive the HRTFs of the user U at a plurality of locations desired by the user U while reducing the burden on the user U, although the accuracy is slightly lower than when actual measurements are taken.

[0042] Furthermore, the acoustic information creation device 100 can also, for example, have the user U take a photo of their own ear and send the image data, and estimate and derive the HRTF of the user U based on the image data of the ear. In such a case, when image data including an image of an ear is input, the acoustic information creation device 100 derives the HRTF of the user U using a learning model that has been machine-trained to output the HRTF corresponding to that ear.

[0043] This allows the acoustic information creation device 100 to estimate and derive the user's HRTF without the user U having to come to a location where the HRTF is to be measured, thereby further reducing the burden on the user U required to measure the HRTF.

[0044] 3 , the description of the audio information 34 will be continued. The location included in the audio information 34 is identification information for identifying a location that a pre-registered user U wants to view while listening to audio content 32. The VR video is identification information for identifying a VR spherical video corresponding to the location included in the audio information 34.

[0045] The acoustic parameters are associated with numerical values ​​indicating reverberation characteristics such as reverberation time and echo characteristics such as the reflection coefficient of sound waves for each of a plurality of sound output positions in each location. The acoustic information creation device 100 measures the actual acoustics at each location and derives the acoustic parameters based on the collected acoustics. This allows the acoustic information creation device 100 to derive accurate acoustic parameters that are appropriate for the actual location. Note that while FIG. 4 conceptually lists the data for each item as "A01" and "B01," in reality, specific data corresponding to each item is stored in the data for each item.

[0046] The acoustic information creation device 100 transmits the created acoustic information 34 to the acoustic processing device 1. The acoustic processing device 1 stores the acoustic information 34 received from the acoustic information creation device 100 in the storage unit 3. Note that although the case where the acoustic information creation device 100 creates the acoustic information 34 has been described here, the acoustic processing device 1 may have the same functions and configuration as the acoustic information creation device 100 and may create the acoustic information 34 by itself and store it in the storage unit 3.

[0047] 2, the control unit 4 will be described. The control unit 4 includes, for example, a microcomputer having a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), input / output ports, and various other circuits.

[0048] The control unit 4 includes an acquisition unit 41, a processing unit 42, and a provision unit 43 that function by the CPU executing various programs (corresponding to an example of an acoustic processing program in the embodiment) stored in the ROM using the RAM as a working area.

[0049] The acquisition unit 41, processing unit 42, and providing unit 43 included in the control unit 4 may each be partially or entirely configured using hardware such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0050] The acquisition unit 41, processing unit 42, and providing unit 43 each realize or execute the information processing functions described below. Note that the internal configuration of the control unit 4 is not limited to the configuration shown in Fig. 2, and may be any other configuration that performs the information processing described below.

[0051] The acquisition unit 41 acquires, for example, a request to provide the video content 31 and the audio content 32 from the user U. Furthermore, when the acquisition unit 41 acquires a request to provide the video content 31 and the audio content 32 from the user U, the acquisition unit 41 acquires, from the user terminal 11, information about a location where the audio content 32 can be viewed by a user who is listening to the audio content 32.

[0052] For example, when the acquisition unit 41 acquires a request to provide movie video content D1 and movie audio content D2 from the user terminal 11 and acquires information indicating a movie theater as location information from the user terminal 11, it outputs the information acquired from the user terminal 11 to the processing unit 42.

[0053] When information acquired from the user terminal 11 is input from the acquisition unit 41, the processing unit 42 converts the acoustic characteristics of the acoustic content 32 based on the acoustic information 34 into acoustic characteristics corresponding to the location corresponding to the location information acquired from the user terminal 11 and outputs them to the provision unit 43.

[0054] At this time, the processing unit 42 applies the HRTF of each user U to convert the acoustic characteristics of the acoustic content 32. This allows the processing unit 42 to convert the acoustic characteristics of the acoustic content 32 so that the acoustic characteristics are optimal for the user U. The processing unit 42 outputs the information acquired from the user terminal 11 to the providing unit 43 together with the acoustic content 32 after the acoustic characteristic conversion.

[0055] The providing unit 43 transmits to the user terminal 11 the audio content 32 input from the processing unit 42, the VR spherical image of the place corresponding to the place information, and the video content 31 requested to be provided by the user.

[0056] As a result, the sound processing device 1 allows the user U to visually recognize not only the video content D1 of a movie projected on a screen, as shown in Figure 1, but also the surrounding environment of the screen, such as the theater's seats, walls, ceiling, etc.

[0057] At the same time, the sound processing device 1 can allow the user to listen to the movie audio content D2 converted to have the acoustic characteristics of a movie theater. Therefore, the sound processing device 1 can give a user U who is watching movie video content at home, for example, a sense of realism as if he or she were watching the movie in a movie theater.

[0058] Up to this point, we have explained the case where the acquisition unit 41 acquires a request to provide video content 31 and audio content 32 from the user terminal 11, but the acquisition unit 41 may also acquire an image of the location of user U along with the request to provide audio content 32 from the user terminal 11.

[0059] In this case, the processing unit 42 converts the acoustic characteristics of the audio content 32 depending on the location of the user U, and the providing unit 43 transmits the converted audio content 32 to the user terminal 11, causing the user terminal 11 to reproduce the sound field. An example of the operation of the audio processing device 1 will be described later with reference to FIGS. 7 and 8.

[0060] The acquisition unit 41 may also acquire, from the creator who created the audio content 32, audio information 34 of the audio content 32 and a request to provide a VR spherical image of a location where the audio content 32 is to be reproduced as a sound field. An example of the operation of the audio processing device 1 in such a case will be described later with reference to FIG.

[0061] (3. Example of operation of sound processing device) Next, an example of the operation of the sound processing device 1 will be described with reference to Fig. 7 to Fig. 10. Fig. 7 to Fig. 10 are explanatory diagrams of an example of the operation of the sound processing device 1 according to the present disclosure.

[0062] As shown in Figure 7, the acquisition unit 41 of the sound processing device 1 may, for example, acquire an image Pic1 of the user's location (here, the interior of the vehicle C) captured by the user terminal 11 when the user U is inside the vehicle C, and a request to provide sound content 32.

[0063] In such a case, the sound processing device 1 predicts the acoustic characteristics of the location from the image Pic1 of the location, converts the acoustic characteristics of the audio content 32 into the predicted acoustic characteristics, and reproduces the sound field on the user terminal 11. For example, the processing unit 42 of the sound processing device 1 determines that the location of the user U is the space inside the cabin of the vehicle C by performing image recognition on the image Pic1.

[0064] The processing unit 42 then estimates the longitudinal length L, lateral length W, and vertical length H of the interior of the vehicle cabin from the image Pic1 to predict the size of the space within the vehicle cabin, and predicts the acoustic characteristics of the space within the vehicle cabin, such as the echo characteristics and reverberation characteristics, based on the size of the space within the vehicle cabin.

[0065] Next, the processing unit 42 converts the acoustic characteristics of the acoustic content 32 into predicted acoustic characteristics, and, for example, places a virtual speaker SpC in a central front position in the vehicle cabin, and places virtual speakers SpL and SpR at positions 30° to the left and right of the center.

[0066] Then, the processing unit 42 converts the acoustic characteristics of the audio content 32 so that it sounds as if it is coming from the three virtual speakers PcC, SpL, and SpR, outputs the converted audio content 32 to the providing unit 43, and causes the providing unit 43 to transmit the audio content 32 to the user terminal 11.

[0067] As a result, when the user U listens to the audio content 32 through earphones, the audio processing device 1 can provide the user U with a sense of realism as if he or she were listening to the audio content 32 through high-quality car audio.

[0068] Also, as shown in FIG. 8, the acquisition unit 41 may acquire an image Pic2 of the user U's location captured by the user U when the user U is in the living room of his or her home, and a request to provide video content 31 and audio content 32.

[0069] At this time, the acquisition unit 41 may also acquire information indicating that, for example, the user U has selected a predetermined area A (here, an area surrounding the television Tv) that includes the center of the user U's field of view from the image Pic2.

[0070] In this case, the processing unit 42 arranges virtual speakers Sp1, Sp2, Sp3, Sp4, Sp5, and Sp6, which are the sound output positions of the audio content 32, so as to surround the predetermined area A. The processing unit 42 then converts the acoustic characteristics of the audio content 32 so that the audio sounds as if they are being output from the virtual speakers Sp1, Sp2, Sp3, Sp4, Sp5, and Sp6, and outputs the converted audio content to the providing unit 43.

[0071] The providing unit 43 transmits the video content 31 requested by the user U and the audio content 32 whose acoustic characteristics have been converted by the processing unit 42 to the user terminal 11, and causes the video content 31 to be displayed and the sound field of the audio content 32 to be reproduced.

[0072] As a result, when the user U listens to the audio content 32 with earphones, the audio processing device 1 can give the user U a sense of realism as if he or she were listening to the audio content 32 with a high-quality audio device while the video content 31 is being displayed on the television Tv.

[0073] 9, for example, the user U may be watching an animation work Vd on a television Tv. In this case, the sound processing device 1 may display augmented reality (AR) images Ca, Cc, and Cd of characters appearing in the work Vd around the user using a head-mounted display Cb. This allows the sound processing device 1 to further improve the sense of realism given to the user U.

[0074] Here, the location of the user U is predicted from an image captured by the user U, but this is just an example. The acquisition unit 41 can also acquire location information of the user U measured by a GPS (Global Positioning System) provided in the user terminal 11, for example.

[0075] In this case, the processing unit 42 predicts the user's location from the user's location information acquired by the acquisition unit, and converts the acoustic characteristics of the audio content 32 into the acoustic characteristics of the predicted location to reproduce a sound field. This allows the processing unit 42 to convert the acoustic characteristics of the audio content 32 into acoustic characteristics that correspond to the accurate location of the user U determined by GPS.

[0076] The acquisition unit 41 can also acquire from the user terminal 11 an image selected from images captured by the user U in the past, or an image viewed by the user U via the communication network N.

[0077] In this case, the processing unit 42 predicts the acoustic characteristics of the location shown in the image acquired by the acquisition unit 41, converts the acoustic characteristics of the audio content into the predicted acoustic characteristics, and reproduces the sound field. In this way, the sound processing device 1 can give the user U a sense of realism as if he or she were listening to the audio content 32 at a memorable place that the user U has visited in the past or a favorite place shown in an image that the user U has viewed in the past.

[0078] Furthermore, the processing unit 42 changes the number of virtual speakers to be arranged as sound output positions of the audio content 32 and the sound output characteristics of the virtual speakers, depending on the size of the space where the user U is predicted from the image. For example, the processing unit 42 increases the number of virtual speakers to be arranged as the size of the predicted space increases.

[0079] Furthermore, if the predicted space size is even larger, the processing unit 42 places virtual speakers, such as surround speakers, that have sound output characteristics such that the audio content 32 can be heard from all directions of 360°. This allows the sound processing device 1 to cause the user terminal 11 to reproduce an optimal sound field according to the size of the location of the user U.

[0080] The acquisition unit 41 may also acquire, for example, from the creator who created the audio content 32, a request to provide audio information 34 of the audio content 32 and a VR spherical image of the location where the audio content 32 is reproduced as a sound field.

[0081] In such a case, as shown in Figure 10, the processing unit 42 of the sound processing device 1 transmits the sound content 32, sound information 34, and VR video information 33 requested to be provided to the creator terminal 101 by the providing unit 43 to the creator terminal 101 used by the creator CR.

[0082] This allows the creator CR to change the audio information 34 based on his or her own creative intentions while viewing, for example, the VR spherical video Vr of a movie theater included in the VR video information 33. For example, the creator CR can change the audio information 34, which currently sounds as if virtual speakers Sp are placed on both sides of the movie theater screen, to audio information 34 that sounds as if the virtual speakers Sp are placed at positions further away from both sides of the screen.

[0083] Furthermore, the creator CR can change the acoustic information 34 to sound as if a new virtual speaker SpU were placed above the screen and a new virtual speaker SpD were placed below the screen. At this time, the creator CR listens to the audio content 32 to which his or her own HRTF has been applied and changes the acoustic information 34.

[0084] Then, the creator CR transmits the changed audio information 34a, audio content 32a, and VR video information 33 from the creator terminal 101 to the audio processing device 1. The audio processing device 1 stores the audio information 34a, audio content 32a, and VR video information 33 received from the creator terminal 101 in the storage unit 3.

[0085] As a result, the next time the sound processing device 1 provides the sound content 32a to the user U, it can reproduce the sound field of the sound content 32a with acoustic characteristics that reflect the creative intention of the creator CR. At this time, the sound processing device 1 provides the user U with the sound content 32a to which the HRTF of the user U has been applied, thereby reproducing the sound field of the sound content 32a with acoustic characteristics that are optimal for the user U.

[0086] (4. Processing performed by the sound processing device) Next, an example of processing executed by the control unit 4 of the sound processing device 1 will be described with reference to Fig. 11 to Fig. 13. Fig. 11 to Fig. 13 are flowcharts showing an example of processing executed by the control unit 4 of the sound processing device 1 according to the present disclosure.

[0087] 11 when receiving a content provision request including audio and video from a user U of the user terminal 11. Specifically, when receiving a content provision request from the user U, the control unit 4 of the sound processing device 1 first obtains content and location information desired by the user U from the user U (step S101).

[0088] Next, the control unit 4 converts the acoustic characteristics of the audio content corresponding to the content desired by the user into acoustic characteristics according to the audio characteristics corresponding to the location information (step S102). After that, the control unit 4 provides the user U with the video content, the audio content with the converted acoustic characteristics, and a VR spherical image that the user U is allowed to view when the audio content and the video content are played, thereby reproducing a sound field (step S103), and ends the process.

[0089] Furthermore, when the control unit 4 receives a request for providing audio content from a user and an image captured by the user showing the user's location, the control unit 4 executes the process shown in Fig. 12. Specifically, when the control unit 4 receives a request for providing content and the captured image from the user U, it first predicts the acoustic characteristics of the space shown in the captured image (step S201).

[0090] Next, the control unit 4 converts the acoustic characteristics of the acoustic content corresponding to the content desired by the user U into the acoustic characteristics of the space predicted in step S201 (step S202). After that, the control unit 4 provides the acoustic content with the converted acoustic characteristics to the user U, plays back the sound field (step S203), and ends the process.

[0091] Furthermore, when a predetermined area A in the captured image is selected by the user, the control unit 4 performs processing to arrange sound output positions of the audio content so as to surround the predetermined area A, and provides the audio content to the user U.

[0092] At this time, the control unit 4 provides the audio content to the user U by changing the number of sound output positions and sound output characteristics of the audio content to be arranged in accordance with the size of the predetermined area A selected by the user.

[0093] In addition, when the control unit 4 receives a request from the user U to provide content including audio and video, and also obtains an image showing the user's location, it can predict the acoustic characteristics of the space shown in the image and provide the user U with audio content converted to the predicted acoustic characteristics.

[0094] Furthermore, when the control unit 4 receives, for example, from the creator CR of the audio content, a request to provide audio information of the audio content and a location for reproducing the sound field of the audio content, the control unit 4 executes the process shown in FIG.

[0095] Specifically, when the control unit 4 receives a request to provide audio information and a location from the creator CR, the control unit 4 first provides the creator CR with the audio information and a VR spherical image of the location where the audio content corresponding to the audio information is reproduced in a sound field (step S301).

[0096] Next, the control unit 4 determines whether or not changed acoustic information has been acquired from the creator CR (step S302). If the control unit 4 determines that acoustic information has not been acquired from the creator CR (step S302, No), the control unit 4 repeats the determination process of step S302 until acoustic information is acquired.

[0097] Then, when the control unit 4 determines that acoustic information has been acquired from the creator CR (step S302, Yes), the control unit 4 stores the acquired acoustic characteristics in association with the VR spherical image provided to the creator CR (step S303), and ends the processing.

[0098] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0099] The present technology can also be configured as follows. (1) an acquisition unit that acquires location information relating to a location that can be seen by a user listening to the audio content; a storage unit that stores acoustic information related to the acoustics at the location; a processing unit that converts the acoustic characteristics of the audio content into acoustic characteristics according to the location based on the audio information and reproduces a sound field; An acoustic processing device having: (2) The storage unit storing a virtual reality spherical image of the location; The processing unit The virtual reality omnidirectional image is displayed to a user while a sound field of the audio content is being reproduced. The sound processing device according to (1) above. (3) The storage unit storing the virtual reality omnidirectional image including a screen on which video content corresponding to the audio content is displayed at the location and an image of the surrounding environment of the screen; The processing unit Displaying the video content on the screen in the virtual reality omnidirectional video. The sound processing device according to (2) above. (4) The storage unit storing the virtual reality spherical image including four screens on which video content corresponding to the audio content is displayed at the location; The processing unit The video content is displayed on one of the four screens in the virtual reality omnidirectional video, and an image of the surrounding environment of the location is displayed on the other three screens. The sound processing device according to (2) above. (5) The acquisition unit acquiring an image of the location taken by the user; The processing unit Predicting acoustic characteristics of the location from the image of the location, converting the acoustic characteristics of the audio content into the predicted acoustic characteristics, and reproducing a sound field. The sound processing device according to (1) above. (6) The processing unit Predicting acoustic characteristics of the space based on the size of the space of the location predicted from the image of the location. The sound processing device according to (5) above. (7) The processing unit Predict the reverberation and reflection characteristics of the space The sound processing device according to (6) above. (8) The processing unit The number of sound output positions and sound output characteristics of the audio content to be arranged are changed according to the spatial size of the location predicted from the image of the location. The sound processing device according to (6) above. (9) The processing unit When the user selects a predetermined area including the center of the user's visual field from the image of the location, the sound output position of the audio content is arranged so as to surround the predetermined area. The sound processing device according to any one of (5) to (8) above. (10) The acquisition unit Acquire an image selected from images previously captured by the user or an image viewed by the user via a communication network; The processing unit The acoustic characteristics of the location shown in the image are predicted, and the acoustic characteristics of the audio content are converted into the predicted acoustic characteristics to reproduce a sound field. The sound processing device according to (1) above. (11) The acquisition unit Acquire location information of the user measured by a GPS (Global Positioning System); The processing unit A location of the user is predicted from the location information of the user, and the acoustic characteristics of the audio content are converted into the acoustic characteristics of the predicted location to reproduce a sound field. The sound processing device according to (1) above. (12) a providing unit that provides the audio content, the virtual reality omnidirectional image of the location where the audio content is reproduced, and the audio information to a creator of the audio content; Furthermore, The acquisition unit obtaining the acoustic information as modified by the creator; The storage unit The audio content provided to the creator and a virtual reality omnidirectional image of a place where the audio content is reproduced in a sound field are stored in association with the audio information changed by the creator. The sound processing device according to (2) above. (13) The storage unit storing the acoustic information generated based on the acoustics measured at the location; The sound processing device according to any one of (1) to (12) above. (14) The storage unit storing a head-related transfer function of the user; The processing unit For each user, converting the acoustic characteristics of the audio content by applying a head-related transfer function of the user. The sound processing device according to any one of (1) to (13) above. (15) The storage unit storing the head-related transfer function derived based on sound recorded by an ear microphone worn by the user; The sound processing device according to (14) above. (16) The storage unit The head-related transfer function derived based on sound recorded by an ear microphone attached to the user and whose sound wave characteristics depend on the user at a time and sound recorded by an ear microphone attached to a doll and whose sound wave characteristics depend on the location is stored. The sound processing device according to (14) above. (17) The storage unit storing the head-related transfer functions derived based on the images of the user's ears; The sound processing device according to (14) above. (18) 1. A computer-implemented method for acoustic processing, comprising: an acquisition step of acquiring location information relating to a location visible to a user listening to the audio content; storing acoustic information relating to the acoustics at the location; a processing step of converting the acoustic characteristics of the audio content into acoustic characteristics according to the location based on the audio information and reproducing a sound field; An acoustic processing method comprising: (19) an acquisition step for acquiring location information relating to a location visible to a user listening to the audio content; a storage procedure for storing acoustic information relating to the acoustics at the location; a processing procedure for converting the acoustic characteristics of the audio content into acoustic characteristics according to the location based on the audio information and reproducing a sound field; An acoustic processing program that causes a computer to execute the above. [Explanation of symbols]

[0100] 1. Sound processing equipment 2. Communications Department 3 Storage section 31 Video Content 32 Audio Content 33 VR video information 34 Acoustic Information 4. Control section 41 Acquisition Department 42 Processing section 43 Providing Department

Claims

1. A computer-implemented acoustic processing method, comprising: an acquisition step of acquiring location information of a location where the sound field of the audio content is to be reproduced; a storage step of storing acoustic characteristics related to the sound, including reverberation characteristics or reflection characteristics corresponding to the location information; a processing step of converting acoustic characteristics of the audio content, including reverberation characteristics or reflection characteristics, into acoustic characteristics according to the location based on the location information; and providing the audio content converted into the audio characteristics according to the location, The acquiring step acquires the changed acoustic characteristics and an image of the location captured by the user, The processing step comprises: Predicting acoustic characteristics of the space based on the size of the space of the location predicted from the image of the location. Acoustic processing methods.

2. The storing step stores the changed acoustic characteristics in association with the acoustic content or the virtual reality spherical image. The acoustic processing method according to claim 1 .

3. The processing step comprises: providing the virtual reality omnidirectional image corresponding to the location during sound field reproduction of the audio content; The acoustic processing method according to claim 2 .

4. The storing step includes: storing the virtual reality omnidirectional image including a screen on which video content corresponding to the audio content is displayed at the location and an image of the surrounding environment of the screen; The processing step comprises: Displaying the video content on the screen in the virtual reality omnidirectional video. The acoustic processing method according to claim 3 .

5. The storing step includes: storing a head-related transfer function of the user; The processing step comprises: For each user, apply a head-related transfer function of the user to convert the acoustic characteristics of the audio content. The acoustic processing method according to claim 1 .

6. The processing step comprises: Converting the acoustic characteristics of the audio content into acoustic characteristics including the head-related transfer functions of the user at each location. The acoustic processing method according to claim 5 .

7. The head-related transfer function of the user at each location is obtained by combining an audio signal waveform of a predetermined time portion whose sound wave characteristics depend on the user and an audio signal waveform of a predetermined time step whose sound wave characteristics depend on the location. The acoustic processing method according to claim 6.

8. The storing step stores the head-related transfer functions at the plurality of locations. The acoustic processing method according to claim 7.

9. The head-related transfer function is derived using a learning model that has been machine-learned to output a head-related transfer function corresponding to the user's ear when an image of the user's ear is input. The acoustic processing method according to claim 5 .

10. The head-related transfer function is derived from an audio signal picked up by an ear microphone worn by the user at the location. The acoustic processing method according to claim 5 .

11. The head-related transfer functions are derived from audio signals acquired at the location. The acoustic processing method according to claim 8.

12. The storing step includes: storing the acoustic characteristics generated based on the acoustics measured at the location; The acoustic processing method according to claim 1 .

13. The acoustic characteristics include acoustic parameters in which a numerical value indicating a reverberation characteristic or a reflection characteristic is associated with each of a plurality of sound output positions in each of the locations. The acoustic processing method according to claim 1 .

14. The processing step comprises: Predict the reverberation and reflection characteristics of the space The acoustic processing method according to claim 1 .

15. The processing step comprises: The number of sound output positions and sound output characteristics of the audio content to be arranged are changed according to the spatial size of the location predicted from the image of the location. The acoustic processing method according to claim 1 .

16. an acquisition unit that acquires location information of a location where sound fields of audio content are to be reproduced; a storage unit that stores acoustic characteristics related to the sound, including reverberation characteristics or reflection characteristics corresponding to the location information; a processing unit that converts acoustic characteristics of the audio content, including reverberation characteristics or reflection characteristics, into acoustic characteristics according to the location, based on the location information; a providing unit that provides audio content converted into audio characteristics according to the location, The acquisition unit Acquire an image of the location taken by the user; The processing unit Predicting acoustic characteristics of the space based on the size of the space of the location predicted from the image of the location. an audio processing device; an acoustic characteristic creating device that creates the acoustic characteristics, transmits the acoustic characteristics to the sound processing device, and stores the acoustic characteristics in the storage unit; a user terminal that receives from the sound processing device the sound content whose acoustic characteristics have been converted and that is provided from the sound processing device, and plays it back; an acoustic treatment system including:

17. an acquisition step for acquiring location information of a location where the sound field of the audio content is to be reproduced; a storage procedure for storing acoustic characteristics relating to the sound; a processing procedure for converting acoustic characteristics of the audio content, including reverberation characteristics or echo characteristics, into acoustic characteristics according to the location, based on the location information; a provision step of providing audio content converted into audio characteristics according to the location; on the computer, The acquisition procedure includes: Acquire an image of the location taken by the user; The processing procedure is as follows: Predicting acoustic characteristics of the space based on the size of the space of the location predicted from the image of the location. Sound processing program.

Images