Acoustic processing device, acoustic processing system, acoustic processing method, and acoustic processing program

Abstract

To provide an acoustic processing device that heightens a sense of realism given to acoustics listeners.SOLUTION: An acoustic processing device 1 comprises an acquisition unit 41, a storage unit 3, a processing unit 42, and a provision unit 43. The acquisition unit 41 acquires place information concerning a place where the sound field of an acoustic content 32 is reproduced. The storage unit 3 stores acoustic information 34 related to acoustics in a location corresponding to the place information. The processing unit 42 converts the acoustic characteristic of the acoustic content 32 into an acoustic characteristic corresponding to the location on the basis of the place information. The provision unit 43 provides the acoustic content 32 and the acoustic information 34 having been converted into the acoustic characteristic corresponding to the location. The acquisition unit 41 acquires the altered acoustic information 34. The storage unit 3 stores the acoustic content 32 and the acoustic information 34 associating with each other.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to an audio processing device. [Background technology]

[0002] There is a sound field collection and generation device that drives a speaker with a drive signal generated by using a wave field synthesis method from sound field signals collected by multiple microphones, and virtually reproduces the sound field at the sound collection location (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2015-171111 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, there are cases where merely reproducing a sound field is not enough to provide a sufficient sense of realism to a listener of the sound. Therefore, the present disclosure proposes a sound processing device that can enhance the sense of realism provided to a listener of the sound. [Means for solving the problem]

[0005] The sound processing device according to the present disclosure includes an acquisition unit, a storage unit, a conversion unit, and a provision unit. The acquisition unit acquires location information of a location where sound content is to be reproduced as a sound field. The storage unit stores sound information relating to sound at a location corresponding to the location information. The conversion unit converts acoustic characteristics of the sound content into acoustic characteristics corresponding to the location based on the location information. The provision unit provides the sound content and the sound information converted into acoustic characteristics corresponding to the location. The acquisition unit acquires the changed sound information. The storage unit stores the sound content and the sound information in association with each other. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is an explanatory diagram showing an overview of sound processing by a sound processing device according to the present disclosure. [Diagram 2] 1 is a block diagram showing an example of a configuration of a sound processing device according to the present disclosure. [Diagram 3] FIG. 1 is an explanatory diagram of a method for creating a VR spherical image according to the present disclosure. [Figure 4] FIG. 2 is an explanatory diagram showing an example of acoustic information according to the present disclosure. [Diagram 5] 1 is an explanatory diagram of a method for measuring HRTF according to the present disclosure. [Figure 6] 1 is an explanatory diagram of a method for measuring HRTF according to the present disclosure. [Figure 7] FIG. 2 is an explanatory diagram of an operation example of the sound processing device according to the present disclosure. [Figure 8] FIG. 2 is an explanatory diagram of an operation example of the sound processing device according to the present disclosure. [Figure 9] FIG. 2 is an explanatory diagram of an operation example of the sound processing device according to the present disclosure. [Figure 10] FIG. 2 is an explanatory diagram of an operation example of the sound processing device according to the present disclosure. [Figure 11] 10 is a flowchart illustrating an example of a process executed by a control unit of the sound processing device according to the present disclosure. [Figure 12] 10 is a flowchart illustrating an example of a process executed by a control unit of the sound processing device according to the present disclosure. [Figure 13] 10 is a flowchart illustrating an example of a process executed by a control unit of the sound processing device according to the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and duplicated descriptions will be omitted.

[0008] (1. Overview of Acoustic Processing) 1 is an explanatory diagram showing an overview of sound processing by a sound processing device 1 according to the present disclosure. The sound processing device 1 is a device that reproduces a sound field in a state where the sound field of a recording location is reproduced by converting the acoustic characteristics of sound content such as movies, live music, music promotion videos, television programs, and music into the acoustic characteristics of the recording location.

[0009] Here, the sound processing device 1 may not be able to provide a sufficient sense of realism to the listener of the sound content by simply reproducing the sound field of the recording location. Specifically, when the measurement location where the acoustic characteristics of the sound content are measured and the playback location of the sound content are the same, the sound processing device 1 can provide the listener with a sense of realism as if they were in that location, but when this is not the case, the sense of realism is reduced by half.

[0010] For example, even if the audio processing device 1 provides a user watching a movie at home with movie audio content that has been converted to acoustic characteristics that reproduce the reverberation and echo characteristics of a movie theater, it is difficult to give the user the feeling that they are in a movie theater because the user is at home.

[0011] Therefore, the sound processing device 1 uses technology such as virtual reality (hereinafter referred to as VR) to reproduce the measurement location where the sound information of the sound content was measured, and then reproduces the sound field of the sound content that reproduces the acoustic characteristics of the measurement location.

[0012] For example, as shown in Fig. 1, the sound processing device 1 stores in advance a movie video content D1, a movie audio content D2, movie theater audio information D3, and a VR spherical image D4 in a movie theater (step S01). The movie theater audio information D3 includes various parameters related to the acoustic characteristics in the movie theater.

[0013] Then, for example, when the sound processing device 1 receives a request for movie content from a user terminal 11 such as a smartphone carried by a user U, it transmits and provides movie video content D1 and movie audio content D2 to the user terminal 11 of the user U.

[0014] At this time, the audio processing device 1 converts the audio characteristics of the movie audio content D2 into the audio characteristics of the movie theater based on the audio information D3 of the movie theater, and further provides the converted audio characteristics to the user terminal 11 together with the VR spherical image D4 in the movie theater (step S02).

[0015] The VR spherical image D4 of the movie theater includes an image of a screen installed in the movie theater and an image of the surrounding environment of the screen, including the audience seats, the walls and ceiling of the movie theater, etc. The sound processing device 1 adds information indicating that the position of the screen in the VR spherical image D4 in the movie theater is the display position of the movie video content D1 to the user terminal 11.

[0016] As a result, the sound processing device 1 can, for example, display a VR spherical image D4 of a movie theater on the head-mounted display 12 worn by the user U, and display the movie video content D1 on a screen within the VR spherical image D4 (step S03).

[0017] At the same time, the sound processing device 1 can reproduce the sound field of the sound content D2 of the video content D1, the acoustic characteristics of which have been converted to those of a movie theater, by, for example, the earphones 13 worn by the user U (step S04).

[0018] In this way, the audio processing device 1 can play audio content D2 to the user U while allowing the user U to visually recognize not only the video content D1 of the movie projected on the screen, but also the surrounding environment of the screen, such as the cinema's seats, walls, ceiling, etc.

[0019] This allows the sound processing device 1 to provide a user U who is watching a movie video content at home with a sense of realism as if he or she were watching the movie in a movie theater. Here, the VR spherical image D4 of the inside of the movie theater is displayed on the head mounted display 12, but the sound processing device 1 may display a 3DCG (Dimensional Computer Graphics) image reproducing the inside of the movie theater instead of the VR spherical image D4 of the inside of the movie theater. Note that the operation of the sound processing device 1 described with reference to FIG. 1 is one example. Other operation examples of the sound processing device 1 will be described later with reference to FIGS. 7 to 10.

[0020] (2. Configuration of Sound Processing Device) Next, an example of the configuration of the sound processing device 1 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the configuration of the sound processing device 1 according to the present disclosure. As shown in Fig. 2, the sound processing device 1 includes a communication unit 2, a storage unit 3, and a control unit 4.

[0021] The communication unit 2 is realized by, for example, a network interface card (NIC) etc. The communication unit 2 is connected to the user terminal 11, the audio information creation device 100, and the creator terminal 101 in a wired or wireless manner via a communication network N such as the Internet so as to be able to communicate information.

[0022] The audio information creation device 100 is a device that creates audio information 34, which will be described later. The creator terminal 101 is a terminal device used by a creator that creates audio content 32 to be provided to a user U by the audio processing device 1.

[0023] The storage unit 3 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 3 stores video content 31, audio content 32, VR video information 33, audio information 34, and the like.

[0024] The video content 31 is a plurality of pieces of video data such as movies, live music performances, music promotional videos, and television programs, and is content data provided to the user terminal 11 by the sound processing device 1.

[0025] The audio content 32 is a plurality of pieces of audio data such as movies, live music, music promotion videos, television programs, and music, and is content data provided to the user terminal 11 by the audio processing device 1.

[0026] The VR video information 33 includes a plurality of VR spherical images captured at various locations. Here, an example of a method for creating a VR spherical image will be described with reference to Fig. 3. Fig. 3 is an explanatory diagram of a method for creating a VR spherical image according to the present disclosure.

[0027] As shown in FIG. 3, when creating a VR spherical image, a 360° camera 102 is installed at each location where the audio content 32 is played back, and the VR spherical image Vr is captured by capturing images in all directions, including front, back, up, down, left and right, of the location using the 360° camera 102.

[0028] As a result, for example, by installing a 360° camera 102 in a movie theater and taking images, it is possible to create a VR spherical image Vr that includes images of the movie theater screen shown in FIG. 1 and the surrounding environment of the screen, such as the movie theater seats, walls, ceiling, etc.

[0029] Note that the example shown in FIG. 1 uses a movie theater with one screen, but this disclosure can also create a VR spherical image Vr of a movie theater with four screens: at the front, on both sides, and at the bottom.

[0030] In this case, the sound processing device 1 displays the video content 31 on one of the four screens, the screen at the front, and displays the surrounding environment of the movie theater on the other three screens. This also allows the sound processing device 1 to improve the sense of realism given to the user.

[0031] Returning to Fig. 2, the acoustic information 34 will be described. The acoustic information 34 includes a plurality of pieces of information related to the acoustics at each location where the acoustic content 32 is reproduced in the sound field. Here, an example of the acoustic information 34 will be described with reference to Fig. 4. Fig. 4 is an explanatory diagram showing an example of the acoustic information 34 according to the present disclosure.

[0032] 4, the acoustic information 34 is provided for each user to whom the video content 31 and the audio content 32 are provided by the audio processing device 1. The acoustic information 34 is information in which a user ID, a user's head-related transfer function (hereinafter, referred to as HRTF), a location, a VR video, and audio parameters are associated with each other.

[0033] The user ID is identification information for identifying each user. The HRTF is function information unique to each user that mathematically represents how sound reaches the user's ears from the sound source. Here, the measurement method of the HRTF will be described with reference to Figs. 5 and 6.

[0034] 5 and 6 are explanatory diagrams of a method for measuring HRTF according to the present disclosure. For example, when measuring the HRTF contained in the sound information D3 of a movie theater described with reference to Fig. 1, as shown in Fig. 5, a user U is asked to wear an ear microphone 14 in a seat in a movie theater Mt, and a test signal Ts is output from a speaker SP of the movie theater.

[0035] Then, the acoustic information creation device 100 acquires an audio signal SL collected by an ear microphone 14 attached to the left ear of the user U, and an audio signal SR collected by an ear microphone 14 attached to the right ear of the user U.

[0036] Then, the acoustic information creating device 100 derives the HRTF of the user U based on the time difference, the signal level (intensity) difference, the difference in reverberation, etc. between the two acquired audio signals SL and SR. In this way, the acoustic information creating device 100 can derive an accurate HRTF of the user U by actually measuring the test signal Ts heard by the user U.

[0037] The HRTF differs depending on the location (environment) where the user U listens to the test signal Ts. For this reason, for example, if there are multiple images of locations that the user wants to view while listening to the audio content 32, the user must visit each location to measure and derive the HRTF, which is a burden on the user.

[0038] Therefore, the acoustic information creating device 100 can derive the HRTFs of the user U in a plurality of locations while reducing the burden on the user U. For example, as shown in Fig. 6, the sound signal collected by the ear microphone 14 attached to the ear of the user U has a period during which the sound wave characteristics depend on the user U at an initial predetermined time portion, and thereafter has a period during which the sound wave characteristics depend on the location.

[0039] For this reason, the acoustic information creating device 100, for example, has the user U come to one location, collects the test signal Ts with the ear microphone 14, and acquires the audio signal waveform of the time portion that depends on the user U. After that, the acoustic information creating device 100 places a dummy doll DM equipped with the ear microphone 14 in multiple locations desired by the user U, and acquires the audio signal waveform of the time portion whose sound wave characteristics depend on the location.

[0040] Then, the acoustic information creation device 100 synthesizes the audio signal waveform of the time portion that is dependent on the user U with the audio signal waveform of the time portion that is dependent on the location obtained using dummy dolls at multiple locations, and derives the HRTF of the user U at each location based on the synthesized signal.

[0041] As a result, the acoustic information creating device 100 can derive the HRTFs of the user U at a plurality of locations desired by the user U while reducing the burden on the user U, although the accuracy is slightly lower than in the case of actual measurement.

[0042] Furthermore, the acoustic information creation device 100 can, for example, have the user U take a picture of his or her own ear and send the image data, and estimate and derive the HRTF of the user U based on the image data of the ear. In such a case, when image data including an image of an ear is input, the acoustic information creation device 100 derives the HRTF of the user U using a learning model that has been machine-learned to output the HRTF corresponding to the ear.

[0043] This allows the acoustic information creation device 100 to estimate and derive the user's HRTF without the user U having to come to a location where the HRTF is to be measured, thereby further reducing the burden on the user U required to measure the HRTF.

[0044] 3, the explanation of the audio information 34 will be continued. The location included in the audio information 34 is identification information for identifying a location that a pre-registered user U wants to see while listening to the audio content 32. The VR video is identification information for identifying a VR spherical video corresponding to the location included in the audio information 34.

[0045] The acoustic parameters are associated with a number indicating reverberation characteristics such as reverberation time and reflection characteristics such as a reflection coefficient of sound waves for each of a plurality of sound output positions in each location. The acoustic information creating device 100 measures the actual acoustics at each location and derives the acoustic parameters based on the collected acoustics. This allows the acoustic information creating device 100 to derive accurate acoustic parameters in accordance with the actual location. Note that while the data for each item is conceptually written as "A01" or "B01" in FIG. 4, in reality, specific data corresponding to each item is stored in the data for each item.

[0046] The acoustic information creation device 100 transmits the created acoustic information 34 to the acoustic processing device 1. The acoustic processing device 1 stores the acoustic information 34 received from the acoustic information creation device 100 in the storage unit 3. Note that, although the case where the acoustic information creation device 100 creates the acoustic information 34 has been described here, the acoustic processing device 1 may have the same functions and configuration as the acoustic information creation device 100 and create the acoustic information 34 by itself and store it in the storage unit 3.

[0047] 2, there will be described the control unit 4. The control unit 4 includes, for example, a microcomputer having a central processing unit (CPU), a read only memory (ROM), a random access memory (RAM), input / output ports, and various other circuits.

[0048] The control unit 4 includes an acquisition unit 41, a processing unit 42, and a provision unit 43 that function by the CPU executing various programs (corresponding to an example of an acoustic processing program in an embodiment) stored in the ROM using the RAM as a working area.

[0049] The acquisition unit 41, the processing unit 42, and the provision unit 43 included in the control unit 4 may each be partially or entirely configured with hardware such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0050] The acquisition unit 41, the processing unit 42, and the provision unit 43 each realize or execute the information processing action described below. Note that the internal configuration of the control unit 4 is not limited to the configuration shown in Fig. 2, and may be other configurations as long as they perform the information processing described below.

[0051] The acquisition unit 41 acquires, for example, a request to provide the video content 31 and the audio content 32 from the user U. Furthermore, when the acquisition unit 41 acquires a request to provide the video content 31 and the audio content 32 from the user U, the acquisition unit 41 acquires, from the user terminal 11, information regarding a location where the user listening to the audio content 32 can be seen.

[0052] For example, when the acquisition unit 41 acquires a request for providing movie video content D1 and movie audio content D2 from the user terminal 11, and acquires information indicating a movie theater as location information from the user terminal 11, it outputs the information acquired from the user terminal 11 to the processing unit 42.

[0053] When information acquired from the user terminal 11 is input from the acquisition unit 41, the processing unit 42 converts the acoustic characteristics of the acoustic content 32 based on the acoustic information 34 into acoustic characteristics corresponding to the location corresponding to the location information acquired from the user terminal 11, and outputs the converted acoustic characteristics to the provision unit 43.

[0054] At this time, the processing unit 42 applies the HRTF of the user U to convert the acoustic characteristics of the acoustic content 32 for each user U. This allows the processing unit 42 to convert the acoustic characteristics of the acoustic content 32 so that the acoustic characteristics become optimal for the user U. The processing unit 42 outputs the information acquired from the user terminal 11 to the providing unit 43 together with the acoustic content 32 after the acoustic characteristic conversion.

[0055] The providing unit 43 transmits to the user terminal 11 the audio content 32 input from the processing unit 42, the VR spherical image of the place corresponding to the place information, and the video content 31 requested to be provided by the user.

[0056] As a result, the sound processing device 1 allows the user U to visually recognize not only the movie video content D1 projected on the screen, as shown in FIG. 1, but also the surrounding environment of the screen, such as the audience seats, walls, and ceiling of the movie theater.

[0057] At the same time, the sound processing device 1 can allow the user to listen to the movie sound content D2 converted to the sound characteristics of the movie theater. Therefore, the sound processing device 1 can provide a user U who watches movie video content at home with a sense of realism as if he or she were watching the movie at a movie theater.

[0058] Up to this point, we have explained the case where the acquisition unit 41 acquires a request to provide video content 31 and audio content 32 from the user terminal 11. However, the acquisition unit 41 may also acquire an image of the location of the user U along with the request to provide audio content 32 from the user terminal 11.

[0059] In this case, the processing unit 42 converts the acoustic characteristics of the acoustic content 32 depending on the location of the user U, and transmits the converted acoustic content 32 to the user terminal 11 via the providing unit 43, and causes the user terminal 11 to reproduce the sound field. An example of the operation of the sound processing device 1 will be described later with reference to Figs. 7 and 8.

[0060] The acquisition unit 41 may also acquire, from a creator who created the audio content 32, audio information 34 of the audio content 32 and a request for providing a VR spherical image of a place where the audio content 32 is to be reproduced as a sound field. An example of the operation of the audio processing device 1 in such a case will be described later with reference to FIG.

[0061] (3. Example of operation of sound processing device) Next, an operation example of the sound processing device 1 will be described with reference to Fig. 7 to Fig. 10. Fig. 7 to Fig. 10 are explanatory diagrams of an operation example of the sound processing device 1 according to the present disclosure.

[0062] As shown in FIG. 7, the acquisition unit 41 of the sound processing device 1 may acquire, for example, an image Pic1 of a location (here, the interior of the vehicle C) captured by the user terminal 11 when a user U is inside a vehicle C, and a request to provide sound content 32.

[0063] In this case, the sound processing device 1 predicts the acoustic characteristics of the location from the image Pic1 of the location, converts the acoustic characteristics of the audio content 32 into the predicted acoustic characteristics, and reproduces the sound field on the user terminal 11. For example, the processing unit 42 of the sound processing device 1 determines that the location of the user U is the space inside the vehicle C by performing image recognition on the image Pic1.

[0064] Then, the processing unit 42 estimates the front-to-rear length L, lateral length W, and height length H of the vehicle interior from the image Pic1 to predict the size of the space within the vehicle interior, and predicts the acoustic characteristics of the space within the vehicle interior, such as the echo characteristics and reverberation characteristics, based on the size of the space within the vehicle interior.

[0065] Next, the processing unit 42 converts the acoustic characteristics of the acoustic content 32 into predicted acoustic characteristics, and, for example, places a virtual speaker SpC in a central front position in the vehicle cabin, and places virtual speakers SpL and SpR at positions 30° to the left and right of the center.

[0066] Then, the processing unit 42 converts the acoustic characteristics of the audio content 32 so that it sounds as if it is coming from the three virtual speakers PcC, SpL, and SpR, outputs it to the providing unit 43, and causes the providing unit 43 to transmit the audio content 32 to the user terminal 11.

[0067] As a result, when the user U listens to the audio content 32 through earphones, the audio processing device 1 can provide the user U with a sense of realism as if the user was listening to the audio content 32 through high-quality car audio.

[0068] Also, as shown in FIG. 8, the acquisition unit 41 may acquire an image Pic2 captured by the user U of his / her location while the user U is in the living room of his / her home, and a request to provide video content 31 and audio content 32.

[0069] At this time, the acquisition unit 41 may acquire information indicating that the user U has selected a predetermined area A (here, an area surrounding the television Tv) including the center of the user U's visual field from the image Pic2.

[0070] In this case, the processing unit 42 arranges virtual speakers Sp1, Sp2, Sp3, Sp4, Sp5, and Sp6, which are the sound output positions of the sound content 32, to surround the predetermined area A. Then, the processing unit 42 converts the acoustic characteristics of the sound content 32 so that the sound is heard as if it is being output from the virtual speakers Sp1, Sp2, Sp3, Sp4, Sp5, and Sp6, and outputs the converted sound to the providing unit 43.

[0071] The providing unit 43 transmits the video content 31 requested by the user U and the audio content 32 whose acoustic characteristics have been converted by the processing unit 42 to the user terminal 11, and causes the video content 31 to be displayed and the sound field of the audio content 32 to be reproduced.

[0072] As a result, when the user U listens to the audio content 32 with earphones, the audio processing device 1 can provide the user U with a sense of realism as if the user were listening to the audio content 32 with a high-quality audio device while the video content 31 is being displayed on the television Tv.

[0073] At this time, for example, as shown in Fig. 9, the user U may be watching an animation work Vd on a television Tv. In this case, the sound processing device 1 may display augmented reality (AR) images Ca, Cc, and Cd of characters appearing in the work Vd around the user using a head-mounted display Cb. This allows the sound processing device 1 to further improve the sense of realism given to the user U.

[0074] Note that, here, the location of the user U is predicted from an image captured by the user U, but this is just an example. The acquisition unit 41 can also acquire location information of the user U measured by a Global Positioning System (GPS) included in the user terminal 11, for example.

[0075] In this case, the processing unit 42 predicts the user's location from the user's location information acquired by the acquisition unit, and converts the acoustic characteristics of the acoustic content 32 into acoustic characteristics of the predicted location to reproduce the sound field. This allows the processing unit 42 to convert the acoustic characteristics of the acoustic content 32 into acoustic characteristics corresponding to the accurate location of the user U determined by GPS.

[0076] The acquisition unit 41 can also acquire from the user terminal 11 an image selected from images captured in the past by the user U, or an image viewed by the user U via the communication network N.

[0077] In this case, the processing unit 42 predicts the acoustic characteristics of the location shown in the image acquired by the acquisition unit 41, converts the acoustic characteristics of the audio content into the predicted acoustic characteristics, and reproduces the sound field. In this way, the audio processing device 1 can provide the user U with a sense of realism as if the user U were listening to the audio content 32 in, for example, a memorable place that the user U has visited in the past or a favorite place shown in an image that the user has viewed in the past.

[0078] Furthermore, the processing unit 42 changes the number of virtual speakers to be arranged as sound output positions of the audio content 32 and the sound output characteristics of the virtual speakers according to the size of the space where the user U is predicted from the image. For example, the processing unit 42 increases the number of virtual speakers to be arranged as the predicted space becomes larger.

[0079] Furthermore, when the predicted space size is larger, the processing unit 42 places virtual speakers with sound output characteristics such that the audio content 32 can be heard from 360° directions, such as surround speakers. This allows the audio processing device 1 to cause the user terminal 11 to reproduce an optimal sound field according to the size of the location of the user U.

[0080] Furthermore, the acquisition unit 41 may acquire, for example, from a creator who created the audio content 32, a request to provide audio information 34 of the audio content 32 and a VR spherical image of a location where the audio content 32 is to be reproduced as a sound field.

[0081] In such a case, as shown in FIG. 10, the processing unit 42 of the sound processing device 1 transmits the sound content 32, sound information 34, and VR video information 33 requested to be provided to the creator terminal 101 by the providing unit 43 to the creator terminal 101 used by the creator CR.

[0082] This allows the creator CR to change the audio information 34 based on his / her own creative intentions while, for example, viewing the VR spherical video Vr of a movie theater included in the VR video information 33. For example, the creator CR can change the audio information 34 that currently sounds as if virtual speakers Sp were placed on both sides of the movie theater screen to audio information 34 that sounds as if the virtual speakers Sp were placed at positions further away from both sides of the screen.

[0083] Furthermore, the creator CR can change the acoustic information 34 to one that sounds as if a new virtual speaker SpU is placed above the screen and a new virtual speaker SpD is placed below the screen. At this time, the creator CR changes the acoustic information 34 by listening to the acoustic content 32 to which his / her own HRTF has been applied.

[0084] Then, the creator CR transmits the changed audio information 34a, audio content 32a, and VR video information 33 from the creator terminal 101 to the audio processing device 1. The audio processing device 1 stores the audio information 34a, audio content 32a, and VR video information 33 received from the creator terminal 101 in the storage unit 3.

[0085] As a result, the sound processing device 1 can reproduce the sound field of the sound content 32a with acoustic characteristics that reflect the creative intention of the creator CR when providing the sound content 32a to the user U next time. At this time, the sound processing device 1 can reproduce the sound field of the sound content 32a with acoustic characteristics that are optimal for the user U by providing the user U with the sound content 32a to which the HRTF of the user U has been applied.

[0086] (4. Processing performed by the sound processing device) Next, an example of processing executed by the control unit 4 of the sound processing device 1 will be described with reference to Fig. 11 to Fig. 13. Fig. 11 to Fig. 13 are flowcharts showing an example of processing executed by the control unit 4 of the sound processing device 1 according to the present disclosure.

[0087] 11 when receiving a content provision request including audio and video from a user U of the user terminal 11. Specifically, when receiving the content provision request from the user U, the control unit 4 of the sound processing device 1 first obtains content and location information desired by the user U from the user U (step S101).

[0088] Next, the control unit 4 converts the acoustic characteristics of the audio content corresponding to the content desired by the user into acoustic characteristics according to the audio characteristics corresponding to the location information (step S102). After that, the control unit 4 provides the video content, the audio content with the converted acoustic characteristics, and a VR spherical image that is visually recognized by the user U when the audio content and the video content are reproduced to reproduce a sound field (step S103), and ends the process.

[0089] Furthermore, when the control unit 4 receives from the user a request for providing audio content and an image captured by the user showing the user's location, the control unit 4 executes the process shown in Fig. 12. Specifically, when the control unit 4 receives a request for providing content and the captured image from the user U, the control unit 4 first predicts the acoustic characteristics of the space shown in the captured image (step S201).

[0090] Next, the control unit 4 converts the acoustic characteristics of the acoustic content corresponding to the content desired by the user U into the acoustic characteristics of the space predicted in step S201 (step S202). After that, the control unit 4 provides the acoustic content with the converted acoustic characteristics to the user U to reproduce the sound field (step S203), and ends the process.

[0091] Furthermore, when a predetermined area A in the captured image is selected by the user, the control unit 4 performs a process of arranging sound output positions of the audio content so as to surround the predetermined area A, and provides the audio content to the user U.

[0092] At this time, the control unit 4 provides the audio content to the user U by changing the number of sound output positions and sound output characteristics of the audio content to be arranged in accordance with the size of the predetermined area A selected by the user.

[0093] In addition, when the control unit 4 receives a request from user U for content including audio and video, and obtains an image showing the user's location, it can predict the acoustic characteristics of the space shown in the captured image and provide user U with audio content that has been converted to have the predicted acoustic characteristics.

[0094] Furthermore, the control unit 4 executes the process shown in FIG. 13 when, for example, receiving from the creator CR of the audio content a request for providing audio information of the audio content and a location for reproducing the sound field of the audio content.

[0095] Specifically, when the control unit 4 receives a request for providing audio information and a location from the creator CR, the control unit 4 first provides the creator CR with the audio information and a VR spherical image of the location where the audio content corresponding to the audio information is reproduced in a sound field (step S301).

[0096] Next, the control unit 4 judges whether or not the changed sound information has been acquired from the creator CR (step S302). If the control unit 4 judges that the sound information has not been acquired from the creator CR (step S302, No), the control unit 4 repeats the judgment process of step S302 until the sound information is acquired.

[0097] Then, when it is determined that the acoustic information has been acquired from the creator CR (step S302, Yes), the control unit 4 stores the acquired acoustic characteristics in association with the VR spherical image provided to the creator CR (step S303), and ends the process.

[0098] It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0099] The present technology can also be configured as follows. (1) An acquisition unit that acquires location information related to a location that can be seen by a user listening to the audio content; A storage unit that stores acoustic information regarding the acoustics at the location; a processing unit that converts the acoustic characteristics of the audio content into acoustic characteristics according to the location based on the audio information and reproduces a sound field; An audio processing device having the above configuration. (2) The storage unit is storing a virtual reality omnidirectional image of the location; The processing unit includes: The virtual reality spherical image is displayed to a user while the sound field of the audio content is being reproduced. The sound processing device according to (1) above. (3) The storage unit is storing the virtual reality omnidirectional image including a screen on which video content corresponding to the audio content is displayed at the location and an image of an environment surrounding the screen; The processing unit includes: Displaying the video content on the screen in the virtual reality spherical video. The sound processing device according to (2) above. (4) The storage unit is storing the virtual reality spherical image including four screens on which video content corresponding to the audio content is displayed at the location; The processing unit includes: The video content is displayed on one of the four screens in the virtual reality spherical video, and an image of the surrounding environment of the location is displayed on the other three screens. The sound processing device according to (2) above. (5) The acquisition unit is acquiring an image of the location captured by the user; The processing unit includes: Predicting acoustic characteristics of the location from an image of the location, and converting the acoustic characteristics of the audio content into the predicted acoustic characteristics to reproduce a sound field. The sound processing device according to (1) above. (6) The processing unit includes: Predicting acoustic characteristics of the space based on the size of the space of the location predicted from the image of the location. The sound processing device according to (5) above. (7) The processing unit includes: Predict the reverberation and reflection characteristics of the space The sound processing device according to (6) above. (8) The processing unit includes: The number of sound output positions and sound output characteristics of the audio content to be arranged are changed according to the spatial size of the location predicted from the image of the location. The sound processing device according to (6) above. (9) The processing unit includes: When the user selects a predetermined area including a center of the user's visual field from the image of the location, the sound output position of the audio content is arranged so as to surround the predetermined area. The sound processing device according to any one of (5) to (8) above. (10) The acquisition unit is Acquire an image selected from images captured in the past by the user or an image viewed by the user via a communication network; The processing unit includes: The acoustic characteristics of a location shown in the image are predicted, and the acoustic characteristics of the audio content are converted into the predicted acoustic characteristics to reproduce a sound field. The sound processing device according to (1) above. (11) The acquisition unit is Acquire location information of the user measured by a Global Positioning System (GPS); The processing unit includes: A location of the user is predicted from the position information of the user, and the acoustic characteristics of the audio content are converted to the acoustic characteristics of the predicted location to reproduce a sound field. The sound processing device according to (1) above. (12) A providing unit that provides the audio content, the virtual reality omnidirectional image of the location where the audio content is reproduced, and the audio information to a creator of the audio content. Further equipped with The acquisition unit is Obtaining the audio information as modified by the creator; The storage unit is The audio content provided to the creator and a virtual reality omnidirectional image of a place where the audio content is reproduced in a sound field are stored in association with the audio information changed by the creator. The sound processing device according to (2) above. (13) The storage unit is storing the acoustic information generated based on the acoustics measured at the location; The sound processing device according to any one of (1) to (12) above. (14) The storage unit is storing a head-related transfer function of the user; The processing unit includes: For each user, converting the acoustic characteristics of the audio content by applying a head-related transfer function of the user. The sound processing device according to any one of (1) to (13) above. (15) The storage unit is Storing the head-related transfer function derived based on sound recorded by an ear microphone worn by the user The sound processing device according to (14) above. (16) The storage unit is The head-related transfer function derived based on sound recorded by an ear microphone attached to the user and whose sound wave characteristics depend on the user at a time and sound recorded by an ear microphone attached to a doll and whose sound wave characteristics depend on the location at a time is stored. The sound processing device according to (14) above. (17) The storage unit is storing the head-related transfer function derived based on the image of the user's ear; The sound processing device according to (14) above. (18) 1. A computer-implemented method for acoustic processing, comprising: acquiring location information relating to a location that is visible to a user listening to the audio content; storing acoustic information relating to the acoustics at the location; a processing step of converting the acoustic characteristics of the audio content into acoustic characteristics according to the location based on the audio information and reproducing a sound field; An acoustic processing method comprising: (19) an acquisition step for acquiring location information relating to a location that is visible to a user listening to the audio content; a storage step for storing acoustic information relating to the acoustics at the location; a processing step of converting the acoustic characteristics of the audio content into acoustic characteristics according to the location based on the audio information and reproducing a sound field; An acoustic processing program that causes a computer to execute the above. [Explanation of symbols]

[0100] 1. Sound Processing Device 2. Communications Department 3 Storage section 31 Video Content 32 Audio Content 33 VR video information 34 Acoustic Information 4. Control section 41 Acquisition Department 42 Processing section 43 Providing Department

Claims

[Claim 1] an acquisition unit that acquires location information of a location where sound field reproduction of audio content is to be performed; A storage unit that stores acoustic information related to an acoustic sound in a place corresponding to the place information; a conversion unit that converts acoustic characteristics of the audio content into acoustic characteristics corresponding to the location based on the location information; a providing unit that provides the audio content and the audio information that have been converted into audio characteristics according to the location, The acquisition unit is Obtaining the modified acoustic information; The storage unit is The audio content and the audio information are stored in association with each other. Sound processing equipment.

Images