Spatial sound restoration device, spatial sound restoration method, and program

Abstract

A spatial sound reconstruction device of one embodiment comprises: a video feature amount calculation unit that calculates a video feature amount on the basis of video information; a sound feature amount calculation unit that calculates a sound feature amount on the basis of sound information, which is monaural audio corresponding to the video information; and a coefficient calculation unit that calculates a high-order ambisonics coefficient on the basis of the video feature amount and the sound feature amount.

Description

[Technical Field]

[0001] The embodiments relate to a spatial sound restoration device, a spatial sound restoration method, and a program. [Background technology]

[0002] Spatial acoustic reconstruction techniques are being researched to virtually reconstruct spatial acoustics formed in real space using headphones or multiple speakers. Examples of spatial acoustic reconstruction techniques include wavefront synthesis and ambisonics. According to wavefront synthesis and ambisonics, spatial acoustics are accurately reconstructed based on the sound field observed at the sound collection point. However, observing an accurate sound field requires a large-scale microphone array. Therefore, accurately observing the sound field can sometimes be difficult.

[0003] As a method for reconstructing spatial acoustics without accurately observing the sound field, a technique has been proposed that uses a neural network to output first-order ambisonic coefficients, taking 360-degree images, optical flow, and monaural audio as inputs. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 10-294999 [Non-patent literature]

[0005] [Non-Patent Document 1] P. Morgado et al., “Self-supervised generation of spatial audio for 360 video”, in proc. NeurIPS 2018, pp. 360-370, 2018. [Overview of the project] [Problems that the invention aims to solve]

[0006] However, in the proposed method, monaural audio is separated into multiple sound sources corresponding to the video, and then each of these separated sound sources is localized to a sound image. As a result, the proposed method presents the following challenges in reconstructing rich spatial sound corresponding to the video. • There is an upper limit to the number of audio sources that can be separated in accordance with the video. • Modeling the effect of reverberation is difficult. • To achieve procedures such as separating sound sources, individual modules are required, which increases the amount of memory needed.

[0007] This invention was made in view of the above circumstances, and its purpose is to provide a means for restoring rich spatial sound corresponding to video. [Means for solving the problem]

[0008] One embodiment of a spatial sound restoration device comprises a video feature calculation unit, an acoustic feature calculation unit, and a coefficient calculation unit. The video feature calculation unit calculates video features based on video information. The acoustic feature calculation unit calculates acoustic features based on acoustic information, which is monaural sound corresponding to the video information. The coefficient calculation unit calculates higher-order ambisonic coefficients based on the video features and acoustic features. [Effects of the Invention]

[0009] According to one embodiment, a means can be provided for restoring rich spatial sound corresponding to video. [Brief explanation of the drawing]

[0010] [Figure 1] Figure 1 shows an example of the configuration of a spatial acoustic restoration system according to the first embodiment. [Figure 2] Figure 2 is a block diagram showing an example of the hardware configuration of a spatial acoustic restoration device according to the first embodiment. [Figure 3]FIG. 3 is a block diagram showing an example of the configuration of the learning function of the spatial acoustic restoration device according to the first embodiment. [Figure 4] FIG. 4 is a block diagram showing an example of the configuration of the restoration function of the spatial acoustic restoration device according to the first embodiment. [Figure 5] FIG. 5 is a flowchart showing an example of the learning operation in the spatial acoustic restoration device according to the first embodiment. [Figure 6] FIG. 6 is a flowchart showing an example of the restoration operation in the spatial acoustic restoration device according to the first embodiment. [Figure 7] FIG. 7 is a block diagram showing an example of the configuration of the learning function of the spatial acoustic restoration device according to the second embodiment. [Figure 8] FIG. 8 is a block diagram showing an example of the configuration of the restoration function of the spatial acoustic restoration device according to the second embodiment. [Figure 9] FIG. 9 is a flowchart showing an example of the learning operation in the spatial acoustic restoration device according to the second embodiment. [Figure 10] FIG. 10 is a flowchart showing an example of the restoration operation in the spatial acoustic restoration device according to the second embodiment. [Figure 11] FIG. 11 is a block diagram showing an example of the configuration of the learning function of the spatial acoustic restoration device according to the third embodiment. [Figure 12] FIG. 12 is a block diagram showing an example of the configuration of the restoration function of the spatial acoustic restoration device according to the third embodiment. [Figure 13] FIG. 13 is a flowchart showing an example of the learning operation in the spatial acoustic restoration device according to the third embodiment. [Figure 14] FIG. 14 is a flowchart showing an example of the restoration operation in the spatial acoustic restoration device according to the third embodiment.

DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments will be described with reference to the drawings. In the following description, components having the same function and configuration are denoted by common reference numerals.

[0012] 1. First Embodiment 1.1 Configuration The configuration of the spatial acoustic restoration device according to the first embodiment will be described below.

[0013] 1.1.1 Spatial Acoustic Restoration System First, the configuration of the spatial sound restoration system, including the spatial sound restoration device according to the first embodiment, will be described. Figure 1 is a block diagram showing an example of the configuration of the spatial sound restoration system according to the first embodiment.

[0014] As shown in Figure 1, the spatial sound restoration system 1 is a system for user U to experience spatial sound in conjunction with video. The spatial sound restoration system 1 comprises multiple speakers SP and a spatial sound restoration device 100.

[0015] Multiple speakers SP are arranged around the user U. The example in Figure 1 shows the case where the multiple speakers SP are arranged away from the user U, but this is not limited to this case. The multiple speakers SP may also be devices worn and used by the user U, such as headphones.

[0016] The spatial sound restoration device 100 is, for example, a terminal. The spatial sound restoration device 100 calculates higher-order ambisonics coefficients based on video information and audio information, which is monaural sound corresponding to the video information. Based on the calculated higher-order ambisonics coefficients, the spatial sound restoration device 100 decodes the output audio information output from multiple speakers SP.

[0017] The higher-order ambisonics coefficient corresponds to the sound field formed by multiple virtual sound sources SS. These multiple virtual sound sources SS are sound sources that are virtually placed in any number and at any position outside of the multiple speakers SP relative to the user U. The multiple virtual sound sources SS do not correspond to the positions and number of actual sound sources identified from the video information. That is, the positions and number of the multiple virtual sound sources SS are determined by the user U independently of the video information.

[0018] 1.1.2 Hardware configuration of the spatial acoustic restoration device Next, the hardware configuration of the spatial acoustic restoration device according to the first embodiment will be described.

[0019] Figure 2 is a block diagram showing an example of the hardware configuration of a spatial acoustic restoration device according to the first embodiment. As shown in Figure 2, the spatial acoustic restoration device 100 includes a control circuit 11, storage 12, communication module 13, interface 14, drive 15, and storage medium 16.

[0020] The control circuit 11 is a circuit that controls all components of the spatial acoustic restoration device 100 as a whole. The control circuit 11 includes a CPU (Central Processing Unit), RAM (Random Access Memory), and ROM (Read Only Memory), etc.

[0021] Storage 12 is an auxiliary storage device for the spatial sound restoration device 100. Storage 12 is, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), or a memory card. Storage 12 stores data used for learning and restoration operations. Storage 12 may also store programs for executing learning and restoration operations.

[0022] The restoration operation is the process of generating acoustic information from reconstructed spatial acoustics. The learning operation is the process of learning the parameters for reconstructing spatial acoustics. Details of the learning and restoration operations will be described later.

[0023] The communication module 13 is a circuit used for sending and receiving data between multiple speakers SP.

[0024] Interface 14 is a circuit for communicating information between user U and control circuit 11. Interface 14 includes input and output devices. Input devices include, for example, a touch panel and operation buttons. Output devices include, for example, an LCD (Liquid Crystal Display) or EL (Electroluminescence) display. Interface 14 converts input from user U into an electrical signal and then transmits it to control circuit 11. Interface 14 outputs the execution result based on the input from user U to user U.

[0025] Drive 15 is a device for reading software stored on the storage medium 16. Drive 15 includes, for example, a CD (Compact Disk) drive and a DVD (Digital Versatile Disk) drive.

[0026] The storage medium 16 is a medium for storing software by electrical, magnetic, optical, mechanical, or chemical means. The storage medium 16 may also store programs for performing learning and retrieval operations.

[0027] 1.1.3 Functional Configuration of the Spatial Acoustic Restoration Device Next, the functional configuration of the spatial sound restoration device according to the first embodiment will be described. The spatial sound restoration device 100 has a learning function for performing learning operations and a restoration function for performing restoration operations.

[0028] 1.1.3.1 Learning Function Figure 3 is a block diagram showing an example of the configuration of the learning function of the spatial acoustic restoration device according to the first embodiment.

[0029] The CPU of the control circuit 11 loads the program related to the learning operation stored in the storage 12 or storage medium 16 into RAM. The CPU of the control circuit 11 then interprets and executes the program loaded into RAM. As a result, the control circuit 11 functions as a computer equipped with a video feature calculation unit 22, an acoustic feature calculation unit 23, a virtual sound source generation unit 24, an encoding unit 25, a decoding unit 26, and an evaluation unit 27. The storage 12 also stores multiple training datasets 21 and trained models 28.

[0030] Multiple training datasets 21 are a collection of datasets used in a single training operation. That is, each of the multiple training datasets 21 is a unit of dataset used in a single training operation. Each of the multiple training datasets 21 includes input video information Ivi, input audio information Iau, input environment information Ien, and teacher audio information Lau.

[0031] The input video information Ivi includes one or more image data. If it includes multiple image data, the input video information Ivi is, for example, multiple image data captured sequentially in a time series. The image data may be color images or monochrome images. The image data may be a 360-degree image or not (for example, a panoramic image).

[0032] The input audio information Iau is monaural audio data associated with the same time series as the input video information Ivi. For example, input audio information Iau is monaural audio data recorded at a recording location that approximately coincides with the shooting location of the input video information Ivi. In the following description, the recording location of input audio information Iau is also referred to as the "center position." The center position corresponds to the location of user U.

[0033] The input environment information Ien is data indicating the playback environment of the spatial sound (sound field) restored by the spatial sound restoration device 100. The input environment information Ien includes, for example, the relative position from the center position of multiple speakers SP, and the number of multiple speakers SP.

[0034] The teacher acoustic information Lau is audio data observed in the playback environment included in the input environment information Ien (i.e., the positions where multiple speakers SP are placed) in the true sound field formed by the actual sound source corresponding to the input video information Ivi.

[0035] The video feature calculation unit 22 includes a neural network having weights and bias terms that function as parameters. The neural network in the video feature calculation unit 22 is configured to calculate video feature quantities Evi using input video information Ivi as input. Video feature quantities Evi include one or more image features. The one or more image features included in video feature quantities Evi correspond, for example, to one or more image data included in the input video information Ivi. That is, the neural network in the video feature calculation unit 22 takes one image data from the input video information Ivi as input and calculates the image feature corresponding to that image data. The video feature calculation unit 22 transmits the video feature quantity Evi, which is a time-series combination of the calculated one or more image features, to the virtual sound source generation unit 24.

[0036] The video feature calculation unit 22 may calculate optical flow based on the input video information Ivi. In this case, the neural network within the video feature calculation unit 22 may calculate video feature Evi using the calculated optical flow as further input.

[0037] The acoustic feature calculation unit 23 includes a neural network having weights and bias terms that function as parameters. The neural network in the acoustic feature calculation unit 23 is configured to calculate acoustic feature Eau using input acoustic information Iau as input. Specifically, for example, the neural network in the acoustic feature calculation unit 23 takes the portion of monaural audio corresponding to one image data within the input video information Ivi as input acoustic information Iau and calculates a feature corresponding to that monaural audio portion. The acoustic feature calculation unit 23 transmits the acoustic feature Eau, which is obtained by combining one or more calculated feature quantities in a time series, to the virtual sound source generation unit 24.

[0038] The virtual sound source generation unit 24 includes a neural network having weights and bias terms that function as parameters. The neural network within the virtual sound source generation unit 24 is configured to calculate virtual sound source information F using video features Evi and acoustic features Eau as inputs. The virtual sound source generation unit 24 transmits the generated virtual sound source information F to the encoding unit 25.

[0039] The virtual sound source information F is defined by the following equation (1).

[0040]

number

[0041] Here, k is the wavenumber, t is time, and X is the number of virtual sound sources SS. Note that the i-th virtual sound source SS is at (θ) from the center position. i ,φ i Let it be located in the direction (1 ≤ i ≤ X). θ is the elevation angle. φ is the azimuth angle. X and (θ i ,φ i ) is determined independently of the input video information Ivi.

[0042] The encoding unit 25 includes an encoder that supports higher-order ambisonics. The encoder in the encoding unit 25 is configured to calculate higher-order ambisonics coefficients A using virtual sound source information F as input. The encoding unit 25 transmits the calculated higher-order ambisonics coefficients A to the decoding unit 26.

[0043] The higher-order ambisonic coefficient A is calculated according to the following equations (2), (3), and (4).

[0044]

number

[0045] Here, Y n m(θ, φ) is a spherical harmonic function of degree n and order m (0 ≤ n ≤ N, -n ≤ m ≤ n). N is the maximum value of the degree. Matrix Y † is the pseudo-inverse matrix of Y.

[0046] The decoding unit 26 includes a decoder corresponding to high-order ambisonics. The decoder in the decoding unit 26 is configured to calculate the output acoustic information F^ with the input environmental information Ien and the high-order ambisonics coefficient A as inputs. The decoding unit 26 transmits the calculated output acoustic information F^ to the evaluation unit 27.

[0047] The output acoustic information F^ is defined by the following equation (5).

[0048]

Equation

[0049] X^ is the number of a plurality of speakers SP. The l-th speaker SP is assumed to be located in the direction of (θ l , φ l )(1 ≤ l ≤ X^) from the center position. The number X^ of the speakers SP and the direction (θ l , φ l ) from the center position are included in the input environmental information Ien.

[0050] The acoustic information F l ^(k, t) reproduced from the l-th speaker SP is calculated according to the following equations (6) and (7).

[0051]

Equation

[0052] The evaluation unit 27 includes an updater for parameter P. The evaluation unit 27 updates parameter P to minimize the error between the output acoustic information F^ and the training acoustic information Lau. Specifically, parameter P consists of bias terms and weights that determine the characteristics of the neural networks provided in the video feature calculation unit 22, the acoustic feature calculation unit 23, and the virtual sound source generation unit 24, respectively. In calculating parameter P, the evaluation unit 27 uses, for example, the backpropagation method.

[0053] Each time parameter P is updated, the evaluation unit 27 applies the updated parameter P to the video feature calculation unit 22, the acoustic feature calculation unit 23, and the virtual sound source generation unit 24, respectively. When the evaluation completion condition is met, the evaluation unit 27 stores the last updated parameter P as the trained model 28 in the storage 12. Hereafter, parameter P as the trained model 28 may be referred to as parameter Pe to distinguish it from parameter P.

[0054] The evaluation termination condition may be, for example, that all of the multiple training datasets 21 are selected. The evaluation termination condition may be, for example, that the update amount of parameter P is less than or equal to a predetermined threshold. The evaluation termination condition may be, for example, that the update of parameter P is repeated a predetermined number of times.

[0055] 1.1.3.2 Recovery Function Figure 4 is a block diagram showing an example of the configuration of the restoration function of the spatial acoustic restoration device according to the first embodiment.

[0056] The CPU of the control circuit 11 loads the program related to the restoration operation stored in the storage 12 or storage medium 16 into RAM. The CPU of the control circuit 11 then interprets and executes the program loaded into RAM. As a result, the control circuit 11 functions as a computer equipped with a video feature calculation unit 22, an acoustic feature calculation unit 23, a virtual sound source generation unit 24, an encoding unit 25, a decoding unit 26, and an output unit 30. The storage 12 stores the trained model 28 and the restoration dataset 29.

[0057] The configuration of the video feature calculation unit 22, acoustic feature calculation unit 23, virtual sound source generation unit 24, encoding unit 25, and decoding unit 26 in Figure 4 is equivalent to the configuration of the video feature calculation unit 22, acoustic feature calculation unit 23, virtual sound source generation unit 24, encoding unit 25, and decoding unit 26 in Figure 3, so a detailed explanation is omitted. The decoding unit 26 transmits the calculated output acoustic information F^ to the output unit 30.

[0058] The trained model 28 is a parameter Pe generated by the training process. The trained model 28 is applied to the neural networks provided in the video feature calculation unit 22, the acoustic feature calculation unit 23, and the virtual sound source generation unit 24 during the restoration process.

[0059] The restoration dataset 29 is the dataset used for the restoration operation. The restoration dataset 29 includes input video information Ivi, input audio information Iau, and input environment information Ien.

[0060] The output unit 30 transmits the output acoustic information F^ to multiple speakers SP. With this configuration, the spatial acoustic restoration device 100 can restore the output acoustic information F^ using the trained model 28.

[0061] 1.2 Operation Next, the operation of the spatial acoustic restoration device according to the first embodiment will be described.

[0062] 1.2.1 Learning process Figure 5 is a flowchart showing an example of the learning operation in the spatial acoustic restoration device according to the first embodiment.

[0063] When the control circuit 11 receives an instruction from user U to perform a learning operation (start), it selects one unselected learning dataset 21 from among multiple learning datasets 21 (S11).

[0064] The video feature calculation unit 22 calculates video feature Evi based on the input video information Ivi in ​​the training dataset 21 selected in the processing of S11 (S12).

[0065] The acoustic feature calculation unit 23 calculates the acoustic feature Eau based on the input acoustic information Iau in the training dataset 21 selected in the processing of S11 (S13).

[0066] The virtual sound source generation unit 24 calculates virtual sound source information F based on the video feature quantity Evi calculated in the processing of S12 and the acoustic feature quantity Eau calculated in the processing of S13 (S14).

[0067] The encoding unit 25 encodes the virtual sound source information F calculated in the processing of S14 into higher-order ambisonic coefficients A (S15).

[0068] The decoding unit 26 decodes the output acoustic information F^ from the higher-order ambisonic coefficients A encoded in the processing of S15, based on the input environment information Ien in the training dataset 21 selected in the processing of S11 (S16).

[0069] The evaluation unit 27 updates the parameter P based on the training acoustic information Lau in the training dataset 21 selected in the processing of S10, and the output acoustic information F^ decoded in the processing of S16 (S17).

[0070] The evaluation unit 27 determines whether or not the evaluation completion conditions have been met (S18).

[0071] If the evaluation termination condition is not met (S18; no), the control circuit 11 selects one unselected training dataset 21 from the multiple training datasets 21 (S11). Then, the control circuit 11 executes the subsequent processes S12 to S18. In this way, processes S11 to S18 are repeatedly executed until the evaluation termination condition is met.

[0072] If the evaluation completion condition is met (S18; yes), the evaluation unit 27 stores the parameter Pe that was last updated in the processing of S17 as the trained model 28 in the storage 12 (S19).

[0073] After processing S19, the learning process ends (end).

[0074] 1.2.2 Restoration Operation Next, the restoration operation in the spatial acoustic restoration device according to the first embodiment will be described.

[0075] Figure 6 is a flowchart showing an example of the restoration operation in the spatial acoustic restoration device according to the first embodiment.

[0076] When user U gives an instruction to perform a restoration operation (start), the video feature calculation unit 22 calculates the video feature Evi based on the input video information Ivi in ​​the restoration dataset 29 (S21).

[0077] The acoustic feature calculation unit 23 calculates the acoustic feature Eau based on the input acoustic information Iau in the reconstruction dataset 29 (S22).

[0078] The virtual sound source generation unit 24 calculates virtual sound source information F based on the video feature quantity Evi calculated in the processing of S21 and the acoustic feature quantity Eau calculated in the processing of S22 (S23).

[0079] The encoding unit 25 encodes the virtual sound source information F calculated in the processing of S23 into higher-order ambisonic coefficients A (S24).

[0080] The decoding unit 26 decodes the output acoustic information F^ from the higher-order ambisonic coefficients A encoded in the processing of S24, based on the input environment information Ien in the reconstruction dataset 29 (S25).

[0081] The output unit 30 outputs the output acoustic information F^, which has been decoded in the processing of S25, to multiple speakers SP (S26).

[0082] Once the S26 process is complete, the restoration operation will end (end).

[0083] 1.3 Effects of the First Embodiment According to the first embodiment, the video feature calculation unit 22 calculates video feature quantity Evi based on the input video information Ivi. The acoustic feature calculation unit 23 calculates acoustic feature quantity Eau based on the input acoustic information Iau, which is monaural audio. The virtual sound source generation unit 24 generates virtual sound source information F based on the video feature quantity Evi and the acoustic feature quantity Eau. The virtual sound source information F is a sound field formed by a plurality of virtual sound sources SS independent of the input video information Ivi. This makes it possible to reproduce a sound field formed by any number of sound sources, regardless of the number of actual sound sources corresponding to the input video information Ivi. Furthermore, it is possible to reproduce reverberation components that are difficult to separate as individual sound sources. Therefore, it is possible to restore rich spatial sound corresponding to the video.

[0084] Furthermore, the encoding unit 25 encodes the virtual sound source information F into higher-order ambisonic coefficients A. This enables sound image localization through deterministic computational processing that does not involve the learning operation of the neural network. Therefore, the increase in the amount of memory required to implement the restoration function can be suppressed.

[0085] Furthermore, the decoding unit 26 decodes the output acoustic information F^ from the higher-order ambisonic coefficient A. The evaluation unit 27 updates the neural network parameters P included in the video feature calculation unit 22, the acoustic feature calculation unit 23, and the virtual sound source generation unit 24, respectively, based on the comparison result between the output acoustic information F^ and the training acoustic information Lau. This improves the estimation accuracy of the video feature Evi, acoustic feature Eau, and virtual sound source information F by the neural network.

[0086] 2. Second Embodiment Next, a spatial sound restoration device according to the second embodiment will be described. The second embodiment differs from the first embodiment in that it calculates the higher-order ambisonic coefficient A from the video feature quantity Evi and the acoustic feature quantity Eau without going through virtual sound source information F. Below, the configuration and operation that differ from the first embodiment will be mainly described. Configurations and operations equivalent to the first embodiment will be omitted from the description as appropriate.

[0087] 2.1 Configuration The configuration of the spatial acoustic restoration device according to the second embodiment will be described below.

[0088] 2.1.1 Learning Function Figure 7 is a block diagram showing an example of the configuration of the learning function of the spatial acoustic restoration device according to the second embodiment. Figure 7 corresponds to Figure 3 in the first embodiment.

[0089] The CPU of the control circuit 11 loads the program related to the learning operation stored in the storage 12 or storage medium 16 into the RAM. The CPU of the control circuit 11 then interprets and executes the program loaded into the RAM. As a result, the control circuit 11 functions as a computer equipped with a video feature calculation unit 22, an acoustic feature calculation unit 23, a decoding unit 26, an evaluation unit 27, and a coefficient calculation unit 31. The storage 12 also stores multiple training datasets 21 and trained models 28.

[0090] The configuration of the multiple training datasets 21, video feature calculation unit 22, acoustic feature calculation unit 23, and decoding unit 26 in Figure 7 is equivalent to the configuration of the multiple training datasets 21, video feature calculation unit 22, acoustic feature calculation unit 23, and decoding unit 26 in Figure 3, so a detailed explanation is omitted. The video feature calculation unit 22 transmits the calculated video feature Evi to the coefficient calculation unit 31. The acoustic feature calculation unit 23 transmits the calculated acoustic feature Eau to the coefficient calculation unit 31.

[0091] The coefficient calculation unit 31 includes a neural network having weights and bias terms that function as parameters. The neural network in the coefficient calculation unit 31 is configured to calculate higher-order ambisonic coefficients A using video feature vector Evi and acoustic feature vector Eau as input. The coefficient calculation unit 31 transmits the generated higher-order ambisonic coefficients A to the decoding unit 26.

[0092] The evaluation unit 27 updates the parameter P to minimize the error between the output acoustic information F^ and the training acoustic information Lau. Specifically, the parameter P consists of a bias term and weights that determine the characteristics of the neural network, provided in the video feature calculation unit 22, the acoustic feature calculation unit 23, and the coefficient calculation unit 31, respectively. In calculating the parameter P, the evaluation unit 27 uses, for example, the backpropagation method.

[0093] Each time parameter P is updated, the evaluation unit 27 applies the updated parameter P to the video feature calculation unit 22, the acoustic feature calculation unit 23, and the coefficient calculation unit 31, respectively. When the evaluation completion condition is met, the evaluation unit 27 stores the last updated parameter Pe as the trained model 28 in the storage unit 12.

[0094] 2.1.2 Recovery Function Figure 8 is a block diagram showing an example of the configuration of the restoration function of the spatial acoustic restoration device according to the second embodiment. Figure 8 corresponds to Figure 4 in the first embodiment.

[0095] The CPU of the control circuit 11 loads the program related to the restoration operation stored in the storage 12 or storage medium 16 into RAM. The CPU of the control circuit 11 then interprets and executes the program loaded into RAM. As a result, the control circuit 11 functions as a computer equipped with a video feature calculation unit 22, an acoustic feature calculation unit 23, a decoding unit 26, an output unit 30, and a coefficient calculation unit 31. The storage 12 stores the trained model 28 and the restoration dataset 29.

[0096] The configurations of the video feature calculation unit 22, acoustic feature calculation unit 23, decoding unit 26, and coefficient calculation unit 31 in Figure 8 are equivalent to those of the video feature calculation unit 22, acoustic feature calculation unit 23, decoding unit 26, and coefficient calculation unit 31 in Figure 7, so their explanation is omitted. Furthermore, the configurations of the reconstruction dataset 29 and output unit 30 in Figure 8 are equivalent to those of the reconstruction dataset 29 and output unit 30 in Figure 4, so their explanation is omitted.

[0097] The trained model 28 is a parameter Pe generated by the training process. The trained model 28 is applied to the neural networks provided in the video feature calculation unit 22, the acoustic feature calculation unit 23, and the coefficient calculation unit 31, respectively, during the restoration process.

[0098] 2.2 Operation Next, the operation of the spatial acoustic restoration device according to the second embodiment will be described.

[0099] 2.2.1 Learning process Figure 9 is a flowchart showing an example of the learning operation in the spatial acoustic restoration device according to the second embodiment. Figure 9 corresponds to Figure 5 in the first embodiment.

[0100] When the control circuit 11 receives an instruction from user U to perform a learning operation (start), it selects one unselected learning dataset 21 from among multiple learning datasets 21 (S31).

[0101] The video feature calculation unit 22 calculates video feature Evi based on the input video information Ivi in ​​the training dataset 21 selected in the processing of S31 (S32).

[0102] The acoustic feature calculation unit 23 calculates the acoustic feature Eau based on the input acoustic information Iau in the training dataset 21 selected in the processing of S31 (S33).

[0103] The coefficient calculation unit 31 calculates the higher-order ambisonics coefficient A based on the video feature quantity Evi calculated in the processing of S32 and the acoustic feature quantity Eau calculated in the processing of S33 (S34).

[0104] The decoding unit 26 decodes the output acoustic information F^ from the higher-order ambisonic coefficient A calculated in the processing of S34, based on the input environment information Ien in the training dataset 21 selected in the processing of S31 (S35).

[0105] The evaluation unit 27 updates the parameter P based on the training acoustic information Lau in the training dataset 21 selected in processing S31 and the output acoustic information F^ decoded in processing S35 (S36).

[0106] The evaluation unit 27 determines whether or not the evaluation completion conditions have been met (S37).

[0107] If the evaluation termination condition is not met (S37; no), the control circuit 11 selects one unselected training dataset 21 from the multiple training datasets 21 (S31). Then, the control circuit 11 executes the subsequent processes S32 to S37. In this way, processes S31 to S37 are repeatedly executed until the evaluation termination condition is met.

[0108] If the evaluation completion condition is met (S37; yes), the evaluation unit 27 stores the parameter Pe that was last updated in the processing of S36 as the trained model 28 in the storage 12 (S38).

[0109] After processing S38, the learning process ends (end).

[0110] 2.2.2 Restoration Operation Next, the restoration operation in the spatial acoustic restoration device according to the second embodiment will be described.

[0111] Figure 10 is a flowchart showing an example of the restoration operation in the spatial acoustic restoration device according to the second embodiment. Figure 10 corresponds to Figure 6 in the first embodiment.

[0112] When user U gives an instruction to perform a restoration operation (start), the video feature calculation unit 22 calculates the video feature Evi based on the input video information Ivi in ​​the restoration dataset 29 (S41).

[0113] The acoustic feature calculation unit 23 calculates the acoustic feature Eau based on the input acoustic information Iau in the reconstruction dataset 29 (S42).

[0114] The coefficient calculation unit 31 calculates the higher-order ambisonics coefficient A based on the video feature quantity Evi calculated in the processing of S41 and the acoustic feature quantity Eau calculated in the processing of S42 (S43).

[0115] The decoding unit 26 decodes the output acoustic information F^ from the higher-order ambisonic coefficient A calculated in the processing of S43, based on the input environment information Ien in the reconstruction dataset 29 (S44).

[0116] The output unit 30 outputs the output acoustic information F^, which has been decoded in the processing of S44, to multiple speakers SP (S45).

[0117] Once the S45 process is complete, the restoration operation will end (end).

[0118] 2.3 Effects according to the second embodiment According to the second embodiment, the coefficient calculation unit 31 calculates a higher-order ambisonics coefficient A based on the video feature quantity Evi and the acoustic feature quantity Eau. This makes it possible to obtain output acoustic information F^ without explicitly defining a virtual sound source SS. Therefore, the increase in the amount of memory required to implement the restoration function can be suppressed.

[0119] Furthermore, the evaluation unit 27 updates the neural network parameters P included in the video feature calculation unit 22, the acoustic feature calculation unit 23, and the coefficient calculation unit 31, respectively, based on the comparison result between the output acoustic information F^ and the training acoustic information Lau. This improves the estimation accuracy of the video feature Evi, acoustic feature Eau, and higher-order ambisonic coefficient A by the neural network.

[0120] 3. Third Embodiment Next, a spatial acoustic restoration device according to the third embodiment will be described. The third embodiment differs from the first and second embodiments in that an auxiliary variable λ is used as an additional input when calculating the higher-order ambisonic coefficient A. The following will mainly describe the configuration and operation that differ from the first and second embodiments. Configurations and operations equivalent to those of the first and second embodiments will be omitted as appropriate.

[0121] 3.1 Configuration The configuration of the spatial acoustic restoration device according to the third embodiment will be described below.

[0122] 3.1.1 Learning Function Figure 11 is a block diagram showing an example of the configuration of the learning function of the spatial acoustic restoration device according to the third embodiment. Figure 11 corresponds to Figure 3 in the first embodiment and Figure 7 in the second embodiment.

[0123] The CPU of the control circuit 11 loads the program related to the learning operation stored in the storage 12 or storage medium 16 into the RAM. The CPU of the control circuit 11 then interprets and executes the program loaded into the RAM. As a result, the control circuit 11 functions as a computer equipped with a video feature calculation unit 22, an acoustic feature calculation unit 23, a decoding unit 26, an evaluation unit 27, a coefficient update unit 32, an auxiliary variable update unit 33, and a virtual sound source update unit 34. The storage 12 also stores multiple training datasets 21 and trained models 28.

[0124] The configuration of the multiple training datasets 21, video feature calculation unit 22, acoustic feature calculation unit 23, and decoding unit 26 in Figure 11 is equivalent to the configuration of the multiple training datasets 21, video feature calculation unit 22, acoustic feature calculation unit 23, and decoding unit 26 in Figure 3, so a detailed explanation is omitted. The video feature calculation unit 22 transmits the calculated video feature Evi to the coefficient update unit 32. The acoustic feature calculation unit 23 transmits the calculated acoustic feature Eau to the coefficient update unit 32.

[0125] The coefficient update unit 32, the auxiliary variable update unit 33, and the virtual sound source update unit 34 update the higher-order ambisonic coefficient A, the auxiliary variable λ, and the virtual sound source information F, respectively. The auxiliary variable λ corresponds to the residual (YA-F) between the product (YA) of the higher-order ambisonic coefficient A and matrix Y and the virtual sound source information F. Each of the higher-order ambisonic coefficient A, the auxiliary variable λ, and the virtual sound source information F is updated once in a single update operation.

[0126] Specifically, the coefficient update unit 32 includes a neural network having weights and bias terms that function as parameters, as well as an updater for higher-order ambisonics coefficients A. The neural network within the coefficient update unit 32 is configured to calculate higher-order ambisonics coefficients A using video feature vector Evi, acoustic feature vector Eau, virtual sound source information F, and auxiliary variable λ as input.

[0127] The coefficient update unit 32 updates the original higher-order ambisonic coefficient A with the calculated higher-order ambisonic coefficient A and stores it in the storage 12. When the update completion condition is met, the coefficient update unit 32 transmits the updated higher-order ambisonic coefficient A as the higher-order ambisonic coefficient Af to the decoding unit 26.

[0128] The update termination condition may be, for example, that the number of update operations exceeds a predetermined threshold. Alternatively, the update termination condition may be that the amount of updates to the higher-order ambisonic coefficient A, auxiliary variable λ, and virtual sound source information F due to the update operation falls below a predetermined threshold.

[0129] The auxiliary variable update unit 33 includes an updater for the auxiliary variable λ. The auxiliary variable update unit 33 updates the auxiliary variable λ based on the following equation (8) and stores it in the storage 12.

[0130]

number

[0131] Here, the higher-order ambisonic coefficient A' is the updated higher-order ambisonic coefficient in the update operation. The auxiliary variables λ and λ' are the auxiliary variables before and after the update operation, respectively. The virtual sound source information F is the virtual sound source information before the update operation. γ1 is a design variable.

[0132] The virtual sound source update unit 34 includes an updater for the virtual sound source information F. The virtual sound source update unit 34 updates the virtual sound source information F based on the following equation (9) and stores it in the storage 12.

[0133]

number

[0134] Here, virtual sound source information F' is the virtual sound source information after the update operation. γ2 is a design variable.

[0135] The evaluation unit 27 updates the parameter P to minimize the error between the output acoustic information F^ and the training acoustic information Lau. Specifically, the parameter P consists of a bias term and weights that determine the characteristics of the neural network, provided in the video feature calculation unit 22, the acoustic feature calculation unit 23, and the coefficient update unit 32, respectively. In calculating the parameter P, the evaluation unit 27 uses, for example, the backpropagation method.

[0136] Each time parameter P is updated, the evaluation unit 27 applies the updated parameter P to the video feature calculation unit 22, the acoustic feature calculation unit 23, and the coefficient update unit 32, respectively. When the evaluation completion condition is met, the evaluation unit 27 stores the last updated parameter Pe as the trained model 28 in the storage unit 12.

[0137] 3.1.2 Recovery Function Figure 12 is a block diagram showing an example of the configuration of the restoration function of the spatial acoustic restoration device according to the third embodiment. Figure 12 corresponds to Figure 4 in the first embodiment and Figure 8 in the second embodiment.

[0138] The CPU of the control circuit 11 loads the program related to the restoration operation stored in the storage 12 or storage medium 16 into RAM. The CPU of the control circuit 11 then interprets and executes the program loaded into RAM. As a result, the control circuit 11 functions as a computer equipped with a video feature calculation unit 22, an acoustic feature calculation unit 23, a decoding unit 26, an output unit 30, a coefficient update unit 32, an auxiliary variable update unit 33, and a virtual sound source update unit 34. The storage 12 stores the trained model 28 and the restoration dataset 29.

[0139] The configuration of the video feature calculation unit 22, acoustic feature calculation unit 23, decoding unit 26, coefficient update unit 32, auxiliary variable update unit 33, and virtual sound source update unit 34 in Figure 12 is equivalent to the configuration of the video feature calculation unit 22, acoustic feature calculation unit 23, decoding unit 26, coefficient update unit 32, auxiliary variable update unit 33, and virtual sound source update unit 34 in Figure 11, so a description is omitted. Also, the configuration of the reconstruction dataset 29 and output unit 30 in Figure 12 is equivalent to the configuration of the reconstruction dataset 29 and output unit 30 in Figure 4, so a description is omitted.

[0140] The trained model 28 is a parameter Pe generated by the training process. The trained model 28 is applied to the neural networks provided in the video feature calculation unit 22, the acoustic feature calculation unit 23, and the coefficient update unit 32 during the restoration process.

[0141] 3.2 Operation Next, the operation of the spatial acoustic restoration device according to the third embodiment will be described.

[0142] 3.2.1 Learning process Figure 13 is a flowchart showing an example of the learning operation in the spatial acoustic restoration device according to the third embodiment. Figure 13 corresponds to Figure 5 in the first embodiment and Figure 9 in the second embodiment.

[0143] When the control circuit 11 receives an instruction from user U to perform a learning operation (start), it selects one unselected learning dataset 21 from among multiple learning datasets 21 (S51).

[0144] The control circuit 11 initializes the update count x of the update operation, the higher-order ambisonic coefficient A, the auxiliary variable λ, and the virtual sound source information F (S52).

[0145] The update count x is initialized to, for example, 0. The higher-order ambisonic coefficient A, the auxiliary variable λ, and the virtual sound source information F are each initialized to, for example, random numbers. The virtual sound source information F may be initialized to a monaural sound (for example, input sound information Iau).

[0146] The video feature calculation unit 22 calculates video feature Evi based on the input video information Ivi in ​​the training dataset 21 selected in the processing of S51 (S53).

[0147] The acoustic feature calculation unit 23 calculates the acoustic feature Eau based on the input acoustic information Iau in the training dataset 21 selected in the processing of S51 (S54).

[0148] The coefficient update unit 32 updates the higher-order ambisonics coefficient A based on the auxiliary variable λ and virtual sound source information F initialized in the S52 process, the video feature quantity Evi calculated in the S53 process, and the acoustic feature quantity Eau calculated in the S54 process (S55).

[0149] The auxiliary variable update unit 33 updates the auxiliary variable λ based on the higher-order ambisonic coefficient A updated in the S55 process and the virtual sound source information F initialized in the S52 process (S56).

[0150] The virtual sound source update unit 34 updates the virtual sound source information F based on the auxiliary variable λ updated in the process of S56 (S57).

[0151] The control circuit 11 determines whether or not the update conditions are met (S58).

[0152] If the update conditions are not met (S58; no), the control circuit 11 increments the update count x (S59).

[0153] After processing in S59, the coefficient update unit 32 updates the higher-order ambisonics coefficient A based on the video feature quantity Evi calculated in processing S53, the acoustic feature quantity Eau calculated in processing S54, the auxiliary variable λ updated in processing S56, and the virtual sound source information F updated in processing S57 (S55).

[0154] The auxiliary variable update unit 33 updates the auxiliary variable λ based on the higher-order ambisonic coefficient A updated in the S55 process and the virtual sound source information F updated in the S57 process (S56).

[0155] The virtual sound source update unit 34 updates the virtual sound source information F based on the auxiliary variable λ updated in the process of S56 (S57).

[0156] In this way, the update operations S55-S57 are repeatedly executed until the update completion conditions are met.

[0157] If the update condition is met (S58; yes), the decoding unit 26 decodes the output acoustic information F^ from the last updated higher-order ambisonic coefficient Af in the processing of S55, based on the input environment information Ien in the training dataset 21 selected in the processing of S51 (S60).

[0158] The evaluation unit 27 updates the parameter P based on the training acoustic information Lau in the training dataset 21 selected in the processing of S31, and the output acoustic information F^ decoded in the processing of S60 (S61).

[0159] The evaluation unit 27 determines whether or not the evaluation completion conditions have been met (S62).

[0160] If the evaluation termination condition is not met (S62; no), the control circuit 11 selects one unselected training dataset 21 from the multiple training datasets 21 (S51). Then, the control circuit 11 executes the subsequent processes S52 to S62. In this way, processes S51 to S62 are repeatedly executed until the evaluation termination condition is met.

[0161] If the evaluation completion condition is met (S62; yes), the evaluation unit 27 stores the parameter Pe that was last updated in the processing of S61 as the trained model 28 in the storage 12 (S63).

[0162] After processing S63, the learning process ends (end).

[0163] 3.2.2 Restoration Operation Next, the restoration operation in the spatial acoustic restoration device according to the third embodiment will be described.

[0164] Figure 14 is a flowchart illustrating an example of the restoration operation in the spatial acoustic restoration device according to the third embodiment. Figure 14 corresponds to Figure 6 in the first embodiment and Figure 10 in the second embodiment.

[0165] When the control circuit 11 receives an instruction from user U to perform a restore operation (start), it initializes the update count x of the update operation, the higher-order ambisonic coefficient A, the auxiliary variable λ, and the virtual sound source information F (S71).

[0166] The video feature calculation unit 22 calculates video feature Evi based on the input video information Ivi in ​​the reconstruction dataset 29 (S72).

[0167] The acoustic feature calculation unit 23 calculates the acoustic feature Eau based on the input acoustic information Iau in the reconstruction dataset 29 (S73).

[0168] The coefficient update unit 32 updates the higher-order ambisonics coefficient A based on the auxiliary variable λ and virtual sound source information F initialized in the S71 process, the video feature quantity Evi calculated in the S72 process, and the acoustic feature quantity Eau calculated in the S73 process (S74).

[0169] The auxiliary variable update unit 33 updates the auxiliary variable λ based on the higher-order ambisonic coefficient A updated in the S74 process and the virtual sound source information F initialized in the S71 process (S75).

[0170] The virtual sound source update unit 34 updates the virtual sound source information F based on the auxiliary variable λ updated in the process of S75 (S76).

[0171] The control circuit 11 determines whether or not the update conditions are met (S77).

[0172] If the update conditions are not met (S77; no), the control circuit 11 increments the update count x (S78).

[0173] After processing in S78, the coefficient update unit 32 updates the higher-order ambisonics coefficient A based on the video feature quantity Evi calculated in processing S72, the acoustic feature quantity Eau calculated in processing S73, the auxiliary variable λ updated in processing S75, and the virtual sound source information F updated in processing S76 (S74).

[0174] The auxiliary variable update unit 33 updates the auxiliary variable λ based on the higher-order ambisonic coefficient A updated in the S74 process and the virtual sound source information F updated in the S76 process (S75).

[0175] The virtual sound source update unit 34 updates the virtual sound source information F based on the auxiliary variable λ updated in the process of S75 (S76).

[0176] In this way, the update operations S74-S76 are repeatedly executed until the update completion conditions are met.

[0177] If the update condition is met (S77; yes), the decoding unit 26 decodes the output acoustic information F^ from the higher-order ambisonic coefficient Af that was last updated in the processing of S74, based on the input environment information Ien in the reconstruction dataset 29 (S79).

[0178] The output unit 30 outputs the output acoustic information F^, which has been decoded in the processing of S79, to multiple speakers SP (S80).

[0179] Once the S80 process is complete, the restoration operation will end (end).

[0180] 3.3 Effects of the Third Embodiment According to the third embodiment, the coefficient update unit 32 calculates and updates the higher-order ambisonics coefficient A based on the video feature quantity Evi, the acoustic feature quantity Eau, the auxiliary variable λ, and the virtual sound source information F. The auxiliary variable update unit 33 updates the auxiliary variable λ based on the updated higher-order ambisonics coefficient A and the virtual sound source information F. The virtual sound source update unit 34 updates the virtual sound source information F based on the updated auxiliary variable λ. This improves the accuracy of the higher-order ambisonics coefficient A applied to the decoding unit 26.

[0181] Furthermore, if the number of update operations by the coefficient update unit 32, the auxiliary variable update unit 33, and the virtual sound source update unit 34 exceeds a predetermined threshold, the update operation is terminated. In this way, by performing multiple update operations, the accuracy of the higher-order ambisonic coefficients A can be sufficiently improved before the subsequent decoding operation by the decoding unit 26 can be performed. As a result, output sound information F^ with richer spatial sound restored can be generated.

[0182] Furthermore, the evaluation unit 27 updates the neural network parameters P included in the video feature calculation unit 22, the acoustic feature calculation unit 23, and the coefficient update unit 32, respectively, based on the comparison result between the output acoustic information F^ and the training acoustic information Lau. This improves the estimation accuracy of the video feature Evi, acoustic feature Eau, and higher-order ambisonic coefficient A by the neural network.

[0183] 4. Others Furthermore, various modifications can be applied to the first, second, and third embodiments described above.

[0184] In the first, second, and third embodiments described above, the case in which the program that performs the learning operation and the restoration operation is executed on the spatial sound restoration device 100 has been described, but it is not limited to this. For example, the program that performs the learning operation and the restoration operation may be executed on computing resources on the cloud.

[0185] Furthermore, while the first, second, and third embodiments described above describe cases where the acoustic feature Eau is calculated after the video feature Evi, the invention is not limited to these cases. For example, the acoustic feature Eau may be calculated before the video feature Evi. Also, for example, the calculation of the video feature Evi and the calculation of the acoustic feature Eau may be performed in parallel.

[0186] Furthermore, while the first, second, and third embodiments described above describe cases in which the parameter P is updated by comparing the output acoustic information F^ with training data, the invention is not limited to this. For example, the parameter P may be updated by comparing the higher-order ambisonic coefficient A with training data. That is, the training acoustic information Lau may be the higher-order ambisonic coefficient that reconstructs the true sound field. In this case, in the process of S17 in Figure 5 of the first embodiment, the evaluation unit 27 updates the parameter P based on the training acoustic information Lau in the training dataset 21 selected in the process of S10 and the higher-order ambisonic coefficient A encoded in the process of S15. In the process of S36 in Figure 9 of the second embodiment, the evaluation unit 27 updates the parameter P based on the training acoustic information Lau in the training dataset 21 selected in the process of S31 and the higher-order ambisonic coefficient A calculated in the process of S34. In the processing of S61 in Figure 13 of the third embodiment, the evaluation unit 27 updates the parameter P based on the teacher acoustic information Lau in the training dataset 21 selected in processing S31, and the higher-order ambisonic coefficient Af that was last updated in processing S55.

[0187] It should be noted that the present invention is not limited to the embodiments described above, and can be modified in various ways during implementation without departing from its essence. Furthermore, each embodiment may be combined as appropriate, and in that case, the combined effects can be obtained. Moreover, the above embodiments include various inventions, and various inventions can be extracted by selecting combinations from the multiple constituent elements disclosed. For example, if the problem can be solved and effects obtained even if some constituent elements are deleted from all the constituent elements shown in the embodiment, then the configuration with these deleted constituent elements can be extracted as an invention. [Explanation of Symbols]

[0188] 1…Spatial Acoustic Restoration System 11…Control circuits 12…Storage 13…Communication module 14… Interface 15…Drive 16...Storage medium 21…Multiple training datasets 22…Video Feature Calculation Unit 23... Acoustic Feature Calculation Unit 24…Virtual sound source generation unit 25...encoding section 26...Decoding section 27…Evaluation Department 28…Trained Model 29…Dataset for restoration 30…Output section 31...Coefficient calculation unit 32...Coefficient update section 33... Auxiliary variable update section 34…Virtual sound source update section 100... Spatial Acoustic Restoration Device SP... Multiple speakers SS... Multiple virtual sound sources U... User

Claims

1. A video feature calculation unit calculates video features based on video information, An acoustic feature calculation unit calculates acoustic features based on acoustic information, which is monaural audio corresponding to the aforementioned video information. A coefficient calculation unit that calculates higher-order ambisonic coefficients based on the aforementioned video features and acoustic features, A spatial acoustic restoration device equipped with the following features.

2. The aforementioned higher-order ambisonic coefficients correspond to virtual sound source information, which is a sound field formed by virtual sound sources independent of the video information. The spatial acoustic restoration device according to claim 1.

3. The coefficient calculation unit, A virtual sound source generation unit generates virtual sound source information based on the aforementioned video features and the aforementioned acoustic features, An encoding unit that encodes the virtual sound source information into the higher-order ambisonic coefficients, including, The spatial acoustic restoration device according to claim 2.

4. The coefficient calculation unit, A coefficient update unit updates the higher-order ambisonics coefficients based on the aforementioned video features, acoustic features, virtual sound source information, and auxiliary variables, An auxiliary variable update unit updates the auxiliary variables based on the updated higher-order ambisonics coefficients and the virtual sound source information, A virtual sound source update unit updates the virtual sound source information based on the updated auxiliary variables, including, The spatial acoustic restoration device according to claim 2.

5. The coefficient calculation unit terminates the update operation if the number of update operations by the coefficient update unit, the auxiliary variable update unit, and the virtual sound source update unit exceeds a predetermined threshold. The spatial acoustic restoration device according to claim 4.

6. A decoding unit that decodes output acoustic information from the aforementioned higher-order ambisonic coefficients, An evaluation unit updates the parameters of the neural networks included in the video feature calculation unit, the acoustic feature calculation unit, and the coefficient calculation unit, based on the comparison result between the output acoustic information or the higher-order ambisonic coefficients and the training acoustic information. It also has the following features: The spatial acoustic restoration device according to claim 1.

7. Calculating video features based on video information, Based on the audio information, which is monaural audio corresponding to the aforementioned video information, acoustic features are calculated. Based on the aforementioned video features and acoustic features, the higher-order ambisonics coefficients are calculated, A spatial acoustic restoration method equipped with [a specific feature].

8. A program for causing a computer to function as a component of the spatial sound restoration device described in any one of claims 1 to 6.

Images