Graph generation device, method, and program

Abstract

This graph generation device comprises: a graph generation unit 1 that generates, for each time interval t∈\{1, …, T\}, a graph gt indicating a contribution ratio wt(m,m'), which is the degree of influence of a sound signal acquired by each microphone m∈\{1, …, M\} on a sound signal acquired in the future by each microphone m' other than the microphone m; a division unit 2 that generates community information ct corresponding to each time interval t by dividing M microphones 1, …, M between communities each composed of a plurality of microphones having a high degree of association with each other; a contribution ratio calculation unit 3 that calculates a contribution ratio wt(mc,mc'), which is the degree of influence of a sound signal acquired by each microphone mc∈\{1, …, Mc\} on a sound signal acquired in the future by each microphone mc' other than the microphone mc among Mc microphones 1, …, Mc; and a selection unit 4 that selects a set of the community information ct and the contribution ratio wt(mc,mc') with which the most appropriate result is obtained when the prediction of the sound signal using the contribution ratio wt(mc,mc') is performed for a plurality of predetermined time intervals.

Description

Graph generation device, method, and program

[0001] The disclosed technology relates to a technology for processing sound signals.

[0002] In time series analysis for predicting general time series signals, state space models that perform linear prediction for stationary signals have long been studied (see, for example, Non-Patent Document 1).

[0003] Although not publicly known, the inventor has also proposed a technique for predicting future observation signals of a specific microphone from past observation signals of surrounding microphones, in which a graph approximate nearest neighbor search method is used to select which of the surrounding microphones to use.

[0004] Jang, Young-Jae, Jaehyun Park, Won-Cheol Lee, and Hong-June Park, 2022, "A Convolution-Neural-Network Feedforward Active-Noise-Cancellation System on FPGA for In-Ear Headphone", Applied Sciences 12, no. 11: 5300.

[0005] However, the graph obtained in the background art is created for each target microphone, and is not intended to simultaneously predict future signals from multiple microphones.

[0006] The disclosed technology aims to provide a graph generation device, method, and program for generating a graph for simultaneously predicting future signals from multiple microphones.

[0007] One aspect of the disclosed technology is to calculate a contribution ratio w , which is the degree to which a sound signal acquired by each microphone m∈{1, ..., M} influences a sound signal acquired by each microphone m′ other than microphone m in the future, where M microphones are 1, ..., M. t Graph g showing (m,m') t for each time interval t∈{1,…,T}, and a graph generator that generates a graph g corresponding to each time interval t. t The contribution rate w tBy dividing M microphones 1, ..., M into communities consisting of multiple microphones that are highly related to each other using (m, m'), the community information c corresponding to each time interval t is obtained. t and the community information c t M included in each community determined by c 1,…,M microphones c As, each microphone m c ∈{1,…,M c} is the sound signal acquired by M c Microphones 1,…,M c microphone m c Each microphone other than m c The contribution ratio w ' is the degree to which the signal will be acquired in the future. t (m c ,m c '), and the community information c corresponding to the time interval t∈{1,…,T} t and contribution rate w t (m c ,m c '), the contribution rate w t (m c ,m c The community information c that gives the best results when predicting sound signals using the t and contribution rate w t (m c ,m c A selection unit for selecting a set of ').

[0008] According to the disclosed technology, it is possible to generate a graph for simultaneously predicting future signals from multiple microphones.

[0009] Fig. 1 is a diagram illustrating an example of the functional configuration of a graph generation device. Fig. 2 is a diagram illustrating an example of a processing procedure of a graph generation method. Fig. 3 is a diagram illustrating an example of processing by a graph generation unit 1. Fig. 4 is a diagram illustrating an example of processing by a division unit 2. Fig. 5 is a diagram illustrating an example of processing by a contribution rate calculation unit 3. Fig. 6 is a diagram illustrating an example of processing by a selection unit 4. Fig. 7 is a diagram illustrating an example of the functional configuration of a computer.

[0010] Hereinafter, embodiments of the disclosed technology will be described with reference to the drawings. Note that components having the same functions in the drawings are given the same reference numerals, and redundant description will be omitted.

[0011] [Graph Generation Device and Method] As shown in FIG. 1, the graph generation device includes, for example, a graph generation unit 1, a division unit 2, a contribution rate calculation unit 3, and a selection unit 4.

[0012] The graph generation method is realized, for example, by each component of the graph generation device performing the processes of steps S1 to S4 shown in FIG.

[0013] <Graph Generator 1> The graph generator 1 receives as input sound signals acquired by each microphone mε{1, . . . , M} in each time interval tε{1, . . . , T}, where M microphones are denoted as 1, .

[0014] The sound signal may be an acoustic signal or a noise signal, for example, a non-stationary noise signal.

[0015] The sound signal may be a time-series signal, a frequency-domain signal, or a feature quantity representing the sound signal.

[0016] The graph generation unit 1 calculates a contribution ratio w , which is the degree to which a sound signal acquired by each microphone m∈{1, ..., M} influences a sound signal acquired by each microphone m′ other than microphone m in the future. t Graph g showing (m,m') t is generated for each time interval tε{1, . . . , T} (step S1).

[0017] The generated graph g t is output to the selection unit 4.

[0018] For example, as shown in FIG. 3, assume that the number of microphones is 16. That is, assume that M=16. In this case, the contribution rate w t (m,m') are associated. Contribution rate w tAs mentioned above, (m, m') represents the degree to which the sound signal acquired by microphone m in time interval t influences the sound signal acquired by microphone m' in the future. In the example of Figure 3, the contribution rate w t Only edges e(m,m') where (m,m') is equal to or greater than a predetermined threshold are indicated by arrows.

[0019] In each time interval t, m×(m-1) contribution ratios w corresponding to m×(m-1) pairs of m, m' are t (m,m') is calculated. Graph g t is the m×(m-1) contribution rate w in each time interval t. t (m,m') is shown. Graph g t is the m×(m-1) contribution rate w in each time interval t. t It can be (m,m') itself. Graph g t is the contribution rate w t It may contain information other than (m,m'). For example, the graph g t may contain information about nodes Mic1,...,M.

[0020] The graph generation unit 1 generates a graph g using sound signals acquired by each microphone m∈{1,...,M} in a time interval t. t In this case, the graph generating unit 1 can use an existing method such as that described in Reference 1.

[0021] [Reference 1] Hilmi E. Egilmez, Eduardo Pavez, and Antonio Ortega, "Graph Learning from Filtered Signals: Graph System and Diffusion Kernel Identification", arXiv, Mar. 2018. In Reference 1, a graph is generated by (1) formulating a graph system to identify it from the sound signals acquired by each microphone, and (2) alternately solving the graph and graph filtering problems.

[0022] <Divider 2> Divider 2 divides the graph g generated by graph generator 1. t is entered.

[0023] The division unit 2 divides a graph g corresponding to each time interval t. t The contribution rate w t By dividing M microphones 1, ..., M into communities consisting of multiple microphones that are highly related to each other using (m, m'), the community information c corresponding to each time interval t is obtained. t is generated (step S2).

[0024] Generated community information c t is output to the contribution rate calculation unit 3 and the selection unit 4.

[0025] The community information is information about each divided community, in other words, information about the microphones that belong to each community.

[0026] For example, the division unit 2 divides the graph g t The contribution rate w t Using (m,m'), the community information c t In this case, the division unit 2 can use an existing method such as that described in Reference 2.

[0027] [Reference 2] M. Girvan and MEJ Newman, "Community structure in social and biological networks", Proceedings of the National Academy of Sciences, vol.99, no.12, pp.7821-7826, 2002. In Reference 2, (1) a process of calculating the betweenness centrality of each edge, (2) a process of deleting the edge with the highest betweenness centrality, and (3) a process of recalculating the betweenness centrality of each edge in the state after deletion are performed. Then, (2) a process of deleting the edge with the highest betweenness centrality, and (3) a process of recalculating the betweenness centrality of each edge in the state after deletion are repeated a predetermined number of times, or until the number of communities generated by edge deletion reaches a predetermined number. Betweenness centrality is a numerical value that indicates the percentage of the edge in question that is included in the shortest path between two different nodes. After these processes, if a contribution rate w t Each community is generated consisting of multiple microphones connected to each other at (m,m').

[0028] In the example of Fig. 4, 16 microphones are divided into a community consisting of 6 microphones, a community consisting of 2 microphones, a community consisting of 4 microphones, and a community consisting of 4 microphones. Information about each of these divided communities is community information. In the example of Fig. 4, the contribution rate w t The arrows indicating (m, m') are omitted.

[0029] The division unit 2 may generate community information using other community detection methods, such as a community detection method using eigenvector centrality.

[0030] <Contribution Rate Calculation Unit 3> The community information generated by the division unit 2 is input to the contribution rate calculation unit 3. In addition, sound signals acquired by each microphone m∈{1, ..., M} in each time interval t∈{1, ..., T} are input to the contribution rate calculation unit 3.

[0031] The contribution rate calculation unit 3 calculates the community information c corresponding to each time interval t∈{1, ..., T}. t M included in each community determined by c 1,…,M microphones c As, each microphone m c ∈{1,…,M c} is the sound signal acquired by M c Microphones 1,…,M c microphone m c Each microphone other than m c The contribution ratio w ' is the degree to which the signal will be acquired in the future. t (m c ,m c ') is calculated (step S3).

[0032] Calculated contribution ratio w t (m c ,m c ') is output to the selection unit 4.

[0033] The contribution ratio calculation unit 3 calculates the contribution ratio w using an appropriate method such as the least squares method or the Ridge method. t (m c ,m c The contribution rate calculation unit 3 calculates the contribution rate w t Using the same method as that used to calculate (m,m'), the contribution rate w t (m c ,m c However, the contribution rate w t (m c ,m c The point where the contribution rate w t This is different from the calculation of (m,m').

[0034] For example, the contribution rate w for a community consisting of four microphones among the four communities in Figure 3 is t (m c ,m cAs shown in FIG. 5, the four microphones that make up this community are microphones 1, 2, 3, and 4. In this case, the contribution rate calculation unit 3 calculates the contribution rate of each microphone m using the sound signals acquired by microphones 1, 2, 3, and 4 in a time period t. c The sound signal acquired at ∈{1, 2, 3, 4} is M c Microphone m among microphones 1, 2, 3, and 4 c Each microphone other than m c The contribution ratio w ' is the degree to which the signal will be acquired in the future. t (m c ,m c ') to calculate

[0035] Specifically, the contribution ratio w is the degree to which the sound signal acquired by microphone 1 influences the sound signals acquired by microphones 1, 2, and 3 other than microphone 1 in the future. t (1,2),w t (1,3),w t (1,4) is calculated. In the example of Figure 5, the contribution rate w t (1,2),w t (1,3),w t The contribution ratio calculation unit 3 performs the same process for each of the microphones 2, 3, and 4, and calculates the contribution ratio w t (2,3),w t (2,4),w t (2,1), contribution rate w t (3,4),w t (3,1),w t (3,2), contribution rate w t (4,1),w t (4,2),w t Calculate (4,3).

[0036] The contribution rate calculation unit 3 performs this process using the community information c t This will be done for each community identified by

[0037] <Selection Unit 4> The selection unit 4 selects the community information c generated by the division unit 2. t and the contribution rate w calculated by the contribution rate calculation unit 3 t (m c ,m c ') is entered.

[0038] The selector 4 selects community information c corresponding to the time interval t∈{1,...,T}. t and contribution rate w t (m c ,m c '), the contribution rate w t (m c ,m c The community information c that gives the best results when predicting sound signals using the t and contribution rate w t (m c ,m c A set of ') is selected (step S4).

[0039] For example, the selection unit 4 selects community information c corresponding to each time interval t. t and contribution rate w t (m c ,m c ') to predict the sound signal of a time interval later than each of the predetermined time intervals, and obtain the community information c corresponding to each time interval t. t and contribution rate w t (m c ,m c ') and calculate the likelihood of the set of community information c t and contribution rate w t (m c ,m c The set of community information and contribution rates that maximizes the likelihood from the set of community information c t and contribution rate w t (m c ,m c ') as a set.

[0040] For example, in the time interval t∈{1, 2, 3}, as shown in FIG. 6, the community information c1 and contribution rate w1(m c ,m c '), the community information c2 and contribution rate w2(m c ,m c '), the community information c3 and contribution rate w3(mc ,m c ') is generated.

[0041] In this case, the selector 4 selects the community information c1 and contribution rate w1(m c ,m c The selection unit 4 selects an evaluation value v , which has a larger value as the predicted sound signal is closer to the sound signal acquired by each microphone in the time interval 2. 12 Similarly, the selector 4 calculates the community information c1 and contribution rate w1(m c ,m c The selection unit 4 selects an evaluation value v , which has a larger value as the predicted sound signal is closer to the sound signal acquired by each microphone in the time interval 3. 13 In this case, the community information c1 and contribution rate w1(m c ,m c The likelihood of a set of v 12 +v 13 is.

[0042] Similarly, the selector 4 selects the community information c2 and contribution rate w2(m c ,m c '), the evaluation value v when predicting the future sound signals acquired by each microphone in time interval 1. 21 and the community information c2 and contribution rate w2(m c ,m c ') to predict the sound signals to be acquired by each microphone in time interval 3. 23 In this case, the community information c2 and contribution rate w2(m c ,m c The likelihood of a set of v 21 +v 23 This becomes:

[0043] Similarly, the selector 4 selects the community information c3 and contribution rate w3(m c,m c '), the evaluation value v when predicting the future sound signals acquired by each microphone in time interval 1. 31 and the community information c3 and contribution rate w3(m c ,m c '), the evaluation value v when predicting the future sound signals acquired by each microphone in time interval 2 32 In this case, the community information c3 and contribution rate w3(m c ,m c The likelihood of a set of v 31 +v 32 This becomes:

[0044] In this case, the selector 4 selects the community information c1 and contribution rate w1(m c ,m c ') the likelihood of the set v 12 +v 13 , community information c2 and contribution rate w2(m c ,m c ') the likelihood of the set v 21 +v 23 , community information c3 and contribution rate w3(m c ,m c ') the likelihood of the set v 31 +v 32 The set of community information and contribution rates with the highest likelihood is selected as the community information c t and contribution rate w t (m c ,m c ') as a set.

[0045] As in this example, the selection unit 4 further uses the sound signals acquired by each microphone in each of a plurality of predetermined time periods to select the community information c that provides the most appropriate result. t and contribution rate w t (m c ,m c ') set may be selected.

[0046] Community Information t and contribution rate wt (m c ,m c ') specifies the graph. Therefore, the community information c t and contribution rate w t (m c ,m c ') is to generate a graph.

[0047] By the above process, a graph can be generated to simultaneously predict future signals from multiple microphones.

[0048] This makes it possible to simultaneously and efficiently predict future signals from multi-channel microphones, for example, noise whose mean, variance, and covariance are non-stationary, using multi-channel microphones.

[0049] [Modifications] The specific configurations of the embodiments of the disclosed technology are not limited to the configurations described above. The specific configurations of the embodiments of the disclosed technology can be appropriately modified in design, etc., within the scope of the spirit of the embodiments of the disclosed technology.

[0050] For example, the graph generating device may include a prediction unit 5 shown by a dashed line in FIG. 1. In this case, the community information c selected by the selection unit 4 t and contribution rate w t (m c ,m c The set of ') is input to the prediction unit 5.

[0051] The prediction unit 5 predicts the sound signals acquired by each microphone m∈{1,...,M} and the input community information c t and contribution rate w t (m c ,m c The prediction unit 5 predicts a sound signal to be acquired in the future at each microphone m ∈ {1, ..., M} using the graph specified by the set of c Future sound signals of 's mc ' can be predicted based on the following formula: s mc is microphone m c If the sound signal is converted into a feature and then processed, mcThe prediction unit 5 performs this process for each community.

[0052] s mc '=Σ mc∈{1,…,Mc} w t (m c ,m c ')s mc In this way, the prediction unit 5 converts the sound signal acquired by the microphone m into the community information c selected by the selection unit 4. t and contribution rate w t (m c ,m c ') is represented by a graph specified by a set of t (m c ,m c '), the microphone m c Future sound signals of 's mc ' may be predicted.

[0053] In this way, by classifying all microphones into those with similar signals and predicting each other's future signals for each community, the accuracy of the predicted signals is improved.

[0054] The various processes described in the embodiments of the disclosed technology may not only be performed chronologically in the order described, but may also be performed in parallel or individually depending on the processing capacity of the device performing the processes or as needed.

[0055] For example, data may be exchanged directly between components of the graph generation device, or may be exchanged via a storage unit (not shown).

[0056] Furthermore, a device (terminal) for using the device, system, or method of the present invention via a network (telecommunications line) may also be provided. The "device (terminal) for use" may be provided with functions (e.g., control function, decoding function, restoration function, input / output function, etc.) necessary to obtain the effects of implementing the device, system, or method of the present invention.

[0057] It goes without saying that other modifications are possible without departing from the spirit of the present invention.

[0058] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[0059] [Program, Recording Medium] The functions realized by the components described in this specification may be implemented in circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), CPUs (Central Processing Units), conventional circuits, and / or combinations thereof, programmed to realize the described functions. A processor includes transistors and other circuits and is considered to be circuitry or processing circuitry. A processor may also be a programmed processor that executes a program stored in a memory.

[0060] In this specification, a circuitry, unit, or means is hardware that is programmed to realize or performs the described functions, which may be any hardware disclosed herein or any hardware known to be programmed to realize or perform the described functions.

[0061] If the hardware is a processor considered to be a type of circuitry, the circuitry, means, or unit is a combination of the hardware and software used to configure the hardware and / or processor.

[0062] The various processes described above can be implemented by loading a program that executes each step of the above method into the recording unit 2020 of the computer 2000 shown in Figure 7, and operating the control unit 2010, input unit 2030, output unit 2040, display unit 2050, etc.

[0063] The program describing the processing contents can be recorded on a computer-readable recording medium, which may be, for example, a magnetic recording device, an optical disk, a magneto-optical recording medium, a semiconductor memory, or any other suitable recording medium.

[0064] The program may be distributed by, for example, selling, transferring, lending, etc. portable recording media such as DVDs and CD-ROMs on which the program is recorded. Furthermore, the program may be stored in a storage device of a server computer, and then transferred from the server computer to other computers via a network, thereby distributing the program.

[0065] A computer that executes such a program may first temporarily store the program recorded on a portable recording medium or transferred from a server computer in its own storage device. Then, when executing a process, the computer reads the program stored on its own recording medium and executes the process in accordance with the read program. Alternatively, the computer may read the program directly from a portable recording medium and execute the process in accordance with the program. Furthermore, the computer may execute the process in accordance with the program each time a program is transferred from a server computer to the computer. Alternatively, the server computer may not transfer the program to the computer, but may instead execute the process through a so-called ASP (Application Service Provider) service, which realizes the processing function by issuing an execution instruction and obtaining the results. Furthermore, the server computer may execute the process at the terminal using a so-called SaaS (Software as a Service) service, which allows users to use part of a server computer along with the program. In this embodiment, the program includes information used for processing by an electronic computer that is equivalent to a program (such as data that is not a direct instruction to a computer but has properties that dictate computer processing).

[0066] Furthermore, in this embodiment, the device is configured by executing a predetermined program on a computer, but at least a part of the processing contents may be realized by hardware.

Claims

1. Let m be the number of microphones, 1,…,M. The contribution ratio w is the degree to which the sound signal acquired by each microphone m∈{1,…,M} influences the sound signal acquired by each microphone m' other than microphone m in the future. t Graph g showing (m,m') t for each time interval t∈{1,…,T}, and a graph generator that generates a graph g corresponding to each time interval t. t The contribution rate w t (m, m') to divide the M microphones 1, ..., M into communities consisting of multiple microphones that are highly related to each other, and obtain community information c corresponding to each time interval t. t and the community information c corresponding to each time interval t∈{1,…,T}. t M included in each community determined by c 1,…,M microphones c As, each microphone m c ∈{1,…,M c } is the sound signal acquired by c Microphones 1,…,M c microphone m c Each microphone other than m c The contribution ratio w ' is the degree to which the signal will be acquired in the future. t (m c ,m c The contribution calculation part calculates the community information c t and contribution rate w t (m c ,m c '), the contribution rate w t (m c ,m c The community information c that gives the best results when predicting sound signals using the t and contribution rate w t (m c ,m c A graph generating device including: a selection unit that selects a set of 2. The graph generating device according to claim 1, wherein the selection unit selects community information c corresponding to each time interval t. t and contribution rate w t (m c ,m c ') to predict a sound signal in a time interval later than each of the predetermined time intervals t, thereby obtaining community information c corresponding to each time interval t. t and contribution rate w t (m c ,m c ') and calculate the likelihood of the set of community information c t and contribution rate w t (m c ,m c The set of community information and contribution rates that maximizes the likelihood from the set of community information c t and contribution rate w t (m c ,m c ') as a set of graph generators.

3. The graph generator calculates the contribution ratio w , which is the degree to which the sound signal acquired by each microphone m∈{1,…,M} influences the sound signal acquired by each microphone m′ other than microphone m, where m is the number of microphones 1,…,M. t Graph g showing (m,m') t A graph generation step generates a graph g for each time interval t∈{1,…,T}. t The contribution rate w t (m, m') to divide the M microphones 1, ..., M into communities consisting of multiple microphones that are highly related to each other, and obtain community information c corresponding to each time interval t. t The contribution calculation unit calculates the community information c corresponding to each time interval t∈{1,…,T}. t M included in each community determined by c 1,…,M microphones c As, each microphone m c ∈{1,…,M c } is the sound signal acquired by c Microphones 1,…,M c microphone m c Each microphone other than m c The contribution ratio w ' is the degree to which the signal will be acquired in the future. t (m c ,m c A contribution calculation step in which the contribution ratio is calculated by calculating the contribution ratio '), and a selection step in which the community information c t and contribution rate w t (m c ,m c '), the contribution rate w t (m c ,m c The community information c that gives the best results when predicting sound signals using the t and contribution rate w t (m c ,m c A selection step of selecting a set of ') and 4. A program for causing a computer to execute each step of the graph generation method of claim 3.

Images