Noise removal device, noise removal method and program

Abstract

To remove noise from a multi-modal signal by solving a noise removal problem while optimizing graph structure from the perspective of noise removal.SOLUTION: A computing equation derived from a first object function that is defined based on the assumption that an original signal becomes smooth on a first graph when the original signal that is formed from an observation signal value in each column of an observation signal matrix and from which noise has been removed is expressed using the first graph, and a second object function that is defined based on the assumption that the original signal becomes smooth on a second graph when the original signal that is formed from the observation signal value in each row of the observation signal matrix and from which noise has been removed is expressed using a second graph, optimizes an adjacency matrix of the first and second graphs, and then applies K observation signal matrices to the computing equation that performs the operation of removing noise from the observation signal based on the optimized adjacency matrix to estimate the original signal for which noise is removed from each of the observation signals.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a noise removal device, a noise removal method, and a program. [Background technology]

[0002] Signal restoration is the process of recovering an original signal from an observed signal that has been degraded by noise or defects. In the real world, observed signals often contain noise and defects, so the signal restoration problem is an important task that continues to be researched. For example, signal restoration problems for audio signals, image signals, video signals, etc. have been widely studied.

[0003] Signals obtained from sensor networks, traffic networks, point clouds, etc. have spatial interconnected relationships. Such connections can be expressed as graphs using vertices and edges. Graph Signal Processing (GSP) technology is known as a signal processing technology for signals that can be expressed as graphs in this way, and by using this technology, it is possible to mathematically analyze graph signals, which are signals that exist in vertex regions. For example, by using this technology, it becomes possible to perform processing that takes into account changes in complex networks.

[0004] Incidentally, observation signals obtained by observation at multiple observation positions in space, more specifically, by measurements using sensors, etc., are often multimodal signals consisting of signals observed in multiple domains, such as light, sound, and distance. It is expected that each modality of a multimodal signal has a correlation with each other. Therefore, the relationship between domains, i.e., between modalities, can be mathematically represented by a modality graph with each modality as a vertex and the relationship between the modalities as an edge. On the other hand, since each observation position has a spatial correlation as described above, the relationship between the observation positions can be mathematically represented by a spatial graph with each observation position as a vertex and the relationship between the observation positions as an edge.

[0005] Like normal signals, multimodal signals are subject to superimposed noise during observation, and this noise must be removed. In the field of graph signal processing technology described above, a problem known as the noise removal problem is known, and this problem aims to obtain an unknown original signal from a noise-containing observed signal by utilizing the connection relationship between samples. A method has been proposed in which this graph signal noise removal problem is set for multimodal signals, and two types of graphs, a spatial graph and a modality graph, are used to remove noise from the multimodal signal and obtain the original signal (see, for example, Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Masatoshi Nagahama and Yuichi Tanaka, “Multimodal Graph Signal Denoising Via Twofold Graph Smoothness Regularization with Deep Algorithm Unrolling”, 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 5862-5866. Summary of the Invention [Problem to be solved by the invention]

[0007] In the above-mentioned problem of removing noise from a graph signal, the graph is often not explicitly given. In such cases, in order to perform an effective analysis, it is necessary to estimate an appropriate graph from the observed signal. However, in the technology disclosed in Non-Patent Document 1, the structure of the graph is given heuristically, in other words, without a clear mathematical basis, and is not optimized from the viewpoint of noise removal.

[0008] The present invention provides a noise removal device, a noise removal method, and a program capable of removing noise from a multimodal signal by solving a noise removal problem while optimizing a graph structure from the viewpoint of noise removal. [Means for solving the problem]

[0009] According to one aspect of the present invention, there is provided a method for generating an observation signal matrix by arranging K observation signal matrices, the number of observation positions x the number of domains x K observation signal values ​​obtained by K observations (where K is an integer equal to or greater than 2) at each of a plurality of observation positions in space, with either the observation positions or the domains as rows and the other as columns; a first objective function determined based on the assumption that when an original signal obtained by removing noise from an observation signal formed by the observation signal values ​​for each column of the observation signal matrix is ​​represented by a first graph having vertices equal to the number of observation signal values ​​included in the columns of the observation signal matrix, the original signal becomes smooth on the first graph; and and a second objective function determined based on the assumption that when an original signal obtained by removing noise from an observed signal formed by the observed signal values ​​for each row of a noise matrix is ​​represented by a second graph having vertices equal to the number of observed signal values ​​included in a row of the observed signal matrix, the original signal will be smooth on the second graph, the second objective function being an arithmetic formula derived from the second objective function determined based on the assumption that the original signal will be smooth on the second graph, the second graph having vertices equal to the number of observed signal values ​​included in a row of the observed signal matrix, the original signal being estimated by applying the K observed signal matrices generated by the matrix generation unit to an arithmetic formula that optimizes an adjacency matrix of each of the first graph and the second graph and performs an operation to remove noise from the observed signal based on the optimized adjacency matrices.

[0010] According to one aspect of the present invention, there is provided a matrix generation step of arranging K observation signal matrices by arranging the number of observation positions x the number of domains x K observation signal values ​​obtained by K observations (where K is an integer equal to or greater than 2) at each of a plurality of observation positions in space, with either the observation positions or the domains as rows and the other as columns; a first objective function determined based on the assumption that when an original signal obtained by removing noise from an observation signal formed by the observation signal values ​​for each column of the observation signal matrix is ​​represented by a first graph having vertices equal to the number of observation signal values ​​included in the columns of the observation signal matrix, the original signal will be smooth on the first graph, and a second objective function determined based on the assumption that when an original signal obtained by removing noise from an observation signal formed by the observation signal values ​​for each row of the observation signal matrix is ​​represented by a second graph having vertices equal to the number of observation signal values ​​included in the rows of the observation signal matrix, the original signal will be smooth on the second graph, the original signal being derived from the first ... being derived from the second objective function determined based on the assumption that

[0011] One aspect of the present invention is a computer comprising: a matrix generation means for generating K observation signal matrices by arranging the number of observation positions × the number of domains × K observation signal values ​​obtained by K observations (where K is an integer equal to or greater than 2) at each of a plurality of observation positions in space, with either the observation positions or the domains as rows and the other as columns; a first objective function determined based on the assumption that when an original signal obtained by removing noise from an observation signal formed by the observation signal values ​​for each column of the observation signal matrix is ​​represented by a first graph having vertices equal to the number of observation signal values ​​included in the columns of the observation signal matrix, the original signal will be smooth on the first graph; and a second objective function defined based on the assumption that when an original signal resulting from removing noise from an observation signal formed by the observation signal values ​​for each row of a signal matrix is ​​represented by a second graph having vertices equal to the number of observation signal values ​​included in the rows of the observation signal matrix, the original signal will be smooth on the second graph, the program causing the program to function as a calculation processing means that applies the K observation signal matrices generated by the matrix generation means to an calculation formula that optimizes the adjacency matrices of the first graph and the second graph and performs a calculation to remove noise from the observation signal based on the optimized adjacency matrices, thereby estimating an original signal resulting from removing noise from each of the observation signals. Effect of the Invention

[0012] According to the present invention, it is possible to remove noise from a multimodal signal by solving a noise removal problem while optimizing a graph structure from the viewpoint of noise removal. [Brief description of the drawings]

[0013] [Figure 1] 1 is a block diagram showing a configuration of a noise removal device according to a first embodiment. [Diagram 2] 4 is a diagram illustrating an observed signal array generated by a matrix generation unit of the first embodiment. FIG. [Diagram 3] 4 is a flowchart showing a process flow of the noise removal device according to the first embodiment. [Figure 4] 5 is a flowchart showing a flow of processing by a modality calculation unit in the first embodiment. [Diagram 5] 5 is a flowchart showing a flow of processing by a spatial calculation unit in the first embodiment. [Figure 6] FIG. 2 is a diagram illustrating a process performed by the noise removal device of the first embodiment in the form of a program. [Figure 7] FIG. 11 is a block diagram showing the configuration of a noise removal device according to a second embodiment. [Figure 8] 10 is a flowchart showing a process flow of a noise removal device according to a second embodiment. [Figure 9] FIG. 11 is a diagram illustrating, in the form of a program, a process performed by the noise removal device according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] (First embodiment) DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described with reference to the drawings. Fig. 1 is a block diagram showing the configuration of a noise removal device 1 according to a first embodiment. The noise removal device 1 includes a matrix generation unit 2 and a calculation processing unit 3.

[0015] The matrix generating unit 2 takes in N×M×K observed signal values. Here, N, M, and K are all integers equal to or greater than 2. The N×M×K observed signal values ​​are, for example, sensors installed at N different positions in space, each of which is capable of measurement in M ​​different domains, and are each measured K times by the sensor. The M different domains are, for example, measurement indices such as light, sound, and distance. Hereinafter, the domains are also referred to as modalities. When light is used as the measurement index, for example, the sensor measures the illuminance around the sensor. When sound is used as the measurement index, for example, the sensor measures the volume around the sensor. When distance is used as the measurement index, for example, the sensor measures the distance from the sensor to a predetermined reference position. Hereinafter, N is referred to as the number of observation positions, M is referred to as the number of modalities, and K is referred to as the number of observations.

[0016] Each of the N×M×K observed signal values ​​taken in by the matrix generation unit 2 is assigned three index values, n, m, and k. For example, when each of the above-mentioned sensors acquires an observed signal value by measurement, it assigns to the acquired observed signal value n indicating the observation position where the observed signal value was obtained, m indicating the modality of the observed signal value, and k indicating the number of times the observed signal value was observed. Therefore, n is any integer value between 1 and N, m is any integer value between 1 and M, and k is any integer value between 1 and K.

[0017] The matrix generator 2 generates K N-row by M-column matrices from the N×M×K observed signal values ​​based on three index values ​​assigned to each of the N×M×K observed signal values. In other words, the matrix generator 2 generates a three-dimensional array with N arrays in the row direction, M arrays in the column direction, and K arrays in the depth direction from the N×M×K observed signal values.

[0018] In the K matrix of N rows by M columns generated by the matrix generating unit 2, there are K×M combinations of N observed signal values ​​for each column. Each of the K×M observed signals including N observed signal values ​​(hereinafter also referred to as "row-wise observed signals") can be represented by a graph having N vertices, and this graph is hereinafter referred to as a spatial graph (hereinafter also referred to as a "first graph"). In the K matrix of N rows by M columns, there are K×N combinations of M observed signal values ​​for each row. Each of the K×N observed signals including M observed signal values ​​(hereinafter also referred to as "column-wise observed signals") can be represented by a graph having M vertices, and this graph is hereinafter referred to as a modality graph (hereinafter also referred to as a "second graph").

[0019] (Definition of terms and symbols) Here, the terms and symbols related to graphs used in the following explanation will be explained. A graph is a mathematical model of a network structure. A graph g having N vertices is represented by the symbols in the following formula (1).

[0020]

number

[0021] In graph g, a vertex set v is represented by the symbols in the following equation (2).

[0022]

number

[0023] In a graph g, an edge set ε is defined by the following equation (3).

[0024]

number

[0025] A graph g is defined by three parameters including a vertex set v and an edge set ε, as shown in the following equation (4).

[0026]

number

[0027] The third parameter in equation (4), W in bold, is the adjacency matrix W. When the number of vertices is N, the adjacency matrix W is an N-row by N-column matrix whose elements are real numbers defined by the following equation (5).

[0028]

number

[0029] Hereinafter, as with the adjacency matrix W, in the formulas, arrays, matrices, and vectors are shown in bold, and in the text, alphanumeric characters representing the arrays, matrices, and vectors are used along with the words "array," "matrix," and "vector."

[0030] An adjacency matrix is ​​a matrix that represents the connection relationship between vertices, that is, the structure of a graph g. When self-loop edges and multiple edges are not taken into consideration, the elements of the adjacency matrix W are defined by the following formula (6).

[0031]

number

[0032] In formula (6), "if(i,j)∈ε" indicates the case where an edge exists between the vertices, and "otherwise" indicates the case where no edge exists between the vertices. ij is the weight of the edge between vertex i and vertex j, and w ij >0. Here, the number of vertices is N, so i = 1, ..., N and j = 1, ..., N.

[0033] The sum of the weights of the edges connected to a vertex is called the degree. The degree of vertex i is d i can be calculated by the following equation (7).

[0034]

number

[0035] The frequency of N items for i=1,…,N is d i A matrix D having diagonal elements is called a frequency matrix. The frequency matrix D is defined by the following equation (8).

[0036]

number

[0037] In equation (8), diag(·) is the symbol for a square matrix whose diagonal elements are “·”. The frequency matrix D is an N-row × N-column matrix. The graph Laplacian matrix L is defined by the following equation (9) using the frequency matrix D and the adjacency matrix W.

[0038]

number

[0039] The graph signal of graph g is a signal vector x defined by the following equation (10), and the domain of the vertex set v is the signal vector x.

[0040]

number

[0041] In other words, a graph signal is a set of signal values ​​corresponding to vertex i, x i Then, the signal values ​​x1, ..., x corresponding to each of the N vertices are N The graph signal is also called a vertex signal.

[0042] If the fluctuation of the graph signal is F(x) (where x in F(x) is a signal vector x), F(x) is defined by the following equation (11) using the signal vector x, which is a graph signal, and the graph Laplacian matrix L.

[0043]

number

[0044] Equation (11) is called the Laplacian quadratic form, and the edge weights w ij and the signal difference between the vertices connected by the edge (x i -x j ) squared. Therefore, F(x) will be a small value when vertices in the graph signal tend to be connected when the difference in signal value x is small. In this way, a state in which vertices in the signal value x tend to be connected when the difference in signal value x is small will be referred to as the graph signal being smooth on the graph g. Therefore, the smaller the value of F(x), the smoother the graph signal, i.e., the signal vector x, will be on the graph g.

[0045] (Noise removal problem setting) Next, we will explain the noise removal problem solved by using the noise removal device 1. Fig. 2 is a diagram showing a three-dimensional array 20 generated by the matrix generation unit 2 from NxMxK ​​observed signal values. Hereinafter, the three-dimensional array 20 will be referred to as observed signal array Y, and the observed signal array Y is defined by the following equation (12).

[0046]

number

[0047] As shown in Fig. 2, the observed signal array Y is formed by K matrices 21-1 to 21-K with N rows and M columns. Each of the matrices 21-1 to 21-K is a so-called multimodal signal. From another perspective, the observed signal array Y can be viewed as including N matrices with M rows and K columns indicated by reference numeral 22-n, or as including M matrices with N rows and K columns indicated by reference numeral 23-m. Here, the observed signal array Y is expressed as in the following equation (13).

[0048]

number

[0049] In equation (13), the first term on the right side is hereinafter referred to as original signal array X, and the second term on the right side is hereinafter referred to as noise array N. Original signal array X is defined by the following equation (14), and noise array N is defined by the following equation (15). That is, like observed signal array Y, original signal array X and noise array N are three-dimensional arrays expressed in the form of three-dimensional array 20 shown in FIG. 2.

[0050]

number

[0051]

number

[0052] Each signal value included in the original signal sequence X is a signal value of an unknown original signal, and each signal value included in the noise sequence N is a signal value of an additive white Gaussian noise signal. The noise removal problem to be solved by using the noise removal device 1 is a problem of estimating the original signal sequence X by removing the noise sequence N from the observed signal sequence Y.

[0053] Here, when the observed signal values ​​in the kth observation are arranged in a matrix format with the observation position as rows and the modality as columns, it becomes a matrix 21-k with N rows and M columns as shown in FIG. 2. This matrix 21-k is hereinafter referred to as the observed signal matrix Y (k) It is expressed as:

[0054]

number

[0055] Observed signal matrix Y (k) The matrix of the original signal corresponding to (k) Let the original signal matrix X (k) is the observed signal matrix Y (k) Since it is an N row × M column matrix like above, it is defined by the following equation (17).

[0056]

number

[0057] The original signal array X is, like the observed signal array Y, an original signal matrix X (k) (k=1,…,K). The original signal matrix X (k) It is assumed that the column vectors of each column of are smooth on the spatial graph, and the row vectors of each row are smooth on the modality graph. As mentioned above, being smooth on the spatial graph means that the original signal matrix X (k) When the m-th column (m=1,...,M) of N signal values ​​is expressed as a spatial graph with N vertices, the difference between the signal values ​​of adjacent vertices in the spatial graph is small and does not change suddenly. Similarly, being smooth on the modality graph means that the original signal matrix X(k) When the M signal values ​​in the nth row (n=1, ..., N) of are represented as a modality graph with M vertices, the difference in signal values ​​between adjacent vertices in the modality graph is small and does not change suddenly.

[0058] Hereinafter, if a spatial graph having N vertices is shown according to the notation of equation (4), it can be expressed as the following equation (18).

[0059]

number

[0060] In equation (18), the parameters on the right hand side are the spatial graph g s A set of vertices in V s , edge set ε s , the adjacency matrix W s and the adjacency matrix W s Since is a matrix with N rows and N columns, it is defined by the following equation (19).

[0061]

number

[0062] Similarly, when a modality graph having M vertices is represented according to the notation of equation (4), it can be expressed as the following equation (20).

[0063]

number

[0064] In equation (20), the parameters on the right side are the modality graph g m A set of vertices in V m , edge set ε m , the adjacency matrix W m and the adjacency matrix W m Since is a matrix of M rows and M columns, it is defined by the following equation (21).

[0065]

number

[0066] In addition, the spatial graph g s and Modality Graph g m is the adjacency matrix W s and the adjacency matrix W m The weights of each ij But as mentioned above, w ij Since >0, it is an undirected graph.

[0067] (Setting the objective function to solve the noise removal problem) The above original signal matrix X (k) The column vectors of each column of the spatial graph g s is smooth on the modality graph g m Using the assumption that the signal is smooth on the graph, we estimate the original signal sequence X from the observed signal sequence Y by graph learning. To do this, we use a spatial graph g s For the modality graph g, an objective function (hereinafter also referred to as the “first objective function”) is set as shown in the following equation (22). m For this, an objective function (hereinafter also referred to as a "second objective function") is set as shown in the following equation (23).

[0068]

number

[0069]

number

[0070] In formula (22) and formula (23), the symbol shown in the following formula (24) is an operator that calculates the square value of the Frobenius norm. tr(·) is an operator that calculates the trace of a matrix, that is, the sum of the diagonal elements of a square matrix.

[0071]

number

[0072] In equation (22), the observed signal array Y s is a three-dimensional array 20 of observed signal array Y, which is a matrix of K observed signals Y (k) (k=1,...,K) and then concatenated while maintaining the format of an N row x M column matrix, as shown in the following equation (25).

[0073]

number

[0074] In equation (22), the original signal array X s is the observed signal array Y s Using the same procedure, the three-dimensional original signal array X is transformed into K original signal matrices X (k) (k=1,…,K) and then concatenated while maintaining the format of an N row × M column matrix.

[0075] In equation (23), the observed signal array Y m is an array generated as follows. The observed signal array Y, which is a three-dimensional array 20, is divided into K observed signal matrices Y (k) (k=1,…,K), and the decomposed K observed signal matrices Y (k) Transpose each of the observation signal matrix Y (k) The transpose of the observed signal matrix Y (k) Since is defined by equation (16), it is defined as the following equation (26).

[0076]

number

[0077] K observed signal matrices Y (k) The transposed matrices of each of the above are combined while maintaining the M row × N column matrix format as shown in the following equation (27), to form the observed signal array Y m become.

[0078]

number

[0079] In equation (23), the original signal array X m is the observed signal array Y m In other words, the original signal array X, which is a three-dimensional array, is divided into K original signal matrices X (k) (k=1,…,K) and decompose the K original signal matrices X (k) (k=1,…,K) to generate a transposed matrix. K original signal matrices X (k) The original signal array X m become.

[0080] The spatial graph g shown in (22) s The graph Laplacian matrix L of s Since is a matrix of N rows and N columns, it is defined by the following equation (28).

[0081]

number

[0082] The modality graph g shown in (23) m The graph Laplacian matrix L of m Since is a matrix of M rows and M columns, it is defined by the following equation (29).

[0083]

number

[0084] The parameter α in the second term in the brackets in equations (22) and (23) s ,α m is a predetermined parameter for the smoothness of the signal, and α s ,α m >0. The third term f(·) in the parentheses of equations (22) and (23) is the function shown in the following equations (30) and (31), respectively.

[0085]

number

[0086]

number

[0087] In equation (30), vector 1 shown in equation (32) s is an N row by 1 column vector whose elements are all 1.

[0088]

number

[0089] In equation (31), vector 1 shown in equation (33) m is an M row by 1 column vector whose elements are all 1.

[0090]

number

[0091] β shown in Equation (30) and Equation (31) s ,β m ,γ s ,γ m is a predetermined parameter, and β s ,β m ,γ s ,γ m >0.

[0092] In equations (22) and (23), the squared value of the Frobenius norm in the first term in the parentheses indicates the difference between the observed signal sequence Y and the original signal sequence X, and the second term indicating the Laplacian quadratic form shown in equation (11) and the third term indicating the characteristics of the adjacency matrix indicating the structure of the graph indicate the smoothness of the original signal. Therefore, equation (22) expresses the spatial graph g sIn the original signal array X s The K original signal matrices X (k) Each of the column vectors in (k=1,…,K) is treated as a space graph g s As we smooth out the spatial graph g s Adjacency matrix W s While optimizing the observed signal array Y s The original signal sequence X that minimizes the difference with s This is the objective function to calculate:

[0093] Similarly, equation (23) expresses the modality graph g m In the original signal array X m The K original signal matrices X (k) Each of the column vectors of the transposed matrix of is expressed as a modality graph g m As shown above, the modality graph g m Adjacency matrix W m While optimizing the observed signal array Y m The original signal sequence X that minimizes the difference with m This is the objective function to calculate:

[0094] Since equations (22) and (23) are generally not convex optimization problems, the original signal array X s ,X m and the adjacency matrix W s ,W m A solution can be found by performing alternating minimization on the following equations (22) and (35), and equation (23) is decomposed into the following equations (36) and (37).

[0095]

number

[0096]

number

[0097]

number

[0098]

number

[0099] Equation (34) is the observed signal array Y s The original signal array X with noise removed from s Calculate the spatial graph g s Equation (35) can be said to be a noise removal formula in the spatial graph g s The optimal adjacency matrix W in s Equation (36) can be said to be an adjacency matrix calculation formula (hereinafter also referred to as the “first adjacency matrix calculation formula”) for calculating the observed signal array Y m The original signal array X with noise removed from m The modality graph g m Equation (37) can be said to be a noise removal formula for the modality graph g m The optimal adjacency matrix W in m (hereinafter also referred to as the "second adjacency matrix calculation formula").

[0100] In equations (35) and (37), the symbol shown in the following equation (38) is an operator for calculating the Hadamard product.

[0101]

number

[0102] In equations (35) and (37), the symbol shown in the following equation (39) is an operator that calculates the sum of the absolute values ​​of each element of the matrix.

[0103]

number

[0104] The matrix Z shown in equation (35) s and the matrix Z shown in equation (37) mare the original signal arrays X s ,X m is a pairwise distance matrix that can be calculated using

[0105] As an example, consider a spatial graph g s The pairwise distance matrix Z corresponding to s The calculation procedure of is explained below. In Fig. 2, the observation position indicated by the reference symbol 23-m is the row, and the observation count is the column, and the matrix X is an N-row x K-column matrix. If the matrix X is expressed as a vector for each column, it can be expressed as the following equation (40).

[0106]

number

[0107] In equation (40), the vector for each column is a column vector with N rows and 1 column, i.e., the signal vector x k (k=1,…,K) and is defined by the following equation (41).

[0108]

number

[0109] On the other hand, when the matrix X is expressed as a vector for each row, it can be expressed as the following equation (42).

[0110]

number

[0111] In the following text, the signal vector x with "-" added to it in equation (42) will be referred to as "signal vector -x". Since equation (42) is added with a symbol indicating a transposed matrix, the signal vector -x n Each of (n=1,...,N) is a column vector with K rows and 1 column, and is defined by the following equation (43).

[0112]

number

[0113] Signal vector x k is a vector containing N original signal values ​​corresponding to the N observed signal values ​​observed by a certain domain at the N observation positions obtained by the kth observation, and is expressed as a spatial graph g s In contrast, the signal vector -x n is a spatial graph g s At the vertex n on the left, it is a vector containing K observed signal values ​​observed by a certain domain. This signal vector -x n By applying the following equation (44), the spatial graph g s Pairwise distance matrix Z for s can be calculated.

[0114]

number

[0115] In addition, the pairwise distance matrix Z s As can be seen from equation (44), the elements of the matrix are positive real numbers, so that it is defined by the following equation (45).

[0116]

number

[0117] Original signal sequence X s There are M matrices X in the original signal array X s M pairwise distance matrices Z from s We obtain the pairwise distance matrix Z s In the spatial graph g s The graph Laplacian matrix L of s and the spatial graph g s Adjacency matrix W s Through this, the relationship shown in the following equation (46) is established.

[0118]

number

[0119] By using this formula (46), the first term in the brackets on the right-hand side of formula (35) is derived from formula (22). m The pairwise distance matrix Z corresponding to m Regarding the spatial graph g, in FIG. 2, the modality indicated by the symbol 22-n is the row, and the number of observations is the column. Let X be a matrix of M rows and K columns. s The pairwise distance matrix Z corresponding to s In this case, the original signal array X m In this case, there are N matrices X, and the original signal array X m to N pairwise distance matrices Z m We obtain the pairwise distance matrix Z m Modality Graph m The graph Laplacian matrix L of m and modality graph g m Adjacency matrix W m Since a relationship similar to that of equation (46) holds through equation (23), the first term in the parentheses on the right-hand side of equation (37) can be derived from equation (23).

[0120] The solution to equation (34) is derived as the following equation (47), and the solution to equation (36) is derived as the following equation (48).

[0121]

number

[0122]

number

[0123] In equation (47), the matrix I sis an N row × N column unit matrix. In equation (47), the original signal array X s and the observed signal array Y s As described above, the matrix Y s can also be said to be an array containing M×K observation signals of N rows and 1 column, which are the observation signals in the row direction of the observation signal array Y. s Since what is being acted on is an N row x N row matrix, equation (47) can also be viewed as an operation that calculates an N row x 1 column original signal from an N row x 1 column observed signal, performed M x K times. In this case, the N row x 1 column original signal calculated by equation (47) based on the N row x 1 column observed signal is also called the first noise removal signal. Equation (47) is also called the first noise removal equation.

[0124] In equation (48), the matrix I m is a unit matrix with M rows and M columns. In equation (48), the original signal array X m and the observed signal array Y m As described above, the observed signal array Y m can also be said to be an array containing N×K observation signals of M rows and 1 column obtained by transposing the observation signals in the column direction of the observation signal array Y. m Since what is being acted on is an M row x M row matrix, equation (48) can also be viewed as an operation of calculating one M row x 1 column original signal from one M row x 1 column observed signal, performed N x K times. In this case, one M row x 1 column original signal calculated by equation (48) from one M row x 1 column observed signal is hereinafter also referred to as the second noise removal signal, and equation (48) is hereinafter also referred to as the second noise removal equation.

[0125] The solutions to equations (35) and (37) can be calculated by any convex optimization method, such as the primal-dual proximity separation method or other methods.

[0126] Returning to FIG. 1, the calculation processing unit 3 includes a calculation control unit 10, a spatial calculation unit 11, and a modality calculation unit 12. The spatial calculation unit 11 takes two input sequences, a first input sequence and a second input sequence, as arguments, and calculates a first graph, i.e., a spatial graph g s The spatial calculation unit 11 performs a calculation using the first noise removal equation of equation (47) and the first adjacency matrix calculation equation of equation (35) corresponding to the above. By this calculation, the spatial calculation unit 11 obtains M×K first noise removed signals and a spatial graph g s Adjacency matrix W s From the calculated M×K first noise-removed signals, spatial operation unit 11 generates an output array that is an array of the same format as three-dimensional array 20, that is, an array formed by K matrices of N rows and M columns.

[0127] The modality calculation unit 12 uses two input sequences, a first input sequence and a second input sequence, as arguments, and calculates a second graph, i.e., a modality graph g m The modality calculation unit 12 performs a calculation using the second noise removal equation of equation (48) and the second adjacency matrix calculation equation of equation (37) corresponding to the above. The modality calculation unit 12 performs a calculation using N×K second noise removed signals and a modality graph g m Adjacency matrix W m The modality calculation unit 12 transposes each of the calculated N×K second noise-removed signals to generate an output array having the same format as the three-dimensional array 20, that is, an array formed by K matrices of N rows and M columns.

[0128] The calculation control unit 10 causes the modality calculation unit 12 and the spatial calculation unit 11 to perform processing alternately while inheriting each other's processing results, and estimates the original signal array X from the observed signal array Y. In the estimation process, the calculation control unit 10 selects an array to be applied to the first input array and the second input array of the spatial calculation unit 11 and the first input array and the second input array of the modality calculation unit 12 from among an initial array, which is a three-dimensional array 20 generated by the matrix generation unit 2, an output array generated by the spatial calculation unit 11, and an output array generated by the modality calculation unit 12, so as to obtain an appropriate original signal array X.

[0129] (Graph Learning Process According to the First Embodiment) 3 to 5, the process performed by the noise removal device 1 of the first embodiment will be described. In the noise removal device 1, the matrix generation unit 2 takes in N×M×K observed signal values ​​provided from the outside (Sa1). The matrix generation unit 2 generates an observed signal array Y, which is a three-dimensional array 20, based on three index values ​​assigned to the N×M×K observed signal values. The matrix generation unit 2 outputs the generated observed signal array Y to the calculation control unit 10 of the calculation processing unit 3. The calculation control unit 10 takes in the observed signal array Y output by the matrix generation unit 2, and records the taken in observed signal array Y as an initial array in an internal storage area (Sa2).

[0130] The calculation control unit 10 calculates the parameter α s ,α m ,β s ,β m ,γ s ,γ m The calculation control unit 10 takes in the taken-in parameter α s ,β s ,γ s is set in the spatial calculation unit 11, and the imported parameter α m ,β m ,γ m is set in the modality calculation unit 12. The calculation control unit 10 records the captured predetermined number of times in an internal storage area (Sa3). Here, the predetermined number of times is an integer equal to or greater than 1, and as the predetermined number of times, for example, a number indicating the number of repetitions to the extent that an original signal sequence X with a required accuracy can be obtained from an observed signal sequence Y is previously determined.

[0131] The arithmetic control unit 10 provides an area for a processing counter in an internal storage area, and initializes the provided processing counter to "0" (Sa4). The arithmetic control unit 10 sets initial sequences stored in an internal storage area for the first input sequence and the second input sequence of the modality calculation unit 12 (Sa5), and causes the modality calculation unit 12 to start processing. The modality calculation unit 12 executes a subroutine of the modality graph learning process shown in Fig. 4 (Sa6).

[0132] (Subroutine for modality graph learning process (initial process)) As shown in Fig. 4, the modality calculation unit 12 takes in the first input sequence and the second input sequence set by the calculation control unit 10. Hereinafter, in order to indicate the first input sequence of the modality calculation unit 12 as a sequence corresponding to the processing counter, the same symbol "X" as the original signal sequence X is used, and further, "t" is used to indicate the number of processing counters, and the first input sequence X will be referred to as the first input sequence X. t Here, the first input array X t is the initial array, and the initial array is the observed signal array Y. The processing counter count is "0". Therefore, the first input array X t The uptake can be expressed as the following equation (49).

[0133]

number

[0134] On the left side of equation (49), the superscript "0" of the array X indicates that the number "t" of the processing counter is "0". t Let us consider the original signal array X m Convert the first input array X into the format m t This can be expressed as the following equation (50), where t=0.

[0135]

number

[0136] In the following description of the process in FIG. 4, the number of the process counter is "t", but in the first case, t=0. The modality calculation unit 12 calculates the first input array X m t From modality graph g m The pairwise distance matrix Z corresponding to m More specifically, the modality calculation unit 12 calculates the first input array X m t , the procedure described with reference to equations (42) to (44) is applied to obtain the modality graph g m The pairwise distance matrix Z corresponding to m Calculate N of these (Sb2).

[0137] The modality calculation unit 12 calculates N pairwise distance matrices Z m The modality calculation unit 12 generates a second adjacency matrix calculation formula shown in Equation (37) using the second adjacency matrix calculation formula (Sb3). m Adjacency matrix W m More specifically, the modality calculation unit 12 calculates N pairwise distance matrices Z m The adjacency matrix W that minimizes the value obtained by the formula in parentheses in (37) is m The adjacency matrix W calculated in the process of Sb4 is m For example, when the number of processing counters is "t", the newly calculated next adjacency matrix W m Therefore, this newly calculated adjacency matrix W m is expressed by the symbol in the following equation (51) with the superscript “t+1” added.

[0138]

number

[0139] The modality calculation unit 12 calculates the adjacency matrix W by using the procedure described with reference to equations (7) to (9). m t+1 From modality graph g m The graph Laplacian matrix L corresponding to m t+1 The modality calculation unit 12 calculates the calculated graph Laplacian matrix L m t+1 Using this, a second noise removal equation shown in equation (48) is generated (Sb5).

[0140] The modality calculation unit 12 converts the second input sequence acquired in the process of Sb1 into the original signal sequence X m The modality calculation unit 12 converts the second input sequence into the format of the original signal sequence X m The array converted to the format is used as the observed signal array Y m As mentioned above, equation (48) is expressed as m The modality calculation unit 12 performs N×K times a calculation to calculate a second noise-removed signal of M rows×1 column corresponding to a single observed signal of M rows×1 column included in the original signal array X m The array converted into the above format is substituted into the generated second noise removal equation shown in equation (48) to calculate N×K new second noise removal signals (Sb6).

[0141] The modality calculation unit 12 calculates the N×K new second noise-removed signals as the original signal array X m The second noise-removed signal sequence is generated by arranging the first and second noise-removed signal sequences in the same format as the first noise-removed signal sequence. The second noise-removed signal sequence is generated by arranging the first and second noise-removed signal sequences in the same format as the first noise-removed signal sequence, which is a candidate for the original signal sequence X and is newly calculated when the processing counter is "t". m Therefore, the second noise elimination signal array X m t+1 It is expressed as:

[0142] The second input sequence of the modality calculation unit 12 is an initial sequence. In the notation used for the first input sequence of the modality calculation unit 12, the first input sequence X 0 Therefore, in the process of Sb6, the modality calculation unit 12 uses the first input array X m 0 Therefore, if the processing of Sb5 and Sb6 is expressed as a generalized formula with the number of processing counters as "t", it can be expressed as the following formula (52).

[0143]

number

[0144] The modality calculation unit 12 generates the second noise-removed signal array X m t+1 Convert the original signal array X to the output array X t+1 The modality calculation unit 12 generates the generated output array X t+1 and the adjacency matrix W calculated in the processing of Sb4 m t+1 to the calculation control unit 10 (Sb7), and the processing of the subroutine is terminated.

[0145] Returning to FIG. 3, the calculation control unit 10 calculates the output array X t+1 and the adjacency matrix W m t+1 The calculation control unit 10 stores the acquired adjacency matrix W m t+1 The calculation control unit 10 records the previous adjacency matrix W m t If is stored, the previous adjacency matrix W m t After eliminating m t+1 The calculation control unit 10 records the output array X t+1, that is, the output array X t+1 (Sa7), and starts the process in the spatial calculation unit 11. The spatial calculation unit 11 executes the subroutine of the spatial graph learning process shown in FIG.

[0146] (Subroutine for spatial graph learning processing) 5, the spatial calculation unit 11 takes in the first input array and the second input array set by the calculation control unit 10. In order to indicate the first input array of the spatial calculation unit 11 as an array corresponding to the processing counter, the same symbol "X" as the original signal array X is used, and further, "t" is used to indicate the number of the processing counter. Hereinafter, the first input array X t The spatial calculation unit 11 expresses the first input array X t Let us consider the original signal array X s and store the converted array as the first input array X s t Here, the first input array X t is the output array X most recently output by the modality calculation unit 12. t+1 Therefore, in consideration of the connection with the most recently performed processing by the modality calculation unit 12, the first input array X t to the first input array X s t The process is as follows: t+1 Let us consider the original signal array X s and store the converted array as the first input array X s t This can be expressed as the following equation (53).

[0147]

number

[0148] The spatial operation unit 11 receives a first input array X s t From the spatial graph g s The pairwise distance matrix Z corresponding to sMore specifically, the original signal array X generated by converting the first input array is calculated. s t Applying the procedure described with reference to equations (42) to (44), we obtain the spatial graph g s The pairwise distance matrix Z corresponding to s Calculate M of these (Sc2).

[0149] The spatial operation unit 11 calculates M pairwise distance matrices Z s The spatial calculation unit 11 calculates the spatial graph g s Adjacency matrix W s More specifically, the spatial calculation unit 11 calculates M pairwise distance matrices Z s The adjacency matrix W that minimizes the value obtained by the formula in parentheses in (35) is s is calculated, for example, by the primal-dual adjacency separation method (Sc4). The adjacency matrix W s For example, when the number of processing counters is "t", the newly calculated next adjacency matrix W s Therefore, this newly calculated adjacency matrix W s is expressed by the symbol in the following equation (54) with the superscript “t+1” added.

[0150]

number

[0151] The spatial calculation unit 11 calculates the adjacency matrix W using the procedure described with reference to equations (7) to (9). s t+1 From the spatial graph g s The graph Laplacian matrix L corresponding to s t+1 The spatial calculation unit 11 calculates the calculated graph Laplacian matrix L s t+1 Using the above, a first noise removal equation shown in equation (47) is generated (Sc5).

[0152] The spatial operation unit 11 converts the second input sequence acquired in the processing of Sc1 into the original signal sequence X s The spatial operation unit 11 converts the second input array into the format of the original signal array X s The array converted into the format is used as the observed signal array Y s As mentioned above, equation (47) is expressed as s The spatial operation unit 11 performs M×K times a calculation to calculate a first noise-removed signal of N rows×1 column corresponding to a single observed signal of N rows×1 column included in the original signal array X s The array converted into the above format is substituted into the generated first noise removal equation shown in equation (47) to calculate M×K new first noise removed signals (Sc6).

[0153] The spatial calculation unit 11 calculates the M×K new first noise-removed signals and stores them in the original signal array X s The first noise-removed signal sequence is generated by arranging the first and second noise-removed signal sequences in the same format as the first noise-removed signal sequence. The first noise-removed signal sequence is generated by arranging the first and second noise-removed signal sequences in the same format as the first noise-removed signal sequence, which is a candidate for the original signal sequence X and is newly calculated when the processing counter is "t". s Therefore, the first noise elimination signal array X s t+1 It is expressed as:

[0154] The second input array of the spatial calculation unit 11 is the output array X t+1 In the notation used in the case of the first input array of the spatial calculation unit 11, the first input array X t Therefore, in the process of Sc6, the spatial calculation unit 11 calculates the first input array X s t Therefore, when the processing of Sc5 and Sc6 is expressed by a generalized formula with the number of processing counters being "t", it can be expressed as the following formula (55).

[0155]

number

[0156] The spatial calculation unit 11 generates the first noise-removed signal array X s t+1 Convert the original signal array X to the output array X t+1 The spatial operation unit 11 generates the output array X t+1 and the adjacency matrix W calculated in the process of Sc4 s t+1 and output it to the calculation control unit 10 (Sc7), and the processing of the subroutine is completed.

[0157] Returning to FIG. 3, the calculation control unit 10 calculates the output array X t+1 and the adjacency matrix W s t+1 The calculation control unit 10 stores the acquired adjacency matrix W s t+1 The calculation control unit 10 records the previous adjacency matrix W s t If is stored, the previous adjacency matrix W s t After eliminating s t+1 The arithmetic control unit 10 reads out the predetermined number of times from the internal storage area. The arithmetic control unit 10 judges whether the number "t" of the processing counter is equal to or greater than (the predetermined number of times-1) (Sa9).

[0158] When the calculation control unit 10 judges that the number of the processing counters "t" is not equal to or greater than (the predetermined number of times - 1) (Sa9, No), it adds 1 to the number of the processing counters in the internal storage area (Sa10). t+1 , that is, the output array X t+1The calculation control unit 10 further sets the initial sequence stored in the internal storage area to the second input sequence of the modality calculation unit 12 (Sa11), and causes the modality calculation unit 12 to start processing. The modality calculation unit 12 executes a subroutine of the modality graph learning process shown in FIG. 4 (Sa12).

[0159] (Subroutine for modality graph learning process (processing from second time onwards)) The difference between the process performed in the subroutine of the modality graph learning process performed at the timing Sa12 and the process performed in the subroutine of the modality graph learning process performed at the timing Sa6 will be described.

[0160] The modality calculation unit 12 receives the first input sequence and the second input sequence set by the calculation control unit 10. Here, the first input sequence of the modality calculation unit 12 is the output sequence X t+1 However, the number of the processing counter is incremented by 1 by the processing of Sa10. Therefore, when the number of the processing counter is adjusted to the incremented number, the superscript changes from "t+1" to "t", so the output array X t+1 is the first input array X t The modality calculation unit 12 calculates the first input array X t Let us consider the original signal array X m Convert the first input array X m t (Sb1). Note that the second input sequence of the modality calculation unit 12 from the second time onwards is the initial sequence, just like the first time. Therefore, the processes from Sb2 to Sb7 thereafter are the same as the processes from Sb2 to Sb7 of the subroutine of the modality graph learning process performed at the timing of Sa6, and the process counter "t" is replaced with the number of the processing counter after the increment.

[0161] 3, in the process of Sa9, it is assumed that the arithmetic control unit 10 judges that the number "t" of the processing counter is equal to or greater than (predetermined number of times - 1) (Sa9, Yes). In this case, the arithmetic control unit 10 t+1 , that is, the output array X t+1 Let X be the original signal sequence, and let W be the adjacency matrix of the spatial graph stored in the internal memory. s t+1 The final adjacency matrix W s Let the adjacency matrix W m t+1 The final adjacency matrix W m Let X be the original signal array and W be the adjacent matrix. s and the adjacency matrix W m Then, the process is terminated by outputting (Sa13).

[0162] FIG. 6 is a diagram showing the processing of FIG. 3 in the form of a program. The "iter" shown in "input" on the first line is a predetermined number of times. Lines 3 and 9 are not lines showing processing, but lines showing comments. The processing on the first line is the processing of Sa6 in FIG. 3, that is, the processing of the first input array X 0 is used as the initial sequence, i.e., the observed signal sequence Y. The processing from the 4th line to the 8th line corresponds to the processing from Sb1 to Sb7 in the subroutine of the modality graph learning processing performed in the processing of Sa6 and Sa12 in FIG. 3, excluding the processing on the 1st line. The processing on the first input sequence of the modality calculation unit 12 included in the processing of Sb1 is shown on the 4th line. The processing of Sb2 to Sb4 is shown on the 5th and 6th lines. The processing of Sb5 and Sb6 is shown on the 7th line, and corresponds to the processing of the second input sequence of the modality calculation unit 12 as the original signal sequence X m The array converted to the format is the array X in the 7th row. m 0 The processing of Sb7 is shown in the 8th line.

[0163] The processing from line 10 to line 14 corresponds to the processing from Sc1 to Sb7 of the spatial graph learning processing subroutine performed in the processing from Sa8 in FIG. 3. The processing from Sc1 is shown on line 10. The processing from Sb2 to Sc4 is shown on lines 11 and 12. The processing from Sc5 and Sc6 is shown on line 13, in which the second input array of the spatial calculation unit 11 is converted into the original signal array X s The array converted to the format is the array X in the 13th line. s t The process of Sc7 is shown in the 14th line. In the process of Sa13 in FIG. 3, the adjacency matrix W s and the adjacency matrix W m and the original signal array X are the spatial graph adjacency matrices W s iter and the adjacency matrix W of the modality graph m iter and the original signal sequence X iter It becomes.

[0164] Second embodiment 7 is a block diagram showing the configuration of a noise removal device 1a according to the second embodiment. In the noise removal device 1a of the second embodiment, the same components as those in the noise removal device 1 of the first embodiment are given the same reference numerals, and different components will be described below. The noise removal device 1a includes a matrix generation unit 2 and a calculation processing unit 3a. The calculation processing unit 3a includes a calculation control unit 10a, a spatial calculation unit 11, and a modality calculation unit 12.

[0165] The calculation control unit 10a makes the modality calculation unit 12 pass on the processing results obtained by performing the iterative process to the spatial calculation unit 11, and makes the spatial calculation unit 11 perform the iterative process to estimate the original signal array X from the observed signal array Y. In the estimation process, the calculation control unit 10a selects an array to be applied to the first input array and the second input array of the spatial calculation unit 11 and the first input array and the second input array of the modality calculation unit 12 from among an initial array, which is the three-dimensional array 20 generated by the matrix generation unit 2, an output array generated by the spatial calculation unit 11, and an output array generated by the modality calculation unit 12, so as to obtain an appropriate original signal array X.

[0166] (Graph Learning Process According to the Second Embodiment) The process performed by the noise removal device 1a of the second embodiment will be described with reference to FIG. The processes of Sd1 and Sd2 are the same as the processes of Sa1 and Sa2 shown in FIG. 3, except that the calculation control unit 10 is replaced with the calculation control unit 10a. The calculation control unit 10a is a s ,α m ,β s ,β m ,γ s ,γ m The calculation control unit 10a takes in the first predetermined number of times and the second predetermined number of times. s ,β s ,γ s is set in the spatial calculation unit 11, and the imported parameter α m ,β m ,γ m is set in the modality calculation unit 12. The calculation control unit 10a records the first and second predetermined times in an internal storage area (Sd3). Here, each of the first and second predetermined times is an integer equal to or greater than 1, and the first and second predetermined times are, for example, predetermined numbers indicating the number of repetitions that can obtain an original signal sequence X with a required accuracy from an observed signal sequence Y.

[0167] The processes of Sd4 and Sd5 are the same as the processes of Sa4 and Sa5 shown in Fig. 3, except that the calculation control unit 10 is replaced with a calculation control unit 10a. The process of the subroutine of the modality graph learning process of Sd6 for the first time is the same as the process of Sa6 shown in Fig. 3, that is, the process of "subroutine of the modality graph learning process (first process)" in the first embodiment described above.

[0168] After the process of Sd6, the calculation control unit 10a calculates the output array X t+1 and the adjacency matrix W m t+1 The calculation control unit 10a stores the acquired adjacency matrix W m t+1 The calculation control unit 10a records the previous adjacency matrix W m t If is stored, the previous adjacency matrix W m t After eliminating m t+1 The calculation control unit 10a reads out the second predetermined number of times from the internal storage area. The calculation control unit 10a judges whether the number "t" of the processing counter is equal to or greater than (the second predetermined number of times-1) (Sd7).

[0169] When the calculation control unit 10a judges that the number of the processing counters "t" is not equal to or greater than the second predetermined number of times - 1 (Sd7, No), it adds 1 to the number of the processing counters in the internal storage area (Sd8). t+1 , that is, the output array X t+1 The calculation control unit 10a further sets the initial sequence stored in the internal storage area to the second input sequence of the modality calculation unit 12 (Sd9), and causes the modality calculation unit 12 to start processing. The modality calculation unit 12 executes the subroutine of the modality graph learning process shown in FIG. 4 (Sd6).

[0170] (Subroutine for modality graph learning process (processing from second time onwards)) The difference between the subroutine processing of the modality graph learning process performed at timing Sd6 from the second time onwards and the subroutine processing of the modality graph learning process performed at timing Sd6 for the first time will be described.

[0171] The modality calculation unit 12 receives the first input sequence and the second input sequence set by the calculation control unit 10a. Here, the first input sequence is the output sequence X t+1 However, the number of the processing counter is incremented by 1 by the processing of Sd8. Therefore, when the number of the processing counter is adjusted to the incremented number, the superscript changes from "t+1" to "t", so the output array X t+1 is the first input array X t The modality calculation unit 12 calculates the first input array X t Let us consider the original signal array X m Convert the first input array X m t (Sb1). Note that the second input sequence of the modality calculation unit 12 from the second time onwards is the initial sequence, just like the first time. Therefore, the processes of Sb2 to Sb7 thereafter are the same as the processes of Sb2 to Sb7 of the subroutine of the modality graph learning process performed at the timing of Sd6 for the first time.

[0172] Returning to FIG. 8, in the process of Sd7, the calculation control unit 10a judges that the number "t" of the processing counter is equal to or greater than (the second predetermined number of times - 1) (Sd7, Yes). In this case, the calculation control unit 10a t+1 , that is, the output array X t+1 In this case, if the second predetermined number of times is represented by "Tm", the output array X t+1 is the output array X TmSimilarly, after the most recent processing of Sd6, the adjacency matrix W recorded in the internal storage area by the calculation control unit 10a is expressed as follows: m t+1 is the adjacency matrix W m Tm It will be expressed as:

[0173] The calculation control unit 10a adds 1 to the number of the processing counter in the internal storage area (Sd10). The calculation control unit 10a calculates the output array X Tm (Sd11) and starts the process in the spatial calculation unit 11. The spatial calculation unit 11 executes the subroutine of the spatial graph learning process shown in Fig. 5 (Sd12).

[0174] (Subroutine for spatial graph learning process (initial process)) 5, the spatial calculation unit 11 takes in the first input array and the second input array set by the calculation control unit 10a. In order to indicate the first input array of the spatial calculation unit 11 as an array corresponding to the processing counter, the same symbol "X" as the original signal array X is used, and further, "t" is used to indicate the number of the processing counter. Hereinafter, the first input array X t Here, the first input array of the spatial calculation unit 11 is the output array X Tm However, the processing counter count has been incremented by 1 by the processing of Sd10. Therefore, the processing counter count after the increment becomes "Tm". Therefore, the output array X Tm is the first input array X Tm The spatial operation unit 11 calculates the first input array X Tm Let us consider the original signal array X s Convert the first input array X s Tm This process can be generalized using the number of processing counters "t" as shown in the following formula (56).

[0175]

number

[0176] The subsequent processing of Sc2 to Sc7 is the same as that of Sc2 to Sc7 in the subroutine of the spatial graph learning processing of the first embodiment. In this case, the second input sequence that the spatial calculation unit 11 takes in the processing of Sc1 and uses in the processing of Sc6 is the output sequence X Tm If this second input array is expressed in the notation used for the first input array, the first input array X Tm Therefore, in the process of Sc6, the spatial calculation unit 11 calculates the first input array X s Tm Therefore, when the processing of Sc5 and Sc6 is expressed by a generalized formula with the number of processing counters being "t", it can be expressed as the following formula (57).

[0177]

number

[0178] Returning to FIG. 8, after the process of Sd12, the calculation control unit 10a calculates the output array X t+1 and the adjacency matrix W s t+1 The calculation control unit 10a stores the acquired adjacency matrix W s t+1 The calculation control unit 10a records the previous adjacency matrix W s t If is stored, the previous adjacency matrix W s t After eliminating s t+1 The calculation control unit 10a reads out the first predetermined number of times and the second predetermined number of times from the internal storage area. The calculation control unit 10a judges whether the number "t" of the processing counter is equal to or greater than (the first predetermined number of times+the second predetermined number of times-1) (Sd13).

[0179] When the calculation control unit 10a judges that the number of the processing counters "t" is not equal to or greater than (the first predetermined number of times + the second predetermined number of times - 1) (Sd13, No), it adds 1 to the number of the processing counters in the internal storage area (Sd14). t+1 , that is, the output array X t+1 The calculation control unit 10a further sets the output array X Tm as the second input array of spatial calculation unit 11 (Sd15), and causes spatial calculation unit 11 to start processing. Spatial calculation unit 11 executes the subroutine of the spatial graph learning process shown in Fig. 5 (Sd12).

[0180] (Subroutine for spatial graph learning process (processing from the second time onwards)) The difference between the processing of the spatial graph learning process subroutine performed at Sd12 from the second time onwards and the processing of the spatial graph learning process subroutine performed at Sd12 for the first time will be described.

[0181] The spatial calculation unit 11 takes in the first input array and the second input array set by the calculation control unit 10a. Here, the first input array is the output array X t+1 However, the number of the processing counter has been incremented by 1 by the processing of Sd14. Therefore, when the number of the processing counter is adjusted to the incremented number, the superscript changes from "t+1" to "t", so the most recent output array X t is the first input array X t The spatial calculation unit 11 calculates the first input array X t Let us consider the original signal array X s t Convert the first input array X s t In addition, the second input array of the spatial calculation unit 11 from the second time onwards is the output array X TmTherefore, the subsequent processing of Sc2 to Sc7 is the same as the processing of Sc2 to Sc7 of the subroutine of the spatial graph learning processing performed at the timing of Sd12 for the first time.

[0182] Returning to FIG. 8, in the process of Sd13, it is assumed that the calculation control unit 10a judges that the number "t" of the processing counter is equal to or greater than (the first predetermined number of times+the second predetermined number of times-1) (Sd13, Yes). In this case, the calculation control unit 10a calculates the number of the input output array X t+1 , that is, the output array X t+1 Let X be the original signal sequence, and let W be the adjacency matrix of the spatial graph stored in the internal memory. s t+1 The final adjacency matrix W s Let the adjacency matrix W m Tm The final adjacency matrix W m Let X be the original signal array and W be the adjacent matrix. s and the adjacency matrix W m Then, the process ends (Sd16).

[0183] FIG. 9 is a diagram showing the processing of FIG. 8 in the form of a program. "Ts" in "Input" on the first line is the first predetermined number of times, and "Tm" is the second predetermined number of times. Lines 2 and 9 are not lines showing processing, but lines showing comments. The processing on the first line is processing Sb1 of the subroutine of the modality graph learning processing performed in the processing of Sd6 in FIG. 8, and is the first input array X 0 is used as the initial sequence, i.e., the observed signal sequence Y. The processing from the 4th line to the 8th line corresponds to the processing from Sb1 to Sb7 of the subroutine of the modality graph learning processing performed in the processing of Sd6 in FIG. 8, excluding the processing on the 1st line. The processing on the first input sequence of the modality calculation unit 12 included in the processing of Sb1 is shown on the 4th line. The processing of Sb2 to Sb4 is shown on the 5th and 6th lines. The processing of Sb5 and Sb6 is shown on the 7th line, and corresponds to the processing of the second input sequence of the modality calculation unit 12 as the original signal sequence X mThe array converted to the format is the array X in the 7th row. m 0 The processing of Sb7 is shown in the 8th line.

[0184] The processing from line 11 to line 15 corresponds to the processing from Sc1 to Sc7 in the spatial graph learning subroutine performed in the processing of Sd12 in FIG. 8. The processing of Sc1 is shown on line 11. The processing of Sc2 to Sc4 is shown on lines 12 and 13. The processing of Sc5 and Sc6 is shown on line 14, in which the second input array of the spatial calculation unit 11 is converted into the original signal array X s The array converted to the format is the array X in the 14th line. s Tm The process of Sc7 is shown in the 15th line. In the process of Sd16 in FIG. 8, the adjacency matrix W s and the adjacency matrix W m and the original signal array X are the adjacency matrix W of the spatial graph obtained after the first predetermined number of times and the second predetermined number of times of processing shown in the “output” in the last row are performed. s Tm+Ts and the adjacency matrix W of the modality graph obtained after the second predetermined number of processes is performed. m Tm and the original signal array X.

[0185] (Actions and Effects of the First and Second Embodiments) In the noise removal devices 1 and 1a of the first and second embodiments described above, the matrix generation unit 2 arranges N×M×K observed signal values ​​obtained by K observations in each of M domains at each of N observation positions in space, with the observation positions as rows and the domains, i.e., modalities, as columns, to generate K observed signal matrices Y (k) In order to solve the noise removal problem for a multimodal signal, the first objective function shown in equation (22) and the second objective function shown in equation (23) are determined in advance under the following two assumptions. The first assumption is that an observed signal matrix Y (k)The second assumption is that when an original signal obtained by removing noise from row-wise observed signals formed by the observed signal values ​​of each column of is represented by a spatial graph, which is a first graph having N vertices, the original signal becomes smooth on the spatial graph. (k) The assumption is that when an original signal, obtained by removing noise from a column-wise observed signal formed by the observed signal values ​​for each row of the matrix, is represented by a modality graph, which is a second graph having M vertices, the original signal will be smooth on the modality graph.

[0186] The calculation processing unit 3 applies K observation signal matrices Y (k) Applying s ,W m Optimize and optimize the adjacency matrix W s ,W m Based on this, the noise removal device 1 performs a calculation to remove noise from the observed signal. More specifically, in the noise removal device 1 of the first embodiment, the calculation control unit 10 causes the modality calculation unit 12 and the spatial calculation unit 11 to perform processing alternately a predetermined number of times while passing on the processing results to each other, thereby estimating the original signal array X from the observed signal array Y. In contrast, the calculation control unit 10a of the second embodiment causes the spatial calculation unit 11 to take over the processing results obtained by causing the modality calculation unit 12 to perform repeated processing a second predetermined number of times, and causes the spatial calculation unit 11 to perform repeated processing a first predetermined number of times, thereby estimating the original signal array X from the observed signal array Y.

[0187] In both the first and second embodiments, the noise reduction device 1, 1a estimates an original signal from which noise has been removed from each of the observed signals. Here, the observed signals include observed signals in the row direction of the observed signal matrix and observed signals in the column direction, and the noise reduction device 1, 1a removes noise from the observed signals in both directions, which means that it removes noise from a multimodal signal. The adjacency matrix W s indicates the graph structure of the spatial graph, and the adjacency matrix W mindicates the graph structure of the modality graph, the noise removal device 1, 1a optimizes the graph structures of both the spatial graph and the modality graph. Therefore, the noise removal device 1, 1a of the first and second embodiments can remove noise from a multimodal signal by optimizing the graph structure from the viewpoint of noise removal and solving the noise removal problem.

[0188] (Another configuration example of the first and second embodiments) In the above first and second embodiments, the observation positions are rows and the modalities are columns. Alternatively, the modalities may be rows and the observation positions are columns. In this case, the first graph is a modality graph and the second graph is a space graph. The first objective function is equation (23) and the second objective function is equation (22). The first adjacency matrix calculation formula is equation (37) and the second adjacency matrix calculation formula is equation (35). The first noise removal formula is equation (48) and the second noise removal formula is equation (47). In the first and second embodiments, the array shown on the right side of equation (25) is referred to as the observed signal array Y s Let the array shown on the right side of equation (27) be the observed signal array Y m However, if the modality is the row and the observation position is the column, the observed signal array Y s is the array shown on the right side of equation (27), and the observed signal array Y m The original signal array X s ,X m For the observation signal array Y s ,Y m Similarly, the structures of the arrays are interchanged. When a calculation unit corresponding to the first adjacency matrix calculation formula and the first noise elimination formula is defined as the first calculation unit, and a calculation unit corresponding to the second adjacency matrix calculation formula and the second noise elimination formula is defined as the second calculation unit, in the first and second embodiments, the first calculation unit corresponds to the spatial calculation unit 11, and the second calculation unit corresponds to the modality calculation unit 12. In contrast, when modalities are defined as rows and observation positions are defined as columns, the first calculation unit corresponds to the modality calculation unit 12, and the second calculation unit corresponds to the spatial calculation unit 11.

[0189] In the first embodiment, as shown in Fig. 3, the calculation control unit 10 performs the processing of the modality calculation unit 12 first and the processing of the spatial calculation unit 11 later, so that the modality calculation unit 12 and the spatial calculation unit 11 perform the processing alternately. In contrast, the calculation control unit 10 may perform the processing of the spatial calculation unit 11 first and the processing of the modality calculation unit 12 later, so that the spatial calculation unit 11 and the modality calculation unit 12 perform the processing alternately. In this case, when the calculation control unit 10 causes the spatial calculation unit 11 to perform the first processing, it sets initial sequences for the first input sequence and the second input sequence of the spatial calculation unit 11, and when the calculation control unit 10 causes the spatial calculation unit 11 to perform the second or subsequent processing, it sets the output sequence outputted most recently by the modality calculation unit 12 to the first input sequence of the spatial calculation unit 11, and sets the initial sequence to the second input sequence of the spatial calculation unit 11. The first input array and the second input array of the modality calculation unit 12 are set to the output array most recently output by the spatial calculation unit 11 .

[0190] In the second embodiment, as shown in Fig. 8, the calculation control unit 10a causes the modality calculation unit 12 to perform processing a second predetermined number of times, and then causes the spatial calculation unit 11 to perform processing a first predetermined number of times. In contrast, the calculation control unit 10a may cause the modality calculation unit 12 to perform processing a second predetermined number of times, and then causes the spatial calculation unit 12 to perform processing a first predetermined number of times. In this case, when the calculation control unit 10a causes the spatial calculation unit 11 to perform processing for the first time, it sets initial arrays for the first input array and the second input array of the spatial calculation unit 11, and when the calculation control unit 10a causes the spatial calculation unit 11 to perform processing for the second or subsequent times, it sets the output array most recently output by the spatial calculation unit 11 to the first input array of the spatial calculation unit 11, and sets the initial array to the second input array of the spatial calculation unit 11. When modality calculation unit 12 is made to perform the first processing, the first input array and the second input array are set to the output array most recently output by spatial calculation unit 11, and when modality calculation unit 12 is made to perform the second or subsequent processing, the first input array of modality calculation unit 12 is set to the output array most recently output by modality calculation unit 12, and the second input array of modality calculation unit 12 is set to the output array most recently output by spatial calculation unit 11.

[0191] For example, the calculation control unit 10a may perform processing according to the method of the second embodiment, and then use the original signal array X obtained when the processing of the spatial calculation unit 11 is completed as an initial array, and perform processing according to the method of the second embodiment again.

[0192] The method of the first embodiment and the method of the second embodiment may be mixed. For example, the arithmetic control unit 10 may also have the configuration of the arithmetic control unit 10a, and the arithmetic control unit 10 may perform processing by the method of the second embodiment using an original signal array X obtained by performing processing by the method of the first embodiment as an initial array of the second embodiment. The arithmetic control unit 10a may also have the configuration of the arithmetic control unit 10, and the arithmetic control unit 10a may perform processing by the method of the first embodiment using an original signal array X obtained by performing processing by the method of the second embodiment as an initial array of the first embodiment.

[0193] The noise removal device 1, 1a in the above-mentioned embodiment may be realized by a computer. In that case, a program for realizing the function may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to realize the function. The term "computer system" here includes hardware such as an OS and peripheral devices. The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into a computer system. The term "computer-readable recording medium" may also include a medium that dynamically holds a program for a short period of time, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, and a medium that holds a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in that case. The above-mentioned program may be a program for realizing a part of the above-mentioned function, or may be a program that can realize the above-mentioned function in combination with a program already recorded in the computer system, or may be a program that is realized using a programmable logic device such as an FPGA (Field Programmable Gate Array).

[0194] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and designs that do not deviate from the gist of the present invention are also included. [Explanation of symbols]

[0195] 1... noise removal device, 2... matrix generation unit, 3... calculation processing unit, 10... calculation control unit, 11... spatial calculation unit, 12... modality calculation unit

Claims

1. a matrix generation unit that generates K observation signal matrices by arranging (the number of observation positions x the number of domains x K) observation signal values ​​obtained by K observations (where K is an integer equal to or greater than 2) at each of a plurality of observation positions in space, with either the observation positions or the domains as rows and the other as columns; a calculation processing unit that applies the K observation signal matrices generated by the matrix generation unit to an arithmetic formula derived from a first objective function determined based on an assumption that when an original signal obtained by removing noise from an observation signal formed by the observation signal values ​​for each column of the observation signal matrix is ​​represented by a first graph having vertices equal to the number of observation signal values ​​included in the columns of the observation signal matrix, the original signal will be smooth on the first graph, and a second objective function determined based on an assumption that when an original signal obtained by removing noise from an observation signal formed by the observation signal values ​​for each row of the observation signal matrix is ​​represented by a second graph having vertices equal to the number of observation signal values ​​included in the rows of the observation signal matrix, the original signal will be smooth on the second graph, the calculation processing unit optimizing an adjacency matrix of each of the first graph and the second graph, and removing noise from the observation signal based on the optimized adjacency matrices, to estimate an original signal obtained by removing noise from each of the observation signals; A noise removal device comprising:

2. The above-mentioned calculation formula is: a first noise removal equation derived from the first objective function, the first noise removal equation being determined by an adjacency matrix of the first graph and configured to calculate a first noise-removed signal obtained by removing noise from the observed signals for each column of the observed signal matrix; a second noise removal equation derived from the second objective function, the second noise removal equation being determined by an adjacency matrix of the second graph and configured to calculate a second noise-removed signal obtained by removing noise from the observation signals for each row of the observation signal matrix; a first adjacency matrix calculation formula derived from the first objective function, the first adjacency matrix calculation formula being determined by the first noise-removed signal and used to calculate an adjacency matrix of the first graph; a second adjacency matrix calculation formula derived from the second objective function, the second adjacency matrix calculation formula being determined by the second noise-removed signal and used to calculate an adjacency matrix of the second graph; That is, The noise removal device according to claim 1 .

3. The arithmetic processing unit is a first calculation unit that generates the first adjacency matrix calculation formula from a first input array, calculates an adjacency matrix of the first graph using the generated first adjacency matrix calculation formula, generates the first noise removal formula based on the calculated adjacency matrix of the first graph, divides the second input array into K matrices having the same format as the observed signal matrix, calculates a new first noise removal signal by substituting each of the first noise removal signals, which are column vectors of each of the divided matrices, into the generated first noise removal formula, and generates an output array that includes the calculated new first noise removal signals as column vectors and has the same format as the array formed by the K observed signal matrices; a second calculation unit that generates the second adjacency matrix calculation formula from a first input array, calculates an adjacency matrix of the second graph using the generated second adjacency matrix calculation formula, divides the second input array into K matrices having the same format as the observed signal matrix, calculates a new second noise-removal signal by substituting each of the second noise-removal signals, which are column vectors of transposed matrices of each of the divided matrices, into the generated second noise-removal formula, and generates an output array having the same format as the array formed by the K observed signal matrices, the output array including, as a row vector, a vector obtained by transposing the calculated new second noise-removal signal; an arithmetic control unit that selects an array to be applied to the first input array and the second input array of the first arithmetic unit and the first input array and the second input array of the second arithmetic unit from among an initial array that is an array formed by the K observation signal matrices, an output array generated by the first arithmetic unit, and an output array generated by the second arithmetic unit, and causes the first arithmetic unit and the second arithmetic unit to perform processing to acquire the original signal; Further comprising: The noise removal device according to claim 2 .

4. The arithmetic and control unit is the first and second input sequences of the second arithmetic unit are set as the initial sequences, and then the second arithmetic unit is caused to perform an initial process, in which, as the process of the first arithmetic unit, an output sequence generated by the second arithmetic unit that has most recently been processed is set as the first input sequence and the second input sequence of the first arithmetic unit; and, in the process of the second arithmetic unit from the second time onwards, the first input sequence of the second arithmetic unit is set as the output sequence generated by the first arithmetic unit that has most recently been processed, and the second input sequence of the second arithmetic unit is set as the initial sequence, by alternately causing the first arithmetic unit and the second arithmetic unit to perform the above process, and an output sequence generated when the first arithmetic unit has performed the process a predetermined number of times is set as an array including the original signal. The noise removal device according to claim 3.

5. The arithmetic and control unit is having the second arithmetic unit perform an initial process using the first input sequence and the second input sequence of the second arithmetic unit as the initial sequence, and then having the second arithmetic unit perform a process using a most recently generated output sequence of the second arithmetic unit as the first input sequence of the second arithmetic unit and the second input sequence of the second arithmetic unit as the initial sequence a second predetermined number of times; the second calculation unit performs the process the second predetermined number of times, and causes the first calculation unit to perform an initial process with the last generated output sequence as the first input sequence and the second input sequence of the first calculation unit, and then causes the first calculation unit to repeatedly perform the process a first predetermined number of times with the output sequence most recently generated by the first calculation unit as the first input sequence of the first calculation unit and the second input sequence of the first calculation unit as the initial second input sequence of the first calculation unit; the first calculation unit performs the process the first predetermined number of times, and a final output array is generated as an array including the original signal; The noise removal device according to claim 3.

6. The first calculation unit is Calculating a pairwise distance matrix corresponding to the first graph from the first input array, and generating the first adjacency matrix calculation formula from the calculated pairwise distance matrix; Calculating an adjacency matrix of the first graph by applying a primal-dual adjacency separation method to the first adjacency matrix calculation formula; The second calculation unit is Calculating a pairwise distance matrix corresponding to the second graph from the first input array, and generating the second adjacency matrix calculation formula from the calculated pairwise distance matrix; Calculating an adjacency matrix of the second graph by applying a primal-dual adjacency separation method to the second adjacency matrix calculation formula; The noise removal device according to any one of claims 3 to 5.

7. a matrix generation step of arranging the number of observation positions x the number of domains x K observation signal values ​​obtained by K observations (where K is an integer equal to or greater than 2) at each of a plurality of observation positions in space, with either the observation positions or the domains as rows and the other as columns, to generate K observation signal matrices; a first objective function determined based on the assumption that when an original signal obtained by removing noise from an observation signal formed by the observation signal values ​​for each column of the observation signal matrix is ​​represented by a first graph having vertices equal to the number of observation signal values ​​included in the columns of the observation signal matrix, the original signal will be smooth on the first graph, and a second objective function determined based on the assumption that when an original signal obtained by removing noise from an observation signal formed by the observation signal values ​​for each row of the observation signal matrix is ​​represented by a second graph having vertices equal to the number of observation signal values ​​included in the rows of the observation signal matrix, the original signal will be smooth on the second graph, the K observation signal matrices generated by the matrix generating step being applied to an arithmetic formula that optimizes an adjacency matrix of each of the first graph and the second graph, and performs an arithmetic operation to remove noise from the observation signal based on the optimized adjacency matrices, to estimate an original signal obtained by removing noise from each of the observation signals; A noise removal method comprising:

8. Computer, a matrix generating means for generating K observation signal matrices by arranging (the number of observation positions x the number of domains x K) observation signal values ​​obtained by K observations (where K is an integer equal to or greater than 2) at each of a plurality of observation positions in space, with either the observation positions or the domains as rows and the other as columns; a calculation formula derived from a first objective function determined based on the assumption that when an original signal obtained by removing noise from an observation signal formed by the observation signal values ​​for each column of the observation signal matrix is ​​represented by a first graph having vertices equal to the number of observation signal values ​​included in the columns of the observation signal matrix, the original signal will be smooth on the first graph, and a second objective function determined based on the assumption that when an original signal obtained by removing noise from an observation signal formed by the observation signal values ​​for each row of the observation signal matrix is ​​represented by a second graph having vertices equal to the number of observation signal values ​​included in the rows of the observation signal matrix, the original signal will be smooth on the second graph, the calculation processing means applying the K observation signal matrices generated by the matrix generation means to a calculation formula that optimizes an adjacency matrix of each of the first graph and the second graph, and performs a calculation to remove noise from the observation signal based on the optimized adjacency matrices, to estimate an original signal obtained by removing noise from each of the observation signals; A program to function as a

Images