Information processing device and electronic device having the same

Abstract

An information processing device (100) for performing signal processing using a neural network comprises: a Fourier transform layer (12) for performing Fourier transform on an input signal (20) and outputting a first amplitude signal (21r) and a first phase signal (21θ); an amplitude connection layer (13A) for multiplying the first amplitude signal (21r) by a first weight matrix (W1) whose value is updated through training and outputting a second amplitude signal (22r); a phase connection layer (13B) for multiplying the first phase signal (21θ) by a second weight matrix (W2) whose value is updated through training and outputting a second phase signal (22θ); a complex activation layer (14) for updating at least the second amplitude signal (22r) of the second amplitude signal (22r) and the second phase signal (22θ) using a complex activation function f in the spatial frequency domain; and an inverse Fourier transform layer (15) for combining the updated signals and performing an inverse Fourier transform.

Description

Technical Field

[0001] The present application relates to an information processing device and an electronic device having the information processing device. Background Art

[0002] In a neural network used for image recognition, etc., images are learned through deep learning to extract feature quantities in the image. In recent years, CNN (convolutional neural network) that uses convolution operations to obtain high recognition performance has attracted attention. The CNN convolves the input image with a weight function learned by the error back propagation algorithm, updates the weight function by the error back propagation algorithm, and associates the input image with the output result. At this time, the following method is known: the input image is expressed as a set of points in a high-dimensional space, the input image is perturbed using a matrix obtained by smoothing the weighting function, the perturbed input image is subjected to discrete Fourier transform or discrete cosine transform to project it into a subspace, and the image subjected to the above transformation is inversely transformed to obtain the perturbed image (for example, Patent Document 1).

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2017-49996 Summary of the invention

[0006] Technical issues to be solved by the invention

[0007] In the above-mentioned neural network using convolution operation, i.e., CNN, the recognition accuracy can be improved by setting the convolution layer as a multi-layer structure. However, for a large image such as an image of 2048 pixels × 2048 pixels, the number of operations of the matrix to be calculated in the convolution layer increases roughly in proportion to the total number of pixels in the image, so the amount of calculation is huge. Therefore, the required calculation cost is high, and it is difficult to simplify the information processing device.

[0008] In addition, the calculations of each layer in the previous CNN were obtained by simulating the brain structure in a mathematical model, and it is difficult to grasp the physical meaning. Therefore, although the constructed CNN as a whole can be evaluated based only on the calculation results of the CNN as a whole, there is a problem that the physical function of each calculation cannot be analyzed based on the input and output signals in each calculation, and the detailed analysis of CNN cannot be performed.

[0009] The present application discloses a technology for solving the above-mentioned technical problems, and aims to provide an information processing device that reduces the amount of calculation of a neural network, obtains high recognition accuracy based on the neural network using a low-cost and simple device structure, and can perform detailed analysis of the neural network.

[0010] Another object is to provide an electronic device that performs high-speed and high-precision control operations based on learning and inference using the information processing device.

[0011] Technical solutions for solving technical problems

[0012] The information processing device disclosed in the present application is an information processing device that processes an input signal using a neural network, and comprises: a Fourier transform layer, which performs Fourier transform on the input signal and outputs a first amplitude signal and a first phase signal; an amplitude connection layer, which multiplies the first amplitude signal by a first weight matrix that updates the values ​​in the matrix through training and outputs a second amplitude signal; a phase connection layer, which multiplies the first phase signal by a second weight matrix that updates the values ​​in the matrix through training and outputs a second phase signal; a complex activation layer, which uses a complex activation function f that is an activation function in the spatial frequency domain to update at least the second amplitude signal of the second amplitude signal and the second phase signal; and an inverse Fourier transform layer, which combines the second amplitude signal updated in the complex activation layer with the second phase signal and performs an inverse Fourier transform.

[0013] The electronic device disclosed in the present application uses the above-mentioned information processing device to perform control operations.

[0014] Effects of the Invention

[0015] According to the information processing device disclosed in the present application, it is possible to reduce the amount of calculation in the neural network, obtain high recognition accuracy based on the neural network with a low-cost and simple device structure, and perform detailed analysis of the neural network.

[0016] Furthermore, according to the electronic device disclosed in the present application, high-speed and high-precision control operations can be performed. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a diagram showing a configuration example of hardware according to the first embodiment.

[0018] Figure 2 A diagram showing a structural example of a CNN as a first comparative example.

[0019] Figure 3 A diagram showing a configuration example of a neural network according to the first embodiment.

[0020] Figure 4 for Figure 3 Some details of the diagram.

[0021] Figure 5 A diagram for explaining processing of an input image in Fourier transform according to the first embodiment.

[0022] Figure 6A diagram for explaining the operation of updating the amplitude value using the complex activation function f in the first embodiment.

[0023] Figure 7 This is a diagram illustrating an example of the reactivation function f according to the first embodiment.

[0024] Figure 8 A diagram for explaining another example of the reactivation function f of implementation mode 1.

[0025] Fig. 9 A diagram for explaining another example of the reactivation function f of implementation mode 1.

[0026] Fig.10 A diagram for explaining another example of the reactivation function f of implementation mode 1.

[0027] Fig.11 This is a diagram showing a configuration example of a neural network according to another example of the first embodiment.

[0028] Fig.12 This is a diagram showing the accuracy of the neural network in embodiment 1.

[0029] Fig.13 A diagram illustrating the speed-up effect of the neural network according to the first embodiment.

[0030] Fig.14 A diagram illustrating the operation of implementation mode 3 using the complex activation function f.

[0031] Fig.15 A diagram showing a partial configuration example of a neural network according to a fifth embodiment.

[0032] Fig.16 A diagram showing a partial configuration example of a neural network according to a sixth embodiment.

[0033] Fig.17 A diagram showing a partial structural example of a neural network according to a seventh embodiment.

[0034] Fig.18 A diagram showing a partial structural example of a neural network according to an eighth embodiment.

[0035] Fig.19 A diagram showing a structural example of a neural network according to a ninth embodiment.

[0036] Fig. 20 This is a diagram showing a structural example of a neural network according to a tenth embodiment.

[0037] Fig.21 A diagram showing a configuration example of an air conditioner according to Embodiment 11.

[0038] Fig. 22A diagram showing a structural example of an electronic device according to another example of embodiment 11.

[0039] Reference numerals

[0040] 2: convolution layer; 10A: amplitude logarithmization layer; 10B: inverse amplitude logarithmization layer; 10C: axis logarithmization layer; 11, 11A: input layer; 12, 12A: Fourier transform layer; 13A: amplitude connection layer; 13B: phase connection layer; 14: complex activation layer; 15: inverse Fourier transform layer; 16: output layer; 18: complex pooling layer; 20: input signal; 21r: 1st amplitude signal; 21θ: 1st phase signal; 22r: 2nd amplitude signal; 22θ: 2nd phase signal; 50: information processing device; 51: infrared sensor; 51a: sensor; 100: hardware; f: complex activation function; Lr, Lθ: minute matrix; W1: 1st weight matrix; W2: 2nd weight matrix. DETAILED DESCRIPTION

[0041] Implementation method 1.

[0042] <Hardware Configuration Example>

[0043] Figure 1 This is a diagram showing the overall structure of hardware 100 as an information processing device used as a neural network (hereinafter referred to as NN) according to the first embodiment of the present application.

[0044] The hardware 100 may be a stand-alone computer, or a server or client of a server-client system using a cloud or the like. In addition, the hardware 100 may be a smart phone or a microcomputer. In addition, in the case of a factory, the computer environment in a closed network in the factory, which is called edge computing, may be used.

[0045] like Figure 1 As shown, the hardware 100 has a built-in CPU (Central Processing Unit) 30, and an input / output interface 35 is connected to the CPU 30 via a bus wiring 34. When an instruction is input to the CPU 30 via the input / output interface 35 due to the input unit 37 being operated by a user using the NN, the CPU 30 executes the program stored in the ROM (Read Only Memory) 31 according to the instruction. Alternatively, the CPU 30 loads the program stored in the hard disk (HDD) 33 or SSD (Solid State Drive, not shown) into the RAM (Random Access Memory) 32, reads and writes and executes it as needed. Accordingly, the CPU 30 performs various processes to make the hardware 100 function as a device with predetermined functions.

[0046] The CPU 30 outputs the results of various processes from the output unit 36, i.e., the output device, via the input / output interface 35 as needed, or sends the results of various processes from the communication unit 38, i.e., the communication device, and causes the hard disk 33 to record the results of various processes. In addition, the CPU 30 receives various information from the communication unit 38 via the input / output interface 35 as needed, and calls out and uses various information from the hard disk 33.

[0047] The input unit 37 includes a keyboard, a mouse, a microphone, a camera, etc. In addition, the output unit 36 ​​includes an LCD (Liquid Crystal Display) or a speaker, etc. In addition, the program executed by the CPU 30 can be pre-recorded in the hard disk 33 or ROM 31 as a recording medium built into the hardware 100. Alternatively, the program can be saved (recorded) in a removable recording medium 40 connected via a drive 39.

[0048] Such a removable recording medium 40 can be provided as so-called packaged software. Examples of the removable recording medium 40 include a floppy disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, and a semiconductor memory.

[0049] In addition, the program can be sent and received by connecting multiple hardware via a system such as WWW (World Wide Web) (Com port) via either or both wired or wireless connections. Furthermore, the training described later can be performed and only the weight function obtained through training can be sent and received using the above method.

[0050] For example, the CPU 30 causes the hardware 100 to function as an information processing device that processes each layer constituting the NN and generates the NN. In this case, the hardware 100 functions as the NN and also functions as a device for generating the NN.

[0051] In addition, in addition to being composed of general-purpose hardware that is good at parallel computing, such as CPU or GPU (Graphics Processing Unit), each layer of NN can also be composed of FPGA (Field-Programmable Gate Array), FFT (Fast Fourier Transform) computing architecture or dedicated hardware.

[0052] In addition, matrix operations such as Fourier transform and inverse Fourier transform described later can be performed by dedicated hardware. In particular, hardware for fast Fourier transform and inverse Fourier transform that can perform Fourier transform and inverse Fourier transform at high speed can be obtained as general hardware. The dedicated hardware can also be built into the CPU or GPU.

[0053] In addition, the hardware 100 may include a sensor for converting physical phenomena such as sound waves, electromagnetic waves including visible light, heat, or vibration into numerical data, or a mechanism for outputting images or calculation results of CAD designed in the hardware 100. Alternatively, the hardware 100 may also include a mechanism for fusing the information of the above sensors with the calculation results of the hardware 100. The hardware 100 may also include a mechanism for driving based on a power cord or an internal battery.

[0054] In addition, the hardware 100 can be configured in multiple units via a communication port, and the training and reasoning described later can be implemented by hardware 100 of different structures. In addition, the hardware 100 can receive sensor signals connected to different hardware 100 via a communication port, or receive multiple sensor signals via a communication port. In addition, multiple virtual hardware environments can be prepared in one hardware 100, and each virtual hardware can be treated as a separate hardware.

[0055] <Data used for learning>

[0056] As data used in NN, supervised learning, unsupervised learning or reinforcement learning data are used. NN is sometimes also called deep learning or perceptron. Regarding perceptron, when the hidden layer described later is one layer, it is called a single perceptron, and when the hidden layer is two or more layers, it is called a multilayer perceptron. This multilayer perceptron is called NN.

[0057] Supervised learning is a method of learning the correct answer labels associated with the learning data. In NN, input signals are associated with output signals.

[0058] On the other hand, unsupervised learning is a method of learning without attaching correct answer labels to learning data. Known methods include stacked auto encoders (SAE) that are multi-layered autoencoders, and deep Boltzmann machines (DBM) that are multi-layered restricted Boltzmann machines.

[0059] Furthermore, unlike supervised learning and unsupervised learning, reinforcement learning often uses DQN (Deep Q Learning), in which the goal of DQN is not to give a correct answer to data that changes all the time, but to maximize the expected value that can be obtained in the future.

[0060] In addition, the present embodiment shows the case where supervised learning is used, but unsupervised learning can also be used. When unsupervised learning is used, multivariate analysis such as data clustering and principal component analysis is performed. In addition, since images of two or more dimensions are usually used as input images, reinforcement learning can also be used.

[0061] In addition, even data for semi-supervised learning in which correct answer labels are not given to some learning data, or data for transfer learning in which a model learned using learning data is applied to other learning data, can be applied to the present embodiment as long as the data can be applied by previous CNNs.

[0062] In addition, as learning, there are batch learning that aggregates and processes the data to be learned and online learning that adds the data to be learned each time it is added. In this embodiment, the case of using batch learning is shown, but online learning can also be used. In any case, this embodiment can be used to reduce the weight and speed up the calculation.

[0063] <CNN Configuration Example as First Comparative Example>

[0064] Since the NN described in this embodiment is an alternative to CNN, before describing the NN of this embodiment, CNN is described below as the first comparative example. CNN is the most representative technology in deep learning, a method with high accuracy in reasoning, and a technology developed by applying pattern recognition called a neurocognitron.

[0065] Figure 2 FIG. 1 is a diagram showing a structural example of a CNN as a first comparative example. Figure 2 As shown, the CNN structure is composed of, starting from input layer 1, convolution layer 2, activation function 3, convolution layer 2, activation function 3, pooling layer 4, convolution layer 2, activation function 3, fully connected layer 5, activation function 3, fully connected layer 5, output layer 6, and output 7.

[0066] As such, in CNN, learning is performed using convolution layer 2 and activation function 3, and with respect to input layer 1 that uses learning data, i.e., an image, as an input signal, after passing through convolution layer 2 and activation function 3, convolution layer 2, activation function 3, pooling layer 4, and fully connected layer 5 are combined multiple times. Then, the output from this combination passes through output layer 6 including an error function such as a Softmax function to obtain a desired output signal (output 7).

[0067] The layer between the input layer 1 and the output layer 6 is called a hidden layer. As mentioned above, a NN has two or more hidden layers, and the NN is composed of a plurality of layers including an input layer and an output layer. In the NN, when observing from a certain layer, the layer on the input side is also called a lower layer, and the layer on the output side is also called an upper layer.

[0068] Conventionally, CNNs having more than 150 hidden layers are known, and in recent years, there has been a trend to increase the number of layers in order to improve accuracy.

[0069] Since the correct answer label is given in supervised learning, the result output through the input layer 1, hidden layer, and output layer 6 is compared with the correct answer label, and the error component is propagated from the output layer 6 to the input layer 1 to optimize the weights and biases used in the hidden layer. For optimization, it is necessary to calculate the differentials of the operations of each layer. The error back propagation algorithm is a method for efficiently calculating the differentials used to express the propagation of the error from the output layer 6 to the input layer 1.

[0070] In the convolution layer 2, a kernel (or filter) is applied to a map and a convolution operation is performed. The kernel is composed of, for example, a 7-pixel × 7-pixel image, i.e., a 7 × 7 matrix, i.e., 49 elements. The kernel is applied to the data of the input layer 1 or the map of the connected lower layer to perform a convolution operation, and the signal is output to the upper layer. The convolution operation has the function of extracting the characteristic image edges shown in the input image from the image using the kernel. Here, the edge refers to the boundary between objects, objects and background images, etc. The process of preparing multiple kernels and updating the matrix constituting the kernel using the above-mentioned error back propagation algorithm is called a learning process.

[0071] Next, the convolution operation is explained. Let the value of the position (i, j) in the input image be xij. i and j are integers, which are integers greater than 0 in this case. In addition, in this case, let each pixel value be an integer greater than 0. Let the kernel be an image smaller than the input image, and let its size be H×H. H is an integer greater than 2. The kernel is represented by the index (p, q) ((p=0,1,2,···,H-1), (q=0,1,2,···,H-1)), and let the value in the image be hpq. At this time, let hpq take an arbitrary real value. Under such conditions, the convolution operation is represented by the following formula (1). In addition, uij is the value in the output image after the convolution operation.

[0072] [Mathematical formula 1]

[0073]

[0074] However, the convolution operation in deep learning is usually defined by the following formula (2). This is called cross-correlation or correlation outside of deep learning. The correlation in this case gives the same result as the convolution operation when the kernel is reversed in the up-down and left-right directions, so the operation represented by formula (2) is called the convolution operation here.

[0075] [Mathematical formula 2]

[0076]

[0077] In the spatial frequency domain after Fourier transformation described later, the difference between the convolution operation and the correlation is equivalent to the complex conjugate of the Fourier transform of one of xij and hpq. However, since the kernel is composed of values ​​obtained by learning, it is not necessary to use the complex conjugate when the initial value of the kernel is randomly set, but it is preferable to use the complex conjugate for the kernel when the weight of the initial value is changed.

[0078] In order to reduce the amount of convolution calculation, a method called stride is sometimes used. When the width of the stride is set to an integer s greater than 2, the stride can be expressed by the following formula (3).

[0079] [Mathematical formula 3]

[0080]

[0081] Based on this, the output image can be reduced to 1 / s, but increasing s means that image features may be missed, which usually leads to worsening performance, so this operation is not performed except when it is essential to speed up learning or reasoning.

[0082] In the convolution operation, calculations are performed and summed while sliding the kernel relative to the input image. In a convolution layer 2, when the size of the image of the lower layer (input image) is set to W×H and the size of the kernel is set to U×V, the size of the output image is (W-2[U / 2])×(H-2[V / 2]). In this case, [] is set as an operator that rounds down the decimal point to an integer. Since [U / 2] and [V / 2] are natural numbers, when a convolution operation is performed, the image after the operation is smaller than the input image before the operation.

[0083] When the images before and after the convolution operation are the same size, most calculations such as the calculations during error back propagation become easier, so images of width [U / 2] and width [V / 2] are added to the outside of the image after the convolution operation to make the output image larger. This method is called padding. The image added to the outside can be any image, but it is usually set to 0, which is specifically called zero padding.

[0084] In the fast Fourier transform and inverse fast Fourier transform described later, the matrix size needs to be a power of 2, so it is preferable to make the size of the input matrix and the size of the output matrix equal by padding.

[0085] Next, activation function 3, which is a nonlinear function, is applied to the output signal after the convolution operation in convolution layer 2. The nonlinear property of activation function 3 is very important for NN (CNN in this case), and by adding a nonlinear function, arbitrary data that cannot be expressed only by a linear function can be expressed.

[0086] As a simple example, let's consider a linear function h(x)=cx with a constant c as the activation function. The function (from x to y) formed by stacking three layers of this linear function h(x) is y=h(h(h(x))). However, this function can also be written as y=c·c·c·x, that is, y=c 3 x, which can also be expressed with 1 layer. This means that the output y can only be expressed as a linear function of the input x, and adding hidden layers is meaningless. Therefore, a nonlinear function is needed as the activation function.

[0087] By using a nonlinear function as activation function 3, as the number of layers increases, the degree of freedom of the function that can be expressed increases, and the relationship between any input signal and the correct answer label can be established with high accuracy.

[0088] As a nonlinear function that becomes activation function 3, the following function can be used: Relu function (normalized linear function)

[0089] g(x)=max(x,0)

[0090] The sigmoid function (also called the logistic sigmoid function) is

[0091] g(x)=1 / (1+exp(-x))

[0092] Or the hyperbolic tangent function is

[0093] g(x)=tanh(x).

[0094] In addition, the following activation functions are known:

[0095] Leaky ReLU

[0096] When x<0, g(x)=ax, when x≥0, g(x)=x,

[0097] Or ReLU with threshold

[0098] When x>θ, g(x)=x, otherwise g(x)=0.

[0099] As an exception, an identity function g(x)=x, which is a linear function, is sometimes used just before the output layer 6. However, in order to express an arbitrary curve, two or more hidden layers using nonlinear functions as activation functions are required, so the identity function is not described here.

[0100] As described above, the error back propagation algorithm is used to update the weights and bias values ​​of the CNN composed of multiple convolutional layers 2 and activation functions 3. This update is called a learning process, or simply learning. At the beginning of learning, pseudo-random numbers are used as weights and biases. In addition, the initial values ​​of Xavier or He described later are sometimes used. When the learning process is completed, the updated weights and biases are output.

[0101] Then, using the updated weights and biases, inference is performed on the input image to estimate the output. The process of performing this inference is called the inference process, or simply inference.

[0102] In typical use, the learning process is performed by a server client system equipped with a GPU or dedicated hardware.

[0103] On the other hand, in reasoning, the learning process to reduce the error with the correct answer label is not necessary. Only the above-mentioned weights and biases are used to calculate and output from the input layer 1 to the output layer 6, so the amount of calculation is small compared to learning. However, among the layers that make up CNN, the amount of calculation of the convolution layer 2 is large and requires calculation time. The reasoning process mostly requires a response speed of less than a few seconds, but in the usual CNN that increases the number of convolution layers 2 to improve the accuracy of learning, it is often difficult to achieve a response within the expected time using simple hardware such as microcomputers. In edge computers that only have computers with low computing power, such as smartphones, dedicated hardware for CNN is sometimes installed, but there will be problems such as increased hardware installation area, complicated processing system, or increased power consumption.

[0104] <Overview>

[0105] In this embodiment, a configuration example of the following hardware 100 (information processing device) is shown: the hardware 100 processes the output signal of a sensor that converts physical phenomena such as sound waves, electromagnetic waves including visible light, heat, or vibration into numerical data, or a signal designed by calculation in the hardware 100, or a signal including both a sensor signal and a calculation result. The calculation in the hardware 100 is a calculation process using NN. The input signal can be any input signal as long as it is one-dimensional or more, and in this embodiment, a two-dimensional image is used as an object for description.

[0106] In the NN of this embodiment, an operation equivalent to the convolution operation is performed in the spatial frequency domain, and a complex activation function as an activation function in the spatial frequency domain is applied to the signal after the operation. That is, the input signal in the spatial domain based on the output signal of the above-mentioned sensor is Fourier transformed to be transformed into a signal in the spatial frequency domain, and an operation equivalent to the convolution operation and an operation using the complex activation function are performed in the spatial frequency domain. After that, the signal in the spatial frequency domain is inversely Fourier transformed to return to a signal in the spatial domain, and the desired output signal is obtained.

[0107] In the convolution operation in the CNN shown in the first comparative example above, one convolution operation requires multiple matrix operations, but one convolution operation in the spatial domain is replaced by one matrix operation in the spatial frequency domain. In this embodiment, by using this feature for learning or reasoning, the number of operations can be greatly reduced, the calculation time, the amount of calculation, the power consumption of the calculation cost can be greatly reduced, and the calculation cost can be greatly reduced. For convenience, the operation equivalent to the convolution operation in the spatial frequency domain is referred to as a complex convolution operation.

[0108] In addition, in the spatial frequency domain, a complex activation function is applied as a nonlinear function to the signal after the operation based on the complex convolution operation. In this way, the complex convolution operation and the operation using the complex activation function are performed in the continuous processing in the spatial frequency domain. According to this, the above-mentioned effect of significantly reducing the number of operations can be achieved reliably without hindrance.

[0109] Assume that the following situation is studied: the convolution operation in the CNN shown in the first comparative example above is replaced by a complex convolution operation in the spatial frequency domain, and then the activation function in the CNN shown in the first comparative example is used. This situation is called the second comparative example. In the case of the second comparative example, it is necessary to perform an inverse Fourier transform after the complex convolution operation, and use an activation function on the output result of the inverse Fourier transform. That is, for each complex convolution operation, it is necessary to repeatedly use the Fourier transform and the inverse Fourier transform before and after it, which hinders a significant reduction in computational cost.

[0110] In this embodiment, since a complex activation function is used after the complex convolution operation, if the Fourier transform and the inverse Fourier transform in the hidden layer are respectively performed once, any number of complex convolution operations and operations using the complex activation function can be performed in the continuous processing in the spatial frequency domain between them. This can achieve a significant reduction in computational cost.

[0111] Figure 3 : is a diagram showing a simplest example of a NN structure according to this embodiment. Figure 3 As shown, the NN is constructed to include, starting from an input layer 11, a Fourier transform layer 12, a connection layer 13 composed of an amplitude connection layer 13A and a phase connection layer 13B for performing complex convolution operations, a complex activation layer 14 for performing operations using a complex activation function f, an inverse Fourier transform layer 15, an output layer 16, and an output 17. Figure 4 for Figure 3 A partial detail diagram of the diagram shows the signals generated at each layer.

[0112] In this NN, a two-dimensional image is used as an input signal 20, and the input layer 11 inputs the input signal 20 to the NN. The Fourier transform layer 12 performs Fourier transform on the input signal 20 in the spatial domain, and outputs a first amplitude signal 21r and a first phase signal 21θ as signals in the spatial frequency domain. In this case, fast Fourier transform is used.

[0113] The amplitude connection layer 13A has a first weight matrix W1 for updating the values ​​in the matrix through training, and multiplies the first amplitude signal 21r by the first weight matrix W1 to output a second amplitude signal 22r. The phase connection layer 13B has a second weight matrix W2 for updating the values ​​in the matrix through training, and multiplies the first phase signal 21θ by the second weight matrix W2 to output a second phase signal 22θ.

[0114] The complex activation layer 14 uses the complex activation function f in the spatial frequency domain to update at least the second amplitude signal 22r of the second amplitude signal 22r and the second phase signal 22θ. In the present embodiment, only the second amplitude signal 22r is updated using the second phase signal 22θ and output as the third amplitude signal 23r, and the second phase signal 22θ is output without being updated.

[0115] The inverse Fourier transform layer 15 combines the third amplitude signal 23r and the second phase signal 22θ to perform inverse Fourier transform, thereby generating a spatial domain signal 25. In this case, inverse fast Fourier transform is used.

[0116] The output layer 16 then transforms the signal 25 from the inverse Fourier transform layer 15 into the desired form, obtaining the output 7 which is output from the NN.

[0117] In addition, Figure 3 and Figure 4 In the figure, a connection layer 13 for performing complex convolution operation and a complex activation layer 14 for performing operation using a complex activation function f are respectively provided between the Fourier transform layer 12 and the inverse Fourier transform layer 15, but usually there are multiple layers of each.

[0118] The number of rows of the first weight matrix W1 is the same as the number of columns of the first amplitude matrix 21r, and the number of rows of the second weight matrix W2 is the same as the number of columns of the first phase matrix 21θ. There is no limit to the number of columns of the first weight matrix W1 and the second weight matrix W2.

[0119] In addition, in the present embodiment, since fast Fourier transform and inverse fast Fourier transform are used, the amplitude matrix and the phase matrix need to be matrices of a power of 2. Therefore, in the amplitude connection layer 13A and the phase connection layer 13B, a first weight matrix W1 and a second weight matrix W2 are used that output matrices of the same size as the amplitude matrix and the phase signal.

[0120] <Input Data>

[0121] As input data, a sensor signal that receives sound waves or electromagnetic waves including visible light, a sensor signal that obtains heat or vibration, or a signal output by operation in hardware 100, or a signal that combines both the sensor signal and the operation result is used. As a sound wave, a signal received by a microphone or an ultrasonic sensor is used. As a sensor that collects electromagnetic waves, a camera that collects visible light, a camera that collects infrared or ultraviolet rays, a light quantity sensor, a near-field antenna, a far-field antenna, a magnetic sensor, an electric field / magnetic field sensor, a current sensor, a voltage sensor, or a radiation sensor is used. In addition, an acceleration sensor, a temperature sensor, a humidity sensor, a gas sensor, a distance sensor, a pressure sensor, or a vibration sensor such as an acceleration or gyroscope can also be used.

[0122] In addition, with regard to sensor signals, it is not necessary to obtain all data at the same time. After the data is obtained, it can be processed at any time and used as one input data. In addition, it can be a sensor that senses by contact or a sensor that senses in a non-contact manner. Furthermore, a signal that combines a signal from a visible light camera with a signal from an infrared camera or the like can be used as input data. For example, it is also possible to use a device that uses an active phased array antenna or a laser beam to measure the condition of the wind in the distance, and to scan electromagnetic waves or wind in space at high speed to obtain input data as an image of two or more dimensions.

[0123] In addition, the data used as input data does not need to be a single data, and two or more data may be combined and used. In this case, by combining the NN using the complex activation function shown in this embodiment with the conventional perceptron for learning, desired learning can be performed.

[0124] The hardware 100 of this embodiment processes an input signal related to an object obtained from a camera that captures images with visible light, for example, and sends an output signal from an output layer. The output signal is a signal obtained by classifying the input signal, an estimate obtained by performing regression analysis, or a result of clustering or multivariate analysis. The type of the output signal is determined based on data (teacher data) to which a correct answer label associated with the input signal is attached.

[0125] For example, in hardware 100 that classifies an input signal of an image into 10 types, data obtained by associating the input image with the correct answer label in a one-to-one correspondence is used as teacher data for learning. The signal captured by the camera and the learned parameters are operated on the parameters obtained by learning (each element of the weight matrix constituting the layer of the NN) to obtain an output signal for the classification.

[0126] In addition, the processing of outputting a regression signal based on an input signal of an image is studied. For example, the safety classification based on the image scene is a simulated signal ranging from 0 to 100 points. In this case, a real number ranging from 0 to 100 points is given to one image as the correct answer label. A plurality of combinations of the image and the correct answer label are prepared as learning data. Then, the parameters of the layer in the NN are determined as the learning result. Then, by taking the image obtained from the camera as the input signal and calculating the above parameters, a regression signal is obtained as an output signal.

[0127] In the above description, a camera that captures images in a frequency band that is visible light is used as a sensor, but the same processing can be performed even with sensors such as antennas that receive infrared light, ultraviolet light, or electromagnetic waves with a frequency lower than infrared light, as long as the sensor signal is given a correct answer label such as classification or regression.

[0128] In addition, although the above description is about supervised learning, the same is true for unsupervised learning without correct answer labels.

[0129] As an example of unsupervised learning, the following describes an autoencoder. An autoencoder is an information processing device configured to perform unsupervised learning using a NN. The goal of the autoencoder is to use the input as data for learning and extract features representing the data, i.e., each element of the matrix in the layer of the NN. Therefore, the goal of the output layer of the NN is to output the same signal as the input layer, so that the size of the matrix of the hidden layer between the input layer and the output layer becomes smaller. In this way, if the results of the input layer and the output layer can be made equal by a small matrix, the file size of the input signal can be reduced. This is called dimensionality reduction.

[0130] In this autoencoder, the image captured by the camera is used as an input signal in the same way as above. In this case, no correct answer label is required, and learning is performed by inputting the output of the camera into the autoencoder.

[0131] When a signal output by operation in the hardware 100 is used as an input signal, operation may be performed using a computer-aided design tool (CAD) or a simulator of electromagnetic field, heat, stress, or the like, or another simulator simulating an environment such as a computer game.

[0132] In addition, regarding the input signal that combines the sensor signal and the calculation result, the sensor signal is input to a simulator, etc. In addition, the sensor type or position information may be appropriately changed based on the output of the simulator.

[0133] In addition, in this embodiment, it is also possible to apply a physical signal obtained by manufacturing or simulating based on CAD data as a correct answer label, or to artificially change the correct answer label. As a physical signal calculated based on CAD data, for example, it includes electromagnetic field distribution, current, voltage, heat, etc. based on electromagnetic waves. Regarding electromagnetic waves, the input of electromagnetic field simulation is CAD data (two-dimensional or three-dimensional image), and the correct answer label is S parameters (Scattering parameters) as simulation results, the amplitude value of the electric field / magnetic field at a certain position in space, voltage / current or Poynting vector, etc. At this time, the S parameters and the amplitude of the electric field / magnetic field are frequency-dependent, so in these cases, there are multiple data as correct answer labels for one input image.

[0134] The above-mentioned input signal based on CAD and simulator has an input image with more than one dimension. Even one-dimensional data such as time series data can be treated as two-dimensional data by setting it as a spectrum diagram with time as the horizontal axis, frequency as the vertical axis, and the amplitude at each time / frequency as the amplitude of the output of Fourier transform. As the Fourier transform when creating a spectrum diagram like this, a short-time Fourier transform that performs Fourier transform per unit time is used. In addition, in order to simplify, two-dimensional data is used for explanation in this embodiment.

[0135] In this embodiment, a grayscale image is assumed, but in the case of a color image, in the case of RGB, the input image is separated into Red, Green, and Blue to form an image with one dimension higher than the input image. This dimension is called a channel, and in the case of RGB, there are three channels. In the case of CMYK, there are four channels: Cyan, Magenta, Yellow, and Key plate.

[0136] When multiple channels are used as input, four channels are usually transformed into one channel by convolution operation using a kernel. In this embodiment, the following methods can be used: a method of setting a convolution layer in the front stage of the Fourier transform layer 12, a method of performing Fourier transform on each channel and transforming it into one channel through a fully connected layer, or a method of simply weighting each channel in advance to set the input signal 20 input to the input layer 11 as one channel.

[0137] The input layer 11 stores input signals 20 to the NN, that is, input data, and delivers the input data to the upper layer of the NN.

[0138] In this embodiment, the learning results using the MNIST (Mixed National Institute of Standards and Technology database) which is commonly used for NN performance evaluation are shown. MNIST is a grayscale image with a vertical × horizontal size of 32 × 32, and has 60,000 learning data and 10,000 test data not used for learning.

[0139] <Fourier transform layer, inverse Fourier transform layer>

[0140] The Fourier transform layer 12 that performs Fourier transform will be described below. In addition, since inverse Fourier transform is the inverse transform of Fourier transform, details of inverse Fourier transform and inverse Fourier transform layer 15 are omitted.

[0141] In Fourier transform, due to its nature, the transform is performed on the premise that a two-dimensional image input for Fourier transform is a two-dimensional plane infinitely connected in the vertical and horizontal directions.

[0142] When the input images are directly connected vertically and horizontally, the image at the end of the image becomes discontinuous on the connected line, and sometimes frequency components that the original input image does not have are generated. Therefore, in the usual Fourier transform, a window function is applied to the vertical and horizontal directions of the image, and the Fourier transform is performed on the signal that makes the end close to 0.

[0143] In such a method using a window function, when the important information that you want to extract with CNN is at the end of the image, the information will be lost. Figure 5 As shown, a method of configuring an image that is vertically, horizontally, and diagonally symmetrical with respect to an input image before Fourier transformation is used.

[0144] When the horizontal axis of the input image is set to the x-axis and the vertical axis is set to the y-axis, an image that is line-symmetric with respect to the boundary line ly parallel to the y-axis is arranged at the end of the x-axis component of the input image, and an image that is line-symmetric with respect to the boundary line lx parallel to the x-axis is arranged at the end of the y-axis component of the input image. Furthermore, an image that is point-symmetric with respect to the intersection of the two boundary lines lx and ly, that is, rotationally symmetric by 180 degrees, is arranged at a position diagonally opposite to the input image.

[0145] By performing Fourier transform on an image consisting of a total of four images obtained by arranging one image symmetrically in the vertical direction, the horizontal direction, and the diagonal direction, calculation can be performed while suppressing the generation of frequency components that the original input image does not have, without using a window function.

[0146] In this embodiment, since the use of fast Fourier transform as the Fourier transform is a premise, the vertical and horizontal sizes of the input image are powers of 2, that is, even numbers. In the above method of configuring an image that is vertically, horizontally, and diagonally symmetrical, the vertical and horizontal sizes of the image composed of four images are also even numbers and have a size that is a power of 2, so fast Fourier transform can be used.

[0147] However, when this method is used, the input image becomes larger, so the processing of the fast Fourier transform becomes larger. Therefore, this method is not applied to the case where there is no information at the ends of the image, such as the MNIST image, and no discontinuity occurs even if the image is connected vertically and horizontally as it is.

[0148] In the inverse Fourier transform, the same processing may be performed before the inverse Fourier transform is performed.

[0149] Fourier transform is a transformation from a signal in the space domain to a signal in the space frequency domain. The signal in the space domain is called a spatial signal, and the signal in the space frequency domain after Fourier transform is called a spatial frequency signal.

[0150] As mentioned above, the convolution operation of the spatial signal is the multiplication of the matrix of the spatial frequency signal. Specifically, the multiplication is the Hadamard product, which is represented by the following mathematical formula. Where F is the Fourier transform, * is the convolution operation, and ◎ is the Hadamard product.

[0151] F[k*u]=F[k]◎F[u]

[0152] The Fourier transform of a signal a(s, t) in the space domain having a size of m×n is represented by the following equation (4). This equation is called discrete Fourier transform. In this case, j represents a complex number, e represents Napier's constant, and π represents pi.

[0153] [Formula 4]

[0154]

[0155] As the Fourier transform, a fast Fourier transform (FFT) is used. In a radix-2 FFT, an N-point signal is divided into two halves, so it can also be considered as a radix-2 Fourier transform with time interval culling.

[0156] When n is the size of the two-dimensional image, the computational order of the common Fourier transform is O(n 3 ), the computational order of convolution operation is also O(n 3 ). Where O() represents the approximate value of the number of calculations.

[0157] The computational order of the fast Fourier transform is O(n 2 ·log2n), the order of magnitude of the calculation obtained by combining the fast Fourier transform and the inverse fast Fourier transform (IFFT) is O(2n 2 ·log2n). In addition, the calculation order of the above-mentioned complex convolution operation as the operation in the connection layer 13 is O(n 2 ), compared with other computational orders of magnitude O(n 3 )、O(n 2 log2n) is small enough to be ignored.

[0158] Therefore, compared with the previous CNN computational level O(n 3 ), the computational order of NN in this embodiment is O(2n 2 ·log2n), the amount of calculation is reduced and the speed can be increased.

[0159] As mentioned above, in a CNN, there are usually multiple convolutional layers 2. When the number of convolutional layers 2 is set to m, the computational order of CNN is O(m·n 3 In addition, in the case of the second comparative example described above, a complex convolution operation is performed instead of a convolution operation. However, since Fourier transform and inverse Fourier transform are repeatedly used before and after each complex convolution operation, when fast Fourier transform and inverse fast Fourier transform are used, the order of calculation is O(2m·n 2 ·log2n). In addition, the number of complex convolution operations in the second comparative example is equal to the number α of the convolutional layers 2 in the CNN.

[0160] Regardless of the number of complex convolution operations corresponding to the number of connection layers 13, the computational order of the NN of this embodiment is O(2n 2 ·log2n), on the other hand, in the CNN as the first comparative example and the above-mentioned second comparative example, the amount of calculation increases sharply whenever the number m of convolution operations or complex convolution operations increases.

[0161] In the case of a small image of about 64×64 with n=64, for example, when m=10, the order of magnitude of the calculation in a CNN using convolution operation is O(m·n 3 )=O(2.6×10 6 ), in the case of the second comparative example, the calculation order is O(2m·n 2 ·log2n)=O(4.9×10 6 ). In contrast, the computational order of the NN in this embodiment is O(2n 2 ·log2n)=O(4.9×10 5 ), which can reduce the amount of calculation and increase the speed by about 10 times.

[0162] In the case of a large image such as n = 1024 (1024 × 1024), when m = 10, the order of computation in a CNN using convolution operations is O(m·n 3 )=O(1.0×10 10 ), in the case of the second comparative example, the calculation order is O(2m·n 2 ·log2n)=O(2.1×10 8 ). In contrast, the computational order of the NN in this embodiment is O(2n 2 ·log2n)=O(2.1×10 7 ). That is, in the NN of this embodiment, the amount of calculation is reduced by 10 times compared with the second comparative example, and the amount of calculation is reduced by about 3 digits compared with the CNN of the first comparative example using convolution operation. Since the amount of calculation is directly related to the calculation time, it means that, for example, a calculation that takes 1 month using CNN can be calculated in about 90 minutes according to the NN of this embodiment.

[0163] As described above, in the present embodiment, the effect of reducing the amount of calculation becomes greater as the number of complex convolution operations corresponding to the convolution operation increases and as the image size becomes larger.

[0164] In CNN, convolution layer 2 is mostly used several to dozens of times. By using the NN of this embodiment, the amount of calculation can be greatly reduced and the speed can be increased. For example, in recent years, the number of pixels in smartphones is sometimes more than 50 million pixels (i.e., output 7000×7000 matrix), and the camera is sometimes more than 100 million pixels (output 10000×10000 matrix). When considering the amount of calculation, it is difficult to perform CNN convolution operations for such a large image without reducing the number of pixels, that is, without degrading the information contained in the image, but this can be achieved by using the NN of this embodiment.

[0165] As described above, according to the NN of this embodiment, it is possible to reduce the amount of calculation and perform operations at a high speed, and also to improve the reliability of operations.

[0166] In the fast Fourier transform, only signals of powers of 2 are used, or in the case of two-dimensional images, a butterfly operation is used, for example, to calculate only images obtained by multiplying powers of 2 by 2 at high speed. Since ordinary signals are not matrices of a size of a power of 2, zero padding is performed on the input signal to make it a size of a power of 2. Based on this, the fast Fourier transform can be used for all signals. The same is true for the inverse fast Fourier transform.

[0167] In addition, the first amplitude signal 21r and the first phase signal 21θ obtained by the Fourier transform layer 12 are subjected to a frequency shift operation in which the order of the images is reversed so that the low-frequency components are concentrated in the center of the data. Accordingly, the low-frequency components are concentrated in the center of the matrix, and the high-frequency components are concentrated in the periphery. Since there are many signals containing feature quantities in the low-frequency signal, such a frequency shift operation is often used. In the NN of this embodiment, since the processing of the image end will produce discontinuity and easily cause errors, the above-mentioned frequency shift operation is more effective. Mathematically, the frequency shift operation has no effect on the result.

[0168] In addition, the Fourier transform in the Fourier transform layer 12 can be separated into a combination of amplitude and phase or real part and imaginary part. For the operation in the subsequent complex activation layer 14, it is separated into amplitude and phase to generate the first amplitude signal 21r and the first phase signal 21θ. At this time, the amplitude is a real number greater than 0. On the other hand, the phase is a real number greater than -π and less than π. In addition, when performing Fourier transform, the above-mentioned padding is performed, usually zero padding, so that the input signal and the output signal of the Fourier transform layer 12 are matrices of the same size.

[0169] In addition, regarding Fourier transform and inverse Fourier transform, a plurality of methods with different coefficients and the like are known, and any method can be used.

[0170] However, the results may sometimes differ depending on the method. This is because, in the relationship between the input and output of the Fourier transform, the total energy (volume) possessed by the spatial domain is different from the total energy possessed by the spatial frequency domain. That is, when an inverse Fourier transform is performed after a Fourier transform, a difference sometimes occurs between the signal before the input and the signal at the output. There is basically no problem in the case of learning and reasoning using a NN with the same hidden layer, but depending on the data, an operation may be performed in accordance with the Parseval identity (or Rayleigh energy theorem) to make the total energy possessed by the spatial domain equal to the total energy possessed by the spatial frequency domain. In addition, the Parseval identity may also be used when the hardware 100 used in learning and reasoning is different, or when rounding errors cause problems. The same is true for the inverse Fourier transform.

[0171] <Connection Layer>

[0172] In this embodiment, a connection layer 13 is used in the spatial frequency domain instead of the convolution layer 2 in the spatial domain. In the connection layer 13, a fully connected layer or a sparsely connected layer is used. In particular, a fully connected layer is used for a layer close to the input layer 11, and a sparsely connected layer is used for a layer close to the output layer 16 of the upper layer, so that overtraining can be prevented. In addition, in the matrix constituting the fully connected layer, the elements of all weight matrices are updated, and in the matrix constituting the sparsely connected layer, there are elements that are not updated with a certain probability.

[0173] In the fully connected layer, the weight matrix between the connected layers is set to W, x is set as the input matrix, u is set as the output matrix, and b is set as the bias vector, and the operation expressed by the following formula is performed.

[0174] u=Wx+b

[0175] In particular, the bias vector may be set to a zero vector at a location close to the output layer 16. As initial values ​​of W and b, pseudo-random values ​​are usually used. In addition, a matrix called Xavier initial values ​​or He initial values ​​may be used as initial values, and it is known that learning progresses quickly by this. This is the same in the case of spatial signals, and the description is omitted.

[0176] The connection layer 13 is composed of an amplitude connection layer 13A and a phase connection layer 13B. The spatial signal input to the Fourier transform layer 12 is separated into a first amplitude signal 21r and a first phase signal 21θ as spatial frequency signals by Fourier transform. Then, the first amplitude signal 21r is input to the amplitude connection layer 13A, and the first phase signal 21θ is input to the phase connection layer 13B, and is multiplied by the first weight matrix W1 and the second weight matrix W2 as the above-mentioned weight matrix, respectively, and the second amplitude signal 22r and the second phase signal 22θ are output.

[0177] That is, in the amplitude connection layer 13A, the first amplitude signal 21r is multiplied by the first weight matrix W1 to output the second amplitude signal 22r. In addition, in the phase connection layer 13B, the first phase signal 21θ is multiplied by the second weight matrix W2 to output the second phase signal 22θ.

[0178] Initial values ​​of the first weight matrix W1 and the second weight matrix W2 include pseudo-random values ​​or the other initial values ​​described above.

[0179] Then, in the amplitude connection layer 13A and the phase connection layer 13B, the values ​​in the first weight matrix W1 and the second weight matrix W2 are updated using the error back propagation algorithm so that the relationship between input and output becomes close. That is, the values ​​in the first weight matrix W1 and the second weight matrix W2 are updated through training.

[0180] Assume that the first weight matrix W1 has only positive real numbers for the amplitude matrix (input matrix) x as the first amplitude signal 21r. Since the amplitude matrix x is a positive real number, the elements of the matrix (W1)x can be transformed into absolute values ​​|(W1)x|, but in this case, the first weight matrix W1 is constrained to be only positive real numbers for learning.

[0181] By setting constraints, the search range can be narrowed during learning, which can reduce the number of operations. In addition, the absolute value transformation operation is no longer required for each element, which can speed up learning.

[0182] In addition, regarding the phase matrix (input matrix) x as the first phase signal 21θ, there is no constraint on the second weight matrix W2. When radians are used as the phase, the matrix (W2)x is subjected to a modulo operation of 2π to make it a value greater than 0 and less than 2π, or a value greater than -π and less than π. However, in the subsequent complex activation layer 14, when a complex activation function f that applies a trigonometric function to the phase matrix is ​​used, a modulo operation is not required.

[0183] Furthermore, a degree, which is a unit of an angle calculated by multiplying a radian by 180 / π, may be used for the phase, in which case a modulo operation of 360° is performed.

[0184] <Reactivate Layer>

[0185] The second amplitude signal 22r and the second phase signal 22θ output from the amplitude connection layer 13A and the phase connection layer 13B are input to the complex activation layer 14, and these signals are operated using the complex activation function f, which is an activation function in the spatial frequency domain. In this case, by the operation using the complex activation function f, the second amplitude signal 22r is updated and output as the third amplitude signal 23, and the second phase signal 22θ is output as it is.

[0186] In the complex activation layer 14, based on the response of the complex activation function f to the phase θ(i) of each point i in the phase matrix constituting the second phase signal 22θ, the value of the amplitude r(i) of the point at the same position as the point i in the amplitude matrix constituting the second amplitude signal 22r is updated.

[0187] Figure 6 This is a diagram for explaining the operation of updating the value of the amplitude r(i) using the complex activation function f. The position of point i in the phase matrix (second phase signal 22θ) is set to (xi, yi), and the value of the complex activation function f is calculated for the phase θ(xi, yi) as the element of point i, and the element in the amplitude matrix (second amplitude signal 22r) at the same frequency and at the same position (xi, yi) as point i in the above phase matrix, i.e., the amplitude r(xi, yi). In addition, the rewritten amplitude matrix is ​​the third amplitude signal 23r.

[0188] The complex activation function f, which is an activation function in the spatial frequency domain, is described below.

[0189] As the complex activation function f, a nonlinear function is used in the same way as the activation function in the spatial domain. When k is an arbitrary constant and x and y are arbitrary variables, the so-called nonlinear function can be defined as a function that does not satisfy any one or both of the following definitions of the linear function g:

[0190] g(x+y)=g(x)+g(y)

[0191] g(k·x)=k·g(x)

[0192] As an example of activation function in the spatial domain, there are logistic function and hyperbolic tangent function. By applying such nonlinear function, there is a difference between forward propagation and backward propagation, which will produce a function that cannot be expressed by only linear function, that is, weighting.

[0193] The complex activation function f of this embodiment is a function for applying the nonlinear function to the spatial frequency signal after Fourier transformation.

[0194] The complex activation function f is different from the previous activation function for spatial signals. In addition, even if a Fourier transform including the activation function is performed, the complex activation function f cannot be generated. This is also obvious from the following formula. Where g is the activation function in the spatial domain.

[0195] F[g(x)]≠F[g]◎F[x]

[0196] That is, the result of applying the activation function g to a value x in the spatial domain and then Fourier transforming it is different from the value obtained by Fourier transforming the activation function g and the value x and then multiplying the two.

[0197] For example, when the Relu function is Fourier transformed, the Fourier transform diverges because the Relu function is monotonically increasing in the region of x ≥ 0. Therefore, the result obtained by Fourier transforming the Relu function cannot be used as an activation function in the spatial frequency domain.

[0198] In this embodiment, the complex activation function f based on the characteristics of the Relu function is called a complex Relu function.

[0199] The Relu function in the spatial domain calculates the same value as the input value when the input value is positive or 0, and calculates 0 when the input value is negative. The complex Relu function is not determined by only one of the amplitude r or the phase θ, and the amplitude component is updated using a function that applies a trigonometric function to the phase component. In this case, a function that applies a trigonometric function to the phase component and then multiplies the amplitude component is used. When any one of the real axis component and the imaginary axis component, for example, the real axis component is positive or 0, the amplitude component is not updated but is the same value. When it is negative, the amplitude component is updated to the calculated value based on the function.

[0200] That is, for the phase θ(i) of each point i in the matrix constituting the second phase signal 22θ, the complex activation function f updates the value of the amplitude r(i) of the point at the same position as the point i in the matrix constituting the second amplitude signal 22r according to different responses when either the real axis component or the imaginary axis component is positive or 0 and when it is negative.

[0201] The following formula (5) shows an example of a complex activation function f using a complex ReLU function.

[0202] [Mathematical formula 5]

[0203]

[0204] In this case, since r|sinθ|=r when θ=−π / 2 or π / 2, the continuity of the function is ensured.

[0205] Figure 7 The above formula (5) is shown in the figure. Figure 7 As shown, for a circle with radius r, when the real axis u and the imaginary axis jv are set, when the u component is negative, the u component is transformed to u=0, and the jv component is transformed to |jv|. In this way, when the component of the real axis u is negative, it is equivalent to replacing the amplitude r with the absolute value of rsinθ.

[0206] In addition, the following formula (6) shows another example of the complex activation function f using the complex ReLU function.

[0207] [Mathematical formula 6]

[0208]

[0209] In this case the continuity of the function is also ensured.

[0210] Figure 8The diagram of the above formula (6) is shown. In this case, when the imaginary axis jv component is negative, it is equivalent to replacing the amplitude r with the absolute value of rcosθ. This is equivalent to substituting (θ+(π / 2)) for θ in the above formula (5), which is superior to the complex activation function f based on the complex Relu function shown in the above formula (5) in terms of ease of programming and fewer comparisons, that is, speeding up calculations. In addition, since the number of comparisons of θ can also be reduced, the amount of calculation can be reduced.

[0211] As long as the continuity of r with respect to θ is ensured, various combinations can be taken as variations of the above two equations (5) and (6). For example, it is possible to consider changing the region of θ by setting θ→(θ / 2), or using hyperbolic secant sechθ instead of |sinθ|.

[0212] In addition, the complex activation function f can be defined by the following equation (7): wherein k is set to a real number greater than 1.

[0213] [Formula 7]

[0214]

[0215] As described above, regarding the complex activation function f, as long as the possible values ​​of the amplitude r are divided into two or more depending on the phase θ, the conditions of θ can be divided into three or more.

[0216] Furthermore, although the case where the complex activation function f is a continuous function has been described, a discontinuous function may also be used.

[0217] Conventionally, in the spatial domain, a step function, which is a discontinuous function, is sometimes used as an activation function. The step function is represented by the following equation (8). Alternatively, it is sometimes represented by equation (9) by considering the singularity when x=0.

[0218] [Mathematical formula 8]

[0219]

[0220] [Mathematical formula 9]

[0221]

[0222] In the spatial frequency domain, the continuity of the complex activation function f is not an essential condition, and it only needs to be a nonlinear function and have a structure that can rewrite the amplitude r based on the output calculated depending on the value of the phase θ.

[0223] The complex activation function f using the complex ReLU function without continuity is expressed by the following equation (10), for example.

[0224] [Formula 10]

[0225]

[0226] The absolute value operation of the above formula (6) is removed from the above formula (10), which loses continuity and does not correspond to the ReLU function in the spatial domain. However, it can reduce the absolute value operation and correspondingly shorten the NN learning time.

[0227] Fig. 9 The above formula (10) is shown in the figure. Fig. 9 As shown, for a circle with radius r, when the component of the imaginary axis jv is negative, it is the result obtained by mapping it to the real axis u where jv=0.

[0228] In addition, the following formula (11) shows another example of a complex activation function f without continuity.

[0229] [Mathematical formula 11]

[0230]

[0231] The absolute value operation of the above formula (5) is removed from the above formula (11). In this case, the continuity is lost and it does not correspond to the ReLU function in the spatial domain, but the absolute value operation can be reduced and the learning time of the NN can be shortened accordingly.

[0232] Fig.10 The above formula (11) is shown in the figure. Fig.10 As shown, for a circle with radius r, when the component of the real axis u is negative, it is the result of mapping to the imaginary axis jv where u=0.

[0233] <Training>

[0234] When updating the weight matrix W (first weight matrix W1, second weight matrix W2) of the NN, in order to minimize the difference between the output of the NN and the teacher data, i.e., the loss L, the optimal value of each element of the weight matrix W is retrieved. The error back propagation algorithm is a means for finding the optimal value, and is based on the gradient descent method.

[0235] In the gradient descent method, the weight matrix W, the learning coefficient α, and the loss L, which is the difference between the inference result and the correct answer label, are used to update the weight matrix W based on the following formula (12).

[0236] [Mathematical formula 12]

[0237]

[0238] Then, the error back propagation algorithm is used to calculate the error that you want to calculate in the gradient descent method. (Perform partial differentiation of the loss L with respect to the weight matrix W) and update the weight matrix W. The process from the input layer to the output layer is called forward propagation, and the process from the output layer to the input layer is called back propagation.

[0239] The operation of updating the weight matrix W is called learning, and the process of learning is called training. When the training is completed, the learning of the NN is completed. In addition, the data used for training is called training data, and is used in the same sense as learning data.

[0240] When training with training data, an early stopping method can be used to stop learning at a point in time when the expected performance predetermined before learning is met, which can prevent over-learning and shorten the learning time. In this regard, the present embodiment is no different from the CNN technology performed in the spatial domain.

[0241] The gradient descent method is described below.

[0242] Gradient descent method refers to an algorithm for searching for a solution used to minimize the objective function (usually optimization). In particular, in NN, the stochastic gradient descent method represented by the above formula (12) is used, which is a method commonly used when the objective function to be minimized is differentiable. At this time, the learning coefficient α is an important parameter for the gradient descent method, and various methods such as AdaGrad method, Adam method, and momentum method are known. These methods are the same in learning in the spatial frequency domain as in learning in the spatial domain, and the detailed methods are omitted.

[0243] However, in general, regarding the minimization method of nonlinear functions, when the objective function is twice differentiable, the Newton method or its derivative method, the quasi-Newton method, converges to the solution faster. In addition, regarding the above-mentioned gradient descent methods such as the AdaGrad method, the Adam method, and the momentum method, the convergence speed and accuracy largely depend on the value of the learning coefficient α, so it is preferred to use the Newton method or the quasi-Newton method. When using complex NNs, it is often difficult to obtain the second derivative, so the gradient descent method is actually used.

[0244] By optimizing the objective function as described above, it is possible to construct a weight matrix W that closely associates the input data with the correct answer label. However, the goal of learning is not to enable correct reasoning for the training data (input data and correct answer label) used in learning, but to enable correct reasoning for unknown samples given after learning. The error for the former training data is called the training error, and the expected value of the error for the latter unknown data is called the generalization error. Although reducing the generalization error is the goal of learning, it cannot be calculated during learning like the training error.

[0245] Therefore, the following method is used: prepare a sample set different from the training data, and use the error calculated for it in the same way as the training error as the benchmark for the generalization error. For example, when using MNIST data, 70% to 80% of the total data is used as training data, and the remaining 20% ​​to 30% of the data is used as test data. After learning, the error for the test data is calculated using the test data, i.e., the test error. Specifically, let the training data be 60,000 and the test data be 10,000. The change in the test error as the weight matrix W is updated is called a learning curve.

[0246] Although the test error is expected to decrease with each repeated training, even if the error for the training data decreases, the error for the test data sometimes increases. This is called overlearning (or overfitting), which is a phenomenon that becomes a problem during NN training. As a method to prevent this phenomenon, weight attenuation (or load attenuation) and dropout are known to prevent overlearning. In this embodiment, these methods are also appropriately used in the same way as in the case of CNN.

[0247] Weight decay is a method of updating weights in a way that the larger the value of the weight matrix is, the closer the weight is to 0. This alleviates the divergence of weights. Various methods such as regularization methods are also known as weight decay, and similar to the spatial domain, there are no constraints when applied to the spatial frequency domain.

[0248] On the other hand, in the dropout method, over-learning can be prevented by setting the components of the amplitude matrix to 0 with a certain probability in the dropout layer. Generally, 20% to 50% is used as the probability.

[0249] In addition, when there is little learning data, over-learning is also likely to occur. In this case, data augmentation is performed. Specifically, geometric deformations such as translation, left-right mirror inversion, and rotation are uniformly applied to an image, as well as changes in depth and color and random noise. By increasing data in this way, over-learning is easily prevented, and the same is true in this embodiment.

[0250] Furthermore, over-learning can be prevented by using a sparsely connected layer instead of a fully connected layer, or by enabling the hidden layer to terminate early or reduce the number of significant figures in the calculation results.

[0251] Next, the error back propagation algorithm is explained.

[0252] Forward propagation is used when inference is performed to estimate the result using the learned weight matrix. In addition, when the weight matrix is ​​updated through training, multiple forward propagations and back propagations are performed.

[0253] In forward propagation, the matrix or function of the hidden layer is applied to the input data at any time. On the other hand, in backward propagation, error information is back-propagated from the immediately following upper layer to the immediately preceding lower layer, and the error information is the difference between the inference value obtained by forward propagation and the correct answer label as the error information.

[0254] Back propagation can be expressed by the chain rule of differentials of forward propagation. The mathematical calculation is complicated and the amount of calculation is large, so the error back propagation algorithm is used. In the error back propagation algorithm, the error component starting from the output layer is propagated from the upper layer to the lower layer, and the differential term of the processing during forward propagation is calculated on the error component to propagate it to the input layer. The value of the weight matrix is ​​updated to reduce the error component.

[0255] The complex activation function f used in this embodiment also needs to be differentiated.

[0256] When the complex activation function f is represented by the above formula (5), that is, for the following complex ReLU function,

[0257] [Mathematical formula 13]

[0258]

[0259] When y = f(r, θ), the partial differential term It is represented by the following formula (14).

[0260] [Formula 14]

[0261] when

[0262] when

[0263] at this time, is the update formula of r from the upper layer to the lower layer, Next, each element of the first weight matrix W1 multiplied by the first amplitude signal 21r and the second weight matrix W2 multiplied by the first phase signal 21θ is updated by substitution.

[0264] In addition, when the complex activation function f is represented by the above formula (11), that is, for the following complex ReLU function,

[0265] [Mathematical formula 15]

[0266]

[0267] When y = f(r, θ), the partial differential term It is represented by the following formula (16).

[0268] [Formula 16]

[0269] when

[0270] when

[0271] In addition, when the complex activation function f is represented by the above formula (6), that is, for the following complex ReLU function,

[0272] [Mathematical formula 17]

[0273]

[0274] When y = f(r, θ), the partial differential term It is represented by the following formula (18).

[0275] [Mathematical formula 18]

[0276] When -π≤θ<0

[0277] When 0≤θ<π……(18)

[0278] In addition, when the complex activation function f is represented by the above formula (10), that is, for the following complex ReLU function,

[0279] [Mathematical formula 19]

[0280]

[0281] When y = f(r, θ), the partial differential term It is represented by the following formula (20).

[0282] [Mathematical formula 20]

[0283] When -π≤θ<0

[0284] When 0≤θ<π……(20)

[0285] As the complex activation function f becomes more complex, it becomes more difficult to express the differential as a mathematical formula. In this case, the change Δy when θ changes by a small amount Δθ, that is, (Δy / Δθ), is used as the numerical differential. If the differential is formulated, the calculation ends by substitution. In contrast, although subtraction or division is required for numerical differential, the amount of calculation becomes large, but the differential can be obtained for any complex activation function f.

[0286] As mentioned above, in the error back propagation algorithm, the loss L is calculated to propagate backwards. Here, we define two functions that transform matrices into matrices, and set them as intermediate values ​​Y and Z. We assume that the size of the input matrix is ​​equal to the size of the matrix to be output. Then, we consider the input X, the weight matrix W, the intermediate values ​​Y and Z as matrices, and the loss L as a scalar.

[0287] In the error back propagation algorithm, when calculating When At this time, the size of each matrix is ​​equal to Z, Y, X, and W. Since they are uniquely determined, the chain rule, which is a general partial differential operation, is used to calculate them.

[0288] This error back-propagation algorithm is very convenient when it comes to incorporating mathematical formulas into a program, and is therefore widely used during training.

[0289] <Output Layer>

[0290] The output layer 16 will be described below. In the output layer 16, an activation function, which is a function that transforms a signal, is used to obtain a desired output 7. In this embodiment, the activation function used in the output layer 16 is referred to as an output activation function.

[0291] In addition, in supervised learning, given the pair of input data and the correct answer label, it is necessary to measure the scale of the output result of the output activation function and the proximity of the teacher data. In this embodiment, this scale is called the error function.

[0292] In general supervised learning, the output required of NN can be roughly divided into three types. The first is regression analysis, the second is multiclass classification, and the third is binary classification. The following will explain them one by one.

[0293] Regression analysis is a method of determining a function that reproduces the training data by taking a function with continuous values ​​as output. In this case, as the output activation function of the NN, a function whose range is consistent with the range of the function as the target is selected. When the range is above -1 and below 1, the hyperbolic tangent function y=tanh(x) is suitable. When the range is between -∞ and ∞, the identity mapping y=x is mostly selected. In addition, the square error is used for the difference between the output result of the output activation function and the correct answer label. When considering the differentiation of backpropagation, the result obtained by multiplying the square error by 1 / 2 is usually used as the error function.

[0294] Next, in multi-classification, the input data is classified into a finite number of classes. For example, MNIST classifies the input handwritten characters with numbers 0 to 9 into 10 categories for multi-classification problems. For such multi-classification, the Softmax function is used as the output activation function. In addition, the cross entropy is used as the error function. The Softmax function and cross entropy are the same as those in the spatial domain.

[0295] Next, in binary classification, the input data is classified into two types. In this case, the logistic function is used as the output activation function, and the same method as the maximum likelihood estimation is used for the error function. However, binary classification can also be regarded as a kind of multivariate classification. As with multivariate classification, the Softmax function can be used as the output activation function, and the cross entropy can be used as the error function.

[0296] As described above, in this embodiment, a complex activation function f is used after the complex convolution operation of multiplying the weight matrix W in the connection layer 13, so that if the Fourier transform and the inverse Fourier transform in the hidden layer are implemented once respectively, the operations in the connection layer 13 and the complex activation layer 14 can be performed any number of times in the continuous processing in the spatial frequency domain between them.

[0297] Fig.11 This is a diagram showing a structural example of a NN when two convolution layers 13 and two reactivation layers 14 are used. The accuracy of the test data when such a NN is used is shown together with the first comparative example, that is, the case of a CNN with two convolution layers 2. Fig.12 .

[0298] In this case, the results of supervised learning when handwritten characters are classified into 10 categories using MNIST as input data are shown as changes in calculation accuracy with respect to the number of training times (number of calculation times).

[0299] Fig.12The solid line in is the case of using the NN of this embodiment, and the dotted line is the case of using the CNN of the first comparative example. In this case, in the NN of this embodiment, the complex activation function f represented by the above formula (6) is used.

[0300] like Fig.12 As shown, the NN of this embodiment can infer the test data with an accuracy of about 95% when the number of calculations exceeds 1500. The CNN of the first comparative example has an accuracy of about 97%, which can be said to be approximately equivalent performance.

[0301] Fig.13 This is a diagram for explaining the speed-up effect of NN based on this embodiment. Fig.13 In the figure, the difference in calculation speed is shown for the first comparative example of a CNN having two convolutional layers 2, the second comparative example of performing two complex convolution operations and performing Fourier transform and inverse Fourier transform before and after each complex convolution operation, and the NN of this embodiment. In this case, the actual calculation using MNIST is performed by calculation based on CPU calculation, and in this case, Fig.12 The number of calculations is shown and the time taken when the calculations are performed 2000 times.

[0302] In addition, as MNIST data, it is assumed that the training data is 60,000 and the test data during inference is 10,000.

[0303] In MNIST, which is a smaller image of 32×32, the calculation time of the first comparison example is 260 seconds, while it is 200 seconds in the second comparison example. The NN according to this embodiment is 75 seconds, which is about 30% of the time compared to the first comparison example, and it can be seen that the speed is significantly improved.

[0304] In addition, by reducing the amount of training data or using a dedicated IC for Fourier transform and inverse Fourier transform that require calculation time, calculations for NNs with a large number of layers can be performed even with a small processing device such as a microcomputer.

[0305] As described above, according to the NN of this embodiment, it is possible to perform operations with roughly the same accuracy as the previous CNN. In addition, since the convolution operation that requires a large amount of calculation is replaced by a complex convolution operation based on a single matrix operation, the amount of calculation can be greatly reduced. Furthermore, by using a complex activation function f, if the Fourier transform and the inverse Fourier transform in the hidden layer are each implemented once, any number of complex convolution operations and complex activation functions f can be used in the continuous processing in the spatial frequency domain between them, and the amount of calculation required for the Fourier transform and the inverse Fourier transform can also be reduced. Therefore, high-speed processing can be performed even for large images, and high recognition accuracy based on NN can be obtained with a low-cost and simple hardware 100 structure.

[0306] In addition, in this embodiment, all operations in each layer between Fourier transform and inverse Fourier transform are operations for continuously processing spatial frequency signals in the spatial frequency domain, and the physical functions of each operation can be analyzed based on the input signal and output signal of each operation. Based on this, a detailed analysis of the NN can be performed, the structure of the NN can be determined based on the analysis results of each layer, and a high-performance NN can be constructed.

[0307] <Application Examples>

[0308] According to this embodiment, two-dimensional images are mainly used as objects, and processing can be performed at a high speed compared to the CNN of the first comparative example, and in particular, a great effect is exerted in a NN having a plurality of connection layers 13 equivalent to the convolution layer 2. In image recognition, in addition to data obtained by CMOS (Complementary Metal-Oxide-Semiconductor) and the like, images obtained by visualizing electromagnetic waves by infrared cameras, ultraviolet cameras, or phased array antennas and the like can also be used as input data.

[0309] In addition, simulators for stress, heat conduction, electric field, magnetic field, electromagnetic wave, piezoelectricity, acoustic wave, fluid, optics, structure, etc. can be used, and analytical models designed by GUI using CAD or other methods with two dimensions or higher can be used as input data, and learning can be performed using the analytical results of the simulator as data labeled as the correct answer.

[0310] Furthermore, as described above, even one-dimensional data can be regarded as two-dimensional data by converting it into a spectrogram, and therefore the method of this embodiment can be used.

[0311] In addition, the NN of this embodiment can also be applied to an information processing device that uses a convolution operation to process input data with time changes such as moving images, such as a generally known convolutional LSTM (Long short-term memory).

[0312] In addition, by transforming into a spectrum diagram, the method of this embodiment can also be applied to the parts of the entire RNN (Recurrent Neural Networks) that require convolution operations.

[0313] In addition, the present embodiment can also be applied to the following method: using a NN that simultaneously learns two NNs, a generator and a discriminator, called a GAN (Generative Adversarial Network), to generate images that are not included in the training data or test data. The GAN usually constructs the generator network and the discriminator network by overlapping a large number of convolutional layers, and since the two NNs are trained simultaneously, the number of convolutional layers increases and the amount of calculation is huge, but by using the method of the present embodiment, it is possible to achieve a significant reduction in calculation costs.

[0314] In calculations such as GAN, tens to tens of thousands of GPU boards (printed circuit boards with GPU computing elements installed) are usually used for learning, so large-scale computers are indispensable, but by applying this embodiment, a speed-up of about 3 to 8 digits can be expected, although it also depends on the size of the image to be generated. Depending on the conditions, learning can be performed even without hardware with parallel processing mechanisms such as GPUs.

[0315] According to GAN, the input data can be an image of two dimensions or more, so it is also possible to input, for example, simulation data and perform desired design.

[0316] In addition, as in the well-known Conditional GAN, the user can also use the learned weight function to determine the conditions and limit the output. In this case, the desired design can be performed using GAN, and by applying this embodiment, the time required to learn the design and the time required for reasoning can be greatly reduced.

[0317] Furthermore, in reinforcement learning, a method of learning using two-dimensional images as input data is also often used, and most of them have multiple convolutional layers. In particular, in reinforcement learning, instead of giving a correct answer label, learning is performed in a process of repeated trial and error by a NN called an agent, so the amount of calculation is huge. For such reinforcement learning, the NN can also be efficiently learned by applying this embodiment.

[0318] The same is true for a stacked auto encoder (SAE) formed by multi-layering an auto encoder for unsupervised learning, a deep Boltzmann machine (DBM) formed by multi-layering a constrained Boltzmann machine, and the like.

[0319] An autoencoder is a NN that learns in such a way that, for example, the same image is output when a two-dimensional image is input to an input layer. Various operations including a convolutional layer are performed between the input layer and the output layer, and the operations are performed so that necessary information is not lost.

[0320] In unsupervised learning, convolution operation is also an indispensable technology, so by applying this embodiment, the calculation cost can be greatly reduced.

[0321] In addition, the input data is not limited to two-dimensional data, and three-dimensional data can also be used, and a combined overview view of a plan view, a cross-sectional view, an image of a CAD or 3D camera (stereo camera), etc. can be used as three-dimensional data as input data.

[0322] Implementation method 2.

[0323] In Embodiment 2, a complex activation function f is used which is different from the complex Relu function used in Embodiment 1. In this embodiment, in the Fourier transform layer 12, it is decomposed into an amplitude component of a real number greater than 0 and a phase component of a real number greater than -π and less than π. The other structures are the same as those in Embodiment 1.

[0324] In the second embodiment, similarly to the first embodiment, the second amplitude signal 22r and the second phase signal 22θ output from the amplitude connection layer 13A and the phase connection layer 13B are input to the complex activation layer 14, and these signals are operated using the complex activation function f, the second amplitude signal 22r is updated and output as the third amplitude signal 23r, and the second phase signal 22θ is output as is.

[0325] In the complex activation layer 14, based on the response of the complex activation function f to the phase θ(i) of each point i in the phase matrix constituting the second phase signal 22θ, the value of the amplitude r(i) of the point at the same position as the point i in the amplitude matrix constituting the second amplitude signal 22r is updated.

[0326] In this embodiment, a complex activation function f based on the characteristics of the logistic function in the space domain is referred to as a complex logistic function and is used.

[0327] The complex activation function f used in the above-mentioned embodiment 1 updates the value of the amplitude r(i) by responding differently according to the size of the phase θ(i), but the complex activation function f used in this embodiment 2 updates the value of the amplitude r(i) according to a constant response based on the same calculation formula regardless of the size of the phase θ(i).

[0328] An example of a complex activation function f using a complex logistic function is represented by the following formula (21).

[0329] [Mathematical formula 21]

[0330]

[0331] Where k is a real number greater than 1. When θ varies between -π and less than π, the output of the complex activation function f varies between 0 and 1. The amplitude r is replaced by this output.

[0332] In addition, in the above formula (21), since ((k 2 -1) / 2) is a constant used to make the maximum value of the output of the complex activation function f 1, so it is not necessary. Therefore, the complex activation function f represented by the following formula (22) can also be used. In this case, the maximum value of the output of the complex activation function f is (2 / (k 2 -1)). The minimum value of the complex activation function f is a real number greater than or equal to 0, and satisfies the above-mentioned condition that the amplitude component is a real number greater than or equal to 0.

[0333] [Mathematical formula 22]

[0334]

[0335] In addition, the following formula (23) represents another example of a complex activation function f using a complex logistic function. The complex activation function f is a symmetric function about θ=0, and the minimum value of the output is 0 and the maximum value is 1.

[0336] [Mathematical formula 23]

[0337]

[0338] In addition, the following formula (24) represents another example of the complex activation function f. The complex activation function f is obtained by deforming the Gaussian error function.

[0339] [Mathematical formula 24]

[0340]

[0341] In this way, based on the two input signals of amplitude r and phase θ, a two-dimensional complex activation function f in the spatial frequency domain is defined, thereby defining an activation function that cannot be expressed by an activation function in the spatial domain. This increases the degree of freedom in expressing the activation function and enables more versatile learning.

[0342] In the error back propagation algorithm that propagates the loss from the output layer as the upper layer to the input layer as the lower layer, the complex activation function f used in this embodiment also needs to be differentiated. However, it is difficult to differentiate the complex logistic function as described above. In this case, differentiation can be performed by performing the above-mentioned numerical differentiation.

[0343] The same effect as in the above-mentioned embodiment 1 is also obtained in the present embodiment 2. That is, calculations with roughly the same accuracy as the previous CNN can be performed. In addition, since the convolution operation that requires a large amount of calculation is replaced by a complex convolution operation based on a single matrix operation, the amount of calculation can be greatly reduced. Furthermore, by using a complex activation function f, if the Fourier transform and the inverse Fourier transform in the hidden layer are respectively implemented once, any number of complex convolution operations and complex activation functions f can be used in the continuous processing in the spatial frequency domain between them, and the amount of calculation required for the Fourier transform and the inverse Fourier transform can also be reduced. Therefore, high-speed processing can be performed even for large images, and high recognition accuracy based on NN can be obtained through the low-cost and simple hardware 100 structure.

[0344] In addition, all operations in each layer between Fourier transform and inverse Fourier transform are operations that continuously process spatial frequency signals in the spatial frequency domain, and the physical functions of each operation can be analyzed based on the input signal and output signal of each operation. Based on this, a detailed analysis of the NN can be performed, the structure of the NN can be determined based on the analysis results of each layer, and a high-performance NN can be constructed.

[0345] In addition, when the calculation speed of the complex logistic function becomes a speed limiting condition in the NN, it is preferable to use a complex activation function f such as the complex Relu function shown in Implementation Example 1, which can theoretically calculate differentials and has a small amount of calculation.

[0346] Implementation method 3.

[0347] In the third embodiment, a different reactivation function f is used from that in the first and second embodiments. The other structures are the same as those in the first embodiment.

[0348] In this third embodiment, similar to the first embodiment, the second amplitude signal 22r and the second phase signal 22θ output from the amplitude connection layer 13A and the phase connection layer 13B are input to the complex activation layer 14, and these signals are operated using the complex activation function f to update the second amplitude signal 22r and the second phase signal 22θ and output them.

[0349] Furthermore, although the second amplitude signal 22r and the second phase signal 22θ are updated in the same manner by the same method, the second phase signal 22θ may be maintained as it is and only the second amplitude signal 22r may be updated.

[0350] Fig.14 A diagram for explaining the operation of updating the second amplitude signal 22r and the second phase signal 22θ using the complex activation function f.

[0351] For simplicity, the second amplitude signal 22 r and the second phase signal 22θ generated in the connection layer 13 are respectively assumed to be an amplitude matrix and a phase matrix which are two-dimensional matrices.

[0352] Each axis of the amplitude matrix represents the frequency axis, and each element represents the value of the amplitude. In the reactivation layer 14, a reduced micro-matrix Lr is generated, where N and M are set to N≥2 and M≥1 respectively, and in the micro-matrix Lr, the frequency axis component, i.e., the frequency component, is 1 / N and the amplitude of each element is 1 / M. The micro-matrix Lr is added to the original amplitude matrix. The matrix generated after the addition is used as the output of the reactivation function f (updated amplitude matrix).

[0353] In addition, regarding the phase matrix, a similar operation of generating a small matrix Lθ and adding them is performed, and the matrix generated after the addition is used as the output of the complex activation function f (the updated phase matrix).

[0354] In addition, the complex activation function f is a nonlinear function in the spatial frequency domain.

[0355] By using such a complex activation function f, the signal components of the input second amplitude signal 22r and the second phase signal 22θ can be effectively concentrated on lower frequency components. When the pooling layer described later is used after using the complex activation function f, the degradation of the main information can be prevented in the pooling layer and the information can be concentrated on the low-frequency components.

[0356] In addition, the pooling layer is disposed at the rear stage of the reactivation layer 14 and is a low-pass filter in electrical engineering, more generally a filter.

[0357] In addition, although the calculation of the reactivation function f in the third embodiment requires more calculation time than the calculation of the reactivation function f shown in the first or second embodiment, it is possible to perform calculations with higher accuracy without degrading information.

[0358] The calculation of the complex activation function f of the third embodiment is combined with the error back propagation algorithm in the learning of the NN. When the operation is repeated multiple times between the upper layer and the lower layer, each element of each matrix is ​​calculated, so the number of calculations is huge. Therefore, it is effective to set N and M to powers of 2 and use shift operations to reduce the calculation weight when generating small matrices Lr and Lθ.

[0359] In the shift operation, it is not necessary to set the number to decimal, but to perform a 2-bit operation which is convenient for calculation, so the calculation cost is low in the operation using the von Neumann computer. For example, in a compiled language such as C language, the calculation cost is about 1 / 10. For operations such as 1 / 2, 1 / 4, 1 / 8, etc., which are raised to the power of 1 / 2, it is sufficient to perform a right shift bit operation.

[0360] As a specific calculation example, the case where N=2 and M=1 will be described below.

[0361] In this case, each element of the amplitude matrix and phase matrix forming the second amplitude signal 22r and the second phase signal 22θ is thinned out one by one. As a result, the size of the rows and columns of each matrix becomes 1 / 2 and becomes a small matrix Lr, Lθ. For the small matrices Lr and Lθ whose sizes have been reduced, the high-frequency components are filled with 0 and processed into the same size as the amplitude matrix and phase matrix before thinning out. By adding the processed small matrices Lr and Lθ to the matrix amplitude matrix and phase matrix before thinning out, the output signal of the simplest complex activation function f can be generated.

[0362] Furthermore, a plurality of small matrices Lr and Lθ may be used respectively, and not only small matrices of 1 / 2 but also small matrices Lr and Lθ of 1 / 4, 1 / 8, etc. may be added.

[0363] In addition, in the present embodiment, since a discrete signal is processed, a mantissa is generated in a case other than M=1. In this case, a method of rounding the mantissa or a method of estimating a value by interpolation is used.

[0364] In the third embodiment, similarly to the first embodiment, the amount of calculation can be greatly reduced by replacing the convolution operation that requires a large amount of calculation with a complex convolution operation based on a single matrix operation. Furthermore, by using the complex activation function f, if the Fourier transform and the inverse Fourier transform in the hidden layer are each performed once, any number of complex convolution operations and the complex activation function f can be used in the continuous processing in the spatial frequency domain between them, and the amount of calculation required for the Fourier transform and the inverse Fourier transform can also be reduced. Therefore, high-speed processing can be performed even for large images, and high recognition accuracy based on NN can be obtained through the low-cost and simple hardware 100 structure.

[0365] In addition, all operations in each layer between Fourier transform and inverse Fourier transform are operations that continuously process spatial frequency signals in the spatial frequency domain, and the physical functions of each operation can be analyzed based on the input signal and output signal of each operation. Based on this, a detailed analysis of the NN can be performed, the structure of the NN can be determined based on the analysis results of each layer, and a high-performance NN can be constructed.

[0366] Implementation method 4.

[0367] In the fourth embodiment, a reactivation function f different from that in the first to third embodiments is used. The other configurations are the same as those in the first embodiment.

[0368] In this fourth embodiment, similar to the above-mentioned first embodiment, the second amplitude signal 22r and the second phase signal 22θ output from the amplitude connection layer 13A and the phase connection layer 13B are input to the complex activation layer 14, and these signals are operated using the complex activation function f to update the second amplitude signal 22r and the second phase signal 22θ and output them.

[0369] Furthermore, although the second amplitude signal 22r and the second phase signal 22θ are updated in the same manner by the same method, the second phase signal 22θ may be maintained as it is and only the second amplitude signal 22r may be updated.

[0370] By the way, in Fourier transform, when g and h are set to arbitrary functions, the following equation holds: Where F is Fourier transform, * is convolution operation, and ◎ is Hadamard product.

[0371] F[g◎h]=F[g]*F[h]

[0372] That is, the Hadamard product in the spatial domain is the convolution in the spatial frequency domain. At this time, F[g] is taken as the input signal, F[h] is considered as the kernel, and the input signal F[g] is convolved with F[h] as the kernel, thereby applying it as an operation of the complex activation function f.

[0373] However, when F[g] is multiplied by an arbitrary kernel F[h] in the spatial frequency domain, the information of the input signal F[g] is degraded. That is, the information of the frequency with large amplitude, which is the dominant frequency component constituting g, is lost. Therefore, by selecting a function with the maximum amplitude (absolute value) at the origin of the kernel F[h], the degradation of information can be prevented.

[0374] An example of such a function is the sinc function. When x is the frequency, the sinc function is represented by (sin(x) / x) and reaches a maximum value at x=0.

[0375] In this way, the complex activation function f uses the function with the largest absolute value at the origin as the reference as the kernel F[h] and performs convolution operation on the input signal (object signal). Thus, a complex activation function f with the same accuracy as the activation function in the spatial domain is obtained.

[0376] The second amplitude signal 22r is convolved using this F[h], and the second phase signal 22θ can be convolved in the same manner.

[0377] In addition, it is preferred that the origin of the kernel F[h] has both positive and negative values ​​in addition to the maximum amplitude. That is, it is preferred that the function be a function that crosses the axis of the kernel F[h] whose amplitude is 0. Accordingly, there is a point where the amplitude of the result (F[g]*F[h]) after the convolution operation is 0, and this point where the amplitude is 0 becomes information in the NN.

[0378] Furthermore, since the output of the second amplitude signal 22r must be a positive real number, it is preferable to convert the output (F[g]*F[h]) after the convolution operation into an absolute value.

[0379] In addition to the above-mentioned sinc function, calculation can also be performed in the spatial frequency domain of the Relu function. However, since Relu(x), which is a common Relu function, increases in proportion to x, the value of the Fourier transform diverges. Therefore, when the axial size (number of pixels) of the image is set to x0, F[h] can be expressed by the following equation (25).

[0380] [Mathematical formula 25]

[0381]

[0382] In this case, when performing calculations of the error back propagation algorithm, the amount of calculations is huge.

[0383] In addition, the sigmoid function represented by the following equation (26) is a convergent function in the spatial frequency domain.

[0384] [Mathematical formula 26]

[0385]

[0386] Therefore, F[h] can be expressed as

[0387] F[h](x)=-jπ·csch(πx)

[0388] To calculate the function.

[0389] In addition, csch(πx) may be a function represented by cosech(x). However, since it is a function that diverges at x=0, it is rounded to a finite value in actual calculation.

[0390] In the present fourth embodiment, a complex activation function f is also used, and thus if the Fourier transform and the inverse Fourier transform in the hidden layer are each performed once, then all the operations of each layer between the Fourier transform and the inverse Fourier transform are operations for continuously processing spatial frequency signals in the spatial frequency domain. Therefore, the physical function of each operation can be analyzed based on the input signal and output signal of each operation. Based on this, a detailed analysis of the NN can be performed, the structure of the NN can be determined based on the analysis results of each layer, and a high-performance NN can be constructed.

[0391] Furthermore, in the fourth embodiment, the complex activation function f performs convolution operation using a kernel in the spatial frequency domain. Therefore, it is possible to perform an operation theoretically equivalent to the CNN of the first comparative example and obtain the same accuracy, but the calculation speed is not improved.

[0392] However, by executing the CNN method using operations that continuously process spatial frequency signals in the spatial frequency domain, a detailed analysis of the CNN as a conventional method can be performed.

[0393] For example, the convolution operation (Hadamard product in the spatial frequency domain) is responsible for emphasizing edges (boundaries between objects). The activation function (complex activation function in the spatial frequency domain) is a nonlinear function and is responsible for generating frequency components different from the input signal (both in the low-frequency band and the high-frequency band). The pooling (complex pooling in the spatial frequency domain) described later plays the role of a filter represented by a low-pass filter. In this way, each operation can be clearly distinguished for analysis, which can contribute to the development of NN including CNN.

[0394] The analysis of operations in each layer of NN is explained in more detail.

[0395] In the convolution operation in the spatial domain, when the horizontal axis of the input image is set to the x-axis and the vertical axis is set to the y-axis, and the kernel is set to, for example, a two-dimensional signal obtained by differentiating a Gaussian distribution in the x-axis direction, by convolving the input image with the kernel, an image that emphasizes the edge of the y-axis component can be output. Similarly, by convolving the input image with a kernel obtained by differentiating a Gaussian distribution in an arbitrary direction, the image edges of the input image in all directions can be extracted.

[0396] In CNN, multiple kernels obtained through training are used to extract edges in various directions and synthesize them to analyze the input image. Image features are extracted by using convolution processing involving multiple operations in the spatial domain. Then, processing with the same physical meaning can be performed in the spatial frequency domain by complex convolution operation without using convolution operation using kernels.

[0397] Next, analysis of the activation function (complex activation function in the spatial frequency domain) will be described.

[0398] In the past, it was known through research on neurocognitive machines or CNN that in spatial domain processing, a structure similar to the discharge of nerve cells in the brain can be expressed by nonlinear functions.

[0399] However, in the analysis of the activation function according to the present embodiment, when combined with the pooling function (complex pooling in the spatial frequency domain) described later, the following result can be derived: the most important role of the activation function in deep learning is to generate frequency components different from the input signal.

[0400] For example, the Relu function in the spatial domain can be attributed to the processing of half-wave rectification in the spatial frequency domain. That is, in the half-wave rectification processing for a triangular wave (usual image) with two or more different periods, part of the energy is transferred to the low-frequency component close to the DC component in addition to the frequency component of the original triangular wave. Therefore, it can be seen that the same effect can be obtained by outputting part of the signal to the low-frequency component side as shown in Implementation 3.

[0401] Furthermore, although the operation in the spatial frequency domain does not produce the same result as the operation in the spatial domain, the difference in the operation results can be adjusted by using training of the complex convolution operation.

[0402] Implementation method 5.

[0403] In the fifth embodiment, a re-pooling layer is provided in the NN of the above-mentioned first to fourth embodiments. The re-pooling layer 18 is a layer in the spatial frequency domain corresponding to the pooling layer 4 in the spatial domain. Fig.15 This is a diagram showing the structure of the NN of the fifth embodiment, which is similar to the structure shown in the first embodiment. Figure 4 The corresponding part of the detailed diagram.

[0404] Similar to the first embodiment, the amplitude connection layer 13A has a first weight matrix W1 and multiplies the first amplitude signal 21r by the first weight matrix W1 to output a second amplitude signal 22r. The phase connection layer 13B multiplies the first phase signal 21θ by a second weight matrix W2 to output a second phase signal 22θ.

[0405] In the reactivation layer 14, only the second amplitude signal 22r of the second amplitude signal 22r and the second phase signal 22θ is updated using the reactivation function f and output as the third amplitude signal 23r, and the second phase signal 22θ is output without being updated.

[0406] The re-pooling layer 18 is provided immediately after the reactivation layer 14, and performs calculation processing on the signal updated in the reactivation layer 14. In this case, since only the amplitude component is updated, the input third amplitude signal 23r is calculated and an amplitude signal 23ra is output.

[0407] In the third and fourth embodiments, the restacking layer 18 is provided, and when the reactivation layer 14 updates both the second amplitude signal 22r and the second phase signal 22θ, the restacking layer 18 performs calculation processing on both updated signals.

[0408] The operation of the pooling layer 4 in the spatial domain is called pooling, and the operation of the complex pooling layer 18 in the spatial frequency domain is called complex pooling.

[0409] Pooling in the spatial domain reduces the sensitivity of the position of the features extracted by the convolutional layer 2, and can recognize an image with the same feature quantity even if the position of the feature quantity of the object changes in the image. This means "blurring" of the image.

[0410] Applying “blurring” in the spatial frequency domain can be easily achieved by removing high-frequency components. High-frequency components are components generated when elements of adjacent pixels change sharply, so by removing high-frequency components in the spatial frequency domain, re-pooling can be achieved.

[0411] The operation in the re-pooling layer 18, i.e., re-pooling, is equivalent to a low-pass filter in the field of signal processing. Then, by performing re-pooling, a structure that is robust to positional deviation and rotation of the input image is formed, over-learning of the NN is prevented, and calculation accuracy is improved.

[0412] Usually in the pooling in the spatial domain, a map is cut for each image size of S×T (S, T is an integer greater than 2), and the maximum value pooling that outputs the maximum value or the average value pooling that outputs the average value is usually used. In addition, Lp pooling is known. It can be considered that the difference in these poolings represents the difference in the shape of the low-pass filter in the spatial frequency domain.

[0413] In the spatial frequency domain, consider a matrix of amplitude signals and phase signals that have been subjected to the following frequency shift: the frequency shift causes the low-frequency components to be in the center of the matrix and the high-frequency components to be in the periphery of the matrix. The function used for complex pooling of these matrices is preferably a two-dimensional Gaussian distribution. This is because, according to the Gaussian distribution, the shape of the filter does not change even if Fourier transform is performed, so there is no need for a filter operation to suppress the generation of high-frequency components that occur during Fourier transform.

[0414] In addition, for complex pooling, a window function obtained by transforming a window function such as a Hann window, a Hamming window, a Hanning window, a Blackman window, a Kaiser window, etc. in accordance with the dimension of the input signal of each hidden layer can also be used. However, when a complex pooling function such as a flat-top window that generates a negative component is calculated for an amplitude signal, the calculation result is converted to an absolute value to avoid generating a negative component. Using this window function, discontinuities when performing Fourier transform and inverse Fourier transform can also be removed, so that noise components generated by the influence of numerical processing can also be removed.

[0415] In addition, it is possible to remove monochromatic information such as background components by using a high-pass filter that removes information that is offset by monochromatic, that is, low-frequency components close to direct current. Specifically, the signals of several pixels in the center are removed for the frequency-shifted signal. In this case, a high-pass filter that suppresses less than 10% of the pixels in the center is usually set. In addition, this method cannot be used for data in which the direct current component is an important factor.

[0416] In actual NN design, it is preferable to use a bandpass filter in which the high-pass filter and the low-pass filter are combined.

[0417] As an example of a bandpass filter, there is a method of using a Gabor filter which is a function represented by the product of a trigonometric function and a Gaussian function as shown by David Hubel and Torsten Wiesel. In addition, an arbitrary bandpass filter may be used by combining a lowpass filter and a highpass filter.

[0418] By the way, the complex activation function f shown in the above-mentioned embodiment 3 or 4 or the activation function used in CNN has the meaning of generating frequency components lower than a specific frequency for NN.

[0419] For example, when the amplitude is negative in CNN, the output of the activation function (Relu function) is 0. The output of 0 means the frequency is 0, that is, the DC component in the spatial frequency domain, and it means half-wave rectification in electrical engineering. In addition, at a specific frequency, in half-wave rectification, the frequency changes continuously from the DC component to the specific frequency, so even if the input signal to the activation function is a single frequency, the output signal has a wide-band frequency component.

[0420] In the fifth embodiment, by providing the complex pooling layer 18 after the complex activation function f is calculated, it is possible to effectively aggregate the main information in the low-frequency component.

[0421] The aggregation of information described here means that high-frequency components are removed using a low-pass filter (eg, a Gaussian filter).

[0422] In the NN having multiple hidden layers, it is preferable that the re-pooling layer 18 is provided not only immediately after the re-activation layer 14 but also in the previous stage of the re-activation layer 14 .

[0423] In addition, when the above-mentioned fourth embodiment is applied, pooling (complex pooling in the spatial frequency domain) can also be analyzed, which is described below.

[0424] As mentioned above, pooling in the spatial domain is roughly divided into maximum pooling and average pooling. For example, it is an operation to cut an image into 2×2 bits and output the maximum value or average value among the 2×2 bits. This operation has a blurring effect. In addition, by blurring, it is possible to produce an effect that the same image can be recognized even if the position of the input image is deviated or rotated.

[0425] On the other hand, it can be seen that when the complex pooling is analyzed in the spatial frequency domain by applying the above-mentioned embodiment 4, the pooling operation is a low-pass filter. In addition, it can be seen that the maximum pooling and the average pooling only differ in the sharpness of the edge at the cutoff frequency of the filter.

[0426] Therefore, although pooling in the spatial domain has an unclear physical meaning, complex pooling can function as an explicit filter in the spatial frequency domain. Based on this, complex pooling can construct a deep learning model with higher inference accuracy than filters in the spatial domain. Furthermore, since arbitrary filters such as bandpass filters that only remove DC components and high-frequency components can be constructed in addition to low-pass filters, it is possible to construct a deep learning model with a high degree of freedom.

[0427] The fact that the complex pooling is a low-pass filter is closely related to the activation function (complex activation function in the spatial frequency domain). As mentioned above, the activation function has the effect of generating low-frequency components. When a low-pass filter is applied to it, it can also be formed to retain the low-frequency components generated by the activation function and remove the original high-frequency components of the input signal. However, this depends on the cutoff frequency of the low-pass filter, that is, the design of the low-pass filter.

[0428] When the above-mentioned Embodiment 4 is applied to the present Embodiment 5 to analyze the contents of CNN, the following results are obtained. CNN is a method that performs convolution with a filter obtained through training to extract the edge of an image, generates a signal in a low-frequency component using a nonlinear function as an activation function, retains frequency components that appear as low-frequency components using pooling, and removes other components to perform learning.

[0429] Implementation method 6.

[0430] In the sixth embodiment, a complex batch normalization layer is provided in the NN of the first to fifth embodiments. Fig.16 This is a diagram showing the structure of the NN according to the sixth embodiment, which is similar to that shown in the first embodiment. Figure 4 The corresponding part of the detailed diagram.

[0431] In this case, an example is shown in which the duplicated normalization layer 19 is provided between the connection layer 13 and the reactivation layer 14 , but it may also be provided at a subsequent stage of the reactivation layer 14 or a preceding stage of the connection layer 13 .

[0432] The situation where the distribution (mean and standard deviation) of the data used for learning is different during training and inference, that is, the situation where there is a deviation is called covariate shift. In NN, the covariate shift between hidden layers during training is called internal covariate shift, and the deviation of internal covariate shift will propagate. As the number of hidden layers of NN increases, the deviation propagates more, and the learning of the weights of the upper layers close to the output layer takes more time. In addition, depending on the situation, learning may not progress.

[0433] Therefore, in the spatial domain, a method called batch normalization is sometimes used to shorten the time taken for training. Batch normalization in the spatial domain performs the following operation: take the mean and standard deviation of the input (usually a matrix) of a hidden layer, subtract the mean from the input and divide it by the standard deviation.

[0434] The complex batch normalization layer 19 of the sixth embodiment is a hidden layer that performs complex batch normalization in the spatial frequency domain corresponding to batch normalization in the spatial domain.

[0435] According to complex batch normalization, by performing the same operation as batch normalization in the spatial domain only on the amplitude signal in the spatial frequency domain, the influence of the internal covariate shift can be reduced, and the time required for training can be shortened.

[0436] In this case, since the complex batch normalization layer 19 is arranged between the connection layer 13 and the reactivation layer 14, the complex batch normalization layer 19 performs complex batch normalization only on the second amplitude signal 22r output from the amplitude connection layer 13A and outputs an amplitude signal 22ra.

[0437] In addition, the disadvantage of calculation accuracy caused by setting the complex batch normalization layer 19 is small, so the complex batch normalization layer 19 can be set before or after the complex pooling layer 18, or before or after the hidden layers such as the amplitude logarithmization layer and the axis logarithmization layer described later.

[0438] In addition, when the computation time of the batch normalization layer 19 itself is affected and learning takes time, it is preferable to reduce the number of batch normalization layers 19 and deal with it by the following method. That is, it is dealt with by changing the complex activation function f, pre-learning the initial value of the weight matrix W, reducing the learning coefficient in the gradient descent method, and using a dropout layer or a sparse connection layer to constrain the degree of freedom of the NN.

[0439] Implementation method 7.

[0440] In the seventh embodiment, the NN of the first to sixth embodiments is provided with an amplitude logarithmization layer and an inverse amplitude logarithmization layer. Fig.17 This is a diagram showing the structure of the NN of the seventh embodiment, showing the case where the amplitude logarithmization layer and the inverse amplitude logarithmization layer are applied to the sixth embodiment described above. Fig.16 Some details of the corresponding NN.

[0441] like Fig.17 As shown, an amplitude logarithmization layer 10A is provided between the Fourier transform layer 12 and the connection layer 13 , and an inverse amplitude logarithmization layer 10B is provided at a stage preceding the inverse Fourier transform layer 15 .

[0442] When the input signal 20 is Fourier transformed in the Fourier transform layer 12 and divided into the first amplitude signal 21r and the first phase signal 21θ, a signal having a large amplitude may be generated in the first amplitude signal 21r at a specific frequency on the frequency axis. In this case, due to the large amplitude signal, other components of the matrix of the amplitude signal may be almost zero, and the characteristics of the image may be lost, and only the large amplitude portion may be learned.

[0443] In the seventh embodiment, the amplitude logarithmization layer 10A calculates the logarithm of the amplitude of the first amplitude signal 21r (amplitude matrix), that is, generates and outputs an amplitude signal 21ra obtained by calculating the logarithm of the amplitude. This can suppress the adverse effects caused by a signal with a large amplitude generated after Fourier transformation and improve the reliability of learning.

[0444] When the input signal is x and the base is a, the output signal y based on the operation of finding the logarithm of the amplitude is:

[0445] y=log a x.

[0446] Here, the base a is usually the natural logarithm e, 2, or 10, but other real numbers may also be used.

[0447] In normal images, the size of the input signal sometimes differs by about 2 to 3 digits. However, when using, for example, base 10, even if there is a difference of 3 digits, it is only a 3-fold change, and learning can be performed in a way that is sensitive to signals with small amplitudes and insensitive to signals with large amplitudes.

[0448] Alternatively, the amplitude signal 21ra may be generated using the following calculation formula by multiplying the logarithm of the signal by a constant b which is a real number greater than 0 and using an error component δ for preventing the input signal from being 0.

[0449] y = b·log a (x+δ)

[0450] When the amplitude matrix does not have any element of 0, the error component δ does not need to be input. When the amplitude matrix has any element of 0, it is preferable to input a value that is one digit smaller than the minimum value other than 0.

[0451] The constant b is used to prevent the output value y from being smaller than the round-off error of the information processing device or from being too large, so 10 or 20 is often used, but other real numbers may also be used.

[0452] In the inverse amplitude logarithmization layer 10B provided at the preceding stage of the inverse Fourier transform layer 15, the amplitude (logarithmized amplitude) of the input amplitude signal (third amplitude signal 23r) is returned to the antilogarithm to generate and output the amplitude signal 23ra. When the input signal is x and the output signal is y, the following equation (27) or equation (28) is used as the operation.

[0453] [Mathematical formula 27]

[0454] y=a x ……(27)

[0455] [Mathematical formula 28]

[0456] y=a x / b -δ……(28)

[0457] Implementation method 8.

[0458] In the eighth embodiment, the NN of the first to seventh embodiments described above includes an axis logarithmization layer and an inverse axis logarithmization layer. Fig.18 This is a diagram showing the structure of the NN of the eighth embodiment, showing the case where the axis logarithmization layer and the inverse axis logarithmization layer are applied to the sixth embodiment described above. Fig.16 Some details of the corresponding NN.

[0459] like Fig.18As shown, an axis logarithmization layer 10C is provided between the Fourier transform layer 12 and the connection layer 13 , and an inverse axis logarithmization layer 10D is provided at the front stage of the inverse Fourier transform layer 15 .

[0460] The axis logarithmization layer 10C will be described below.

[0461] In this embodiment, for simplicity, input data for learning is also two-dimensional data. One axis of the two-dimensional data is called an X-axis, and the other axis is called a Y-axis.

[0462] The X-axis and Y-axis of the first amplitude signal 21r and the first phase signal 21θ after the Fourier transform in the Fourier transform layer 12 are antilogarithmic like the input data. The first amplitude signal 21r and the first phase signal 21θ are input to the axis logarithmization layer 10C, and the axis logarithmization layer 10C uses logarithms for the X-axis and Y-axis of the first amplitude signal 21r and the first phase signal 21θ, that is, generates and outputs the amplitude signal 21rb and the phase signal 21θb after the axis logarithmization.

[0463] In addition, the base only needs to be a real number greater than or equal to 0.

[0464] This allows the amplitude signal 21rb and the phase signal 21θb to be two-dimensional data with low-frequency components emphasized on the X-axis and the Y-axis. This can be considered as dynamic range expansion, and learning can be performed without missing small changes in low-frequency components. In addition, such a method cannot be used in previous CNNs.

[0465] In the present embodiment, it is possible to emphasize information on low-frequency components having a large amount of information and suppress information on high-frequency components having a small amount of information, thereby enabling efficient and highly reliable learning.

[0466] In the inverse axis logarithmization layer 10D provided at the front stage of the inverse Fourier transform layer 15, the X-axis and Y-axis of the input amplitude signal (third amplitude signal 23r) and the phase signal (second phase signal 22θ) are returned to antilogarithms to generate and output the amplitude signal 23rb and the phase signal 22θb. However, as long as the inverse Fourier transform can be performed in the subsequent stage inverse Fourier transform layer 15, the inverse axis logarithmization of returning the X-axis and the Y-axis to antilogarithms is not necessarily required.

[0467] The amplitude signal 21rb and the phase signal 21θb may become images with wide sampling intervals on the low-frequency side due to the axis logarithmization. In such a case, interpolation is performed. As interpolation, a well-known method such as linear interpolation, polynomial interpolation, and spline interpolation is used.

[0468] In addition, when the physical quantities on the X-axis and Y-axis are different, as in a spectrum diagram, only one axis may be logarithmized. In addition, even if the physical quantity is the same, but the learning angles on the X-axis and Y-axis are different, only one axis may be logarithmized.

[0469] Furthermore, when the eighth embodiment is applied to the seventh embodiment, when the amplitude is set as the Z axis, all the axes of the X axis, the Y axis, and the Z axis are logarithmized, that is, fully logarithmized. The amplitude logarithmization layer 10A and the inverse amplitude logarithmization layer 10B of the seventh embodiment can be combined with the axis logarithmization layer 10C and the inverse axis logarithmization layer 10D of the eighth embodiment to form a full logarithmization layer and an inverse full logarithmization layer that perform full logarithmization and inverse full logarithmization. After the full logarithmization layer, the rows, columns, and elements of the matrix of the amplitude signal (first amplitude signal 21r) are all logarithmic, and the rows and columns of the matrix of the phase signal (first phase signal 21θ) are all logarithmic.

[0470] Implementation method 9.

[0471] Fig.19 This is a diagram showing the structure of NN according to the ninth embodiment.

[0472] like Fig.19 As shown, the NN is constructed to include, starting from the input layer 11A, a connection layer 13 composed of an amplitude connection layer 13A and a phase connection layer 13B for performing complex convolution operations, a complex activation layer 14 for performing operations using a complex activation function f, a complex pooling layer 18, an inverse Fourier transform layer 15, an output layer 16 and an output 17.

[0473] Then, separately from the NN, there is a Fourier transform layer 12A, which is a calculation unit that performs Fourier transform in the preprocessing of the NN. The Fourier transform layer 12A performs Fourier transform on the input signal 20 in the spatial domain, and outputs the first amplitude signal 21r and the first phase signal 21θ as signals in the spatial frequency domain.

[0474] In the ninth embodiment, the input layer 11A of the NN receives the first amplitude signal 21r and the first phase signal 21θ, which are signals in the spatial frequency domain, as input signals and communicates with the connection layer 13 as an upper layer.

[0475] Therefore, Fourier transform is not required in NN, and repeated calculations in the Fourier transform layer are not required during learning. In addition, back propagation calculations are also not required, so the calculation time can be shortened.

[0476] In this case, an example including the re-pooling layer 18 is shown. This embodiment can also be applied to the NN shown in the above-mentioned embodiments 1 to 8 to separate the Fourier transform layer 12A from the NN, so that Fourier transform can be performed in the pre-processing of the NN.

[0477] Implementation method 10.

[0478] Fig. 20 This is a diagram showing the structure of NN according to the tenth embodiment.

[0479] like Fig. 20 As shown, the NN is composed of a convolution layer 2 for performing convolution operation, an activation function 3, a Fourier transform layer 12, a connection layer 13 composed of an amplitude connection layer 13A and a phase connection layer 13B for performing complex convolution operation, a complex activation layer 14 for performing operation using a complex activation function f, a complex pooling layer 18, an inverse Fourier transform layer 15, an output layer 16, and an output 17. This is an example in which the convolution layer 2 and the activation function 3 are inserted in the latter stage of the input layer 11 of the NN of the above-mentioned embodiment 5.

[0480] The input signal 20, which is a spatial signal, is convolved using a kernel in the spatial domain at the convolution layer 2 and then operated using the activation function 3. The spatial signal after the operation based on the activation function 3 is divided into a first amplitude signal 21r and a first phase signal 21θ at the Fourier transform layer 12. The subsequent operations are the same as those in the fifth embodiment described above.

[0481] In the present embodiment, the convolution layer 2 and the activation function 3 as a hidden layer in the spatial domain are provided before the Fourier transform layer 12. Therefore, a NN combining the spatial domain and the spatial frequency domain is formed.

[0482] For example, for an input signal 20 composed of a combination of RGB, a method of dividing the input data according to each color in the input layer is used in CNN. In the case of a color image, it is usually a three-dimensional shape in the channel direction as the RGB color dimension in addition to the vertical and horizontal information. The three-dimensional shape contains spatial information such as many similar values ​​between pixels close in space. In addition, the three-dimensional shape sometimes includes essential information of the image such as a close relationship between each channel of RGB, or pixels far apart are almost unrelated to each other.

[0483] Convolutional layer 2 is able to extract and maintain this information.

[0484] In this embodiment, it is possible to use methods unique to the spatial frequency domain, such as complex convolution operations, complex activation functions f, and complex pooling, and to perform calculations that cannot be performed in the spatial domain. In particular, it is difficult to ascertain the local and global changes in an image in the spatial domain, but in this embodiment, learning and reasoning can be performed while maintaining this information. With respect to the physical positional relationship of the feature quantities lost due to Fourier transform, information can be retained by setting the convolution layer 2 and the activation function 3 near the input layer 11. Therefore, learning and reasoning can be performed with high precision.

[0485] In the above example, only one convolution layer 2 and one activation function 3 are provided, but NN is usually constructed by multi-layering, and a CNN having a small number of layers of about 1 to 5 convolution layers 2 is inserted between the input layer 11 and the Fourier transform layer 12 to construct the overall NN. Then, between the Fourier transform layer 12 and the output layer 16, training is performed in the same manner as in the above-mentioned first embodiment.

[0486] Based on this, the speed can also be increased for multi-dimensional input data above three dimensions.

[0487] In addition, compared with the first comparative example of CNN in the spatial domain only, the calculation is lightweight, so more images can be processed by one learning. Therefore, when the amount of calculation is considered to be the same, higher accuracy can be expected compared with the case of using the first comparative example.

[0488] In the above-mentioned embodiment 10, it is shown that the convolution layer 2 and the activation function 3 as the hidden layer in the spatial domain are arranged in the front stage of the Fourier transform layer 12, but the convolution layer 2 and the activation function 3 can also be arranged between the inverse Fourier transform layer 15 and the output layer 16, and can also be arranged in both the front stage of the Fourier transform layer 12 and the back stage of the inverse Fourier transform layer 15.

[0489] Implementation method 11.

[0490] In the present eleventh embodiment, an electronic device that performs a control operation using the NN of the information processing device described in each of the first to tenth embodiments will be described.

[0491] The information processing device of the present application is used to process sensor information mounted on an air conditioner, sensor information of a servo system used in a factory for factory automation, or sensor information installed inside or outside a vehicle. In the past, in order to use CNN in these processes, it was necessary to prepare a GPU, ASIC (application specific integrated circuit) or FPGA dedicated to neural network processing. On the other hand, according to this embodiment, information from air conditioners, servo systems, vehicle-mounted sensors, etc. can be processed by general-purpose hardware including existing CPU microcomputers and memories.

[0492] Fig.21 1 is a diagram showing a configuration example of an air conditioner as an electronic device according to Embodiment 11. The air conditioner includes an information processing device 50 of the present application, an infrared sensor 51 for identifying an object 58, and an air supply unit 52. The information processing device 50 includes an input unit 53 for inputting data, an analysis unit 54 for analyzing the data input from the input unit 53, a storage unit 55 for recording the analysis result, a determination unit 56 for determining the analysis result as the object 58, and a control unit 57 for controlling each unit.

[0493] The information processing device 50 controls at least the wind direction, wind volume, and temperature of the air supply unit 52 according to the control unit 57 .

[0494] In this case, the analyzing unit 54, the determining unit 56, and the controlling unit 57 as a whole have Figure 1 The storage unit 55 has the functions of the CPU 30 of the hardware 100 shown in the figure, and has the functions of the ROM 31 and the RAM 32 of the hardware 100.

[0495] Moreover, the information processing device 50 has: a learning process, which uses the output signal of the infrared sensor 51 as an input signal to learn the position and temperature changes of the biological body according to the input signal; and a reasoning process, which performs reasoning based on the input signal according to the information obtained through the learning process, and the information processing device 50 performs control work based on the reasoning process.

[0496] Next, the operation of the information processing device 50 using the NN will be described.

[0497] In the information processing device 50, the NN has a process for inferring the position of the organism and the temperature change of the organism based on the input signal from the infrared sensor 51. The inference of the position of the organism is performed by detecting the temperature of the organism using the information processing device 50. In addition, the information processing device 50 performs processing to predict the temperature suitable for each organism based on the temperature change of the organism.

[0498] Before using the air conditioner, the following methods are used: a method of making the NN learn in advance and storing the learning result of the NN, that is, the weight matrix W, in the storage unit 55, or a method of learning a part of the weight matrix W of the NN in accordance with the environment after the air conditioner is installed. When making the NN learn in advance, the information processing device 50 only has the inference process of the learning process and the inference process, and calls the weight matrix W in the storage unit 55 during inference to calculate the input signal from the infrared sensor 51. When learning a part of the weight matrix W in accordance with the environment, the training of the NN described in the above-mentioned embodiments 1 to 10 is performed.

[0499] In particular, in the case of an air conditioner, the correct answer label is set to, for example, data that grasps the structure of the house, and a mechanism that grasps the distance between each organism and the air conditioner, or the temperature and air volume suitable for each organism is set to control. Regarding the structure of the house, for example, those with constant temperature are identified as houses, furniture, etc., and those with changing temperature are identified as doors, organisms, etc.

[0500] In addition, regarding the temperature and air volume suitable for each organism, a mechanism is provided to grasp the position of the controller (for example, a component absorbing or reflecting infrared rays is provided at the front end of the controller), and information when each organism operates the controller of the air conditioner is used as the correct answer label. At this time, NN is used to extract and recognize feature quantities such as movement, body temperature, and contour of different organisms.

[0501] In addition, NN can also be used to determine whether it is a living body or an object such as a cooker based on the temperature change when cold air or warm air is blown to the object 58. Furthermore, NN can also be used to identify each living body to determine whether to blow air / not blow air to each living body, and in addition, it can be determined that the cooker is not blowing air.

[0502] The learned NN weight matrix W is stored in the storage unit 55, and is read from the storage unit 55 and used when the air conditioner is started next time.

[0503] In addition, before inputting into NN, preprocessing such as normalizing the image data so that the maximum value is 1 and the minimum value is 0, standardizing the image data so that the mean is 0 and the variance is 1, batch learning can be performed only at startup, or online learning can be used to perform learning at any time during startup according to the situation.

[0504] In addition, Fig.21In the embodiment, the air supply unit 52 of the air conditioner is set to one, but a plurality of air supply units 52 may be provided. For example, two air supply units 52 capable of outputting air at different temperatures and air volumes may be provided, and a mechanism for delivering air at a temperature and air volume suitable for each organism may be provided according to the processing result of the NN. In addition, the two air supply units 52 may be used to appropriately mix the air in the room for temperature adjustment.

[0505] In addition, the learning of NN can also identify each organism, so that the information can be combined to form an air conditioner with various functions. In addition, even if multiple organisms exist at the same time, each air supply unit 52 can fix the blades that adjust the wind direction and continuously send out wind with a temperature and air volume suitable for each organism.

[0506] Generally speaking, in order to identify objects, people, door switches, etc. in the room, infrared sensors and further distance sensors are used in air conditioners. This system becomes a NN that uses sensor signals of infrared sensors as input signals and objects, people, door switches, etc. in the room as output signals. In recent years, due to the improvement of the performance of infrared sensors, the amount of information (number of pixels) obtained has increased, and real-time response is required. Therefore, correspondingly, the large-scale and high-powered hardware for processing information is becoming indispensable.

[0507] According to the air conditioner of this embodiment, it is possible to suppress the increase in size and power consumption and speed up the processing, thereby reducing power consumption and eliminating the need for an additional structure for heat dissipation, thereby achieving a smaller indoor unit and higher efficiency.

[0508] Furthermore, when large-scale calculations are required, the following operations have been performed in the past: the calculations are performed via computers such as computer servers that are good at large-scale calculations through information networks such as the WWW, and the results of the computers are returned to the air conditioner through the WWW. According to the air conditioner of this embodiment, it is not necessary to use information networks such as the WWW, and the power consumption and time delay of these communications can be reduced, and a more comfortable air conditioner system can be provided to users.

[0509] Fig. 22 1 is a diagram showing the structure of an electronic device constituting a servo system as another example of an electronic device of the present embodiment 11. The electronic device includes an information processing device 50 of the present application, a sensor 51a for detecting electromagnetic waves, and a working unit 52a. The information processing device 50 includes an input unit 53 for inputting data, an analysis unit 54 for analyzing the data input from the input unit 53, a storage unit 55 for recording the analyzed result, a determination unit 56 for determining the analyzed result, and a control unit 57 for controlling each unit.

[0510] The information processing device 50 controls at least one of stopping the operation of the operating portion 52 a and removing foreign matter by the control unit 57 .

[0511] In this case, the analyzing unit 54, the determining unit 56, and the controlling unit 57 as a whole have Figure 1 The storage unit 55 has the functions of the CPU 30 of the hardware 100 shown in the figure, and has the functions of the ROM 31 and the RAM 32 of the hardware 100.

[0512] Moreover, the information processing device 50 has: a learning process, which takes the output signal of the sensor 51a as an input signal and learns at least one of the position change, electric field, magnetic field, and temperature of the object based on the input signal; and an inference process, which performs inference based on the input signal according to the information obtained through the learning process, and the information processing device 50 performs control work based on the inference process.

[0513] In this case, the NN can also be trained in advance, and the information processing device 50 has at least an inference process among the learning process and the inference process, and the weight matrix W in the storage unit 55 is called out for use during inference.

[0514] Generally speaking, in an information processing device installed in a servo system, the position of an item produced using the servo system is monitored by using NN calculations, and the characters, colors, barcodes, and defects recorded on the items are analyzed. The NN is input with a signal read by a CCD or CMOS image sensor such as a camera, a near-field antenna, or a far-field antenna.

[0515] Since the servo system works at high speed, instantaneous judgment is required. In the past, CNN has been used to reduce the search range and miniaturize the image in order to increase the speed. By using the electronic device of this embodiment for the servo system, the same processing speed can be maintained for large images, so that a wide range of information can be processed with fewer sensors, and a NN with a large number of layers can be used to achieve improved accuracy.

[0516] In the servo system of this embodiment, it is not necessary to use an information network such as the WWW, and the power consumption and time delay of these communications can be reduced, so that a servo system that responds more quickly can be provided to users.

[0517] For the servo system, real-time performance is important because it is required to control multiple servo motors in linkage. In addition, it is necessary to quickly determine the situation when the operation is abnormal, and stop and restart as needed. In the servo system of this embodiment, due to the rapid response, it is possible to quickly avoid manufacturing and working in abnormal times. Therefore, the waste of waste can be reduced, and the failure caused by collision between devices due to the device working in an abnormal posture can be reduced.

[0518] In addition, as an electronic device constituting the above-mentioned servo system, there is an autonomous robot as shown below. The autonomous robot has at least one of a CCD, a CMOS image sensor such as a camera, a near-field antenna, and a far-field antenna as a sensor 51a, and can monitor the position of an object and recognize and determine at least one of a character, a color, and a barcode recorded on the object.

[0519] The sensor 51a is directly mounted on the autonomous robot or mounted around it. The information processing device 50 has: a learning process, based on the input signal input from the sensor 51a, to learn the presence or absence of defects of characters, colors, barcodes, etc. to which the noise of the sensor 51a itself or the noise depending on the use environment of the sensor 51a is applied; and an inference process, based on the information obtained through the learning process, inference is performed based on the input signal, and the information processing device 50 performs control work based on the inference process.

[0520] In this case as well, the information processing device 50 only needs to have at least the inference process among the learning process and the inference process. In this case, the weight matrix W in the storage unit 55 is called out and used during inference.

[0521] In addition, as an electronic device having the information processing device 50 of the present application, there is also a collision prevention device as a vehicle-mounted device for automatic driving of a vehicle. Fig. 22 The situation shown is the same, and the same effect is obtained, namely, low power and quick response.

[0522] The collision prevention device includes an information processing device 50 of the present application, a sensor 51a, and an operating unit 52a. The information processing device 50 includes an input unit 53 for inputting data, an analysis unit 54 for analyzing the data input from the input unit 53, a storage unit 55 for recording the analyzed result, a determination unit 56 for determining the analyzed result, and a control unit 57 for controlling each unit.

[0523] The sensor 51a detects the external environment information as information for controlling the operation, and uses at least one of a CCD such as a camera, a CMOS image sensor, a radar, and an ultrasonic sonar. In the case of a radar, a radar using a laser beam such as a laser radar (Light Detection and Ranging), a radar using a millimeter wave, etc. are used. In addition to a single sensor, a combination of different sensors or a plurality of the same sensors can also be used to improve the safety during driving.

[0524] The information processing device 50 includes: a learning process for learning at least one of the position of a living body outside the vehicle, the position and traveling direction of surrounding vehicles, traffic light information, and lane information based on the input signal input from the sensor 51a; and an inference process for inferring based on the input signal according to the information obtained through the learning process, and the information processing device 50 performs control operations based on the inference process. In this case, at least the inference process is sufficient.

[0525] The collision prevention device can control the steering wheel, accelerator and brake, and as the output from the control unit 57 of the information processing device 50, it is roughly classified into two types: issuing a warning to the driver in an abnormal situation, or controlling the vehicle without issuing a warning to the driver. As a specific example of abnormal situation, there is a situation where the vehicle is detected to deviate from the lane, the distance between the vehicle and the vehicle in front is close when driving, and there is a person or object in the vehicle's forward direction (front and back, left and right) and a collision is expected, etc., and a warning is given to the driver through sound, light or vibration to convey the abnormality. When it is determined that it is too late to warn the driver due to the high speed of the vehicle, the working unit 52a directly controls the vehicle according to the instruction from the control unit 57 of the information processing device 50. Specifically, the information processing device 50 electrically controls the steering wheel to avoid deviating from the lane, drives the accelerator or brake to control the distance with the vehicle in front or objects in front, back, left and right, etc. In addition, the control such as grasping the situation around the vehicle to stop the vehicle in a safe position is performed.

[0526] In addition, the above-mentioned collision prevention device only has the sensor 51a for detecting the external environment information of the vehicle, but a sensor such as a camera based on visible light or infrared rays can also be installed in the vehicle to judge the physical condition of the driver to detect abnormalities. In this case, the output signal of the sensor in the vehicle is also used as an input signal, and the information processing device 50 is used for calculation to judge the physical condition of the driver, such as sleeping state, drinking state, etc. Based on this judgment, a warning is given to the driver through sound, light or vibration, or the condition around the vehicle is grasped to stop the vehicle in a safe position.

[0527] In addition, as another example of an electronic device having the information processing device 50 of the present application, the following is a vehicle-mounted device that can identify and authenticate the driver to prevent vehicle theft, driving by an unlicensed driver, etc. Fig. 22 The situation shown is the same.

[0528] The sensor 51a is mounted in the vehicle, detects in-vehicle environment information as information for controlling the operation, and uses at least one of a CCD or CMOS image sensor such as a camera.

[0529] The information processing device 50 includes: a learning process for learning at least the face of the driver's face and body shape based on the input signal input from the sensor 51a; and an inference process for inferring based on the input signal according to the information obtained through the learning process, and the information processing device 50 performs control work based on the inference process.

[0530] In this case, the information processing device 50 has both a learning process and an inference process, and can learn a specific driver, identify and authenticate the driver, and set it to allow only the driver to operate the vehicle. However, considering that the face and body shape of a person change with age, online learning is used to add the data to learn every time the data as a learning object is added, and the information is updated regularly.

[0531] According to the in-vehicle device of this embodiment, it is not necessary to use an information network such as the WWW, and the driver can be quickly identified and authenticated using an energy-saving, simple and inexpensive structure regardless of the connection environment with these networks.

[0532] Furthermore, a surveillance camera can also be applied as an electronic device having the information processing device 50 of the present application. For example, an image of visible light from a camera as a sensor 51a is input to the information processing device 50, and the information processing device 50 outputs, for example, an ID assigned to each individual (in Japan, a personal number (My Number), a passport number, etc.). In addition, the behavior of each individual can be monitored in real time using a surveillance camera.

[0533] In this embodiment, it is not necessary to upload the images of the surveillance camera via an information network such as the WWW, and the amount of information can be reduced by performing distributed processing by the information processing device 50. For example, each surveillance camera previously processed 1MB / s of information, but in this embodiment, only 1kB / s of information is required, which can significantly reduce the amount of information. In this way, it is possible to easily and reliably monitor personal behavior.

[0534] In the above-mentioned vehicle-mounted equipment for identifying and authenticating the driver and the electronic devices such as the surveillance cameras for monitoring personal behavior, the input signal read by the CCD or CMOS image sensor is used, and the behavior, state, etc. of the person are used as the output signal. When the image is used as the input information, the multi-layered NN can increase the speed and improve the accuracy. In this case, compared with the above-mentioned electronic devices such as the air conditioner or servo system, there are fewer restrictions on the use of electricity and computers, and by using the electronic devices of this embodiment, the speed and accuracy can be further increased. In particular, by using an electronic device to which the NN of embodiment 10 is applied, the accuracy is effectively improved by using operations in both the spatial domain and the spatial frequency domain.

[0535] In addition, Fig.21and Fig. 22 , an electronic device having sensors 51 and 51a is shown, but the information processing device 50 of the present application can also be applied to an electronic device without sensors 51 and 51a. In this case, a signal generated by calculation in the information processing device 50 is used as an input signal. In this case, at least the inference process of a learning process for learning based on the input signal and an inference process for inferring based on the input signal according to information obtained through the learning process is provided, and the control operation is performed based on the inference process.

[0536] In addition to the above contents, the information processing device of the present application can also be applied to electronic devices that infer at least one of characters, colors, light intensity, size, shape, position, angle, position change of an object caused by speed or acceleration, electric field, magnetic field, and temperature.

[0537] The present application describes various exemplary embodiments and examples, but various features, methods, and functions described in one or more embodiments are not limited to application to specific embodiments, but can be applied to the embodiments alone or in various combinations.

[0538] Therefore, numerous modifications not shown in the examples are conceivable within the technical scope disclosed in the present application, including, for example, the modification, addition or omission of at least one constituent element, and the extraction and combination of at least one constituent element with constituent elements of other embodiments.

Claims

1. An information processing device for processing an input signal using a neural network, wherein the input signal is an image, and the information processing device comprises: A Fourier transform layer, performing Fourier transform on the input signal, and outputting a first amplitude signal and a first phase signal; an amplitude connection layer that multiplies the first amplitude signal by a first weight matrix that updates the values ​​in the matrix through training, and outputs a second amplitude signal; a phase connection layer that multiplies the first phase signal by a second weight matrix that updates the values ​​in the matrix through training to output a second phase signal; a complex activation layer, which updates at least the second amplitude signal of the second amplitude signal and the second phase signal based on the values ​​in the matrix constituting the second phase signal, using a complex activation function f which is an activation function in the spatial frequency domain; as well as The inverse Fourier transform layer combines the second amplitude signal updated in the reactivation layer with the second phase signal to perform inverse Fourier transform.

2. The information processing device according to claim 1, wherein: At least one of the amplitude connection layer, the phase connection layer and the complex activation layer is provided between the Fourier transform layer and the inverse Fourier transform layer, and signal processing in the spatial frequency domain is continuously performed between the Fourier transform layer and the inverse Fourier transform layer.

3. The information processing device according to claim 1 or 2, wherein: The complex activation layer updates the value of the amplitude r(i) of the point at the same position as the point i in the matrix constituting the second amplitude signal according to the response of the complex activation function f to the value, i.e., the phase θ(i) of each point i in the matrix constituting the second phase signal, outputs the updated second amplitude signal, and outputs the second phase signal without updating.

4. The information processing device according to claim 1 or 2, wherein: The complex activation function f used in the complex activation layer uses a function with the largest absolute value at the origin as a reference as a kernel to perform a convolution operation on the object signal, wherein the object signal is at least the second amplitude signal among the second amplitude signal and the second phase signal.

5. The information processing device according to claim 3, wherein: The complex activation function f is a complex Relu function that updates the value of the amplitude r(i) in the second amplitude signal according to different responses when either the real axis component or the imaginary axis component of the phase θ(i) in the second phase signal is positive or 0 and when it is negative.

6. The information processing device according to claim 5, wherein: When the real axis component is positive or 0, (-π / 2)≤θ(i)<(π / 2), the complex activation function f maintains the value of the amplitude r(i). When the real axis component is negative, -π≤θ(i)<(-π / 2) or (π / 2)≤θ(i)<π, the complex activation function f changes the value of the amplitude r(i) to (r(i)・|sinθ(i)|) or (r(i)・sinθ(i)).

7. The information processing device according to claim 5, wherein: When the imaginary axis component is positive or 0, 0≤θ(i)<π, the complex activation function f maintains the value of the amplitude r(i). When the imaginary axis component is negative, −π≤θ(i)<0, the complex activation function f changes the value of the amplitude r(i) to (r(i)・|cosθ(i)|) or (r(i)・cosθ(i)).

8. The information processing device according to claim 3, wherein: The complex activation function f is a complex logistic function that updates the value of the amplitude r(i) in the second amplitude signal according to a constant response using the same calculation expression regardless of the magnitude of the phase θ(i) in the second phase signal.

9. The information processing device according to claim 4, wherein: The complex activation function f uses the sinc function as the kernel function, performs a convolution operation on the object signal, and then calculates the absolute value.

10. An information processing device for processing an input signal using a neural network, wherein the input signal is an image, the information processing device comprising: A Fourier transform layer, performing Fourier transform on the input signal, and outputting a first amplitude signal and a first phase signal; an amplitude connection layer that multiplies the first amplitude signal by a first weight matrix that updates the values ​​in the matrix through training, and outputs a second amplitude signal; a phase connection layer that multiplies the first phase signal by a second weight matrix that updates the values ​​in the matrix through training to output a second phase signal; A complex activation layer, using a complex activation function f as an activation function in the spatial frequency domain, generates a tiny matrix for an object signal and adds the tiny matrix to a matrix constituting the object signal, thereby updating the object signal, wherein the object signal is at least the second amplitude signal of the second amplitude signal and the second phase signal, in which N and M are respectively set to integers of N≥2 and M≥1, and the frequency component of the axial component of the matrix constituting the object signal is 1 / N, and each element is 1 / M; as well as The inverse Fourier transform layer combines the second amplitude signal updated in the reactivation layer with the second phase signal to perform inverse Fourier transform.

11. The information processing device according to claim 10, wherein: At least one of the amplitude connection layer, the phase connection layer and the complex activation layer is provided between the Fourier transform layer and the inverse Fourier transform layer, and signal processing in the spatial frequency domain is continuously performed between the Fourier transform layer and the inverse Fourier transform layer.

12. The information processing device according to claim 10 or 11, wherein: The complex activation function f uses a plurality of the tiny matrices to update the object signal.

13. The information processing device according to claim 10 or 11, wherein: The N and M are respectively powers of 2, and the micro-matrix is ​​calculated by performing a shift operation.

14. The information processing device according to claim 1 or 10, wherein: A re-pooling layer is provided immediately after the re-activation layer. The re-pooling layer is a low-pass filter or a band-pass filter for the signal updated by the re-activation function f in the second amplitude signal and the second phase signal.

15. The information processing device according to claim 1 or 10, wherein: An amplitude logarithmization layer for calculating the logarithm of the amplitude of the first amplitude signal is provided at a subsequent stage of the Fourier transform layer, and an inverse amplitude logarithmization layer for canceling the logarithmization is provided at a preceding stage of the inverse Fourier transform layer.

16. The information processing device according to claim 1 or 10, wherein: An axis logarithmization layer for logarithmizing the axes of the first amplitude signal and the first phase signal is provided at a subsequent stage of the Fourier transform layer.

17. An electronic device comprising the information processing device according to any one of claims 1 to 16 and performing a control operation, wherein: The electronic device includes a sensor for detecting information used for the control operation. The information processing device has at least the reasoning process among the learning process and the reasoning process, and performs the control work based on the reasoning process, wherein in the learning process, the output signal of the sensor is used as the input signal, and learning is performed based on the input signal, and in the reasoning process, reasoning is performed based on the input signal according to the information obtained through the learning process.

18. An electronic device comprising the information processing device according to claim 17, wherein: The electronic device is an air conditioner, which has an infrared sensor as the sensor and can control wind direction, wind volume and temperature. In the learning process, the position and temperature changes of the biological body are learned.

19. An electronic device comprising the information processing device according to claim 17, wherein: The electronic device can detect electromagnetic waves using the sensor and control at least one of stopping operation and removing foreign matter. In the learning process, at least one of the position change amount, electric field, magnetic field, and temperature of the object is learned.

20. An electronic device comprising the information processing device according to any one of claims 1 to 16 and performing a control operation, wherein: The information processing device has at least the reasoning process among the learning process and the reasoning process, and performs the control work based on the reasoning process, wherein in the learning process, the signal generated by the calculation is used as the input signal, and learning is performed based on the input signal, and in the reasoning process, reasoning is performed based on the input signal according to the information obtained through the learning process.

Images