Learning device, conversion device, learning method and program

Abstract

The one aspect of the present invention is a learning device comprising: a neural network that converts an acoustic time series, which is a time series representing sound, to feature quantity data expressed in a prescribed format requested by a downstream task; and an update unit that updates the neural network on the basis of a result of an execution of the downstream task using the feature quantity data, wherein the neural network comprises: a feature extraction unit that converts an inputted acoustic time series to an intermediate feature quantity tensor, which is a third-order tensor that represents feature quantities of the inputted acoustic time series and has time, frequency, and channel; and an intermediate network that executes a process of converting the representation of the intermediate feature quantity tensor to a representation of a second-order tensor having time and a direct product quantity, which is a quantity representing the direct product of the frequency and the channel, and a process of acquiring, as the feature quantity data, a one-dimensional vector representing, for each direct product quantity in the second-order tensor, a statistical quantity of each direct product quantity in a time axis direction.

Description

[Technical Field]

[0001] The present invention relates to a learning device, a conversion device, a learning method, and a program. [Background technology]

[0002] There is known a technology that uses machine learning to generate a mathematical model that converts an input acoustic time series into a predetermined format required by a downstream task. Here, the acoustic time series refers to a time series that represents sound, such as an acoustic signal. The downstream task refers to a process that uses feature data, and the content of the process may be any predetermined content. The feature data is information about the sound obtained based on the acoustic time series and expressed in a target format.

[0003] The downstream task may be, for example, a process of removing noise from a sound represented by input feature data, a process of synthesizing a sound represented by input feature data with a sound represented by another acoustic time series, or a process of classifying a sound represented by input feature data. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] S. Hershey et al., "CNN architectures for large-scale audio classification," 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017, pp. 131-135, doi: 10.1109 / ICASSP.2017.7952132. [Non-patent document 2] Kong et al., "PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition," in IEEE / ACM Transactions on Audio, Speech, and Language Processing, vol. 28, pp. 2880-2894, 2020, doi: 10.1109 / TASLP.2020.3030497. Summary of the Invention [Problem to be solved by the invention]

[0005] However, when downstream tasks are executed using the results of such mathematical models, the execution results may not satisfy a predetermined standard. For example, in the case of noise removal as described above, the noise may not be removed to a degree that satisfies the predetermined standard. Also, in the case of sound classification as described above, the classification accuracy may not satisfy the predetermined standard. In other words, the probability that the classification result is incorrect may be higher than the predetermined standard.

[0006] In view of the above circumstances, an object of the present invention is to provide a technique for reducing the frequency with which the result of execution of a downstream task does not satisfy a predetermined standard. [Means for solving the problem]

[0007] One aspect of the present invention is a learning device comprising: a neural network that converts an acoustic time series, which is a time series representing sound, into feature data, which is information about sound obtained based on the acoustic time series and expressed in a predetermined format required by a downstream task; and an update unit that updates the neural network based on a result of executing the downstream task using the feature data. The neural network comprises: a feature extraction unit that converts an input acoustic time series into an intermediate feature tensor, which is a third-order tensor indicating features of the acoustic time series and which has time, frequency, and channel; a non-equivalent second-order tensorization process that converts the representation of the intermediate feature tensor into a representation of a planar tensor, which is a second-order tensor having time and a Cartesian product, which is a quantity indicating the Cartesian product of frequency and channel; and a vectorization process that acquires, for each Cartesian product of the planar tensor, a one-dimensional vector indicating a statistical quantity in the time axis direction of each Cartesian product.

[0008] One aspect of the present invention is a neural network that converts an input acoustic time series into feature data, which is information about sound obtained based on the acoustic time series and expressed in a predetermined format required by a downstream task; and an update unit that updates the neural network based on a result of execution of the downstream task using the feature data. The neural network converts the input acoustic time series into an intermediate feature tensor, which is a third-order tensor that indicates features of the acoustic time series and has time, frequency, and channel. a feature extraction unit that converts the representation of the intermediate feature tensor into a representation of a planar tensor that is a second-order tensor having a Cartesian product, which is a quantity indicating a Cartesian product of frequency and channel, and time; and a vectorization process that acquires, for each Cartesian product of the planar tensor, a one-dimensional vector indicating a statistical quantity of each Cartesian product in the time axis direction as the feature data.

[0009] One aspect of the present invention includes a neural network that converts an acoustic time series, which is a time series representing sound, into feature data, which is information about sound obtained based on the acoustic time series and expressed in a predetermined format required by a downstream task; and an update unit that updates the neural network based on a result of execution of the downstream task using the feature data. The neural network includes a feature extraction unit that converts an input acoustic time series into an intermediate feature tensor, which is a third-order tensor indicating features of the acoustic time series and has time, frequency, and channel; a non-equivalent second-order tensorization process that converts the representation of the intermediate feature tensor into a representation of a plane tensor, which is a second-order tensor having a Cartesian product, which is a quantity indicating a Cartesian product of frequency and channel, and time; and an update unit that updates each of the plane tensors. and a vectorization process for acquiring, as feature data, one-dimensional vectors that indicate statistics of each Cartesian product quantity in the time axis direction for each Cartesian product quantity of the planar tensor. The learning method includes: a feature extraction step for converting an input acoustic time series into an intermediate feature tensor, which is a third-order tensor that indicates features of the acoustic time series and has time, frequency, and channel; a non-equivalent second-order tensorization process execution step for converting a representation of the intermediate feature tensor into a representation of a planar tensor that is a second-order tensor that has time and a Cartesian product quantity that indicates the Cartesian product of frequency and channel; and a vectorization process execution step for acquiring, as feature data, one-dimensional vectors that indicate statistics of each Cartesian product quantity in the time axis direction for each Cartesian product quantity of the planar tensor.

[0010] One aspect of the present invention is a program for causing a computer to function as the learning device described above.

[0011] One aspect of the present invention is a program for causing a computer to function as the above-described conversion device. [Effects of the Invention]

[0012] The present invention makes it possible to reduce the frequency with which the results of the execution of downstream tasks do not satisfy a predetermined standard. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an acoustic time series conversion system according to an embodiment. [Figure 2] FIG. 2 is an explanatory diagram illustrating an example of the configuration of a feature extraction network according to the embodiment. [Figure 3] FIG. 2 is a diagram illustrating an example of a hardware configuration of a learning device according to an embodiment. [Figure 4] FIG. 2 is a diagram showing an example of the configuration of a control unit included in the learning device according to the embodiment. [Figure 5] 10 is a flowchart showing an example of a flow of processing executed by a learning device according to an embodiment. [Figure 6] FIG. 2 is a diagram illustrating an example of a hardware configuration of a conversion device according to an embodiment. [Figure 7] FIG. 2 is a diagram showing an example of the configuration of a control unit in the embodiment. [Figure 8] 10 is a flowchart showing an example of a flow of processing executed by a conversion device according to an embodiment. [Figure 9] FIG. 10 is a first explanatory diagram illustrating an experimental result in the embodiment. [Figure 10] FIG. 10 is a second explanatory diagram illustrating the experimental results in the embodiment. [Figure 11] FIG. 10 is a third explanatory diagram illustrating experimental results in the embodiment. [Figure 12] FIG. 10 is a diagram showing an example of experimental results in a modified example. [Figure 13] FIG. 10 is a diagram showing an example of the configuration of a learning device including an acoustic conversion network according to a modified example. [Figure 14] 10 is a flowchart showing an example of the flow of processing executed by a learning device including an acoustic conversion network in a modified example. [Figure 15] FIG. 10 is a first diagram showing an example of experimental results in a modified example. [Figure 16] FIG. 2 is a second diagram showing an example of experimental results in the modified example. DETAILED DESCRIPTION OF THE INVENTION

[0014] (Embodiment) FIG. 1 is a diagram showing an example of the configuration of an acoustic time series conversion system 100 according to an embodiment. The acoustic time series conversion system 100 includes a learning device 1 and a conversion device 2. The learning device 1 updates the content of a process (hereinafter referred to as "acoustic conversion process") for converting an input acoustic time series into data in a predetermined format (hereinafter referred to as "target format") through learning. Hereinafter, information about sound obtained based on the acoustic time series and expressed in the target format will be referred to as feature data. The acoustic conversion process is a type of learning model. The acoustic time series is a time series representing sound such as an acoustic signal.

[0015] More specifically, an audio time series is information in which sound is expressed as a tensor. An audio time series may be, for example, a second-order tensor (i.e., a matrix) indicating the frequency at each time and the intensity of the frequency component, with the element value indicating the intensity of the frequency component. An audio time series may be, for example, a third-order tensor indicating a set of a channel, a frequency, and the intensity of the frequency component at each time, with the element value indicating the intensity of the frequency component.

[0016] An acoustic time series may be, for example, a first-order tensor (i.e., vector) that indicates the intensity of the sound at each time. If the tensor is a second-order or higher tensor, the values ​​of the tensor elements indicate the intensity of the frequency components.

[0017] The target format is specifically a predetermined format required by a downstream task. The downstream task refers to a process that uses input feature data, and the content of the process may be any predetermined content.

[0018] The downstream task may be, for example, a process of removing noise from a sound represented by input feature data, or a process of synthesizing a sound represented by input feature data with a sound represented by another acoustic time series. The downstream task may be, for example, a process of classifying a sound represented by input feature data. For example, if the downstream task is a 10-class classification task, the target format is a format of 10 32-bit floating-point numbers.

[0019] The learning device 1 includes an acoustic conversion network 110, a downstream task execution unit 120, and an update unit 130. The acoustic conversion network 110 is a neural network that includes a feature extraction network 111 and an intermediate network 112, and converts an input acoustic time series into feature data.

[0020] A neural network is a circuit such as an electronic circuit, an electrical circuit, an optical circuit, or an integrated circuit that represents a learning model. A neural network is updated through machine learning. More specifically, being updated through machine learning means that the parameters of the neural network are appropriately adjusted based on the value of the loss function (i.e., the loss). The parameters of a neural network are the parameters of the learning model that it represents. Furthermore, the parameters of a neural network are the parameters of the circuits that make up the neural network.

[0021] The feature extraction network 111 receives an input of an acoustic time series. The feature extraction network 111 converts the input acoustic time series into a third-order tensor that indicates the feature of the input acoustic time series and has time, frequency, and channel. Hereinafter, the third-order tensor that indicates the feature of the input acoustic time series and has time, frequency, and channel will be referred to as an intermediate feature tensor.

[0022] 2 is an explanatory diagram illustrating an example of the configuration of a feature extraction network 111 in an embodiment. The feature extraction network 111 is a neural network including one or more pairs of a convolutional layer and a pooling layer (hereinafter referred to as "paired layers"). In each paired layer, the convolutional layer is located before the pooling layer. That is, in each paired layer, the pooling layer is located after the convolutional layer. Therefore, in each paired layer, the output of the convolutional layer is input to the pooling layer.

[0023] When the feature extraction network 111 includes multiple pair layers, the pair layers are arranged one-dimensionally. Being arranged one-dimensionally means that the arrangement of the pair layers satisfies a first arrangement condition, a second arrangement condition, and a third arrangement condition. The first arrangement condition is that each pair layer is connected to one or more other pair layers, and a pair layer connected to two pair layers is connected to one other pair layer on the input side and to one other pair layer different from the pair layer connected to the input side on the output side. The second arrangement condition is that there are two pair layers connected to one other pair layer. The third arrangement condition is that for two pair layers connected to each other, the output of the pooling layer of the previous pair layer is input to the convolution layer of the subsequent pair layer.

[0024] The intermediate feature tensor is the output of the last pair layer.

[0025] Returning to the explanation of Figure 1, the intermediate network 112 receives an input of an intermediate feature tensor. The intermediate network 112 converts the representation of the input intermediate feature tensor from a third-order tensor to a first-order tensor using a non-equivalent second-order tensorization process and a vectorization process. Note that a first-order tensor is a one-dimensional vector.

[0026] <Non-equivalent second-order tensorization processing> The non-equivalent second-order tensorization process is a process of converting an intermediate feature tensor representation expressed as a third-order tensor having time, frequency, and channel into a second-order tensor representation having time and a direct product quantity. The direct product quantity is a quantity that indicates the direct product of frequency and channel. In other words, the direct product quantity is an identifier that identifies each direct product of frequency and channel. Hereinafter, a second-order tensor having time and a direct product quantity will be referred to as a planar tensor.

[0027] To simplify the explanation, some terms and expressions are defined here. Hereinafter, a third-order tensor having time, frequency, and channel is referred to as an acoustic third-order tensor. The intermediate feature tensor is an example of an acoustic third-order tensor. Hereinafter, the number of time elements of a tensor is represented as a, the number of frequency elements is represented as b, and the number of channel elements is represented as c. Hereinafter, the size of a third-order tensor whose number of time elements is a, the number of frequency elements is b, and the number of channel elements is c is represented as (a, b, c). Hereinafter, an acoustic third-order tensor of size (a, b, c) is represented as acoustic third-order tensor (a, b, c).

[0028] Hereinafter, a second-order tensor having time and a direct product quantity is referred to as an acoustic second-order tensor. A plane tensor is an example of an acoustic second-order tensor. Hereinafter, the number of elements in a tensor's direct product quantity is represented as d. Therefore, the size of an acoustic second-order tensor is represented as (a, d). Since a direct product quantity is the direct product of frequency and channel, d = b × c. The symbol × simply means multiplication, not a direct product. Hereinafter, an acoustic second-order tensor of size (a, d) will be represented as an acoustic second-order tensor (a, d).

[0029] In the following, a first-order tensor of size e will be referred to as an acoustic first-order tensor, and its size will be represented as (e). In other words, an acoustic first-order tensor of size e will be represented as acoustic first-order tensor (e).

[0030] Using this expression, the non-equivalent second-order tensorization process can be expressed as a process of converting intermediate feature tensor (a, b, c) into a plane tensor (a, d).

[0031] <Vectorization processing> The vectorization process is a process of obtaining a statistical vector, which is a one-dimensional vector indicating the statistical quantity of each direct product quantity in the time axis direction for each direct product quantity of a plane tensor.

[0032] The vectorization process is a process of obtaining a statistical vector by performing, for example, average pooling in the time axis direction.

[0033] <Time axis average pooling> Time axis direction average pooling is a process of obtaining an average vector based on a planar tensor. The average vector is a one-dimensional vector that indicates the average value of each direct product of the planar tensor in the time axis direction. Therefore, if the size of the direct product of the planar tensor is d, the number of average vectors required is d. In this way, time axis direction average pooling is a process of converting a planar tensor (a, d) into an average vector (d). A statistical vector is, for example, the average vector obtained in this way.

[0034] The vectorization process is a process of obtaining a statistical vector by performing, for example, maximum pooling in the time axis direction.

[0035] <Maximum pooling in time axis direction> Time axis direction max pooling is a process of obtaining a maximum vector based on a planar tensor. The maximum vector is a one-dimensional vector that indicates the maximum value of each direct product of the planar tensor in the time axis direction. Therefore, if the size of the direct product of the planar tensor is d, the number of maximum vectors required is d. In this way, time axis direction max pooling is a process of converting a planar tensor (a, d) into a maximum vector (d). A statistical vector is, for example, the maximum vector obtained in this way.

[0036] The vectorization process is a process of acquiring a statistical vector by performing, for example, time-axis average pooling, time-axis maximum pooling, and vector sum acquisition process. The vector sum acquisition process is a process of acquiring a one-dimensional vector expressed as the vector sum of an average vector and a maximum vector.

[0037] Since the size of the average vector and the size of the maximum vector obtained based on the same plane tensor are the same, it is possible to define a vector sum between the average vector and the maximum vector. Therefore, the vector sum of the average vector and the maximum vector can be obtained by the vector sum obtaining process. The size of the one-dimensional vector expressed by the vector sum of the average vector and the maximum vector is d, which is the same as the size of the average vector or the maximum vector.

[0038] In this way, the statistical quantity of each direct product quantity in the time axis direction in the vectorization process is, for example, either an average value or a maximum value. Also, in this way, the statistical vector may be a one-dimensional vector that is the vector sum of a one-dimensional vector that indicates the average value of each direct product quantity in the time axis direction for each direct product quantity of the planar tensor, and a one-dimensional vector that indicates the maximum value of each direct product quantity in the time axis direction for each direct product quantity of the planar tensor.

[0039] The statistical vector obtained in this manner is the feature data. Therefore, the predetermined format required by the downstream task is specifically a format that satisfies the condition of a one-dimensional vector. Therefore, the vectorization process is a process of acquiring, as feature data, a one-dimensional vector that indicates the statistical quantity of each direct product quantity in the time axis direction for each direct product quantity of a planar tensor. Furthermore, since the statistical vector is the feature data, the statistical vector is input to the downstream task execution unit 120 as the feature data. In other words, the statistical vector is input to the downstream task.

[0040] The acoustic conversion network 110, which includes the feature extraction network 111 and the intermediate network 112, is updated through learning so that the acoustic conversion network 110 can convert an acoustic time series into feature data in a manner that reduces the frequency with which the results of the execution of downstream tasks do not satisfy a predetermined standard.

[0041] The process of causing the acoustic conversion network 110 to convert an acoustic time series into feature data is the acoustic conversion process.

[0042] Note that an update in which the acoustic time series is converted into feature data in the acoustic conversion network 110 so as to reduce the frequency with which the results of the execution of downstream tasks do not satisfy a predetermined standard is, for example, an update that reduces the value of the acoustic loss function.

[0043] The acoustic loss function is a loss function whose value indicates the difference between the result of the execution of the downstream task and a predetermined result that satisfies a predetermined criterion. The value indicating the difference between the result of the execution of the downstream task and the predetermined result that satisfies a predetermined criterion is, for example, the difference between the result of the execution of the downstream task and the predetermined result that satisfies the predetermined criterion.

[0044] The downstream task execution unit 120 executes a predetermined downstream task using the input feature data. The update unit 130 updates the acoustic conversion network 110 based on the results of the downstream task execution by the downstream task execution unit 120. For example, the update is performed by decreasing the value of the acoustic loss function. Learning is performed until a predetermined termination condition is met. The predetermined termination condition (hereinafter referred to as the "learning termination condition") is, for example, a condition that a predetermined number of updates have been performed. The update unit 130 also executes a process to determine whether the learning termination condition is met.

[0045] In this way, the learning device 1 converts acoustic time series into statistical vectors and updates the acoustic transformation network 110 based on the results of executing downstream tasks using the statistical vectors. Therefore, the learning data used in the learning performed by the learning device 1 includes acoustic time series.

[0046] The conversion device 2 converts the target audio time series into feature data using a trained acoustic conversion network 110. The trained acoustic conversion network 110 is the acoustic conversion network 110 at the time when a learning termination condition is satisfied. The conversion device 2 is provided with the trained acoustic conversion network 110 by replacing it from the learning device 1.

[0047] When the conversion device 2 is provided with the acoustic conversion network 110 in advance, the conversion device 2 may acquire network parameter information from the learning device 1 and obtain the trained acoustic conversion network 110 based on the acquired network parameter information. Specifically, the conversion device 2 may obtain the trained acoustic conversion network 110 by converting the acoustic conversion network 110 provided in advance into the trained acoustic conversion network 110 based on the acquired network parameter information. The network parameter information is information indicating the values ​​of each parameter of the trained acoustic conversion network 110.

[0048] In this way, the conversion device 2 includes the same neural network as the trained acoustic conversion network 110 included in the training device 1.

[0049] 3 is a diagram showing an example of the hardware configuration of a learning device 1 according to an embodiment. The learning device 1 includes a control unit 11 having a processor 91, such as a CPU (Central Processing Unit), and a memory 92 connected via a bus, and executes a program. By executing the program, the learning device 1 functions as a device including the control unit 11, an input unit 12, a communication unit 13, a memory unit 14, and an output unit 15.

[0050] More specifically, the processor 91 reads out a program stored in the storage unit 14 and stores the read out program in the memory 92. When the processor 91 executes the program stored in the memory 92, the learning device 1 functions as a device including a control unit 11, an input unit 12, a communication unit 13, a storage unit 14, and an output unit 15.

[0051] The control unit 11 controls the operation of various functional units included in the learning device 1. The control unit 11 includes an acoustic conversion network 110, a downstream task execution unit 120, and an update unit 130. The control unit 11 controls the operation of, for example, the output unit 15. The control unit 11 records, in the storage unit 14, various pieces of information generated by the operations of, for example, the acoustic conversion network 110, the downstream task execution unit 120, and the update unit 130.

[0052] Input unit 12 includes input devices such as a mouse, keyboard, and touch panel. Input unit 12 may be configured as an interface that connects these input devices to learning device 1. Input unit 12 accepts input of various types of information to learning device 1.

[0053] The communication unit 13 includes a communication interface for connecting the learning device 1 to an external device. The communication unit 13 communicates with the external device via wired or wireless communication. The external device is, for example, a device that transmits an acoustic time series. The external device is, for example, the conversion device 2.

[0054] The storage unit 14 is configured using a computer-readable storage medium device such as a magnetic hard disk drive or a semiconductor storage device. The storage unit 14 stores various information related to the learning device 1. The storage unit 14 stores information input via, for example, the input unit 12 or the communication unit 13. The storage unit 14 stores various information generated by the operations of, for example, the acoustic conversion network 110, the downstream task execution unit 120, and the update unit 130.

[0055] Output unit 15 outputs various types of information. Output unit 15 includes a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or an organic EL (Electro-Luminescence) display. Output unit 15 may be configured as an interface that connects these display devices to learning device 1. Output unit 15 outputs information input to input unit 12, for example.

[0056] 4 is a diagram showing an example of the configuration of the control unit 11 in an embodiment. The control unit 11 includes an acoustic time series acquisition unit 101, an acoustic conversion network 110, a downstream task execution unit 120, an update unit 130, a memory control unit 140, a communication control unit 150, and an output control unit 160. The acoustic time series acquisition unit 101 acquires an acoustic time series input to the communication unit 13. The memory control unit 140 records various information in the memory unit 14. The communication control unit 150 controls the operation of the communication unit 13. The output control unit 160 controls the operation of the output unit 15.

[0057] 5 is a flowchart showing an example of the flow of processing executed by the learning device 1 in this embodiment. The acoustic time series acquisition unit 101 acquires an acoustic time series (step S101). Next, the feature extraction network 111 converts the acoustic time series into intermediate feature tensors (step S102). Next, the intermediate network 112 executes a non-equivalent second-order tensorization process to acquire a plane tensor based on the intermediate feature tensors (step S103).

[0058] Next, the intermediate network 112 executes a vectorization process to obtain statistical vectors based on the plane tensors (step S104). The obtained statistical vectors are feature data. Next, the downstream task execution unit 120 executes a predetermined downstream task using the feature data (step S105). Next, the update unit 130 in step S105 updates the acoustic conversion network 110 based on the results of the execution of the downstream task (step S106).

[0059] Next, the update unit 130 determines whether the learning termination condition is satisfied (step S107). If the learning termination condition is not satisfied (step S107: NO), the process returns to step S101. If the learning termination condition is satisfied (step S107: YES), the process ends. The acoustic conversion network 110 at the time the process ends is the trained acoustic conversion network 110.

[0060] 6 is a diagram showing an example of the hardware configuration of the conversion device 2 according to the embodiment. The conversion device 2 includes a control unit 21 having a processor 93 such as a CPU and a memory 94 connected via a bus, and executes a program. By executing the program, the conversion device 2 functions as a device including the control unit 21, an input unit 22, a communication unit 23, a storage unit 24, and an output unit 25.

[0061] More specifically, the processor 93 reads out the program stored in the storage unit 24 and stores the read out program in the memory 94. When the processor 93 executes the program stored in the memory 94, the conversion device 2 functions as a device including the control unit 21, the input unit 22, the communication unit 23, the storage unit 24, and the output unit 25.

[0062] The control unit 21 controls the operation of various functional units included in the conversion device 2. The control unit 21 includes a trained acoustic conversion network 110. The control unit 21 executes the trained acoustic conversion network 110. The control unit 21 controls, for example, the operation of the output unit 25. The control unit 21 records, for example, various pieces of information generated by the execution of the trained acoustic conversion network 110 in the storage unit 24.

[0063] The input unit 22 includes input devices such as a mouse, a keyboard, and a touch panel. The input unit 22 may be configured as an interface that connects these input devices to the conversion device 2. The input unit 22 accepts input of various information to the conversion device 2.

[0064] The communication unit 23 includes a communication interface for connecting the conversion device 2 to an external device. The communication unit 23 communicates with the external device via wired or wireless communication. The external device is, for example, a device that transmits an acoustic time series to be converted. The external device is, for example, the learning device 1.

[0065] The storage unit 24 is configured using a computer-readable storage medium device such as a magnetic hard disk drive or a semiconductor storage device. The storage unit 24 stores various information related to the conversion device 2. The storage unit 24 stores information input via, for example, the input unit 22 or the communication unit 23. The storage unit 24 stores, for example, the results of execution of the trained acoustic conversion network 110.

[0066] The output unit 25 outputs various types of information. The output unit 25 includes a display device such as a CRT display, a liquid crystal display, or an organic EL display. The output unit 25 may be configured as an interface that connects these display devices to the conversion device 2. The output unit 25 outputs, for example, information input to the input unit 22. The output unit 25 may also display, for example, the execution result of the trained acoustic conversion network 110.

[0067] 7 is a diagram showing an example of the configuration of the control unit 21 in the embodiment. The control unit 21 includes a conversion target acquisition unit 210, a conversion unit 220, a memory control unit 230, a communication control unit 240, an output control unit 250, and a downstream task execution unit 260. The conversion target acquisition unit 210 acquires an acoustic time series input to the communication unit 23.

[0068] Specifically, the conversion unit 220 is the trained acoustic conversion network 110. The conversion unit 220 converts the acoustic time series acquired by the conversion target acquisition unit 210 into feature data using the trained acoustic conversion network 110.

[0069] The memory control unit 230 records various information in the memory unit 24. The communication control unit 240 controls the operation of the communication unit 23. The output control unit 250 controls the operation of the output unit 25. The downstream task execution unit 260 executes a predetermined downstream task that is the same as the downstream task executed by the learning device 1.

[0070] 8 is a flowchart showing an example of the flow of processing executed by the conversion device 2 in the embodiment. The conversion target acquisition unit 210 acquires an acoustic time series input to the communication unit 23 (step S201). Next, the conversion unit 220 acquires a statistical vector using the trained acoustic conversion network 110 based on the acoustic time series acquired in step S201 (step S202). Next, the downstream task execution unit 260 executes a predetermined downstream task using the statistical vector acquired in step S202 as feature data (step S203). After step S203, the output control unit 250 controls the operation of the output unit 25 to cause the output unit 25 to output the result of the execution of the downstream task (step S204).

[0071] <Experimental Results> The results of multiple experiments will be explained using Figures 9 to 11. Figure 9 is a first explanatory diagram explaining the experimental results in the embodiment. Figure 10 is a second explanatory diagram explaining the experimental results in the embodiment. More specifically, Figure 10 is an explanatory diagram explaining VGGish, an encoder CNN that converts the image classification model VGG to acoustic signals and learns by classifying sounds, and a process in which the flattening process of VGGish is replaced with a non-equivalent second-order tensorization process and a vectorization process. Note that in the experiment, the vectorization process was specifically a process of obtaining a one-dimensional vector that is the vector sum of an average vector and a maximum vector.

[0072] Fig. 11 is a third explanatory diagram illustrating experimental results in an embodiment. More specifically, Fig. 11 is an explanatory diagram illustrating Cnn14, a learning model proposed for acoustic signals and performing supervised learning for sound classification similar to VGGish, and a process in which the Cnn14 process is replaced with a non-equivalent second-order tensorization process and a vectorization process. Note that in the experiment, the vectorization process specifically was a process of obtaining a one-dimensional vector that is the vector sum of an average vector and a maximum vector.

[0073] "US8K" in Figure 9 indicates UrbanSound8k, a task for classifying environmental sounds such as horns and barking dogs. "SPCV2" indicates Speech commands V2, a task for classifying voice commands. "VoxCeleb1" indicates VoxCeleb1, a task for classifying speakers. "NSynth" indicates NSynth, a task for classifying musical instrument sounds. "Surge" indicates Pitch Audio Dataset (Surge synthesizer), a task for classifying musical scales.

[0074] Result E101 in Figure 9 is an example of the result of existing VGGish processing. Specifically, the processing involved inputting a spectrogram with 96 frames of time, 64 frequency bins, and one channel, and packing the input spectrogram in the channel direction as it passed through the convolutional layer. The data size changed as follows:

[0075] (96, 64, 1) changed to (96, 64, 64), then to (48, 32, 128), then to (24, 16, 256), then to (12, 6, 512). This continues until the acoustic third-order tensor (6, 4, 512) is reached. The process then performs a flatten on the acoustic third-order tensor (6, 4, 512), producing the acoustic first-order tensor (6x4x512). Image G101 in Figure 10 shows this flattening process.

[0076] Result E102 in Figure 9 is an example of the result of replacing the VGGish flattening process with a non-equivalent second-order tensorization process and vectorization process. Image G102 in Figure 10 shows the result of replacing the VGGish flattening process with a non-equivalent second-order tensorization process and vectorization process.

[0077] Results E101 and E102 in FIG. 9 indicate that result E102 has a higher frequency of the results of the execution of downstream tasks meeting a predetermined criterion.

[0078] Result E103 in Figure 9 is an example of the processing result of the existing CNN14. In the experiment, the final convolutional layer output of CNN14 was (3, 2, 2048). In the experiment, CNN14 then performed global pooling.

[0079] Specifically, two frequency bins are first averaged and converted into a (3,2048) matrix. Then, the average and maximum values ​​are calculated in the time direction, and the feature is output as a 2048-dimensional vector. In other words, in the experiment, Cnn14 converted the feature tensor (3,2,2048) into an acoustic second-order tensor (3,2048). Image G103 in Figure 11 shows this global pooling process.

[0080] Result E104 in Figure 9 is an example of the result of replacing the processing of the existing CNN14 with non-equivalent second-order tensorization processing and vectorization processing. Image G104 in Figure 11 shows the processing of replacing the processing of the existing CNN14 with non-equivalent second-order tensorization processing and vectorization processing. Specifically, the processing involves using non-equivalent second-order tensorization processing to expand the frequency and channel components of the feature tensor (3,2,2048) and transforming it into a matrix of (3,2 x 2048 = 4096), and then using vectorization processing to calculate a one-dimensional vector that is the vector sum of the average vector and the maximum vector, and outputting 4096-dimensional feature data.

[0081] Result E103 shows that the existing Cnn14 simply averages the high and low frequency components, losing information, and therefore cannot achieve good performance, especially in tasks where pitch determination is important. Result E103 and result E104 show that result E104 more frequently meets the specified criteria for the results of downstream tasks.

[0082] The learning device 1 configured in this way executes non-equivalent second-order tensorization processing and vectorization processing. As a result, the learning device 1 does not treat frequency, channel, and time without distinction, but calculates frequency and channel statistics on the time axis to obtain features that eliminate the time component while not mixing frequency and channel components. Therefore, the reduction in the amount of information is smaller than when treating frequency, channel, and time without distinction. Therefore, the learning device 1 can reduce the frequency with which the results of downstream task execution do not satisfy a predetermined standard.

[0083] Furthermore, the conversion device 2 configured in this manner performs conversion using the trained feature extraction network 111 obtained by training by the training device 1. Therefore, the conversion device 2 can reduce the frequency with which the results of the execution of downstream tasks do not satisfy a predetermined standard.

[0084] (Variation) The non-equivalent second-order tensorization processing execution unit 121 may convert the obtained planar tensor into a planar tensor in which at least a portion is changed using a machine learning mathematical model. That is, the non-equivalent second-order tensorization processing execution unit 121 may obtain a new planar tensor by updating the obtained planar tensor using a machine learning mathematical model. In such a case, the machine learning mathematical model that updates the planar tensor is updated when the acoustic conversion network 110 is updated through learning.

[0085] The feature extraction network 111 is an example of a feature extraction unit. The process of converting an input acoustic time series into an intermediate feature tensor does not necessarily have to be performed by a neural network such as that shown in Figure 2. Here, Figure 12 shows an example of the process performed by the feature extraction network 111 and experimental results.

[0086] 12 is a diagram showing an example of experimental results in a modified example. The process of converting an input acoustic time series into an intermediate feature tensor may be, for example, a process of converting the input acoustic time series into a linear scale spectrogram. That is, instead of the process of FIG. 2, the feature extraction network 111 may execute a process of converting the input acoustic time series into a linear scale spectrogram. In this case, the intermediate feature tensor is a linear scale spectrogram.

[0087] Result E201 in Figure 12 is an example of the results of an experiment using a process that converts an input acoustic time series into a linear-scale spectrogram. Result E201 shows that even a process that converts an input acoustic time series into a linear-scale spectrogram can achieve sufficient performance for downstream tasks where pitch is important. Specifically, the vectorization process in the experiment involved obtaining a one-dimensional vector that is the vector sum of the average vector and the maximum vector.

[0088] When the acoustic time series input to the feature extraction network 111 is a spectrogram, the feature extraction network 111 may have one or two pairs of layers.

[0089] Results E202 and E203 in Figure 12 are examples of experimental results when the feature extraction network 111 has one or two pair layers. Results E202 and E203 indicate that high-performance results can be obtained with 768 or 2544 parameters. Parameters are the weight values ​​of the neural network. For example, when the Conv layer is 3 × 3 and the convolution has 64 output channels, the total number of parameters is 640, which is 10 times 64, with 3 × 3 = 9 weights and 1 bias. Note that the above number 768 means that 128 parameters are consumed by another layer, BatchNorm, which has parameters included in the feature extraction network 111 with a single pair layer, resulting in a total of 768. Note that in the experiment, the vectorization process specifically involved obtaining a one-dimensional vector that is the vector sum of the average vector and the maximum vector.

[0090] In addition, the non-equivalent second-order tensorization process and vectorization process may be applied to self-supervised learning such as BYOL (Bootstrap Your Own Latent) for Audio.

[0091] The weights of the acoustic conversion network 110 before the start of learning by the learning device 1 may be weights of a trained acoustic conversion network 110 that has been trained in advance using a downstream task different from the downstream task executed by the learning device 1. In other words, the learning of the acoustic conversion network 110 by the learning device 1 may be transfer learning.

[0092] As described above, the feature extraction network 111 in the learning device 1 of the embodiment is a neural network including one or more paired layers. Furthermore, in the feature extraction network 111 in the learning device 1 of the embodiment, the intermediate feature tensor is the single output of the last paired layer. Then, a plane tensor is obtained based on the single intermediate feature tensor obtained.

[0093] However, if the feature extraction network 111 is a neural network having a plurality of paired layers, the intermediate feature tensor does not necessarily have to be the single output of the last paired layer.

[0094] For example, the feature extraction network 111 may output the outputs of a plurality of different pairs of layers as one intermediate feature tensor, i.e., the feature extraction unit may convert the input acoustic time series into a plurality of intermediate feature tensors.

[0095] Therefore, the feature extraction network 111 that outputs the outputs of each of a plurality of mutually different pairs of layers as one intermediate feature tensor will be defined as a feature extraction network 111a and will be described below with reference to FIG.

[0096] 13 is a diagram showing an example of the configuration of a learning device 1 that includes an acoustic conversion network 110a in a modified example instead of the acoustic conversion network 110. In the example of FIG. 13, the learning device 1 includes an acoustic conversion network 110a instead of the acoustic conversion network 110.

[0097] The acoustic conversion network 110a includes a feature extraction network 111a and an intermediate network 112a. As described above, the feature extraction network 111a is a neural network including a plurality of paired layers. The feature extraction network 111a acquires a plurality of intermediate feature tensors. More specifically, the feature extraction network 111a outputs the outputs of a plurality of predetermined, mutually different paired layers as one intermediate feature tensor. The plurality of paired layers may be, for example, two paired layers.

[0098] The intermediate network 112a performs time unification processing on each intermediate feature tensor output by the feature extraction network 111a. The multiple intermediate feature tensors output by the feature extraction network 111a do not necessarily have the same number of elements in the time direction because they are output from different pair layers. The time unification processing is processing that targets a set of multiple tensors. The time unification processing is processing that unifies the number of elements in the time direction of tensors included in the set that is the target of execution.

[0099] For example, if the multiple intermediate feature tensors output by the feature extraction network 111 are tensor A1, tensor A2, and tensor A3, by executing the time unification process, the number of elements in the time direction of tensor A1, the number of elements in the time direction of tensor A2, and the number of elements in the time direction of tensor A3 will all be the same.

[0100] The process of unifying the number of elements in the time direction is, for example, a process of executing an element number increase process or an element number reduction process. The element number increase process is a process of increasing the number of elements for one or more tensors included in the set to be executed and inputting values ​​obtained by interpolation into the increased elements. The element number reduction process is a process of reducing the number of elements for one or more tensors included in the set to be executed.

[0101] The following formula (1) expresses an example of the time unification process.

[0102]

number

[0103] Equation (1) is the tensor z l This shows that the tensor on the left side is obtained by converting the number of elements in the time direction to a predetermined number of elements T0. In this way, the time unification process is a process of normalization in the time direction. Therefore, the tensor z l is a third-order tensor representing the features of an acoustic time series and having time, frequency, and channel, then the tensor after conversion is also a third-order tensor representing the features of an acoustic time series and having time, frequency, and channel. In other words, if the tensor to be converted is an intermediate feature tensor, then the tensor after conversion by the time unification process is also an intermediate feature tensor.

[0104] The intermediate network 112a performs non-equivalent second-order tensorization on each of the intermediate feature tensors whose lengths in the time direction have been unified as a result of the time unification process. As a result of each non-equivalent second-order tensorization process, a planar tensor is obtained for each non-equivalent second-order tensorization process.

[0105] The intermediate network 112a executes a combining process. The combining process combines multiple planar tensors obtained as a result of each non-equivalent second-order tensorization process. For example, if three planar tensors, tensor B1, tensor B2, and tensor B3, are obtained as a result of each non-equivalent second-order tensorization process, the combining process combines tensor B1, tensor B2, and tensor B3 on a predetermined axis. The predetermined axis is, for example, an axis formed by the direct product of frequency and channel. The combining process obtains a single planar tensor that is larger in size than each planar tensor before being combined.

[0106] The intermediate network 112a performs vectorization processing on the combined planar tensor.

[0107] 14 is a flowchart showing an example of the flow of processing executed by a learning device 1 equipped with an acoustic conversion network 110a in a modified example. For simplicity of explanation, processing similar to that described in FIG. 5 is assigned the same reference numerals as in FIG. 5, and description thereof will be omitted.

[0108] The acoustic time series acquisition unit 101 acquires an acoustic time series (step S101). Next, the feature extraction network 111a acquires a plurality of intermediate feature tensors based on the acoustic time series (step S102a). Next, the intermediate network 112a executes time unification processing (step S108). By executing the time unification processing, the number of elements in the time direction of the intermediate feature tensors obtained in step S102a is unified.

[0109] Next, the intermediate network 112a performs non-equivalent second-order tensorization processing on each intermediate feature tensor whose number of elements in the time direction has been unified by the time unification processing (step S103a). By executing the processing of step S103a, a plane tensor is obtained for each intermediate feature tensor.

[0110] Next, the intermediate network 112a executes a combining process (step S109). By executing the combining process, the multiple planar tensors obtained in step S103a are combined. As a result of the combining, a single tensor that is larger than before the combining is obtained.

[0111] After step S109, the processes of steps S104 to S107 are executed. The plane tensor that is the target of the vectorization process of step S104 is the single large tensor obtained after the combination by the process of step S109.

[0112] Note that time unification processing does not necessarily have to be performed. For example, if the number of elements in the time direction of the intermediate feature amount vectors obtained in step S102a is the same, time unification processing does not necessarily have to be performed.

[0113] When the time unification process is not performed, the intermediate network 112a performs a non-equivalent second-order tensorization process on each of the intermediate feature tensors output by the feature extraction network 111a. As a result of each non-equivalent second-order tensorization process, a planar tensor is obtained for each non-equivalent second-order tensorization process. In other words, the intermediate network 112a obtains a planar tensor for each of the intermediate features obtained by the feature extraction network 111a.

[0114] Note that even when the learning device 1 performs the time unification process, the time unification process does not necessarily have to be performed after step S102a. The time unification process may be performed on each planar tensor obtained as a result of each non-equivalent second-order tensorization process. In other words, the time unification process may be performed on multiple planar tensors that are the targets of the combination process. In such a case, multiple planar tensors that have the same number of elements in the time direction as a result of the time unification process are combined.

[0115] As with the time unification process for the intermediate feature tensors described above, if the tensor to be transformed in the time unification process is a planar tensor, the tensor transformed by the time unification process is also a planar tensor.

[0116] In this way, the time unification process does not necessarily have to be performed, and may be performed at any timing as long as it is performed before the process of combining the planar tensors and after the acquisition of multiple intermediate feature tensors.

[0117] <Experimental results for modified examples> An example of the experimental results using the learning device 1 equipped with the acoustic conversion network 110a will be described with reference to FIG.

[0118] Fig. 15 is a first diagram showing an example of experimental results for a modified example. In the experiment, the image classification model VGGish, Cnn14, a learning model that performs supervised learning for sound classification, and an Audio Spectrogram Transformer (AST) model were used. Each value in Fig. 15 indicates the estimation accuracy.

[0119] "VGGish" means VGGish. "VGGish-Fusion#10#15" means processing in which the processing of the feature extraction network 111 and intermediate network 112 of VGGish is replaced with the feature extraction network 111a and intermediate network 112a, and the outputs of the 10th and 15th paired layers of the feature extraction network 111a are used. "CNN14" means CNN14. "CNN14-Fusion#3#6" means processing in which the processing of the feature extraction network 111 and intermediate network 112 of CNN14 is replaced with the feature extraction network 111a and intermediate network 112a, and the outputs of the 3rd and 6th paired layers of the feature extraction network 111a are used.

[0120] "AST" stands for AST. "AST-Fusion#5#12" means processing in which the processing of the feature extraction network 111 and the intermediate network 112 of the AST is replaced with the processing of the feature extraction network 111a and the intermediate network 112a, and the outputs of the fifth and twelfth pair layers of the feature extraction network 111a are used.

[0121] "ESC-50" is an environmental sound classification task. "SER tasks" refer to acoustic event recognition tasks. "ESC-50" and "US8K" are all types of "SER tasks." "VC1" refers to VoxCeleb1, a speaker identification task. "VF" refers to VoxForge, a spoken language identification task. "CRM-D" refers to CREMA-D, a spoken speech emotion recognition task. "NOSS tasks" refer to non-semantic speech tasks. "SPCV2," "VC1," "VF," and "CRM-D" are all types of "NOSS tasks." "GTZAN" refers to a music genre classification task. "Music tasks" refer to music-related tasks. "GTZAN," "NSynth," and "Surge" are all types of "Music tasks."

[0122] "Avg." indicates the average estimation accuracy. The results in Fig. 15 show that for "VGGish," "CNN14," and "AST," the estimation accuracy is higher when the feature extraction network 111a and the intermediate network 112a are used than when they are not used.

[0123] FIG. 16 is a second diagram showing an example of experimental results for a modified example. The horizontal axis of FIG. 16 indicates downstream tasks. The vertical axis of FIG. 16 indicates paired layers included in "VGGish-Fusion#10#15" used in the experiment. FIG. 16 shows the estimation accuracy of downstream tasks for each paired layer when only the output of the target paired layer is used as the intermediate feature tensor. The closer the estimation accuracy is to 1, the higher the accuracy. FIG. 16 shows that, for example, when only the output of the paired layer of the 15th intermediate feature tensor is used, the estimation accuracy of "ESC-50" is 85%.

[0124] As shown in Fig. 16, the paired layer that outputs the intermediate feature tensor with high estimation accuracy differs for each downstream task. Therefore, Fig. 16 shows that it is effective to use the intermediate feature tensors of multiple paired layers to obtain a mathematical model that achieves an estimation accuracy above a predetermined level for multiple downstream tasks.

[0125] For example, Fig. 16 shows that a mathematical model with high estimation accuracy for three downstream tasks, "ESC-50," "US8K," and "GTZAN," uses the intermediate feature tensor of the 15th paired layer. Fig. 16 also shows that a mathematical model with high estimation accuracy for six downstream tasks, "SPCV2," "VF," "VC1," "CR-D," "NSynth," and "Surge," uses the intermediate feature tensor of the 10th paired layer.

[0126] In this way, the use of intermediate feature tensors for multiple pairs of layers improves the accuracy of estimation of multiple downstream tasks. Therefore, the learning device 1 equipped with the acoustic conversion network 110a in this modified example can reduce the frequency with which the results of the execution of multiple downstream tasks do not satisfy a predetermined standard.

[0127] It should be noted that statistical vectors do not necessarily need to be obtained between the acoustic conversion network 110 and the acoustic conversion network 110a. Even if statistical vectors are not obtained between the acoustic conversion network 110 and the acoustic conversion network 110a as long as planar tensors are obtained, it is possible to reduce the frequency with which the results of the execution of downstream tasks do not satisfy a predetermined standard, because there is no need to calculate statistics when handling only one element on the time axis in a task of processing a short-term acoustic signal.

[0128] The following explains in more detail why there is no need to calculate statistics when dealing with only one time axis element in a task that processes short-term acoustic signals. For example, if the downstream task is a task that searches for an event that occurs at a certain time, classification processing is performed for each time frame, so statistical processing is not required. Thus, for example, if the downstream task is a task that handles short-term acoustic signals, the time axis element is 1, so there is no need to take statistics. In this way, as long as a planar tensor is obtained, it is not necessary to obtain a statistical vector. Therefore, if a planar tensor is obtained, the frequency with which the results of the execution of the downstream task do not meet the specified criteria is reduced, even if a statistical vector is not obtained.

[0129] The feature extraction network 111a is also an example of a feature extraction unit.

[0130] The learning device 1 and the conversion device 2 may each be implemented using a plurality of information processing devices communicably connected via a network. In this case, the functional units of the learning device 1 and the conversion device 2 may be distributed and implemented among the plurality of information processing devices.

[0131] The learning device 1 and the conversion device 2 do not necessarily have to be implemented as different devices, but may be implemented as a single device that combines the functions of both devices.

[0132] Note that all or part of the functions of the acoustic time series conversion system 100, the learning device 1, and the conversion device 2 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as flexible disks, magneto-optical disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. The program may be transmitted via a telecommunications line.

[0133] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention. [Explanation of symbols]

[0134] 100...acoustic time series conversion system, 1...learning device, 2...conversion device, 11...control unit, 12...input unit, 13...communication unit, 14...memory unit, 15...output unit, 101...acoustic time series acquisition unit, 110...acoustic conversion network, 111, 111a...feature extraction network, 112, 112a...intermediate network, 120...downstream task execution unit, 130...update unit, 140...storage control unit, 150...communication control unit, 160...output control unit, 21...control unit, 22...input unit, 23...communication unit, 24...memory unit, 25...output unit, 210...conversion target acquisition unit, 220...conversion unit, 230...memory control unit, 240...communication control unit, 250...output control unit, 91...processor, 92...memory, 93...processor, 94...memory

Claims

1. a neural network that converts an acoustic time series, which is a time series representing a sound, into feature data, which is information about the sound obtained based on the acoustic time series and expressed in a predetermined format required by a downstream task; an update unit that updates the neural network based on a result of execution of the downstream task using the feature amount data; Equipped with The neural network a feature extraction unit that converts an input acoustic time series into an intermediate feature tensor, which is a third-order tensor indicating a feature of the acoustic time series and has time, frequency, and channel; an intermediate network that executes a non-equivalent second-order tensorization process that converts the representation of the intermediate feature tensor into a representation of a planar tensor that is a second-order tensor having a Cartesian product, which is a quantity indicating a Cartesian product of a frequency and a channel, and time, and a vectorization process that acquires, as the feature data, a one-dimensional vector that indicates a statistic of each Cartesian product in a time axis direction for each Cartesian product of the planar tensor; A learning device comprising:

2. The statistic is either an average value or a maximum value. The learning device according to claim 1 .

3. The feature data is a one-dimensional vector of a vector sum of a one-dimensional vector indicating an average value of each direct product quantity of the planar tensor in the time axis direction and a one-dimensional vector indicating a maximum value of each direct product quantity of the planar tensor in the time axis direction. The learning device according to claim 1 .

4. The non-equivalent second-order tensorization processing execution unit updates the plane tensor using a machine learning mathematical model. The learning device according to claim 1 .

5. a conversion object acquisition unit that acquires an acoustic time series that is a time series representing a sound; a neural network that converts an input acoustic time series into feature data, the feature data being information about sound obtained based on the acoustic time series and expressed in a predetermined format required by a downstream task; and an update unit that updates the neural network based on a result of execution of the downstream task using the feature data, wherein the neural network is a trained neural network obtained by learning using a learning device that includes: a feature extraction unit that converts the input acoustic time series into an intermediate feature tensor, the intermediate feature tensor being a third-order tensor indicating a feature of the acoustic time series, the intermediate feature tensor being a third-order tensor having time, frequency, and channel; a non-equivalent second-order tensorization process that converts the representation of the intermediate feature tensor into a representation of a planar tensor that is a second-order tensor having time and a Cartesian product, the Cartesian product being a quantity indicating a Cartesian product of frequency and channel; and a vectorization process that acquires, as the feature data, a one-dimensional vector indicating a statistical quantity of each Cartesian product in the time axis direction for each Cartesian product of the planar tensor; A conversion device comprising:

6. a neural network that converts an acoustic time series, which is a time series representing sound, into feature data, which is information about sound obtained based on the acoustic time series and expressed in a predetermined format required by a downstream task; and an update unit that updates the neural network based on a result of execution of the downstream task using the feature data, wherein the neural network comprises an intermediate network that performs: a feature extraction unit that converts an input acoustic time series into an intermediate feature tensor, which is a third-order tensor indicating a feature of the acoustic time series and has time, frequency, and channel; a non-equivalent second-order tensorization process that converts the representation of the intermediate feature tensor into a representation of a planar tensor that is a second-order tensor having time and a Cartesian product, which is a quantity indicating a Cartesian product of frequency and channel; and a vectorization process that acquires, for each Cartesian product of the planar tensor, a one-dimensional vector indicating a statistic of each Cartesian product in the time axis direction as the feature data, a feature extraction step of converting an input acoustic time series into an intermediate feature tensor, which is a third-order tensor indicating features of the acoustic time series and has time, frequency, and channel; a non-equivalent second-order tensorization processing execution step of converting the representation of the intermediate feature tensor into a representation of a plane tensor that is a second-order tensor having a Cartesian product that is a quantity indicating a Cartesian product of frequency and channel and time; a vectorization processing execution step of acquiring, as the feature amount data, a one-dimensional vector indicating a statistical amount of each direct product of the plane tensor in a time axis direction; A learning method that has

7. A program for causing a computer to function as the learning device according to any one of claims 1 to 4.

8. A program for causing a computer to function as the conversion device according to claim 5.

9. a neural network that converts an acoustic time series, which is a time series representing a sound, into feature data, which is information about the sound obtained based on the acoustic time series and expressed in a predetermined format required by a downstream task; an update unit that updates the neural network based on a result of execution of the downstream task using the feature amount data; Equipped with The neural network a feature extraction unit that acquires an intermediate feature tensor, which is a third-order tensor indicating a feature of the input acoustic time series and has time, frequency, and channel, based on the input acoustic time series; an intermediate network that performs a non-equivalent second-order tensorization process that converts the representation of the intermediate feature tensor into a representation of a plane tensor that is a second-order tensor having a direct product quantity that is a quantity indicating a direct product of frequency and channel and time; A learning device comprising:

10. the feature extraction unit acquires a plurality of intermediate feature tensors; The intermediate network acquires the plane tensor for each of the plurality of intermediate feature tensors. The learning device according to claim 9 .

11. The intermediate network includes a combination process for combining a plurality of the planar tensors; a vectorization process for acquiring, as the feature data, a one-dimensional vector indicating a statistical quantity of each direct product quantity in the time axis direction for each direct product quantity of the planar tensor after the combination process; The learning device according to claim 10.

12. the intermediate network performs a process of making the number of elements in the time direction of the plurality of intermediate feature tensors the same. The learning device according to claim 10 or 11.

13. The intermediate network performs a process of making the number of elements in the time direction of the plurality of planar tensors that are targets of the combining process the same. The learning device according to claim 10 or 11.

Images