Learning device, conversion device, learning method, conversion method, and program

Abstract

An embodiment of the present invention is a learning device provided with a control unit that obtains, through learning, a target representation conversion process for converting the representation of data to be converted into a target representation, which is a predetermined specific representation, wherein the control unit performs: a first representation conversion process in which the representation of a processing target, which is first data obtained by removing mask data from 0th data that is data represented by a tensor, is converted into a target representation; a mask data representation prediction process in which the target representation of the mask data is predicted on the basis of the result of the first representation conversion process; a second representation conversion process in which the representation of second data, which is part or all of the mask data, is converted into a target representation; and an update process in which the content of the first representation conversion process is updated so as to reduce the difference between the result of the mask data representation prediction process and the result of the second representation conversion process. The first representation conversion process at the point in time when a prescribed condition for ending the updating is satisfied is said target representation conversion process.

Description

[Technical Field]

[0001] The present invention relates to a learning device, a conversion device, a learning method, a conversion method, and a program. Regarding. [Background technology]

[0002] A technique is known that uses machine learning to generate a mathematical model that converts the representation of data to be converted into a predetermined representation. The predetermined representation is, for example, a representation required by a downstream task. Note that converting the representation of data to be converted into a predetermined representation refers to an encoding process that converts the data to be converted into data expressed in the predetermined representation. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Kaiming He, et al., "Masked autoencoders are scalable vision learners," arXiv preprint arXiv:2111.06377, 2021. [Non-patent document 2] A. Baevski, et al., “data2vec: A general framework for self-supervised learning in speech, vision and language,” in arXiv preprint arXiv:2202.03555, 2022. Summary of the Invention [Problem to be solved by the invention]

[0004] To date, MAE (Masked Autoencoders) and data2vec have been proposed as such technologies. Both of these mask part of the input information and use the results for training. However, both of these technologies sometimes have poor conversion accuracy.

[0005] In view of the above circumstances, an object of the present invention is to provide a technique for improving the accuracy of converting the representation of data to be converted into a predetermined representation. [Means for solving the problem]

[0006] One aspect of the present invention is a learning device that includes a control unit that acquires, through learning, a target representation conversion process, which is a process of converting a representation of data to be converted into a target representation that is a predetermined representation. The control unit executes the following: a first representation conversion process, which takes as a processing target first data, which is data obtained by removing mask data that is a part of zeroth data, which is data expressed by tensors, from the representation of the target data, a mask data representation prediction process, which predicts a target representation of the mask data based on a result of the first representation conversion process; a second representation conversion process, which converts a representation of the second data into a target representation based on second data that is all or part of the mask data; and an update process, which updates the contents of the first representation conversion process, the mask data representation prediction process, and the second representation conversion process so as to reduce a difference between a result of the mask data representation prediction process and a result of the second representation conversion process; and the first representation conversion process is the target representation conversion process when a predetermined condition for ending the update is satisfied.

[0007] One aspect of the present invention includes a conversion target acquisition unit that acquires data that is a target of conversion of an expression into a target expression that is a predetermined expression, and a control unit that acquires, through learning, a target expression conversion process that is a process of converting an expression of the data to be converted into a target expression that is a predetermined expression, wherein the control unit performs a first expression conversion process that converts the expression of the target to the target expression using first data, which is data obtained by removing mask data that is part of the zeroth data, which is data expressed by a tensor, as a processing target, and a mask data expression prediction process that predicts a target expression of the mask data based on a result of the first expression conversion process. The conversion device includes a representation conversion unit that converts the representation of the second data into a target representation based on second data that is part or all of the mask data, and an update process that updates the contents of the first representation conversion process, the mask data representation prediction process, and the second representation conversion process so as to reduce the difference between the result of the mask data representation prediction process and the result of the second representation conversion process, and a learning device in which the first representation conversion process is the target representation conversion process at the time a predetermined condition for ending the update is satisfied.

[0008] One aspect of the present invention is a learning method including a control step of obtaining, by learning, a target representation conversion process, which is a process of converting a representation of data to be converted into a target representation that is a predetermined representation, wherein the control step executes the following: a first representation conversion process, which uses first data as a processing target, which is data obtained by removing mask data that is a part of zeroth data, which is data expressed by tensors, from the zeroth data; a mask data representation prediction process, which predicts a target representation of the mask data based on a result of the first representation conversion process; a second representation conversion process, which converts a representation of the second data into a target representation based on second data that is all or part of the mask data; and an update process, which updates contents of the first representation conversion process, the mask data representation prediction process, and the second representation conversion process so as to reduce a difference between a result of the mask data representation prediction process and a result of the second representation conversion process; and the first representation conversion process is the target representation conversion process when a predetermined condition for ending the update is satisfied.

[0009] One aspect of the present invention includes a conversion target acquisition step of acquiring data to be converted into a target representation, which is a predetermined representation; and a control step of acquiring, by learning, a target representation conversion process, which is a process of converting the representation of the data to be converted into a target representation, which is a predetermined representation. The control step includes a first representation conversion process, which converts the representation of the target representation into a target representation, using first data, which is data obtained by removing mask data, which is part of zeroth data, from zeroth data, which is data expressed by a tensor, as the processing target; and a mask data representation prediction process, which predicts the target representation of the mask data based on a result of the first representation conversion process. a learning method that executes: a second representation conversion process that converts a representation of the second data into a target representation based on second data that is a part or all of the mask data; and an update process that updates the contents of the first representation conversion process, the mask data representation prediction process, and the second representation conversion process so as to reduce a difference between a result of the mask data representation prediction process and a result of the second representation conversion process, and in which the first representation conversion process is the target representation conversion process when a predetermined condition for ending the update is satisfied.

[0010] One aspect of the present invention is a program for causing a computer to function as either the learning device or the conversion device described above. [Effects of the Invention]

[0011] According to the present invention, it is possible to improve the accuracy of converting the representation of data to be converted into a predetermined representation. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an expression conversion system according to an embodiment. [Figure 2] FIG. 3 is an explanatory diagram illustrating a patch according to the embodiment. [Figure 3] FIG. 3 is an explanatory diagram illustrating an outline of a processing flow executed by a learning unit in the embodiment. [Figure 4] 10 is a flowchart showing an example of the flow of a learning process executed by a learning unit in the embodiment. [Figure 5] FIG. 2 is a diagram illustrating an example of a hardware configuration of a learning apparatus according to an embodiment. [Figure 6] FIG. 2 is a diagram showing an example of the configuration of a control unit in the embodiment. [Figure 7] FIG. 2 is a diagram illustrating an example of a hardware configuration of a conversion device according to an embodiment. [Figure 8] FIG. 2 is a diagram showing an example of the configuration of a control unit in the embodiment. [Figure 9] 10 is a flowchart showing an example of a flow of processing executed by a conversion device according to an embodiment. [Figure 10] FIG. 10 is a diagram showing an example of a result of an experiment according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] (Embodiment) 1 is a diagram illustrating an example of the configuration of an expression conversion system 100 according to an embodiment. The expression conversion system 100 includes a learning device 1 and a conversion device 2. The learning device 1 acquires, through learning, a process (hereinafter referred to as a "target expression conversion process") for converting an expression of data to be converted into a predetermined expression (hereinafter referred to as a "target expression").

[0014] Note that converting a data representation into a target representation means an encoding process that converts data to be encoded into data expressed in a target representation. Therefore, converting a representation of data to be converted into a target representation means an encoding process that converts data to be converted into data expressed in a target representation.

[0015] The target representation is, for example, a representation embedding. The target representation is, for example, a representation of 768 floating-point values. The target representation may be, for example, a representation of 1024 floating-point values. The target representation may be, for example, a representation of 2048 floating-point values. The representation conversion process is a type of learning model.

[0016] The data to be converted may be any data obtained based on image data, acoustic signal data, natural language data, or general time-series data, as long as it is expressed as a tensor. Therefore, the data to be converted may be, for example, image data, a spectrogram of acoustic signal data, or a series of word samples in which each word is expressed as an M-dimensional vector (M is a natural number greater than or equal to 1) (hereinafter referred to as a "natural language series"). The data to be converted may be, for example, general time-series data in which symbols other than letters and numbers are expressed as vectors.

[0017] A spectrogram of the acoustic signal data is obtained based on the acoustic signal data. A natural language sequence is obtained based on the natural language data.

[0018] The learning device 1 includes a learning unit 10. The learning unit 10 executes a first representation conversion process, a mask data representation prediction process, a second representation conversion process, and an update process. Note that the contents of the first representation conversion process, the mask data representation prediction process, and the second representation conversion process may have been updated in advance by, for example, transfer learning.

[0019] The first representation conversion process is a process of converting a representation of a processing target into a target representation. When the learning unit 10 executes the first representation conversion process, the processing target of the first representation conversion process is first data.

[0020] The first data is data obtained by removing the mask data, which is part of the 0th data, from the 0th data. The 0th data is data represented by a tensor.

[0021] The 0th data is, for example, a tensor that expresses the original data in a state where it has been divided into patches, and also has data position information that indicates the position of each patch, and is a tensor whose elements are tensors. The 0th data is, for example, a matrix whose elements are the vectors of each patch resulting from the patch division of the tensor to be divided. Patch division is a process of dividing the object to be divided into parts called patches.

[0022] The original data is data that corresponds to the object of conversion and is expressed as a tensor. Therefore, if the object of conversion is image data, the original data is image data.

[0023] When the data to be converted is data obtained based on acoustic signal data, the original data is data obtained based on the acoustic signal data and expressed as a tensor, such as a spectrogram of the acoustic signal data.

[0024] When the data to be converted is data obtained based on natural language data, the original data is data obtained based on the natural language data and expressed as a tensor. Such data is, for example, a natural language sequence. When the data to be converted is data obtained based on general time-series data, the original data is data obtained based on general time-series data and expressed as a tensor. Such data is, for example, a time series of numerical values ​​such as stock prices and temperature.

[0025] In this way, the original data is either image data, data obtained based on acoustic signal data and expressed in tensors, data obtained based on natural language data and expressed in tensors, or data obtained based on general time series data and expressed in tensors.

[0026] One element of the tensor in the 0th data indicates information about one patch. One patch contains one or more elements of the tensor that represents the original data. One element is data expressed as a 768-dimensional vector, for example. The position of a patch is the position within the 0th data of the element corresponding to each patch.

[0027] The division into patches and the assignment of data position information may be performed by the learning device 1. That is, the process of obtaining the 0th data based on the original data (hereinafter referred to as the "0th data generation process") may be performed by the learning device 1, or may be performed by a device other than the learning device 1. The 0th data may be, for example, the original data itself. In such a case, it is not necessary to perform the 0th data generation process.

[0028] Fig. 2 is an explanatory diagram illustrating a patch in an embodiment. Specifically, Fig. 2 is an explanatory diagram illustrating a patch in an example where the conversion target is acoustic signal data. More specifically, in the example of Fig. 2, it is an explanatory diagram illustrating a patch in an example where the acoustic signal data is a spectrogram configured with a frequency axis and a time axis.

[0029] To explain patches, Figure 2 shows a spectrogram divided into multiple rectangular sections of the same size on both the frequency and time axes. A patch is data that represents each of these sections as a vector. For example, one patch is the data that represents one section of area D1 in Figure 2.

[0030] A patch is, for example, a vector indicating the pixel values ​​of each pixel included in the corresponding partition. The number of elements of a vector representing a patch is the same regardless of the vector. The number of elements of a vector representing a patch may be the same regardless of the vector, and may be proportional to the number of pixels included in the partition, for example. The proportionality coefficient may or may not be 1. If the proportionality coefficient is not 1, the value of each element may be a value obtained by, for example, interpolation. For example, if one patch contains 256 pixels and the coefficient is 3, the patch is a 768-dimensional vector.

[0031] The data position information is, for example, a vector with the same number of dimensions as the vector representing the patch. Each element of the 0th data is, for example, a vector expressed as the vector sum of the vector representing the patch and the vector indicating the data position information.

[0032] In the example of Figure 2, the position of each patch is indicated by its position in the frequency axis direction and the time axis direction. Therefore, in the example of Figure 2, the 0th data is a matrix with vectors as elements. In the example of Figure 2, if the patch is a vector with the number of elements proportional to the number of pixels, the 0th data is a matrix with first-order tensors as elements. It goes without saying that a first-order tensor is a vector and a second-order tensor is a matrix. Incidentally, a 0th-order tensor is a scalar.

[0033] When the proportional coefficient is greater than 1, it is possible to suppress an increase in information loss when data is encoded in various processes such as the first representation conversion process.

[0034] In this way, the 0th data is obtained based on the original data. The value of one element of the tensor representing the 0th data is a vector representing each section when the tensor representing the original data is divided according to a predetermined rule. The dimension of the vector representing the section is the number of elements included in the section, which is, for example, larger than the number of elements of the tensor representing the original data.

[0035] Returning to the explanation of Figure 1, the mask data representation prediction process is a process for predicting a target representation of mask data based on the result of the first representation conversion process. The second representation conversion process is a process for converting a representation of second data, which is part or all of the mask data, into a target representation based on the second data. The update process is a process for updating the contents of the first representation conversion process, the mask data representation prediction process, and the second representation conversion process so as to reduce the difference between the result of the mask data representation prediction process and the result of the second representation conversion process (hereinafter referred to as "conversion error").

[0036] The conversion error may be the MSE (mean square error) between the result of the mask data representation prediction process and the result of the second representation conversion process, or may be L1, which is the mean absolute value of the difference.

[0037] In this way, the learning unit 10 performs learning of the first representation conversion process and the mask data representation prediction process so as to reduce the difference between the target representation of the mask data obtained based on data excluding the mask data and the target representation of the mask data obtained based on some or all of the mask data.

[0038] The conversion device 2 converts the representation of the data to be processed using the first representation conversion process at the time of update until a predetermined condition regarding the end of the update (hereinafter referred to as an "update end condition") is satisfied.

[0039] Here, an example of the flow of processing executed by the learning unit 10 will be described with reference to the drawings. FIG. 3 is an explanatory diagram outlining the flow of processing executed by the learning unit 10 in the embodiment. In FIG. 3, data x is an example of the 0th data. In the example of FIG. 3, the learning unit 10 generates first data and second data based on the 0th data x. In the example of FIG. 3, data D101 is an example of the first data, and data D102 is an example of the second data.

[0040] Data D101 is data in which a portion of the 0th data has been masked. Masking refers to a process that prevents data to be processed from being processed by a predetermined other process. In other words, masking refers to a process that restricts information access. The predetermined other process is, for example, a mask data representation prediction process. An example of the result of the masking process will be explained by showing an example of how the masked data is handled in the next stage. For example, if input data is a series of "010101010101" and is converted by masking into data of the series "****0101****", the data "*" will not be subject to processing in the next stage. This is the result of masking. In this example, "*" will not be subject to processing in the next stage. In other words, in this example, "*" is an example of data with restricted information access.

[0041] In FIG. 3, solid patches represent masked patches. Non-solid patches in FIG. 3 represent unmasked patches. Patch P1 is an example of a solid patch. Patch P2 is an example of a non-solid patch. A collection of masked patches is an example of mask data.

[0042] 3, the mask data in the first data is the non-mask data in the second data, so data including the non-mask data in the first data and the non-mask data in the second data includes the 0th data.

[0043] However, as described in the second representation conversion process, it is not necessary that all of the mask data in the first data be non-mask data in the second data. The non-mask data in the second data may be part of the mask data in the first data.

[0044] The determination of which data of the 0th data will be used as the mask data in the first data may be performed in any manner, for example, randomly. Hereinafter, the process of determining which data of the 0th data will be used as the mask data in the first data will be referred to as the first mask data determination process.

[0045] As described above, the second data is, for example, data determined as mask data in the first data. However, as described above, all of the data determined as mask data in the first data does not necessarily have to be second data. An example of how to determine the second data when only some, but not all, of the data determined as mask data in the first data is second data will be described below. In such a case, the determination of which of the data determined as mask data in the first data will be used as the mask data may be made in any manner, for example, randomly.

[0046] Hereinafter, the process of determining which of the data determined as mask data in the first data should be used as mask data will be referred to as the second mask data determination process. As can be seen from the explanation so far, if the second data is all of the data determined as mask data in the first data, the second mask data determination process does not necessarily need to be performed. This is because the data determined as mask data in the first data is the second data.

[0047] In the example of Fig. 3, the first mask data determination process and the second mask data determination process are executed by the learning unit 10. However, the first mask data determination process and the second mask data determination process do not necessarily have to be executed by the learning unit 10. For example, a device other than the learning device 1 may execute the process up to generating the first data and the second data, and the generated first data and second data may be executed by the learning unit 10. In such a case, the learning unit 10 does not execute the first mask data determination process and the second mask data determination process. Note that if the second data is all of the mask data in the first data, the second mask data determination process does not need to be executed.

[0048] In the example of Fig. 3, the learning unit 10 performs a first conversion process on the first data. θ (·) represents the first conversion process. θ represents the result of the addition process. The addition process is a process of adding mask tokens to the result of the first conversion process on the first data, the number of which is the same as the number of elements belonging to the mask data. The mask token is information indicating whether or not an element in the 0th data belongs to the mask data.

[0049] The elements belonging to the mask data are elements of the tensor representing the 0th data that belong to the mask data in the 1st data. Therefore, the elements belonging to the mask data are, for example, patches that are determined to be mask data in the first mask data determination process.

[0050] That is, the addition process is a process of adding a vector obtained by adding information indicating the position within the 0th data of patch P1 and information indicating that it is a mask token to the result of the first conversion process. In this way, the result of the addition process is an example of a result based on the result of the first conversion process. When the addition process is executed as in the example of FIG. 3, the addition process is executed by, for example, the learning unit 10.

[0051] In the example of Fig. 3, the learning unit 10 performs a mask data representation prediction process on the result of the addition process. In this way, the addition process is performed after the first representation conversion process and before the mask data representation prediction process. The function q in Fig. 3 θ (·) represents the mask data representation prediction process. θ represents the result of the mask data representation prediction process.

[0052] In the example of Fig. 3, the learning unit 10 performs the second conversion process on the second data. ξ (·) represents the second transformation process. The function f θ and the function f representing the second transformation process ξ Both of these are parameterized functions, and the parameter values ​​are updated by the update process.

[0053] Function f θ The symbol means the function f with the parameter value θ, and the function f ξ The symbol means the function f with the parameter value ξ. Therefore, the function f θ The symbol and the function f ξ The symbol z' in Figure 3 indicates that the functions of the first and second conversion processes are the same except for the difference in parameters. ξ represents the result of the second conversion process.

[0054] The result of the mask data representation prediction process is a target representation of the mask data. The result of the second conversion process is a target representation of all or part of the mask data. Therefore, by updating the contents of the first representation conversion process, the mask data representation prediction process, and the second representation conversion process based on the conversion error so as to reduce the conversion error, the accuracy of converting the representation of the data to be converted into a predetermined representation is improved.

[0055] 3 indicates that the contents of the first representation conversion process, the mask data representation prediction process, and the second representation conversion process are updated so as to reduce the conversion error. In other words, the "Maximize agreement" in FIG. 3 indicates that the learning unit 10 executes the update process.

[0056] "Stop gradient" in Figure 3 indicates that backpropagation is not performed when updating the content of the second representation conversion process. Specifically, when updating the content of the process, the parameter values ​​included in the function executed in the process are updated. Therefore, the function f ξ In the example of the second conversion process expressed as (1), updating the value of the parameter ξ is equivalent to updating the content of the second conversion process. In the example of Fig. 3, the content of the second representation conversion process is updated not by backpropagation but by another update process. The other update process is, for example, a predetermined exponential moving average process based on the content of the first representation conversion process.

[0057] An example of exponential moving average processing will be described. The contents of the first conversion processing and the second conversion processing change by changing the parameter values ​​of the parameterized functions that express each processing. The parameterized function that expresses the first conversion processing is, for example, a function f θ The parameterized function representing the second transformation process is, for example, the function f ξ In this case, the value of the parameter ξ after the t-th update obtained by the exponential moving average process is, for example, ξ[t]=βξ[t-1]+(1-β)θ[t-1], where β is a predetermined constant. For example, β is 0.99.

[0058] In fact, ξ[t]=βξ[t-1]+(1-β)θ[t-1] is the sum of the values ​​of θ at each update, weighted by a predetermined amount. Therefore, the exponential moving average process is the average value of the values ​​of θ at each update. The average value of the values ​​of θ at each update converges as the update progresses and accuracy increases. Therefore, the exponential moving average process enables updating to ξ with high conversion accuracy. Then, if the value of ξ converges to a value with high conversion accuracy, the value of θ will also converge to a value with high conversion accuracy as a result of the update process that attempts to reduce the conversion error.

[0059] 4 is a flowchart showing an example of the flow of the learning process executed by the learning unit 10 in the embodiment. The learning unit 10 acquires first data and second data (step S101). Next, the learning unit 10 executes a first conversion process (step S102). Next, the learning unit 10 executes a mask data expression prediction process (step S103). Next, the learning unit 10 executes a second conversion process (step S104).

[0060] Next, the learning unit 10 executes the update process (step S105). Next, the learning unit 10 determines whether the update termination condition is satisfied (step S106). If the update termination condition is satisfied (step S106: YES), the process ends. On the other hand, if the update termination condition is not satisfied (step S106: NO), the process returns to step S101.

[0061] The first representation conversion process at the time when the update termination condition is satisfied is the target representation conversion process. During learning by the learning unit 10, the processing target of the first representation conversion process was the first data. However, the mask data is not predetermined and is not the same for each learning. Therefore, the first representation conversion process at the time when the update termination condition is satisfied (i.e., the target representation conversion process) can convert the representation of a processing target that does not include mask data into a target representation with high accuracy. Note that the processes of steps S102 to S103 and the process of step S104 may be executed in parallel.

[0062] 5 is a diagram showing an example of the hardware configuration of a learning device 1 according to an embodiment. The learning device 1 includes a control unit 11 having a processor 91, such as a CPU (Central Processing Unit), and a memory 92 connected via a bus, and executes a program. By executing the program, the learning device 1 functions as a device including the control unit 11, an input unit 12, a communication unit 13, a storage unit 14, and an output unit 15.

[0063] More specifically, the processor 91 reads out a program stored in the storage unit 14 and stores the read out program in the memory 92. When the processor 91 executes the program stored in the memory 92, the learning device 1 functions as a device including a control unit 11, an input unit 12, a communication unit 13, a storage unit 14, and an output unit 15.

[0064] The control unit 11 controls the operation of various functional units included in the learning device 1. The control unit 11 includes a learning unit 10. Therefore, the control unit 11 executes, for example, a first representation conversion process, a mask data representation prediction process, a second representation conversion process, and an update process. The control unit 11 may further execute a first mask data determination process, a second mask data determination process, or an additional process.

[0065] The control unit 11 controls, for example, the operation of the output unit 15. The control unit 11 records, in the storage unit 14, various pieces of information generated by the execution of various processes, such as the first representation conversion process, the mask data representation prediction process, the second representation conversion process, and the update process.

[0066] Input unit 12 includes input devices such as a mouse, keyboard, and touch panel. Input unit 12 may be configured as an interface that connects these input devices to learning device 1. Input unit 12 accepts input of various types of information to learning device 1.

[0067] The communication unit 13 includes a communication interface for connecting the learning device 1 to an external device. The communication unit 13 communicates with the external device via wired or wireless communication. The external device is, for example, the device that sent the zeroth data. When the external device is the device that sent the zeroth data, the communication unit 13 acquires the zeroth data by communicating with the device that sent the zeroth data.

[0068] The external device is, for example, a device that has transmitted original data. When the external device is the device that has transmitted original data, the communication unit 13 acquires the original data by communicating with the device that has transmitted the original data. The external device is, for example, a device that has transmitted acoustic signal data.

[0069] The external device is, for example, a device that has transmitted acoustic signal data. When the external device is the device that has transmitted acoustic signal data, the communication unit 13 acquires the acoustic signal data by communicating with the device that has transmitted the acoustic signal data.

[0070] The external device is, for example, a device that has transmitted natural language data. When the external device is a device that has transmitted natural language data, the communication unit 13 acquires the natural language data by communicating with the device that has transmitted the natural language data.

[0071] The external device is a device that transmits general time series data in which symbols and numbers other than characters are expressed as vectors. When the external device is a device that transmits general time series data, the communication unit 13 acquires the general time series data by communicating with the device that transmits the general time series data.

[0072] The various information input to the communication unit 13 may be input to the input unit 12 instead of the communication unit 13 .

[0073] The external device is, for example, the conversion device 2. The communication unit 13 communicates with the conversion device 2 to transmit to the conversion device 2 information indicating the content of the first expression conversion process (i.e., the target expression conversion process) at the time when the update termination condition is satisfied.

[0074] The storage unit 14 is configured using a computer-readable storage medium device (non-transitory computer-readable recording medium) such as a magnetic hard disk drive or a semiconductor storage device. The storage unit 14 stores various information related to the learning device 1. The storage unit 14 stores information input via the input unit 12 or the communication unit 13, for example. The storage unit 14 stores various information generated by the operation of the control unit 11, for example.

[0075] The output unit 15 outputs various types of information. The output unit 15 includes a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or an organic EL (Electro-Luminescence) display. The output unit 15 may be configured as an interface that connects these display devices to the learning device 1. The output unit 15 outputs information input to the input unit 12, for example. The output unit 15 may also display the results of processing by the control unit 11, for example.

[0076] 6 is a diagram showing an example of the configuration of the control unit 11 in the embodiment. The control unit 11 includes a learning unit 10, a data acquisition unit 110, a storage control unit 120, a communication control unit 130, and an output control unit 140.

[0077] The data acquiring unit 110 acquires data to be transmitted to the learning unit 10. The data to be transmitted to the learning unit 10 may be the zeroth data or a set of the first data and the second data. When the data acquiring unit 110 transmits a set of the first data and the second data to the learning unit 10 and acquires the zeroth data, the data acquiring unit 110 executes a first mask data determination process and a second mask data determination process.

[0078] When the input unit 12 or the communication unit 13 acquires acoustic signal data, the data acquisition unit 110 acquires, as original data, data expressed in tensors that are obtained based on the acoustic signal data acquired by the input unit 12 or the communication unit 13. The data acquisition unit 110 then executes a 0th data generation process to acquire 0th data based on the acquired original data.

[0079] When the input unit 12 or the communication unit 13 acquires natural language data, the data acquisition unit 110 acquires, as original data, data expressed in tensors that are obtained based on the natural language data acquired by the input unit 12 or the communication unit 13. The data acquisition unit 110 then executes a 0th data generation process to acquire 0th data based on the acquired original data.

[0080] When the input unit 12 or the communication unit 13 acquires general time-series data in which symbols and numbers other than characters are expressed as vectors, the data acquisition unit 110 acquires, as original data, data expressed as tensors that are obtained based on the general time-series data acquired by the input unit 12 or the communication unit 13. The data acquisition unit 110 then executes a 0th data generation process to acquire 0th data based on the acquired original data.

[0081] When the input unit 12 or the communication unit 13 acquires the 0th data, the data acquisition unit 110 acquires the 0th data acquired by the input unit 12 or the communication unit 13. When the input unit 12 or the communication unit 13 acquires a pair of first data and second data, the data acquisition unit 110 acquires the pair of first data and second data acquired by the input unit 12 or the communication unit 13.

[0082] The storage control unit 120 records various information in the storage unit 14. The communication control unit 130 controls the operation of the communication unit 13. The output control unit 140 controls the operation of the output unit 15.

[0083] 7 is a diagram showing an example of the hardware configuration of the conversion device 2 according to the embodiment. The conversion device 2 includes a control unit 21 having a processor 93 such as a CPU and a memory 94 connected via a bus, and executes a program. By executing the program, the conversion device 2 functions as a device including the control unit 21, an input unit 22, a communication unit 23, a storage unit 24, and an output unit 25.

[0084] More specifically, the processor 93 reads out the program stored in the storage unit 24 and stores the read out program in the memory 94. When the processor 93 executes the program stored in the memory 94, the conversion device 2 functions as a device including the control unit 21, the input unit 22, the communication unit 23, the storage unit 24, and the output unit 25.

[0085] The control unit 21 controls the operation of various functional units included in the conversion device 2. The control unit 21 acquires, for example, information obtained by the learning device 1, indicating the content of the first expression conversion process (i.e., the target expression conversion process) at the time when the update termination condition is satisfied, and records this information in the storage unit 24.

[0086] The control unit 21 executes the target expression conversion process. The execution of the target expression conversion process by the control unit 21 is executed, for example, by the control unit 21 reading and executing information indicating the content of the target expression conversion process recorded in the storage unit 24. The control unit 21 controls, for example, the operation of the output unit 25. The control unit 21 records, for example, various pieces of information generated by the execution of the target expression conversion process in the storage unit 24.

[0087] The input unit 22 includes input devices such as a mouse, a keyboard, and a touch panel. The input unit 22 may be configured as an interface that connects these input devices to the conversion device 2. The input unit 22 accepts input of various information to the conversion device 2.

[0088] The communication unit 23 includes a communication interface for connecting the conversion device 2 to an external device. The communication unit 23 communicates with the external device via wired or wireless communication. The external device is, for example, a device that transmits data to be converted into a target expression. The communication unit 23 acquires data to be converted into a target expression by communicating with such an external device. The external device is, for example, the learning device 1. The communication unit 23 acquires information indicating the content of the target expression conversion process by communicating with the learning device 1.

[0089] The various information input to the communication unit 23 may be input to the input unit 22 instead of the communication unit 23 .

[0090] The storage unit 24 is configured using a computer-readable storage medium device (non-transitory computer-readable recording medium) such as a magnetic hard disk device or a semiconductor storage device. The storage unit 24 stores various information related to the conversion device 2. The storage unit 24 stores information input via the input unit 22 or the communication unit 23, for example. The storage unit 24 stores various information generated by the operation of the control unit 21, for example. The storage unit 24 stores the contents of the target expression conversion process, for example.

[0091] The output unit 25 outputs various types of information. The output unit 25 is, for example, a communication interface connected to be able to communicate with a device that executes a downstream task. The output unit 25 may be configured to include a display device such as a CRT display, a liquid crystal display, or an organic EL display. The output unit 25 may be configured as an interface that connects these display devices to the conversion device 2. The output unit 25 outputs, for example, information input to the input unit 22. The output unit 25 may also output, for example, the execution result of the target expression conversion process.

[0092] 8 is a diagram showing an example of the configuration of the control unit 21 in the embodiment. The control unit 21 includes a conversion target acquisition unit 210, an expression conversion unit 220, a memory control unit 230, a communication control unit 240, and an output control unit 250. The conversion target acquisition unit 210 acquires data that is input to the communication unit 23 and is the target of conversion of an expression into a target expression.

[0093] The expression conversion unit 220 uses a target expression conversion process to convert the expression of the data acquired by the conversion target acquisition unit 210. The expression conversion unit 220 may execute various processes executed by the data acquisition unit 110, such as the zeroth data generation process, according to the data acquired by the conversion target acquisition unit 210, so as to be able to execute the target expression conversion process.

[0094] The storage control unit 230 records various information in the storage unit 24. The communication control unit 240 controls the operation of the communication unit 23. The output control unit 250 controls the operation of the output unit 25.

[0095] 9 is a flowchart showing an example of the flow of processing executed by the conversion device 2 in the embodiment. The conversion target acquisition unit 210 acquires data that is input to the communication unit 23 and is the target of conversion of the expression into a target expression (step S201). Next, the expression conversion unit 220 converts the expression of the data acquired in step S201 using a target expression conversion process (step S202). Next, the output control unit 250 controls the operation of the output unit 25 to cause the output unit 25 to output the result of step S202 (step S203). Note that the destination of the output by the output unit 25 may be, for example, a device that executes a downstream task.

[0096] <Experimental Results> The results of an experiment in which downstream tasks were performed on data whose expression was converted using the target expression conversion process will be explained using Fig. 10. The downstream tasks performed in the experiment were environmental sound classification, voice command word identification, speaker identification, spoken language classification, classification of emotions contained in voice, music genre classification, musical instrument classification of musical sounds, and musical pitch classification of musical sounds.

[0097] FIG. 10 shows an example of the results of an experiment in an embodiment. "ESC50" and "US8K" both indicate a task of environmental sound classification. "SPCV2" indicates a task of voice command word identification. "VC1" indicates a task of speaker identification. "VF" indicates a task of spoken language classification. "CRM-D" indicates a task of emotion classification contained in voice. "GTZAN" indicates a task of music genre classification. "NSynth" indicates a task of musical instrument classification. "Surge" indicates a task of musical pitch classification.

[0098] "MAE" refers to the existing technology, MAE (Masked Autoencoders). "MABL" refers to target representation conversion processing. The results in Figure 9 show that the accuracy of downstream tasks using the results of representation conversion by target representation conversion processing is higher than the accuracy of downstream tasks using the results of representation conversion by MAE for all downstream tasks.

[0099] For example, for “ESC50”, the accuracy of the downstream task “ESC50” using the results of expression conversion by MAE is 87.35%, while the accuracy of the downstream task “ESC50” using the results of expression conversion by the target expression conversion process is 89.03%.

[0100] For example, for “VC1”, the accuracy of the downstream task “VC1” using the results of expression conversion by MAE is 54.64%, while the accuracy of the downstream task “VC1” using the results of expression conversion by target expression conversion processing is 58.96%.

[0101] The learning device 1 configured in this manner learns the first representation conversion process so as to reduce the difference between the representation of mask data obtained based on data excluding the mask data and the representation of mask data obtained based on some or all of the mask data.

[0102] This differs from MAE, which reconstructs the patch image of the masked portion from the representation of the non-masked patch output by the model and calculates the loss using the difference between the input signal and the reconstructed signal. In the case of MAE, reconstruction (i.e., decoding) is performed after the representation conversion, and therefore errors may occur in the information due to the reconstruction. Therefore, unlike MAE, the learning device 1, which does not perform reconstruction during learning, can improve the accuracy of converting the representation of the data to be converted into a predetermined representation (i.e., target representation).

[0103] Furthermore, learning by the learning device 1 differs from data2vec. In the case of data2vec, all patches are input to obtain a target representation of the moving average model, and only the masked portion of the target representation is used as the training signal. Therefore, the target representation of the masked portion contains information about the unmasked portion. Note that the masked portion refers to the masked data, and the unmasked portion refers to the data in the 0th data other than the masked data. As a result, in data2vec, learning relies on information about the unmasked portion, and learning that improves the accuracy of representation conversion may not be performed.

[0104] On the other hand, in learning by the learning device 1, a target representation of part or all of the masked data is obtained in the second representation conversion process without using data that is not masked data among the 0th data. The result of the second representation conversion process is then compared with the result of the masked data representation prediction process. That is, in learning by the learning device 1, the result of the second representation conversion process is used as a teacher signal. As described above, the result of the second representation conversion process does not include information on the unmasked portion. Therefore, unlike data2vec, the learning device 1 can improve the accuracy of converting the representation of the data to be converted into a predetermined representation.

[0105] (Variation) In the first data, the proportion of mask data may be 50% of the data included in the 0th data. When the proportion of mask data is 50%, the accuracy of converting the expression of the data to be converted into a predetermined expression is improved compared to when the proportion is not 50% for the following reason (hereinafter referred to as "first reason").

[0106] <1st reason> The target representations output by the first representation conversion process and the second representation conversion process each have a small conversion error when the entire data, including the respective mask data, is modeled. Modeling is easy when there is little mask data, and difficult when there is a lot of mask data. Target representations processed with different levels of difficulty are heterogeneous, with different amounts of information available for modeling. By setting the proportion of mask data to 50%, the first and second data become homogeneous target representations processed using the same amount of information, and the conversion error is optimal for learning better modeling.

[0107] In the first data, the proportion of data that is not mask data may be 50% or less of the data included in data 0, and the proportion of data that is part of the mask data and is used in the second representation conversion process may also be 50% or less of the data included in data 0. In such a case, the difficulty of modeling increases, and therefore the accuracy of converting the representation of the data to be converted into a predetermined representation improves compared to other cases.

[0108] The learning device 1 and the conversion device 2 may each be implemented using a plurality of information processing devices communicably connected via a network. In this case, the functional units of the learning device 1 and the conversion device 2 may be distributed and implemented among the plurality of information processing devices.

[0109] The learning device 1 and the conversion device 2 do not necessarily have to be implemented as different devices, but may be implemented as a single device that combines the functions of both the learning device 1 and the conversion device 2, for example.

[0110] All or part of the functions of the representation conversion system 100, the learning device 1, and the conversion device 2 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. The program may be transmitted via a telecommunications line.

[0111] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention. [Explanation of symbols]

[0112] 100...representation conversion system, 1...learning device, 2...conversion device, 10...learning unit, 11...control unit, 12...input unit, 13...communication unit, 14...memory unit, 15...output unit, 110...data acquisition unit, 120...memory control unit, 130...communication control unit, 140...output control unit, 21...control unit, 22...input unit, 23...communication unit, 24...memory unit, 25...output unit, 210...conversion target acquisition unit, 220...representation conversion unit, 230...memory control unit, 240...communication control unit, 250...output control unit, 91...processor, 92...memory, 93...processor, 94...memory

Claims

1. a control unit that acquires, through learning, a target expression conversion process, which is a process of converting a representation of data to be converted into a target expression that is a predetermined representation; Equipped with The control unit a first representation conversion process for converting a representation of the target data to a target representation, the first representation conversion process being data obtained by removing mask data that is a part of the zeroth data from the zeroth data that is data expressed by a tensor; a mask data representation prediction process for predicting a target representation of the mask data based on a result of the first representation conversion process; a second representation conversion process for converting a representation of the second data to a target representation based on second data that is a part or all of the mask data; and an update process for updating contents of the first representation conversion process, the mask data representation prediction process, and the second representation conversion process so as to reduce a difference between a result of the mask data representation prediction process and a result of the second representation conversion process; the first representation conversion process at the time when a predetermined condition regarding the end of the update is satisfied is the target representation conversion process; Learning device.

2. the control unit executes an addition process after execution of the first representation conversion process and before the mask data representation prediction process, in which mask tokens indicating whether or not elements in the zero data belong to mask data are added to a result of the first representation conversion process, the number of mask tokens being the same as the number of elements belonging to the mask data; The processing target of the mask data representation prediction processing is the result of the additional processing. The learning device according to claim 1 .

3. a ratio of mask data in the first data to data included in the 0 data is 50%; The learning device according to claim 1 .

4. In the first data, a ratio of data that is not mask data to a ratio of data included in the 0 data is 50% or less, and a ratio of data that is part of the mask data and is used in the second representation conversion process is also 50% or less to a ratio of data included in the 0 data. The learning device according to claim 1 .

5. a conversion target acquisition unit that acquires data that is a target of expression conversion into a target expression that is a predetermined expression; a control unit that acquires, by learning, a target representation conversion process, which is a process of converting a representation of data to be converted into a target representation that is a predetermined representation, wherein the control unit executes: a first representation conversion process, which takes first data as a processing target, which is data obtained by excluding mask data that is a part of zeroth data, which is data expressed by a tensor, from the zeroth data, the first representation conversion process, and converts the representation of the processing target into a target representation; a mask data representation prediction process that predicts a target representation of the mask data based on a result of the first representation conversion process; a second representation conversion process that converts a representation of second data into a target representation based on second data that is a part or all of the mask data; and an update process that updates contents of the first representation conversion process, the mask data representation prediction process, and the second representation conversion process so as to reduce a difference between a result of the mask data representation prediction process and a result of the second representation conversion process; and a representation conversion unit that converts the representation of data acquired by the conversion target acquisition unit using the target representation conversion process acquired by a learning device in which the first representation conversion process at the time when a predetermined condition for completion of the update is satisfied is the target representation conversion process; A conversion device comprising:

6. a control step of acquiring, by learning, a target expression conversion process, which is a process of converting a representation of data to be converted into a target expression that is a predetermined representation; and The control step a first representation conversion process for converting a representation of the target data to a target representation, the first representation conversion process being data obtained by removing mask data that is a part of the zeroth data from the zeroth data that is data expressed by a tensor; a mask data representation prediction process for predicting a target representation of the mask data based on a result of the first representation conversion process; a second representation conversion process for converting a representation of the second data to a target representation based on second data that is a part or all of the mask data; and an update process for updating contents of the first representation conversion process, the mask data representation prediction process, and the second representation conversion process so as to reduce a difference between a result of the mask data representation prediction process and a result of the second representation conversion process; the first representation conversion process at the time when a predetermined condition regarding the end of the update is satisfied is the target representation conversion process; How to learn.

7. a conversion target acquisition step of acquiring data to be converted into a target expression, which is a predetermined expression; a control step of acquiring, by learning, a target representation conversion process that converts a representation of data to be converted into a target representation that is a predetermined representation, wherein the control step executes: a first representation conversion process that converts a representation of a target representation into a target representation, using first data as a processing target, the first data being data obtained by excluding mask data that is a part of the zeroth data, the zeroth data being data expressed by a tensor; a mask data representation prediction process that predicts a target representation of the mask data based on a result of the first representation conversion process; a second representation conversion process that converts a representation of the second data into a target representation based on second data that is a part or all of the mask data; and an update process that updates contents of the first representation conversion process, the mask data representation prediction process, and the second representation conversion process so as to reduce a difference between a result of the mask data representation prediction process and a result of the second representation conversion process, and wherein the first representation conversion process at the time a predetermined condition for ending the update is satisfied is the target representation conversion process; A conversion method having the following structure:

8. A program for causing a computer to function as either the learning device according to any one of claims 1 to 4 or the conversion device according to claim 5.

Images