Training device and inference device

Abstract

This learning device is provided with a control unit that performs learning of a learning target model using a pair of acoustic-side data, which is data relating to an acoustic signal, and explanatory sentence data, which is text data for explaining the content of the acoustic-side data, as training data. The learning target model executes: a first feature quantity inference process for inferring a feature quantity of masked data, which is a part of the acoustic-side data, on the basis of the masked data; an intermediate acoustic feature quantity inference process for inferring an unmasked intermediate acoustic feature quantity, which is a feature quantity of at least a part of the acoustic-side data different from the masked data, on the basis of said part; a second feature quantity inference process for inferring a feature quantity of masked data on the basis of the unmasked intermediate acoustic feature quantity; a first text semantic feature quantity inference process for inferring a text semantic feature quantity, which is a feature quantity of the explanatory sentence data, on the basis of the explanatory sentence data; and a second text semantic feature quantity inference process for inferring a text semantic feature quantity on the basis of the feature quantity inferred from the first feature quantity inference process or the unmasked intermediate acoustic feature quantity.

Description

Learning and inference devices

[0001] The present invention relates to a learning device and an inference device.

[0002] There has been growing interest in techniques for obtaining, through learning, mathematical models that estimate features to be used in downstream tasks (see Non-Patent Documents 1 and 2). Examples of downstream tasks include image-related processing such as image classification and adding explanatory text to images (see Non-Patent Document 2), and sound-related processing such as emotion recognition based on speech and music genre classification (see Non-Patent Document 1). In such sound-related downstream tasks, processing such as emotion recognition and genre classification is performed based on sound features.

[0003] Jinsen, et al., "Masked Modeling Duo: Learning Representations by Encouraging Both Networks to Model the Input," ICASSP, 2023.Mu et al., "SLIP: Self-supervision Meets Language-Image Pre-training," in ECCV 2022. https: / / arxiv.org / abs / 2112.12750

[0004] However, until now, depending on the downstream task, it was sometimes necessary to change the mathematical model used to obtain features, which could impose a heavy burden on users because it was necessary to change the technology used to obtain features depending on the downstream task.

[0005] In view of the above circumstances, an object of the present invention is to provide a technique for reducing the burden required to acquire features used in downstream tasks.

[0006] One aspect of the present invention is a learning object model, which is a mathematical model to be learned, comprising a control unit that performs learning using pairs of acoustic side data, which is data related to an acoustic signal, and explanatory sentence data, which is text data explaining the content of the acoustic side data, as training data, and the learning object model includes a first feature amount inference process that infers features of masked data, which is a part of the acoustic side data included in the training data, based on the masked data, an intermediate acoustic feature amount inference process that infers unmasked intermediate acoustic features, which are features of at least a part of the acoustic side data different from the masked data, based on the part, a second feature amount inference process that infers features of the masked data based on the unmasked intermediate acoustic features, and an explanatory sentence data paired with the acoustic side data. and a second text semantic feature inference process that infers text semantic features based on features inferred from the first feature inference process or the unmasked intermediate acoustic features, and in the learning, the first feature inference process, the intermediate acoustic feature inference process, the second feature inference process, and the second text semantic feature inference process are updated so as to reduce a difference between the features inferred in the first feature inference process and the features inferred in the second feature inference process, and a difference between the text semantic features inferred in the first text semantic feature inference process and the text semantic features inferred in the second text semantic feature inference process.

[0007] One aspect of the present invention is a learning of a learning object model, which is a mathematical model to be learned, comprising a control unit that performs learning using pairs of acoustic side data, which is data related to an acoustic signal, and explanatory sentence data, which is text data explaining the content of the acoustic side data, as training data, wherein the learning object model includes a first feature inference process that infers features of masked data, which is a part of the acoustic side data included in the training data, based on the masked data; an intermediate acoustic feature inference process that infers unmasked intermediate acoustic features, which are features of at least a part of the acoustic side data different from the masked data, based on the part; a second feature inference process that infers features of the masked data based on the unmasked intermediate acoustic features; and a first feature inference process that infers text semantic features, which are features of explanatory sentence data paired with the acoustic side data, based on the explanatory sentence data. and an inference unit that executes a text semantic feature inference process and a second text semantic feature inference process that infers text semantic features based on features inferred using the first feature inference process or the unmasked intermediate acoustic features, wherein the first feature inference process, the intermediate acoustic feature inference process, the second feature inference process, and the second text semantic feature inference process are updated in the learning process so as to reduce a difference between the features inferred in the first feature inference process and the features inferred in the second feature inference process, and a difference between the text semantic features inferred in the first text semantic feature inference process and the text semantic features inferred in the second text semantic feature inference process.

[0008] The present invention makes it possible to reduce the burden required to acquire feature amounts used in downstream tasks.

[0009] An explanatory diagram for explaining an information processing system according to an embodiment. An explanatory diagram for explaining an example of a learning object model according to an embodiment. A diagram showing an example of the hardware configuration of a learning device according to an embodiment. A flowchart showing an example of the flow of processing executed by the learning device according to an embodiment. A diagram showing an example of the hardware configuration of an inference device according to an embodiment. A flowchart showing an example of the flow of processing executed by the inference device according to an embodiment. A first diagram showing an example of experimental results of an experiment according to an embodiment. A second diagram showing an example of experimental results of an experiment according to an embodiment.

[0010] 1 is an explanatory diagram illustrating an information processing system 100 according to an embodiment. The information processing system 100 includes a learning device 1 and an inference device 2.

[0011] The learning device 1 includes a control unit 11 including a processor 91 such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an NPU (Neural Network Processing Unit), and a memory 92, which are connected via a bus.

[0012] The control unit 11 performs a learning process. The learning process is a process in which learning of a mathematical model to be learned (hereinafter referred to as a "learning target model") is performed until a predetermined condition for terminating learning (hereinafter referred to as a "learning termination condition") is satisfied. The learning termination condition may be any condition related to the termination of learning, and may be, for example, a condition that the learning target has been updated a predetermined number of times, or a condition that the change in the learning target due to the update is smaller than a predetermined change. The state of the mathematical model or process at the time when the learning termination condition is satisfied is what is called a "learned" state.

[0013] As is well known, training data is used in the learning stage of machine learning. In the learning of the model to be trained, pairs of acoustic training data and text training data are used as the training data.

[0014] The acoustic training data is acoustic side data included in the training data. The acoustic side data is data related to an acoustic signal. The acoustic side data is, for example, the acoustic signal itself. The acoustic side data may be, for example, data obtained from the acoustic signal. The data obtained from the acoustic signal is, for example, a spectrogram of the acoustic signal.

[0015] The text training data is explanatory sentence data included in the training data and explaining the content of the paired audio training data. The explanatory sentence data is character string data explaining the content of the audio data and is composed of a series of character codes. The character code of the explanatory sentence data may be, for example, an English character code. In other words, the explanatory sentence data is text data indicating explanatory sentences explaining the audio data.

[0016] For example, if the audio data is a recording of the sound of rain, the content of the audio data is "rain." Therefore, in this case, the explanatory data explaining the content of the audio data is, for example, character string data indicating the text "It's raining." For example, if the audio data is a recording of a conversation, the content of the audio data is "conversation." Therefore, in this case, the explanatory data explaining the content of the audio data is, for example, character string data indicating the text "A conversation is taking place." For example, if the audio data is a recording of environmental sounds including music and a dog barking, the content of the audio data is "complex environmental sounds consisting of music and a dog barking." Therefore, in this case, the explanatory data explaining the content of the audio data is, for example, character string data indicating the text "A dog barked while the music was playing."

[0017] The relationship between the audio data and the explanatory sentence data included in the training data is such that the explanatory sentence data is text data that explains the content of the corresponding audio data.

[0018] The learning target model executes a first feature amount inference process, an intermediate acoustic feature amount inference process, a second feature amount inference process, a first text semantic feature amount inference process, and a second text semantic feature amount inference process.

[0019] The first feature inference process is a process of inferring features of mask data based on the mask data. The features of the mask data are part of the acoustic training data. Hereinafter, the features of the mask data inferred by the first feature inference process will be referred to as "first features."

[0020] The intermediate acoustic feature inference process is a process of inferring unmasked intermediate acoustic features, which are features of at least a portion of the acoustic training data that is different from the masked data (hereinafter referred to as "non-masked data"), based on the portion (i.e., the non-masked data). The at least a portion of the acoustic training data that is different from the masked data is, in other words, at least a portion of the acoustic training data that is not masked data.

[0021] The second feature inference process is a process of inferring features of masked data based on unmasked intermediate acoustic features. Hereinafter, the features of masked data inferred by the second feature inference process will be referred to as second features.

[0022] The first text semantic feature inference process is a process of inferring the feature quantities of the text training data (hereinafter referred to as "text semantic features") based on the text training data.

[0023] The second text semantic feature inference process is a process of inferring text semantic features based on unmasked intermediate acoustic features or features of masked data inferred in the first feature inference process. That is, the second text semantic feature inference process is a process of inferring text semantic features based on a predetermined one or both of the unmasked intermediate acoustic features and the features of masked data inferred in the first feature inference process.

[0024] The features inferred by the first text semantic feature inference process and the features inferred by the second text semantic feature inference process are both features of text training data. However, the processes for obtaining these features and the information on which they are based are different. Therefore, the features inferred by the first text semantic feature inference process and the features inferred by the second text semantic feature inference process are not necessarily the same.

[0025] In the learning, the first feature inference process, the intermediate acoustic feature inference process, the second feature inference process, and the second text semantic feature inference process are updated so as to reduce the difference between the first feature and the second feature, and the difference between the text semantic feature inferred in the first text semantic feature inference process and the text semantic feature inferred in the second text semantic feature inference process.

[0026] In addition, during the learning, the first text semantic feature inference process may also be updated so as to reduce the difference between the first feature and the second feature, and the difference between the text semantic feature inferred by the first text semantic feature inference process and the text semantic feature inferred by the second text semantic feature inference process.

[0027] Note that updating a process means updating the parameter values ​​of a parameterized function that represents the process. The process may be performed by, for example, running a neural network, in which case updating the process means updating the parameter values ​​of the neural network.

[0028] The inference device 2 executes an inference process. The inference process is a process of inferring at least the feature quantities of the acoustic data of the feature inference target using the trained intermediate acoustic feature inference process and the trained second text semantic feature inference process obtained by the learning device 1. The feature quantities of the acoustic data obtained by the intermediate acoustic feature inference process are an example of feature quantities that can be used in downstream tasks by transfer learning. Furthermore, semantic features obtained by the second text semantic feature inference process based on these feature quantities are an example of feature quantities that can be used in downstream tasks by zero-shot learning.

[0029] The inference device 2 may, for example, execute downstream tasks such as zero-shot learning in addition to the inference processing. The downstream task may be, for example, a process of performing inference based on text data, such as zero-shot learning. In such a case, the inference device 2 may further execute, for example, a trained first text semantic feature inference process in the inference processing. When the trained first text semantic feature inference process is executed in the inference processing, the trained first text semantic feature inference process is executed, and therefore the inference device 2 also infers features of text data indicating explanatory sentences that explain the acoustic data that is the subject of feature inference.

[0030] 2 is an explanatory diagram illustrating an example of a training model in an embodiment. Fig. 2 illustrates a mathematical model including Masked Modeling Duo (M2D) and Contrastive Language-Audio Pre-training (CLAP) as an example of the training model. In Fig. 2, CLAP is indicated as "CLAP Extension."

[0031] Data D101 shown in Fig. 2 is an example of acoustic data. The acoustic data in the example of Fig. 2 is a spectrogram of an acoustic signal. Data D102 shown in Fig. 2 is an example of a first feature. Data D103 shown in Fig. 2 is an example of a second feature.

[0032] In the example of FIG. 2, "A saxophone gently plays a jazzy tune" is an example of description data. The text indicated by this description data is a caption that explains the content of the sound indicated by data D101. "Text Encoder" shown in FIG. 2 represents the execution of text semantic feature inference processing. Therefore, output D104 of "Text Encoder" shown in FIG. 2 is an example of text semantic features.

[0033] Data D105 shown in Fig. 2 is an example of masked data, data D106 shown in Fig. 2 is an example of unmasked data, and data D107 shown in Fig. 2 is an example of unmasked intermediate acoustic features.

[0034] The example in Figure 2 shows the loss L indicated as "Online (M2D) loss" in training. ｍ２ｄ and the loss L denoted as "Semantic (CLAP) loss" ｃｌｉｐ In the example of FIG. ｍ２ｄ Specifically, the loss L indicates the difference between the first feature amount and the second feature amount. ｃｌｉｐ Specifically, the loss indicates the difference between the text semantic features obtained by executing the intermediate acoustic feature inference process and the second text semantic feature inference process and the text semantic features obtained by executing the first text semantic feature inference process.

[0035] In the example of Fig. 2, the process of obtaining data D102 based on data D105 is an example of a first feature inference process. In the example of Fig. 2, the process of obtaining data D107 based on data D106 is an example of an intermediate acoustic feature inference process. In the example of Fig. 2, the process of obtaining data D103 based on data D107 is an example of a second feature inference process. In the example of Fig. 2, the first feature inference process, intermediate acoustic feature inference process, and second feature inference process are processes executed in M2D.

[0036] <Effects of the Learning Process> The following describes the effects of the learning process executed by the control unit 11. As described above, the inference device 2 uses a trained intermediate acoustic feature inference process and a trained second text semantic feature inference process, which are one of the processes included in the learning target model. Based on the output of the trained intermediate acoustic feature inference process, the features output by the trained second text semantic feature inference process are used in downstream tasks including zero-shot learning.

[0037] The learning target model is trained by a training process, in which the first feature inference process, intermediate acoustic feature inference process, second feature inference process, and second text semantic feature inference process are updated based on the first feature, the second feature, the text semantic feature inferred by the first text semantic feature inference process, and the text semantic feature inferred by the second text semantic feature inference process. Therefore, the trained second text semantic feature inference process outputs features that also reflect the training results of the correspondence between explanatory sentence data and acoustic signals.

[0038] In the case of the techniques described in Non-Patent Document 1 and Non-Patent Document 2, the correspondence between the description data and the audio signal is not learned. Since there is no correspondence, it is natural that the features obtained by the techniques of Non-Patent Document 1 and Non-Patent Document 2 cannot be applied to zero-shot learning of the audio signal.

[0039] By executing the learning process in this way, unlike the techniques described in Non-Patent Document 1 and Non-Patent Document 2, a process for obtaining features used in zero-shot learning that matches sounds with explanatory text that explains those sounds (hereinafter referred to as "sound zero-shot learning") is obtained (i.e., a learned intermediate acoustic feature inference process and a learned second text semantic feature inference process).

[0040] When using the trained intermediate acoustic feature inference process and the trained second text semantic feature inference process obtained in this way, the user can enjoy, for example, the following benefits. The task targeted by the sound zero-shot learning is when a downstream task is configured with a pair of a sound and a sound to be searched for. Therefore, the features obtained using the trained process obtained in the learning process can be applied to any downstream task configured with such a pair of a sound and a sound to be searched for. Therefore, even when the user changes the type of sound zero-shot learning to be performed, the user does not need to change the mathematical model used to acquire the features. This is one of the benefits the user can enjoy. Therefore, by executing the learning process, the burden required to acquire features to be used in downstream tasks related to sound is reduced. Therefore, by executing the learning process, the burden required to acquire features to be used in downstream tasks is reduced.

[0041] 3 is a diagram showing an example of the hardware configuration of the learning device 1 according to an embodiment. The learning device 1 is equipped with a control unit 11, which is a control unit including a processor 91 such as a CPU, GPU, or NPU, and a memory 92, which are connected via a bus, and executes a program. By executing the program, the learning device 1 functions as a device including the control unit 11, an interface unit 12, and a storage unit 13.

[0042] More specifically, the processor 91 reads the program stored in the storage unit 13 and stores the read program in the memory 92. The processor 91 executes the program stored in the memory 92, causing the learning device 1 to function as a device including the control unit 11, the interface unit 12, and the storage unit 13.

[0043] The control unit 11 controls the operation of each functional unit included in the learning device 1. The control unit 11 executes, for example, a learning process. The control unit 11 acquires, for example, information stored in the memory unit 13. Specifically, the process of acquiring information stored in the memory unit 13 is reading.

[0044] The interface unit 12 includes a communication interface for connecting the learning device 1 to an external device. The interface unit 12 communicates with the external device via a wired or wireless connection.

[0045] The external device is, for example, a device that transmits signals or information used for learning. The interface unit 12 acquires the signals or information used for learning by communicating with the device that transmits the signals or information used for learning. The signals or information used for learning are, for example, acoustic data. The signals or information used for learning are, for example, explanatory data.

[0046] The external device may be, for example, the inference device 2. In such a case, the inference device 2 can execute inference processing of trained intermediate acoustic features obtained by executing a training process and inference processing of trained second text semantic features through communication via the interface unit 12. Furthermore, in such a case, if the inference device 2 also performs inference based on text data such as zero-shot training, the inference device 2 may also be able to execute inference processing of trained first text semantic features obtained by executing a training process through communication via the interface unit 12.

[0047] The interface unit 12 may be configured to include input devices such as a mouse, keyboard, or touch panel. The interface unit 12 may be configured as an interface that connects these input devices to the learning device 1. In this way, the input devices of the interface unit 12 accept input of various information to the learning device 1 via wired or wireless connections. Note that signals or information do not necessarily have to be input to the communication interface of the interface unit 12, but may also be input to the input devices of the interface unit 12.

[0048] The interface unit 12 outputs, for example, various types of information. The interface unit 12 includes a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or an organic EL (Electro-Luminescence) display, as well as a speaker. The interface unit 12 may be configured as an interface that connects these display devices or speakers to the learning device 1. Therefore, the interface unit 12 may output, for example, information input to an input device of the interface unit 12 as an image or sound.

[0049] The storage unit 13 is configured using a computer-readable storage medium (non-transitory computer-readable recording medium) such as a magnetic hard disk drive or a semiconductor storage device. The storage unit 13 stores various information related to the learning device 1. The storage unit 13 stores various information generated by the operation of the control unit 11, for example. The storage unit 13 may exist on a cloud, for example.

[0050] 4 is a flowchart showing an example of the flow of processing executed by the learning device 1 in the embodiment. The control unit 11 executes the learning processing (step S101).

[0051] 5 is a diagram showing an example of the hardware configuration of the inference device 2 in an embodiment. The inference device 2 is equipped with a control unit 21 including a processor 93 such as a CPU, GPU, or NPU, and a memory 94, which are connected by a bus, and executes a program. By executing the program, the inference device 2 functions as a device equipped with the control unit 21, an interface unit 22, and a memory unit 23.

[0052] More specifically, the processor 93 reads the program stored in the storage unit 23 and stores the read program in the memory 94. When the processor 93 executes the program stored in the memory 94, the inference device 2 functions as a device including the control unit 21, the interface unit 22, and the storage unit 23.

[0053] The control unit 21 controls the operation of each functional unit included in the inference device 2. The control unit 21 executes, for example, an inference process. The control unit 21 may execute, for example, a learned first text semantic feature inference process. The control unit 21 acquires, for example, information stored in the memory unit 23. Specifically, the process of acquiring information stored in the memory unit 23 is reading.

[0054] The interface unit 22 includes a communication interface for connecting the inference device 2 to an external device. The interface unit 22 communicates with the external device via wired or wireless communication.

[0055] The external device is, for example, a device that transmits the processing target of the inference processing. Therefore, the external device is, for example, a device that transmits the processing target of the trained intermediate acoustic feature inference processing. The processing target of the trained intermediate acoustic feature inference processing is, specifically, the acoustic side data of the inference target of the feature by the inference processing. The processing target of the trained second text semantic feature inference processing is, specifically, the feature output by the intermediate acoustic feature inference processing. The interface unit 22 acquires the processing target of the inference processing by communicating with the device that transmits the processing target of the inference processing. The interface unit 22 acquires the processing target of the trained intermediate acoustic feature inference processing by communicating with the device that transmits the processing target of the trained intermediate acoustic feature inference processing.

[0056] For example, when the inference process executes a downstream task, the external device may be a device that transmits data used in the downstream task, such as text data. The interface unit 22 acquires the data used in the downstream task by communicating with the device that transmits the data used in the downstream task.

[0057] The external device may be, for example, the learning device 1. In such a case, the inference device 2 can execute, by communication via the interface unit 22, the inference process of the trained intermediate acoustic feature obtained by the learning device 1, the inference process of the trained second text semantic feature, and the inference process of the trained first text semantic feature.

[0058] The interface unit 22 may be configured to include input devices such as a mouse, keyboard, touch panel, etc. The interface unit 22 may be configured as an interface that connects these input devices to the inference device 2. In this way, the input devices of the interface unit 22 accept input of various information to the inference device 2 via wired or wireless connections. Note that signals or information do not necessarily have to be input to the communication interface of the interface unit 22, but may also be input to the input devices of the interface unit 22.

[0059] The interface unit 22 outputs, for example, various types of information. The interface unit 22 is configured to include, for example, a display device such as a CRT display, a liquid crystal display, or an organic EL display, and a speaker. The interface unit 22 may be configured as an interface that connects these display devices or speakers to the inference device 2. Therefore, the interface unit 22 may output, for example, information input to an input device of the interface unit 22 as an image or sound.

[0060] The storage unit 23 is configured using a computer-readable storage medium device (non-transitory computer-readable recording medium) such as a magnetic hard disk device or a semiconductor storage device. The storage unit 23 stores various information related to the inference device 2. The storage unit 23 stores various information generated by the operation of the control unit 21, for example. The storage unit 23 may exist on a cloud, for example.

[0061] 6 is a flowchart showing an example of the flow of processing executed by the inference device 2 in the embodiment. The control unit 21 executes the inference processing (step S201).

[0062] <Example of Experimental Results> Examples of experimental results are shown below regarding the accuracy of the execution results of downstream tasks using features obtained by the trained intermediate acoustic feature inference process and the trained second text semantic feature inference process, and the accuracy of the execution results of downstream tasks using features obtained using a comparative technology (hereinafter referred to as the "comparative technology").

[0063] 7 is a first diagram showing an example of experimental results by transfer learning in an embodiment. In FIG. 7, "ESC-50" and "US8K" both indicate a task of classifying environmental sounds. "CRM-D" indicates a task of classifying emotions contained in audio. "NSynth" indicates a task of classifying musical instruments. "Surge" indicates a task of classifying musical intervals.

[0064] In FIG. ２０２２" refers to the CLAP technology described in Reference 1 below. "M2D / 0.7" refers to the technology described in Non-Patent Document 1. "M2D-CLAP / 0.7" refers to the inference technology used by the inference device 2.

[0065] Reference 1: Alec Radford et al., “Learning Transferable Visual Models From Natural Language Supervision”, arXiv:2103.00020v1 [cs.CV] 26 Feb 2021

[0066] The results in Figure 7 show that ２０２２ The figures show the inference accuracy when the downstream tasks of "ESC-50", "US8K", "CRM-D", "NSynth", and "Surge" are executed using "ESC-50" and "M2D / 0.7".

[0067] FIG. 7 shows the accuracy of inference when the downstream tasks of "ESC-50", "US8K", "CRM-D", "NSynth", and "Surge" are executed by "M2D-CLAP / 0.7".

[0068] The results in FIG. 7 indicate that the larger the numerical value, the better the accuracy of the inference. Therefore, FIG. 7 shows that the inference results by the inference device 2 are better than those of the technology being compared (i.e., "CLAP ２０２２ " and "M2D / 0.7").

[0069] 8 is a first diagram showing an example of experimental results of an experiment in an embodiment. In FIG. 8, "AS" indicates a classification task using zero-shot learning of Audio Set environmental sounds. "FSD" indicates a classification task using zero-shot learning of FDS50K. "GTZ" indicates a classification task using zero-shot learning of GTZAN. "NS" indicates a classification task using zero-shot learning of NSynth.

[0070] The results of Figure 8 show that ２０２２8 shows the inference accuracy when zero-shot learning for each of "AS", "FSD", "GTZ", and "NS" is performed using "M2D-CLAP / 0.7" and "M2D-CLAP / 0.7". Note that, unlike FIG. 7, the results for "M2D / 0.7" are not shown in FIG. 8 because zero-shot learning cannot be performed using the features obtained using the technology described in Non-Patent Document 1.

[0071] The results in FIG. 8 indicate that the larger the numerical value, the better the accuracy of the inference. Therefore, the results in FIG. 8 show that the inference results by the inference device 2 are better than those of the technology to be compared (i.e., "CLAP ２０２２ ") is better than

[0072] The learning device 1 configured in this way executes the learning process, which reduces the burden required to acquire features used in downstream tasks, as described in <Effects of the Learning Process>.

[0073] Furthermore, the inference device 2 configured in this manner performs inference using the learning results obtained by the learning device 1. Therefore, the inference device 2 can reduce the burden required to acquire features to be used in downstream tasks.

[0074] Furthermore, the information processing system 100 configured in this manner includes the learning device 1. Therefore, the information processing system 100 can reduce the burden required to acquire features used in downstream tasks.

[0075] (Modification) The learning device 1 may be implemented using a plurality of information processing devices connected to each other via a network so that they can communicate with each other. In this case, the processes executed by the control unit 11 may be distributed among the plurality of information processing devices.

[0076] The inference device 2 may be implemented using a plurality of information processing devices connected to each other via a network so that they can communicate with each other. In this case, the processes executed by the control unit 21 may be distributed among the plurality of information processing devices.

[0077] Note that all or part of the functions of the information processing system 100, the learning device 1, and the inference device 2 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. The program may be transmitted via a telecommunications line.

[0078] The control unit 21 is an example of an inference unit.

[0079] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention.

[0080] REFERENCE SIGNS LIST 100... Information processing system, 1... Learning device, 2... Inference device, 11... Control unit, 12... Interface unit, 13... Storage unit, 21... Control unit, 22... Interface unit, 23... Storage unit, 91... Processor, 92... Memory, 93... Processor, 94... Memory

Claims

1. A control unit that performs training of a learning model, which is a mathematical model to be learned, using pairs of acoustic data, which is data related to an acoustic signal, and explanatory text data, which is text data explaining the content of the acoustic data, as training data, wherein the learning model executes the following: a first feature inference process that infers features of masked data, which is a part of the acoustic data included in the training data, based on the masked data; an intermediate acoustic feature inference process that infers unmasked intermediate acoustic features, which are features of at least a part of the acoustic data different from the masked data, based on the part; a second feature inference process that infers features of the masked data based on the unmasked intermediate acoustic features; a first text semantic feature inference process that infers text semantic features, which are features of explanatory text data paired with the acoustic data, based on the explanatory text; and a second text semantic feature inference process that infers text semantic features based on features inferred from the first feature inference process or the unmasked intermediate acoustic features, the learning device updates the first feature inference process, the intermediate acoustic feature inference process, the second feature inference process, and the second text semantic feature inference process so as to reduce a difference between the feature inferred in the first feature inference process and the feature inferred in the second feature inference process, and a difference between the text semantic feature inferred in the first text semantic feature inference process and the text semantic feature inferred in the second text semantic feature inference process.

2. The learning device according to claim 1, wherein the learning further updates the first text semantic feature inference process.

3. The learning device according to claim 1 or 2, wherein the first feature inference process, the intermediate acoustic feature inference process, and the second feature inference process are processes executed in Masked Modeling Duo (M2D) on the acoustic data.

4. A control unit is provided for training a learning model, which is a mathematical model to be learned, using pairs of acoustic data, which is data related to an acoustic signal, and explanatory sentence data, which is text data explaining the content of the acoustic data, as training data, wherein the learning model includes a first feature inference process for inferring features of masked data, which is a part of the acoustic data included in the training data, based on the masked data; an intermediate acoustic feature inference process for inferring unmasked intermediate acoustic features, which are features of at least a part of the acoustic data different from the masked data, based on the part; a second feature inference process for inferring features of the masked data based on the unmasked intermediate acoustic features; and a first feature inference process for inferring text semantic features, which are features of explanatory sentence data paired with the acoustic data, based on the explanatory sentence data. an inference unit that executes a text semantic feature inference process and a second text semantic feature inference process that infers text semantic features based on features inferred using the first feature inference process or the unmasked intermediate acoustic features, wherein the first feature inference process, the intermediate acoustic feature inference process, the second feature inference process, and the second text semantic feature inference process are updated in the learning process so as to reduce a difference between the features inferred in the first feature inference process and the features inferred in the second feature inference process, and a difference between the text semantic features inferred in the first text semantic feature inference process and the text semantic features inferred in the second text semantic feature inference process; and an inference unit that infers features of acoustic data that are a target of feature inference using the trained intermediate acoustic feature inference process and the trained second text semantic feature inference process obtained by the learning device.

Images