Sound processing apparatus, system, method, and computer program product

Abstract

The invention relates to a sound processing apparatus, a sound processing system, a sound processing method, and a computer program product. The purpose of the present invention is to efficiently execute tasks relating to sound processing. This sound processing device is provided with a task execution unit that executes a task pertaining to sound processing on the basis of a learned model that learns using sound data and a tag obtained by converting a feature vector extracted from the sound data in accordance with a predetermined rule.

Description

Technical Field

[0001] This invention relates to sound processing apparatus, sound processing system, sound processing method, and computer program products. Background Technology

[0002] Techniques for performing sound processing-related tasks based on machine learning are known. For example, Patent Document 1 discloses an information processing device that acquires sound data, extracts sound features from the sound data, obtains sound representation from the sound features, inputs the sound representation into a sound recognition unit, and acquires text data.

[0003] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2023-64381 Summary of the Invention

[0005] However, existing technologies have room for efficient execution of sound processing-related tasks. For example, most existing sound recognition technologies learn sound recognizers through supervised learning using sound data and transcribed text, but preparing transcribed text from large amounts of sound data requires enormous costs.

[0006] The purpose of one embodiment of the present invention is to efficiently perform tasks related to sound processing.

[0007] One embodiment of the present invention relates to a sound processing apparatus including a task execution unit that performs sound processing-related tasks based on a learned model. The learned model is learned using sound data and labels, wherein the labels are obtained by transforming feature vectors extracted from the sound data according to a predetermined rule.

[0008] The effects of the present invention are explained below:

[0009] According to one embodiment of the present invention, tasks related to sound processing can be performed efficiently. Attached Figure Description

[0010] Figure 1 This is a block diagram illustrating an example of the overall structure of a sound processing system according to an embodiment of the present invention.

[0011] Figure 2 This is a block diagram illustrating an example of the hardware configuration of a computer according to one embodiment of the present invention.

[0012] Figure 3 This is a block diagram illustrating an example of the functional structure of a sound processing system according to an embodiment of the present invention.

[0013] Figure 4This is a diagram illustrating an example of a tag transformation process according to an embodiment of the present invention.

[0014] Figure 5 This is a flowchart illustrating an example of model learning processing according to one embodiment of the present invention.

[0015] Figure 6 This is a flowchart illustrating an example of a tag transformation process according to an embodiment of the present invention.

[0016] Figure 7 This is a flowchart illustrating an example of task execution processing according to one embodiment of the present invention.

[0017] The symbols in the diagram are explained as follows:

[0018] 10: Model Learning Device

[0019] 20: Sound processing device

[0020] 101: Tagless Data Storage Department

[0021] 102: Tag-based Data Storage Department

[0022] 110: Feature Extraction Department

[0023] 120: Label Transformation Section

[0024] 130: Model Generation Department

[0025] 140: Additional Study Department

[0026] 201: Model Storage Department

[0027] 210: Voice Input Section

[0028] 220: Mission Execution Department

[0029] 230: Result Output Section

[0030] 1000: Sound processing system. Detailed Implementation

[0031] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Furthermore, components with the same function are labeled with the same numbers in the drawings, and repeated descriptions are omitted.

[0032] [Implementation Mode]

[0033] One embodiment of the present invention is an information processing system that performs tasks related to sound processing. Hereinafter, the information processing system involved in this embodiment will be referred to as a "sound processing system". In this embodiment, the sound processing system can perform any task related to sound processing (hereinafter simply referred to as a "sound processing task"). As an example, a sound processing task may include sound recognition, sound synthesis, sound enhancement, speaker recognition, speaker authentication, emotion recognition, sound region detection, etc.

[0034] Voice recognition, as an example of a voice processing task, is a technology that converts sound data, including human speech, voices, or conversations, into text data. Voice recognition technology is widely used in business settings, such as in displaying captions in meetings, taking meeting minutes, or creating reports. Compared to humans using keyboards to convert sound into text, voice recognition technology facilitates textification and data input into systems. Therefore, voice recognition technology is expected to be an effective input method related to business efficiency.

[0035] Previously, in the learning of voice recognition machines, supervised learning (teacher-supervised learning) was generally used, where voice data and its corresponding transcribed text were paired to learn their correspondence. However, in teacher-supervised learning, obtaining a large amount of transcribed voice data is required to develop a voice recognition machine with high accuracy, making the acquisition of such data costly and a significant challenge.

[0036] Reference 1 discloses a method for learning a voice recognizer through simulated teacher learning, also known as semi-teacher learning. In the method of Reference 1, the voice recognizer is learned using a small number of transcribed sounds. The learned voice recognizer is then used to generate text from a large number of untranscribed sounds. Pairs of input sounds with high confidence in the inference process with the output text are used as new transcribed sounds. The learned voice recognizer is then updated using semi-teacher learning based on the above results. However, in generating transcribed text for semi-teacher learning, the accuracy of the text inferred by the learned voice recognizer and its confidence level as learning data is not guaranteed. If semi-teacher learning is performed using incorrect transcribed text, it can actually hinder the learning of the voice recognizer.

[0037] Reference 1: Japanese Patent No. 7052866

[0038] Reference 2 proposes another method utilizing a large amount of untranscribed audio. Reference 2 applies masked language models, used in machine learning processes such as large-scale language models, to audio performance learning. In the method of Reference 2, a large amount of untranscribed audio is first transformed into acoustic features, and then an N-level classification machine learning model is generated through self-teacher learning. N-level classification means that, based on a certain proportion of masked frames, the model predicts which of the N predetermined acoustic feature values ​​is closest to the masked frame based on the context of the preceding and following unmasked frames. Then, using the parameters of the pre-learned machine learning model as initial values, a small amount of transcribed audio is used for transfer learning to the audio recognition task. According to Reference 2, compared to audio recognizers using only a small amount of transcribed audio, performance can be improved without incurring transcription costs.

[0039] Reference 2: Wei-Ning Hsu, Benjamin Bolte, Yao-Hung HuBERT Tsai, KushalLakhotia, Ruslan Salakhutdinov, Abdelrahman Mohamed, "HuBERT T: Self-SupervisedSpeech Representation Learning by Masked Prediction of Hidden Units," IEEE / ACMTransactions on Audio, Speech, and Language Processing, vol.29, pp.3451-3460, 2021.

[0040] As in Reference 2, in order to perform pre-learning for the classification task based on masked frames, a finite number of positive solution labels are required. On the other hand, unlike masked language models that take discrete vectors such as text information as input, the input signal of a voice recognizer is a continuous vector. Therefore, in Reference 2, in order to perform pre-learning for the classification task based on masked frames of input voice, the audio features are pre-quantized into N categories, and a finite number of positive solution labels (hereinafter referred to as "self-teacher labels") are created for self-teacher learning.

[0041] Reference 2, as a quantization method, classifies the acoustic features of all frames of untranscribed audio into N categories using the k-means method, and uses the classified category number as the teacher label for each frame. Furthermore, Reference 2 proposes a method using Cartesian product quantization to generate classification labels. However, in these quantization methods, based on the statistical distribution of untranscribed audio as the population, the centroids of each category only fall within a partial space constituted by the population. Therefore, when new untranscribed audio with different statistical distributions is added to the pre-learned model, it cannot be guaranteed that appropriate categories exist in these partial spaces. In such cases, continuing the pre-learning model promotes imbalance in the classification model, and the performance may not be sufficiently improved when applied to the transfer learning to the audio recognition model.

[0042] One embodiment of the present invention aims to efficiently perform sound processing tasks. Therefore, in this embodiment, sound processing tasks are performed based on a learned model obtained by learning sound data and labels obtained by transforming feature vectors extracted from the sound data according to predetermined rules.

[0043] In this implementation, the statistical inference factor is excluded from the self-teacher label generation process during the pre-learning model's learning process, thus eliminating the dependence of self-teacher labels on data distribution. Furthermore, since self-teacher labels are derived solely from language information (phonemes) based on sound data, self-teacher learning with high affinity for sound processing tasks is possible.

[0044] According to this embodiment, instead of using the statistical distribution of the dataset, deterministic operations according to predetermined rules are used to derive the teacher's label, thus enabling efficient performance of audio processing tasks. Furthermore, according to this embodiment, a pre-learning model capable of supplementary learning using a small amount of transcribed text is generated, thus enabling efficient performance of various audio processing tasks.

[0045] <Overall Structure of the Sound Processing System>

[0046] Reference Figure 1 The overall structure of the sound processing system is explained. Figure 1 This is a block diagram illustrating an example of the overall structure of a sound processing system according to an embodiment of the present invention.

[0047] like Figure 1 As shown, the sound processing system 1000 includes a model learning device 10 and a sound processing device 20. The model learning device 10 and the sound processing device 20 are connected to a communication network N. The communication network N is configured such that the connected devices can communicate with each other.

[0048] Communication networks N, such as the Internet, LAN (Local Area Network), or

[0049] WAN (Wide Area Network) and other network constructions based on wired communication. Communication network N can include not only wired communication, but also wireless communication such as wireless LAN or short-range wireless communication, or mobile communication such as WiMAX (Worldwide Interoperability for Microwave Access), LTE (Long Term Evolution), or 5G (5th Generation).

[0050] The model learning device 10 is an example of an information processing device that generates a machine learning model for performing sound processing tasks. As an example, the model learning device 10 can be a computer such as a personal computer, workstation, or server.

[0051] As an example, a machine learning model can be a neural network. As further examples, a neural network can be a deep learning-based deep neural network, a recurrent neural network, an attention mechanism model, an autoregressive model (such as the Transformer), etc.

[0052] The model learning device 10 pre-stores audio data that serves as the learning object. The audio data includes untranscribed audio and transcribed audio. The transcribed audio may be less than the untranscribed audio. The model learning device 10 generates a pre-learning model based on the untranscribed audio. The model learning device 10 further learns from the pre-learning model based on the transcribed audio to generate a fully learned machine learning model (hereinafter also referred to as the "fully learned model").

[0053] The sound processing device 20 is an example of an information processing device that performs sound processing tasks based on a learned model. As an example, the sound processing device 20 can be a computer such as a personal computer, workstation, or server.

[0054] The learned model is stored in the sound processing device 20. The learned model can also be a model generated by the model learning device 10. The sound processing device 20 accepts input sound data that is to be processed. The sound processing device 20 performs a sound processing task by inputting the input sound data into the learned model. The sound processing device 20 outputs the execution result of the sound processing task.

[0055] The model learning device 10 or the sound processing device 20 can be any device with communication capabilities, and is not limited to a computer. Examples of the model learning device 10 or the sound processing device 20 include image forming devices (printers, fax machines, MFPs (Multifunction Peripheral / Product / Printers), scanning devices), PJs (projectors), IWBs (Interactive Whiteboards), digital signage output devices, HUD (Head-Up Display) devices, industrial machinery, camera devices, sound collection devices, medical devices, networked appliances, connected cars, laptops (Personal Computers), mobile phones, smartphones, tablets, game consoles, PDAs (Personal Digital Assistants), digital cameras, wearable PCs, or desktop PCs, etc.

[0056] Figure 1 The structure of the sound processing system 1000 shown is one example, but other structures are also possible. For example, the model learning device 10 or the sound processing device 20 can be implemented by a single information processing device, or it can be a system implemented by multiple information processing devices. The sound processing system 1000 includes various devices for at least one of inputting and outputting electronic data, and these devices can also utilize various services provided by the sound processing system 1000.

[0057] <Hardware Structure of the Sound Processing System>

[0058] Reference Figure 2 The hardware structure of each device included in the sound processing system 1000 will be described. As an example, the model learning device 10 or the sound processing device 20 included in the sound processing system 1000 can also be implemented by a computer. Figure 2 This is a block diagram illustrating an example of the hardware configuration of a computer according to an embodiment of the present invention.

[0059] like Figure 2 As shown, computer 500 includes CPU 501, ROM 502, RAM 503, HD 504, HDD (hard disk drive) controller 505, display 506, external device connection I / F (interface) 507, network I / F 509, bus 510, keyboard 511, indicator device 512, DVD-RW (rewritable digital universal disc) drive 514, and media I / F 516.

[0060] The CPU 501 controls the overall operation of the computer. ROM 502 stores IPL and other programs used to drive the CPU 501. RAM 503 serves as the working area for the CPU 501. HD 504 stores programs and other data. Under the control of the CPU 501, the HDD controller 505 controls the reading of various data from HD 504 and the writing of various data to HD 210.

[0061] The monitor 506 displays various information such as the cursor, menus, windows, text, or images. The external device connection I / F508 is an interface for connecting various external devices. Examples of external devices in this case include USB (Universal Serial Bus) storage devices, printers, etc. Network.

[0062] I / F509 is an interface used for data communication via communication network N1. Bus 510 is used for electrical connection. Figure 2 The CPU501 and other components shown include the address bus, data bus, etc.

[0063] Keyboard 511 is an input device that includes multiple keys for inputting characters, numbers, various instructions, etc. Indicator device 512 is an input unit for selecting and executing various instructions, selecting processing objects, moving the cursor, etc. DVD-RW drive 514 controls the reading of various data from and writing various data to DVD-RW 513, which is an example of a removable recording medium. However, it is not limited to DVD-RW; it can also be DVD-R, etc. Media I / F 516 controls the reading of data from or writing (storing) data to recording medium 515, such as flash memory.

[0064] <Functional Structure of a Sound Processing System>

[0065] Reference Figure 3 The functional structure of the sound processing system 1000 is explained. Figure 3 This is a block diagram illustrating an example of the functional structure of a sound processing system according to an embodiment of the present invention.

[0066] Model Learning Device

[0067] like Figure 3 As shown, the model learning device 10 includes an unlabeled data storage unit 101, a labeled data storage unit 102, a feature extraction unit 110, a label transformation unit 120, a model generation unit 130, and an additional learning unit 140.

[0068] For example, using the tagless data storage unit 101 and the tagged data storage unit 102 Figure 2 The HD504 implementation is shown. For example, data stored in the HD504 is read or written via the HDD controller 505.

[0069] The feature extraction unit 110, label transformation unit 120, model generation unit 130, and additional learning unit 140, for example, by using... Figure 2 The program shown is expanded from HD504 onto RAM503 and executed by CPU501.

[0070] Unlabeled data is pre-stored in the unlabeled data storage unit 101. Unlabeled data is data that has not been assigned a correct answer label. In this embodiment, unlabeled data may also be untranscribed audio data (i.e., untranscribed audio). A sufficient amount of unlabeled data is pre-stored in the unlabeled data storage unit 101.

[0071] Tag data is pre-stored in the tagged data storage unit 102. Tag data is data that has been assigned a correct answer label. In this embodiment, the tagged data may also be a pair of sound data and text data transcribed from the sound data (i.e., transcribed sound). Only a very small amount of tagged data needs to be stored in the tagged data storage unit 102.

[0072] Sound data is electronic data based on human voices. Sound data can also be a time-domain sound signal that records human voices. Sound data can also be data that transforms a time-domain sound signal into a frequency-domain signal. In this embodiment, sound data is a sequence of frames of a sound signal transformed into a log-Mel spectrum. The dimension of the log-Mel spectrum can be arbitrary, but in this embodiment, as an example, it is set to 80 dimensions.

[0073] The text data included in labeled data only needs to represent the speech content contained in the audio data. The text data included in labeled data does not necessarily have to be written by a human. For example, the text data included in labeled data can also be the speech recognition results of the audio data.

[0074] The feature extraction unit 110 extracts feature vectors from the unlabeled data. Alternatively, the feature extraction unit 110 can generate a feature vector sequence by extracting feature vectors from each frame of the unlabeled data. In this embodiment, the feature vectors are set as Mel-Frequency Cepstrum Coefficients (MFCCs). For example, the feature extraction unit 110 can also transform an 80-dimensional log-Mel spectrum into 80-dimensional Mel-Frequency Cepstrum Coefficients by applying a discrete cosine transform to the MFCC.

[0075] The label transformation unit 120 transforms feature vectors into its own teacher labels. The label transformation unit 120 can also transform feature vectors extracted by the feature extraction unit 110 into its own teacher labels. The label transformation unit 120 can also transform feature vectors extracted from unlabeled data into its own teacher labels. The label transformation unit 120 can also generate a sequence of its own teacher labels by transforming the feature vectors corresponding to each frame of unlabeled data into its own teacher labels.

[0076] The label transformation unit 120 transforms the feature vector into its own teacher label according to a predetermined transformation rule. The transformation rule is a rule that uniquely and deterministically derives the teacher label from the feature vector itself, independent of the statistical distribution of the unlabeled data.

[0077] The tag transformation unit 120 can also quantize the feature vector into a predetermined number of integers. The tag transformation unit 120 can quantize a portion of the feature vector into an integer. The tag transformation unit 120 can also obtain elements representing language information from the feature vector as part of the feature vector. The tag transformation unit 120 can also obtain a predetermined d-dimensional element from the feature vector as part of the feature vector. The tag transformation unit 120 can also obtain elements with less than d dimensions from the feature vector as part of the feature vector. Here, d is an integer less than the dimension of the feature vector. In this embodiment, d is an integer of 4 or more and less than 80. d can be set to any integer, for example, 10.

[0078] Mel frequency cepstral coefficients are generated by performing a discrete cosine transform on a logarithmic Mel spectrum. Therefore, sound signals exhibit the characteristic that linguistic information (speech) is stored in low-dimensional elements, while paralinguistic and non-linguistic information is stored in high-dimensional elements. Non-linguistic information includes, for example, timbre, prosody, and noise. Thus, by extracting low-dimensional elements from the feature vector and discarding high-dimensional elements, it is possible to quantize the sound signal into a smaller number of integers while preserving its linguistic information.

[0079] The label transformation unit 120 can also transform each dimension of the feature vector into a 1-bit β-ary number, and transform the integer base of the β-ary number concatenated with each bit into a decimal number, thereby quantizing it into a decimal integer. Here, β is an integer greater than or equal to 2.

[0080] Specifically, the label transformation unit 120 first normalizes the feature vector to a mean of 0 and a variance of 1. Then, the label transformation unit 120 transforms each dimension of the normalized feature vector into a 1-digit beta number. Alternatively, the label transformation unit 120 can transform the feature vector into a beta number by comparing each dimension with β-1 thresholds.

[0081] For example, the label transformation unit 120 transforms the nth dimension element xn of the feature vector into a base-β number according to equation (1). Here, n is an integer greater than or equal to 1 and less than d, and λ1, λ2, ..., λn are integers. β-1 The threshold is predetermined.

[0082] [Formula 1]

[0083]

[0084] As an example, when the number is converted to binary (i.e., β = 2), the tag conversion unit 120 can also set the threshold to λ1 = 0 and calculate formula (2).

[0085] [Formula 2]

[0086]

[0087] As another example, when the number is converted to ternary (i.e., β = 3), the tag conversion unit 120 can also set the threshold to λ1 = -0.5 and λ2 = 0.5, and calculate formula (3).

[0088] [Formula 3]

[0089]

[0090] The tag transformation unit 120 concatenates the beta numbers corresponding to each dimension of the feature vector according to their dimensions. This generates an integer represented as a d-bit beta number. The tag transformation unit 120 then transforms the d-bit beta number into a decimal number. The tag transformation unit 120 obtains the decimal integer and uses it as its own teacher tag.

[0091] The eigenvectors (Mel frequency cepstral coefficients) of a continuous vector are quantized into β. d There are several categories. For example, when transforming features with fewer than 10 dimensions in the feature vector into binary numbers, since β = 2 and d = 10, the energy can be converted to β. d =2 10 =1024 types. Furthermore, for example, when transforming elements with less than 6 dimensions in the eigenvector into ternary numbers, since β = 3 and d = 6, the energy can be converted to β. d =3 6 =729 types.

[0092] Reference Figure 4 The tag transformation process performed by the tag transformation unit 120 will be explained. Figure 4 This is a diagram illustrating an example of a label transformation process in one embodiment of the present invention.

[0093] Figure 4 The x1~x shown NIt is the feature vector corresponding to frames 1 to N of the sound signal. n (where n is an integer greater than 1 and less than N) is an 80-dimensional real vector R. 80 The label transformation unit 120 transforms the feature vector x... n The side with the lower dimension obtains a vector^x containing d-dimensional elements. n ^x n R is a d-dimensional real vector. d .

[0094] The label transformation unit 120 generates the vector ^x n Normalized vectors ～ x n The label transformation unit 120 will convert the vector... ～ x n The d-dimensional elements are transformed into beta numbers, and the d-digit beta numbers are further transformed into decimal numbers. In this way, the tag transformation unit 120 transforms the feature vectors x1 to x2 corresponding to frames 1 to N into... N Transform them into their own teacher labels c1 to c1 respectively. N Teacher's own label c n It is greater than or equal to 0 and less than β. d Integers.

[0095] The model generation unit 130 generates a pre-learning model based on unlabeled data and its own teacher labels. The model generation unit 130 can also input unlabeled data into the pre-learning model during learning, and update the parameters of the pre-learning model based on the error between the output of the pre-learning model and its own teacher labels. For example, the model generation unit 130 can update the weights of the intermediate layers of the neural network included in the pre-learning model based on the backpropagation method.

[0096] The supplementary learning unit 140 performs supplementary learning on the pre-learning model based on labeled data, thereby generating a fully learned model. The supplementary learning unit 140 can also fine-tune the pre-learning model through supplementary learning. The supplementary learning unit 140 can also perform supplementary learning on the pre-learning model through transfer learning.

[0097] The additional learning unit 140 can also perform additional learning on the pre-learning model, enabling it to perform various sound processing tasks. As an example, sound processing tasks may include sound recognition, sound synthesis, sound enhancement, speaker recognition, speaker authentication, emotion recognition, and sound region detection.

[0098] Sound Processing Device

[0099] like Figure 3 As shown, the sound processing device 20 includes a model storage unit 201, a sound input unit 210, a task execution unit 220, and a result output unit 230.

[0100] Model storage unit 201, for example, uses Figure 2 The HD504 implementation is shown. For example, data stored in the HD504 is read or written via the HDD controller 505.

[0101] The sound input unit 210, the task execution unit 220, and the result output unit 230, for example, are connected via a source... Figure 2 The program shown is expanded from HD504 onto RAM503, enabling the CPU501 to execute the processing.

[0102] The learned model is stored in the model storage unit 201. The learned model stored in the model storage unit 201 can also be generated by the model learning device 10. The learned model stored in the model storage unit 201 can also be a model obtained by supplementing the learning of a pre-learned model that has been learned using unlabeled data with labeled data.

[0103] The voice input unit 210 accepts input of voice data that is to be processed. The voice input unit 210 can also accept voice data input via a microphone connected to an external device connection I / F provided with the voice processing device 20. The voice input unit 210 can also receive voice data from a terminal device equipped with a microphone via a communication network N.

[0104] The audio input unit 210 can also accept audio signal input. The audio input unit 210 can also accept a log-Mel spectrum obtained by transforming an audio signal. When receiving an audio signal input, the audio input unit 210 can also transform each frame of the audio signal into a log-Mel spectrum. The dimension of the log-Mel spectrum can be arbitrary, but in this embodiment, as an example, it is set to 80 dimensions.

[0105] The task execution unit 220 performs sound processing tasks. The task execution unit 220 can also perform sound processing tasks based on the learned model read from the model storage unit 201. The task execution unit 220 can also perform sound processing tasks based on sound data input to the sound input unit 210. The task execution unit 220 can also perform sound processing tasks by inputting sound data into the learned model.

[0106] The task execution unit 220 can also perform various sound processing tasks. The task execution unit 220 can also perform sound recognition tasks on sound data input to the sound input unit 210. The tasks performed by the task execution unit 220 are not limited to sound recognition; for example, they may include sound synthesis, sound emphasis, speaker recognition, speaker authentication, emotion recognition, and sound region detection. When the task execution unit 220 performs a task that does not require sound input (e.g., sound synthesis), the sound processing device 20 may not need to include the sound input unit 210.

[0107] The result output unit 230 outputs the execution result of the sound processing task. For example, if the sound processing task is sound recognition, the execution result of the sound processing task includes the recognition result of the sound data. The recognition result of the sound data may also include text data representing the speech content contained in the sound data.

[0108] For example, when the sound processing task is sound synthesis, the result includes the synthesized sound signal of the text data. For example, when the sound processing task is sound enhancement, the result includes the enhanced sound signal. For example, when the sound processing task is speaker recognition, the result includes speaker identification information. For example, when the sound processing task is speaker authentication, the result includes the speaker authentication result. For example, when the sound processing task is emotion recognition, the result includes emotion tags. For example, when the sound processing task is sound region detection, the result includes information representing sound regions in the sound data.

[0109] The result output unit 230 can also display the execution result of the sound processing task on the display 506 of the sound processing device 20. The result output unit 230 can also send the execution result of the sound processing task to a terminal device equipped with a display via the communication network N.

[0110] <Sound processing steps>

[0111] Reference Figures 5 to 7 The sound processing methods performed by the sound processing system 1000 are described. These methods may include model learning processing (see [reference]). Figure 5 ) and task execution processing (refer to Figure 7 ).

[0112] Model Learning Processing

[0113] Model learning processing is the process of generating a learned model for performing sound processing tasks. Figure 5This is a flowchart illustrating an example of the learning process involved in one embodiment of the present invention.

[0114] In step S101, the feature extraction unit 110 of the model learning device 10 reads unlabeled data from the unlabeled data storage unit 101. The feature extraction unit 110 may also read one or more unlearned unlabeled data items from the unlabeled data stored in the unlabeled data storage unit 101.

[0115] In step S102, the feature extraction unit 110 of the model learning device 10 extracts feature vectors from the unlabeled data read in step S101. Specifically, the feature extraction unit 110 transforms the unlabeled data, which is a log-Mel spectrum, into Mel frequency cepstral coefficients by applying a discrete cosine transform to the unlabeled data. The feature extraction unit 110 then sends the extracted feature vectors to the label transformation unit 120.

[0116] In step S103, the label transformation unit 120 of the model learning device 10 receives feature vectors from the feature extraction unit 110. The label transformation unit 120 transforms the feature vectors into its own teacher labels according to a predetermined transformation rule. The label transformation unit 120 then sends its own teacher labels to the model generation unit 130.

[0117] In step S104, the model generation unit 130 of the model learning device 10 reads the unlabeled data read by the feature extraction unit 110 in step S101 from the unlabeled data storage unit 101. The model generation unit 130 receives its own teacher label from the label transformation unit 120. Based on the unlabeled data and its own teacher label, the model generation unit 130 generates a pre-learning model. The model generation unit 130 sends the pre-learning model to the supplementary learning unit 140.

[0118] Specifically, the model generation unit 130 inputs unlabeled data into the pre-learning model. The pre-learning model performs a sound processing task on the input unlabeled data and outputs the result of the sound processing task. The model generation unit 130 calculates the error between the output of the pre-learning model and its own teacher label. Based on the error between the output of the pre-learning model and its own teacher label, the model generation unit 130 updates the parameters of the pre-learning model.

[0119] The model learning device 10 can also repeatedly execute steps S101 to S104. For example, the model learning device 10 can repeatedly update the parameters of the pre-learned model until a termination condition for ending pre-learning is met. The termination condition may be, for example, that the number of parameter updates exceeds a predetermined threshold. Alternatively, the termination condition may be, for example, that the amount of parameter updates converges.

[0120] In step S105, the additional learning unit 140 of the model learning device 10 reads tagged data from the tagged data storage unit 102. The additional learning unit 140 may also read one or more unlearned tagged data items from the tagged data stored in the tagged data storage unit 102.

[0121] In step S106, the additional learning unit 140 of the model learning device 10 receives the pre-learning model from the model generation unit 130. Based on the labeled data read in step S105, the additional learning unit 140 performs additional learning on the pre-learning model. Thus, a fully learned model is generated.

[0122] The model learning device 10 may also repeatedly execute the processes of steps S105 to S106. For example, the model learning device 10 may repeatedly update the parameters of the pre-learned model until the termination condition for ending supplementary learning is met. The termination condition for ending supplementary learning may be the same as the termination condition for ending pre-learning, or it may be a different condition.

[0123] In step S107, the additional learning unit 140 of the model learning device 10 outputs the learned model. The additional learning unit 140 may also send the learned model to the sound processing device 20. The sound processing device 20 may also receive the learned model from the model learning device 10 and store it in the model storage unit 201.

[0124] The additional learning unit 140 can also store the learned model in a storage device such as an HD504 in the model learning device 10. The learned model stored in the storage device of the sound processing device 20 can also be read by the sound processing device 20. In addition, the model learning device 10 can also send the learning device stored in the storage device to the sound processing device 20 according to a request from the sound processing device 20.

[0125] Label Transformation Processing

[0126] Reference Figure 6 Label transformation processing ( Figure 5 Step S103 will be explained in more detail. Figure 6 This is a flowchart illustrating an example of a tag transformation process according to one embodiment of the present invention.

[0127] In step S131, the tag transformation unit 120 acquires elements of dimension d or lower from the feature vector. Specifically, the tag transformation unit 120 acquires elements from the 1st to the dth dimension from the 80-dimensional Mel frequency cepstral coefficients. In this embodiment, d is set to 10.

[0128] In step S132, the label transformation unit 120 normalizes the d-dimensional feature vector obtained in step S131 to have a mean of 0 and a variance of 1. Specifically, the label transformation unit 120 divides the value of each dimension of the d-dimensional feature vector after subtracting the minimum value by the difference between the maximum and minimum values.

[0129] In step S133, the tag transformation unit 120 transforms each dimension of the d-dimensional feature vector normalized in step S132 into a one-bit β-ary number. For example, when transforming each dimension of the feature vector into a binary number, the tag transformation unit 120 sets λ1 = 0 and calculates equation (2).

[0130] In step S134, the tag transformation unit 120 concatenates the d beta numbers transformed in step S132 according to the dimension. This generates an integer represented as a d-bit beta number. The tag transformation unit 120 then transforms the d-bit beta number base into a decimal number. The tag transformation unit 120 obtains the integer represented as a decimal number as its own teacher tag.

[0131] Regarding the relationship between the feature vector and its own teacher label, a specific example is given. The first example assumes d = 10 and β = 2, transforming the 10-dimensional feature vector x into a 10-bit binary number ^x, resulting in the teacher label C as a decimal integer. The threshold is set to λ1 = 0. In this case, x, ^x, and C are as follows.

[0132] ·x=[0.8,-0.4,0.3,0.8,0.2,-0.5,-0.1,0.6,-0.3,0.1]

[0133] ·^x=[1,0,1,1,1,0,0,1,0,1]

[0134] C = {1011100101}² = 741

[0135] The second example is assuming d = 6 and β = 3, transforming the 6-dimensional feature vector x into a 6-bit ternary number ^x, resulting in the teacher label C as a decimal integer. The thresholds are set to λ1 = -0.5 and λ2 = 0.5. In this case, x, ^x, and C are as follows.

[0136] ·x=[0.8,-0.4,0.3,0.8,0.2,-0.5]

[0137] ·^x=[2,1,1,2,1,1]

[0138] ·C={211211}3=616

[0139] The third example is assuming d = 5 and β = 4, transforming the 5-dimensional feature vector x into a 5-bit quaternary number ^x, resulting in the teacher label C as a decimal integer. The thresholds are set to λ1 = -0.5, λ2 = 0, and λ3 = 0.5. In this case, x, ^x, and C are as follows.

[0140] ·x=[0.8,-0.4,0.3,0.8,0.2]

[0141] ·^x=[3,1,2,3,2]

[0142] C = {31232}4 = 878

[0143] Task Execution Processing

[0144] The task execution process is based on the learned model to perform sound processing tasks. Figure 7 This is a flowchart illustrating an example of task execution processing according to an embodiment of the present invention.

[0145] In step S201, the sound input unit 210 of the sound processing device 20 accepts input sound data that is to be processed. If the sound data is a time-domain sound signal, the sound input unit 210 converts the sound signal into a log-Melogram. The sound input unit 210 then sends the sound data to the task execution unit 220.

[0146] In step S202, the task execution unit 220 of the sound processing device 20 receives sound data from the sound input unit 210. The task execution unit 220 reads the learned model from the model storage unit 201.

[0147] In step S203, the task execution unit 220 of the sound processing device 20 executes a sound processing task based on the sound data input in step S201 and the learned model read in step S202. Specifically, the task execution unit 220 inputs sound data into the learned model. The learned model performs the sound processing task on the input sound data and outputs the execution result of the sound processing task. The task execution unit 220 obtains the execution result output from the learned model. The task execution unit 220 sends the execution result of the sound processing task to the result output unit 230.

[0148] In step S204, the result output unit 230 of the sound processing device 20 receives the execution result of the sound processing task from the task execution unit 220. The task execution unit 220 may also display the execution result of the sound processing task on the display 506 of the sound processing device 20. The task execution unit 220 may also send the execution result of the sound processing task to a terminal device equipped with a display via the communication network N.

[0149] <Effects of Implementation>

[0150] The sound processing apparatus 20 of this embodiment performs sound processing tasks based on a learned model. The learned model learns using sound data and labels obtained by transforming feature vectors extracted from the sound data according to predetermined rules. In one aspect, according to this embodiment, instead of using the statistical distribution of the dataset, deterministic operations according to predetermined rules are used to derive labels for its own teacher learning, thus enabling efficient performance of sound processing tasks.

[0151] The rule can quantize the feature vector into a predetermined number of integers. The rule could be to set β to an integer greater than 2, transforming each element of the feature vector into a 1-bit β-ary number, thereby quantizing it into integers. In one aspect, according to this embodiment, since the integers obtained by quantizing the elements of the feature vector are used as the teacher's label, the teacher's label can be derived with less computation.

[0152] The rule can quantize a portion of the feature vector into an integer. Alternatively, a portion of the feature vector can have 'd' set to an integer less than the number of features in the feature vector, including features with dimensions less than 'd' in the feature vector. In one aspect, according to this embodiment, the types of teacher labels can be adjusted, thus enabling efficient generation of pre-learning models.

[0153] A portion of the feature vector may include elements representing linguistic information within the feature vector. The feature vector may be Mel-frequency cepstral coefficients. In one aspect, according to this embodiment, Mel-frequency cepstral coefficients preserve linguistic information in low-dimensional elements, thus enabling the generation of pre-learning models suitable for sound processing.

[0154] The learned model can also be further learned using the sound data and text data representing the speech content contained in the sound data. In one aspect, according to this embodiment, it is possible to efficiently generate learned models for performing various sound processing tasks.

[0155] The sound processing device 20 can also perform sound recognition tasks by inputting sound data into the learned model. In one aspect, according to this embodiment, sound recognition can be performed efficiently.

[0156] According to this embodiment, it can perform self-learning with high affinity for sound processing tasks, thus enabling high-precision recognition of sounds containing noise or reverberation far from the microphone, as well as unclear speech between people. Therefore, it can be flexibly applied in business settings where accuracy is crucial. For example, in workplaces where many people share work, it can support various work modes as a tool for automating sound communication processing, such as automatically creating meeting minutes or reports, displaying captions during meetings, and voice interaction with AI agents. When applied to voice dialogue with AI agents, it can instantly recognize and analyze customer speech, dynamically generate the next question, and thus accurately capture customer needs and provide precise advice.

[0157] [Replenish]

[0158] Each function of the above-described embodiments can be implemented by one or more processing circuits. Here, "processing circuit" in this specification includes processors programmed to perform each function by software, such as processors installed by electronic circuits, ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field-programmable gate arrays), existing circuit modules, and other devices designed to perform the functions described above.

[0159] The apparatus group described in the embodiments represents only one of a plurality of computing environments for implementing embodiments of the present disclosure. In some embodiments, the model learning apparatus 10 or the sound processing apparatus 20 includes a plurality of computing devices, such as a server cluster. The plurality of computing devices are configured to communicate with each other via any type of communication link (including networks, shared memory, etc.) to perform the processing of the present disclosure.

[0160] As an example of the present invention, it is described below.

[0161] (Postscript 1)

[0162] A sound processing device, characterized in that:

[0163] The sound processing device includes a task execution unit that performs sound processing-related tasks based on the learned model.

[0164] The learned model uses sound data and labels for learning. The labels are obtained by transforming the feature vectors extracted from the sound data according to a predetermined rule.

[0165] (Postscript 2)

[0166] The sound processing apparatus according to Appendix 1 is characterized in that:

[0167] The rule quantizes the feature vector into a predetermined number of integers.

[0168] (Note 3)

[0169] The sound processing apparatus according to Appendix 2 is characterized in that:

[0170] The rule sets β to an integer greater than 2, and quantizes each element of the feature vector into a 1-bit β-ary number.

[0171] (Postscript 4)

[0172] The sound processing apparatus according to Appendix 2 or 3 is characterized in that:

[0173] The rule quantizes a portion of the feature vector into the integer.

[0174] (Note 5)

[0175] The sound processing apparatus according to Appendix 4 is characterized in that:

[0176] A portion of the feature vector includes elements representing language information within the feature vector.

[0177] (Note 6)

[0178] The sound processing apparatus according to Appendix 4 or 5 is characterized in that:

[0179] A portion of the feature vector includes elements of dimension d or less in the feature vector, wherein d is set to an integer less than the number of elements in the feature vector.

[0180] (Note 7)

[0181] The sound processing apparatus according to any one of Appendices 1 to 6 is characterized in that:

[0182] The eigenvectors are Mel frequency cepstral coefficients.

[0183] (Postscript 8)

[0184] The sound processing apparatus according to any one of Appendices 1 to 7 is characterized in that:

[0185] The learned model then performs additional learning using the audio data and text data representing the speech content contained in the audio data.

[0186] (Note 9)

[0187] The sound processing apparatus according to Appendix 8 is characterized in that:

[0188] It further includes a voice input unit that accepts the input of second voice data.

[0189] The task execution unit performs the task of recognizing the second sound data by inputting the second sound data into the learning model.

[0190] (Postscript 10)

[0191] A sound processing system, comprising a model learning device and a sound processing device, characterized in that:

[0192] The sound processing device includes a task execution unit that performs sound processing-related tasks based on the learned model.

[0193] The model learning device includes:

[0194] The feature extraction unit extracts feature vectors from the sound data;

[0195] The label transformation unit transforms the feature vector into labels according to a predetermined rule; and

[0196] The model generation unit uses the sound data and the labels to generate the learned model.

[0197] (Postscript 11)

[0198] A sound processing method, characterized in that:

[0199] The computer performs the following steps: based on the learned model, it performs tasks related to sound processing.

[0200] The learned model uses sound data and labels derived from feature vectors extracted from the sound data according to predetermined rules to learn.

[0201] (Postscript 12)

[0202] A program, characterized in that:

[0203] The computer performs the following steps: based on the learned model, it performs tasks related to sound processing.

[0204] The learned model uses sound data and labels derived from feature vectors extracted from the sound data according to predetermined rules to learn.

[0205] (Postscript 13)

[0206] A recording medium stores program code for instructing a computer system to execute sound processing methods, enabling the computer to perform the following steps: based on a learned model, performing sound processing-related tasks.

[0207] The learned model uses sound data and labels derived from feature vectors extracted from the sound data according to predetermined rules to learn.

[0208] (Postscript 14)

[0209] A computer program product, characterized in that:

[0210] It includes computer instructions that, when executed by a processor, implement the steps of the sound processing method as described in Appendix 11.

[0211] The above description illustrates various embodiments, but these embodiments are merely examples and the present invention is not limited to them. The embodiments described can be implemented in various other ways, and various combinations, omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments or their variations are included in the scope or spirit of the invention, and are also included in the scope of the invention as described in the claims and its equivalents.

Claims

1. A sound processing apparatus characterized by comprising a task execution section that executes a task related to sound processing based on a learned model, the learned model being learned using sound data and a label obtained by transforming a feature vector extracted from the sound data according to a predetermined rule.

2. The sound processing apparatus according to claim 1, characterized in that the rule quantizes the feature vector into a predetermined number of integers.

3. The sound processing apparatus according to claim 2, characterized in that the rule sets β to an integer of 2 or more and quantizes the feature vector into the integer by transforming each element of the feature vector into a β-ary number of 1 bit.

4. The sound processing apparatus according to claim 2, characterized in that the rule quantizes a part of the feature vector into the integer.

5. The sound processing apparatus according to claim 4, characterized in that the part of the feature vector includes an element of the feature vector that represents language information.

6. The sound processing apparatus according to claim 4, characterized in that the part of the feature vector includes an element of the feature vector that is d-dimensional or less, where d is set to an integer smaller than the number of elements of the feature vector.

7. The sound processing apparatus according to any one of claims 1 to 6, characterized in that the feature vector is a mel-frequency cepstral coefficient.

8. The sound processing apparatus according to any one of claims 1 to 6, characterized in that the learned model is additionally learned using sound data and text data that represents a content of speech included in the sound data.

9. The sound processing apparatus according to claim 8, characterized in that further comprising a sound input section that accepts input of second sound data, the task execution section executes the task of performing sound recognition on the second sound data by inputting the second sound data to the learned model.

10. A sound processing system including a model learning apparatus and a sound processing apparatus, characterized in that the sound processing apparatus includes a task execution section that executes a task related to sound processing based on a learned model, the model learning apparatus includes: a feature extraction section that extracts a feature vector from sound data; a label transformation section that transforms the feature vector into a label according to a predetermined rule; and a model generation section that generates the learned model using the sound data and the label.

11. A sound processing method characterized by comprising the steps of: a computer executing a task related to sound processing based on a learned model, the learned model being learned using sound data and a label obtained by transforming a feature vector extracted from the sound data according to a predetermined rule. computer instructions that, when executed by a processor, implement the steps of the sound processing method of claim 11.

12. A computer program product comprising computer instructions that, when executed by a processor, implement the steps of the sound processing method of claim 11. ​ 12. A computer program product, characterised in that : ​

Images