Speech separation method and device using transformer model based on auditory working memory storing global context information

Abstract

A speech separation device according to one embodiment can perform the operations of: acquiring an audio file including speeches of a plurality of speakers; encoding the audio file into an audio vector of a predetermined format; dividing the audio vector into sequence tokens of a preset unit to be used as input of a transformer model, and embedding same; connecting, to each divided sequence token, a memory token of a predetermined size, allocated from a short-term memory, so as to generate a global information token, and inputting the global information token into the transformer model performing speech masking, so as to perform masking on vector information of each speaker; generating, on the basis of the masked vector information in the audio vector, audio vector information in which the speech of each speaker is separated out; and decoding the audio vector information in which the speech of each speaker is separated out, so as to generate an audio file in which the speech of each speaker is separated out.

Description

A method and device for voice separation using a transformer model based on auditory working memory that stores global contextual information

[0001] The present invention relates to a technique for separating the voices of multiple speakers contained in an audio file using a transformer model based on auditory working memory that stores global contextual information.

[0002] Voice separation technology is essential in various speech-based applications, including speech recognition, speaker recognition, and machine translation. Recent advancements in transformer models have led to remarkably high accuracy in single-channel voice separation.

[0003] However, existing Transformer models require high computational loads, memory usage, dual-path processing inefficiencies, and fixed processing paths to achieve this high level of separation. Consequently, voice separation techniques utilizing existing Transformer models are difficult to apply to resource-constrained mobile and IoT devices. An example of these issues is shown in Figure 1.

[0004] Figure 1 is a performance comparison table for performing voice separation on a given audio file by applying various voice separation models based on the CPU and GPU used in a mobile device.

[0005] Referring to Figure 1, in the case of Sepformer (Subakan, C., Ravanelli, M., Cornell, S., Bronzi, M., & Zhong, J. "Attention is all you need in speech separation." IEEE ICASSP. 13 May 2021.), among the latest transformer models, it can be confirmed that when performing speech separation on an audio file with an 8 kHz, 0.5 second input using the Samsung Galaxy S20 CPU, it took 2.53 seconds, showing performance that is not suitable for application to real-time mobile speech processing.

[0006] The inefficiency of these existing transformer models is mainly evident in two aspects.

[0007] First, the dual-path operation process of the existing transformer model has a structure that divides audio data into groups of a certain size (chunking), repeats information processing within each group (intra-chunk processing), and information processing between groups (inter-chunk processing), but has the characteristic of generating unnecessary computational overhead during the information processing within each group.

[0008] Second, the fixed information processing path of the existing transformer model has the characteristic of limiting the flexibility of the model and hindering efficient resource utilization.

[0009] Due to the above characteristics, there is a need for an improved transformer model structure that can shorten voice processing delay time and reduce computing resource usage.

[0010] The problem to be solved by the present invention is to propose a technology for increasing resource efficiency by implementing an improved transformer model that utilizes auditory working memory to extract the global context of speech and separate speech from an audio file in order to overcome the limitations of the above-described transformer model.

[0011] Meanwhile, the technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by those skilled in the art from the description below.

[0012] A method performed by a voice separation device operated by a processor according to one embodiment may include: obtaining an audio file including voices of multiple speakers; encoding the audio file into an audio vector of a predetermined format; dividing the audio vector into sequence tokens of a preset unit to be used as inputs of a transformer model and embedding the same; generating a global information token by connecting a memory token of a predetermined size allocated from a short-term memory to each of the divided sequence tokens, and inputting the global information token into a transformer model that performs voice masking to perform masking on vector information of each speaker; generating audio vector information in which the voice of each speaker is separated based on the masked vector information from the audio vector; and decoding the audio vector information in which the voice of each speaker is separated to generate an audio file in which the voice of each speaker is separated.

[0013] In addition, the operation of performing the masking may include an operation of generating a global information token by connecting a memory token of a predetermined size allocated from a short-term memory to each segmented sequence token, and inputting the global information token into a transformer model that performs masking for each speaker based on the global information token, thereby performing masking for vector information of each speaker.

[0014] In addition, the operation of performing the masking may include an operation of performing masking on vector information of each speaker by performing an Intra-Chunk Transformer process that processes information within the token based on the structure of the Sepformer model and an Inter-Chunk Transformer process that processes information between the tokens.

[0015] Additionally, the memory token may be characterized in that information about the global context included in the audio file is stored by an intra-chunk transform process according to the Sepformer model.

[0016] Additionally, the Inter-Chunk Transform operation for the above memory token can be performed by short-term memory.

[0017] In addition, the Intra-Chunk Transformer process and the Inter-Chunk Transformer process may be performed recursively a preset number of times, and in each recursive process, information processing in the layer may be stopped for a token determined to be pruned among the global information tokens in the transformer layer included in the Intra-Chunk Transformer process and the Inter-Chunk Transformer process.

[0018] In addition, the Intra-Chunk Transformer process and the Inter-Chunk Transformer process are recursively performed a preset number of times, and in each recursive process, a transformer layer included in the Intra-Chunk Transformer process and the Inter-Chunk Transformer process can determine whether to stop or continue information processing for each global information token based on a probability calculated based on the following [Mathematical Formula 1] for the global information token.

[0019]

[0020] ( Is Input sequence tokens according to conditions, is the transformer layer, is a differentiable estimator, is the output token according to the ith recursion, , , : preset threshold, n: number of recursion)

[0021] In addition, when the operation processing for a token determined to have been pruned according to the above [Mathematical Formula 1] is stopped, the number of tokens and the layer depth of the transformer layer included in the Intra-Chunk Transform process and the Inter-Chunk Transform process may be dynamically reduced.

[0022] Additionally, the audio vector may include a two-dimensional spectrogram including intensity values ​​according to frequency band.

[0023] According to one embodiment, a voice separation device includes a memory including a command; and a processor for performing a predetermined operation based on the command, wherein the operation of the processor may include: obtaining an audio file including voices of multiple speakers; encoding the audio file into an audio vector in a predetermined format; dividing the audio vector into sequence tokens of a preset unit to be used as inputs of a transformer model and embedding the same; generating a global information token by connecting a memory token of a predetermined size allocated from a short-term memory to each of the divided sequence tokens, and inputting the global information token into a transformer model that performs voice masking to perform masking on vector information of each speaker; generating audio vector information in which the voice of each speaker is separated based on the masked vector information from the audio vector; and decoding the audio vector information in which the voice of each speaker is separated to generate an audio file in which the voice of each speaker is separated.

[0024] The present invention can extract global information of the context contained in speech while resolving the inefficiency of the dual-path process by utilizing auditory working memory, and can increase the efficiency of computation by removing redundant or unnecessary tokens according to the input content.

[0025] Through this, the present invention can shorten voice processing delay time and reduce computing resource usage, and thus can be expected to find wider use in embedded systems such as smartphones, AR devices, and AI robots in the future.

[0026] Meanwhile, the effects of the present invention are not limited to those mentioned above, and other technical effects not mentioned can be clearly understood by those skilled in the art from the description below.

[0027] Figure 1 is a performance comparison table for performing voice separation on a given audio file by applying various transformer models based on the CPU used in a mobile device.

[0028] Figure 2 is a configuration diagram of a voice separation device according to one embodiment.

[0029] FIG. 3 is a flowchart illustrating steps of operations performed by a voice separation device according to one embodiment.

[0030] Fig. 4 is a conceptual diagram showing the execution configuration of each operation according to the embodiment of Fig. 3 as a functional block.

[0031] FIG. 5 is a conceptual diagram of an operation of Sepformer, a transformer model according to one embodiment, to separate a speaker's voice from an audio file.

[0032] Fig. 6 is an experimental result that checks whether overhead occurs in the Intra-Chunk Transformer and Inter-Chunk Transformer processes when using Sepformer, a transformer model according to one embodiment.

[0033] Figure 7 is a conceptual diagram of an operation of AWM, a transformer model according to one embodiment, to separate a speaker's voice from an audio file.

[0034] Figure 8 is a block diagram of layers constituting AWM according to one embodiment.

[0035] Figure 9 is an exemplary diagram conceptualizing an adaptive token removal technique according to one embodiment.

[0036] FIGS. 10 to 13 are flowcharts showing the processing process of data passing through the layers of AWM from the perspective of data according to the embodiments of FIGS. 8 and 9.

[0037] Figure 14 is a comparison table of the performance of a voice separation technique using a conventional transformer and the performance of a voice separation technique (Papez, Papez + DM) using AWM according to an embodiment of this document.

[0038] The purpose, technical configuration, and resulting operational effects of the present invention will be more clearly understood through the following detailed description based on the drawings attached to the specification of the present invention. The following describes embodiments of the present invention in detail with reference to the attached drawings.

[0039] The embodiments disclosed herein should not be construed or used to limit the scope of the present invention. Those skilled in the art will readily appreciate that the descriptions herein, including the embodiments, have a wide range of applications. Therefore, any embodiments described in the detailed description of the present invention are intended to serve as illustrative examples to better illustrate the present invention and are not intended to limit the scope of the present invention to the embodiments.

[0040] The functional blocks depicted in the drawings and described below are merely examples of possible implementations. Other implementations may utilize other functional blocks without departing from the spirit and scope of the detailed description. Furthermore, while one or more functional blocks of the present invention are depicted as individual blocks, one or more of the functional blocks of the present invention may be a combination of various hardware and software configurations that perform the same function.

[0041] Additionally, the expression “including certain components” is an “open” expression, simply indicating the presence of those components, and should not be construed as excluding additional components.

[0042] Furthermore, when it is said that a component is “connected” or “connected” to another component, it should be understood that it may be directly connected or connected to that other component, but there may also be other components in between.

[0043] Hereinafter, various embodiments of the present invention will be described with reference to the accompanying drawings. However, this is not intended to limit the present invention to specific embodiments, and it should be understood that various modifications, equivalents, and / or alternatives of the embodiments of the present invention are included.

[0044] Voice separation technology is essential in various speech-based applications, including speech recognition, speaker recognition, and machine translation. Recent advancements in transformer models have led to remarkably high accuracy in single-channel voice separation.

[0045] To achieve this high level of separation, existing Transformer models require high computational loads, memory usage, inefficiencies in dual-path processing, and fixed processing paths. Consequently, voice separation techniques utilizing existing Transformer models are difficult to apply to resource-constrained mobile and IoT devices. An example of these issues is shown in Figure 1.

[0046] Figure 1 is a performance comparison table for performing voice separation on a given audio file by applying various transformer models based on the CPU and GPU used in a mobile device.

[0047] Referring to Figure 1, in the case of Sepformer (Subakan, C., Ravanelli, M., Cornell, S., Bronzi, M., & Zhong, J. "Attention is all you need in speech separation." IEEE ICASSP. 13 May 2021.), among the latest transformer models, it can be confirmed that when performing speech separation on an audio file with an 8 kHz, 0.5 second input using the Samsung Galaxy S20 CPU, it took 2.53 seconds, showing performance that is not suitable for application to real-time mobile speech processing.

[0048] The inefficiency of these existing transformer models is mainly evident in two aspects.

[0049] First, the dual-path operation process of the existing transformer model has a structure that divides audio data into groups of a certain size (chunking), repeats information processing within each group (intra-chunk processing), and information processing between groups (inter-chunk processing), but has the characteristic of generating unnecessary computational overhead during the information processing within each group (inter-chunk processing).

[0050] Second, the fixed information processing path of the existing transformer model has the characteristic of limiting the flexibility of the model and hindering efficient resource utilization.

[0051] To improve the above-described characteristics, the embodiments of this document propose a technique for increasing resource efficiency by implementing an improved transformer model that extracts the global context of speech by utilizing auditory working memory to separate speech from an audio file, together with FIGS. 2 to 11 below.

[0052] FIG. 2 is a configuration diagram of a voice separation device (100) (hereinafter referred to as 'device (100)') according to one embodiment.

[0053] Referring to FIG. 2, a device (100) according to one embodiment may each include a memory (110), a processor (120), an input / output interface (130), and a communication interface (140).

[0054] The memory (110) can store data acquired from an external device or data generated by the processor (120). The memory (110) can store commands that can perform operations of the processor (120). For example, the memory (110) can store a transformer model and commands for controlling the transformer model, which will be described later.

[0055] The processor (120) is a computing device that controls overall operations. The processor (120) can execute instructions stored in the memory (110). The operation of the device (100) according to the embodiment of this document can be understood as an operation performed by the processor (120).

[0056] The input / output interface (130) may include a hardware interface or software interface for inputting or outputting information.

[0057] The communication interface (140) enables information to be transmitted and received via a communication network. To this end, the communication interface (140) may include a wireless communication module or a wired communication module.

[0058] The device (100) may be implemented in various forms of devices capable of performing operations via a processor (120) and transmitting and receiving information via a network. For example, the device may be implemented in the form of a server, a computer device, a portable communication device, a smart phone, a portable multimedia device, a laptop, a tablet PC, etc., but is not limited to these examples.

[0059] FIG. 3 is a flowchart of operations performed by a device (100) according to one embodiment, and FIG. 4 is a conceptual diagram showing the performance configuration of each operation according to the embodiment of FIG. 3 as a functional block.

[0060] The operation of the device (100) according to the embodiment of FIG. 3 and the operation of each functional block according to the embodiment of FIG. 4 can be understood as an operation performed by the processor (120).

[0061] Each step and configuration disclosed in FIGS. 3 and 4 are merely preferred embodiments for achieving the purpose of the present invention, and some steps may be added or deleted as needed, and any one step may be included and performed within another step. The order of each operation disclosed in FIG. 3 is merely arranged for the convenience of understanding, and this order is not limited to a chronological order, and the order may be changed and performed differently depending on the designer's choice.

[0062] Referring to FIGS. 3 and 4, at step S1010, the device (100) may obtain an audio file (e.g., Mixture in FIG. 4). For example, the audio file may include a file in which audio is played over time and may include the voices of one or more speakers.

[0063] In step S1020, the device (100) can encode the audio file into an audio vector of a predetermined format. For example, the device (100) can perform a Fourier transform on the audio file to encode the audio file into an audio vector of a two-dimensional spectrogram (2D Spectrogram) having a format including intensity values ​​according to frequency bands.

[0064] At step S1030, the device (100) may perform an embedding operation to divide the audio vector into sequence tokens of preset units to be used as inputs for a given transformer model. For example, if the total size of the audio vector is 256 (unit omitted), it may be divided (chunked) into tokens of preset units (e.g., 4) to generate 256 / 4 = 64 sequence tokens.

[0065] In step S1040, the device (100) can generate a global information token by connecting a memory token of a predetermined size allocated from a short-term memory (e.g., Memory slots of FIG. 4) to each segmented sequence token, and input the global information token into a transformer model according to an embodiment of the present document to perform masking on vector information of each speaker. The transformer model can recursively repeat an operation of receiving input data, passing it through all layers, and then setting the output data as input data again, thereby passing it through all layers again, for a preset number of times.

[0066] At step S1050, the device (100) can generate audio vector information in which each speaker's voice is separated based on masked vector information from the encoded audio vector.

[0067] At step S1060, the device (100) can decode audio vector information in which each speaker's voice is separated to generate an audio file in which each speaker's voice is separated (e.g., Source 0, Source 1 in FIG. 4).

[0068] Hereinafter, an embodiment of a transformer model (hereinafter referred to as 'AWM') according to an embodiment of this document that separates voices of an audio file while extracting the global context of the voice by utilizing auditory working memory through the above-described S1040 step (ex. AWM Transformer of FIG. 4) is specifically presented.

[0069] Meanwhile, before explaining the AWM according to the embodiment of this document, for the convenience of understanding, the structure of Sepformer (Subakan, C., Ravanelli, M., Cornell, S., Bronzi, M., & Zhong, J. "Attention is all you need in speech separation." IEEE ICASSP. 13 May 2021.), a transformer model that is the target of improvement of AWM, is explained with FIGS. 5 and 6, and the differences of AWM corresponding to the improvement points for Sepformer are explained with FIGS. 7 to 10.

[0070] FIG. 5 is a conceptual diagram of an operation of Sepformer, a transformer model according to one embodiment, to separate a speaker's voice from an audio file, and FIG. 6 is an experimental result that checks whether overhead occurs in the Intra-Chunk Transformer and Inter-Chunk Transformer processes when Sepformer, a transformer model according to one embodiment, is used.

[0071] Referring to Fig. 5, Sepformer has a structure that repeats information processing in two directions: first processing information within the group (intra-chunk processing) and then processing information between chunks (inter-chunk processing) after grouping and cutting audio vectors into groups of a certain length. This is called a dual-path process. The experimental results according to Fig. 6 show that as the Transformer Layer Index increases, the Sepformer process quickly generates unnecessary computational overhead in the inter-chunk processing process.

[0072] In addition, when applying the adaptive token pruning technique according to [Mathematical Formula 1] described below to Sepformer, it can be seen that in inter-chunk processing, a large number of tokens have already been pruned (=a state in which information processing is no longer required) in the front part of the neural network. On the other hand, in the case of intra-chunk processing, it can be confirmed that almost no tokens have been pruned in any part of the neural network. This shows that an inefficient amount of computation is being consumed in the inter-chunk processing process of Sepformer.

[0073] Accordingly, audio files contain parts with silence, parts with only one person's voice, etc., and since these parts are easier to separate than other parts, the AWM of the present invention can significantly reduce the amount of computation for a certain part of audio data according to the global context of the audio file, and the contents of the following Figures 7 to 14 will be explained based on this.

[0074] Figure 7 is a conceptual diagram of an operation of AWM, a transformer model according to one embodiment, to separate a speaker's voice from an audio file.

[0075] Referring to Fig. 7, AWM proposes a small-sized auditory working memory that replaces the inter-chunk transformer to reduce the amount of inter-chunk processing computation of the existing Sepformer. The auditory working memory is a short-term memory (e.g., RAM, etc.) that creates a region of memory tokens to be linked to each segmented sequence token. The global information token (= memory token + sequence token) generated by linking the memory token to the sequence token (e.g., linking to the front of the sequence token) is used as the input of the transformer model. In addition, the latency and model size can be optimized by sharing weights between layers through recursive configuration.

[0076] Furthermore, AWM utilizes an adaptive token elimination technique, described later in [Mathematical Formula 1], to determine whether to suspend or continue information processing for each global information token at each iteration of data passing through the layers within the AWM. This allows AWM to determine, based on probability, whether to suspend or continue computational processing for tokens that do not require additional information processing, depending on the situation. Accordingly, the width (number of tokens) and depth (layer repetition) of computations can be dynamically reduced.

[0077] Fig. 8 is a block diagram of the layers that constitute an AWM (e.g., the AWM Transformer of Fig. 4) according to one embodiment. Fig. 9 is an exemplary diagram conceptualizing an adaptive token removal technique according to one embodiment. Figs. 10 to 13 are flowcharts illustrating the data processing process as it passes through the layers of the AWM from the perspective of data according to the embodiments of Figs. 8 and 9.

[0078] Here, the order of FIGS. 10 to 13 will be described with reference to FIGS. 8 and 9 together. At this time, the memory token and sequence token described in the above-described step S1030 correspond to the Memory Tokens and Sequence Tokens blocks located at the bottom of FIG. 8, and in FIG. 8, the memory token and sequence token can be understood as data being processed while passing through the layers from the bottom to the top of the layers constituting the AWM. The flowchart of FIG. 10 can be understood as corresponding to the flowchart of FIG. 3, and any duplicate description according to the description in FIG. 3 will be omitted. Here, the operation of the auditory working memory transformer neural network described in the 4th step of FIG. 10 can be described as a process from the beginning to the end of FIGS. 11 to 13.

[0079] First, the first step of Fig. 11 is, according to the starting step of the flowchart as an internal operation of AWM, a sequence token including N length H vectors and a memory token including M length H vectors can be prepared.

[0080] The second step of FIG. 11 corresponds to the lower Embedding block of FIG. 4, and at this time, the device (100) can embed sequence tokens and memory tokens using a depth 2 MLP neural network. Accordingly, in the second step of FIG. 11, output 1: embedded sequence tokens (N vectors of length H) and embedded memory tokens (M vectors of length H) can be output.

[0081] In the third step of Fig. 11, the device (100) can generate output tokens and stopping probabilities having the same number of vectors as the sequence tokens and initialize them all to 0 (the number of transformer iterations = 1). Accordingly, in the third step of Fig. 11, output 2: sequence tokens (N vectors of length H), memory tokens (M vectors of length H), and output tokens (N vectors of length 1) can be output.

[0082] The fourth step of Fig. 11 corresponds to the chunking block of Fig. 8, and at this time, the device (100) can divide the vectors of the sequence token into groups of a certain chunk length K. (K=150, C=ceil(N / K*2)). Accordingly, the sequence token can be a group of C vectors, and each group can be a vector of length H of K. Accordingly, the memory token output in the fourth step of Fig. 11 can be a vector of length H of M.

[0083] The fifth step of Fig. 11 corresponds to the duplicate and concat (bottom) blocks of Fig. 8, and at this time, the device (100) can copy the memory token and add it to each group of sequence tokens. Accordingly, the memory + sequence token (= global information token) output in the fifth step of Fig. 11 can be C vector groups, and each group can be K+M vectors of length H.

[0084] The 6th step of FIG. 11 corresponds to the multi-head attention block of FIG. 8, and at this time, the device (100) can perform a multi-head attention deep learning operation on memory + sequence tokens. Accordingly, the memory + sequence tokens (= global information tokens) output in the 6th step of FIG. 11 can be C vector groups, and each group can be K+M vectors of length H.

[0085] The first step of Fig. 12 corresponds to the split (bottom) and average blocks of Fig. 8, and at this time, the device (100) can extract a memory token from each group of memory + sequence tokens and then take an average to update the memory token value. Accordingly, in the first step of Fig. 12, sequence tokens (C vector groups, each group consisting of K vectors of length H) and memory tokens (M vectors of length H) can be output.

[0086] The second step of Fig. 12 corresponds to the overlap-add block of Fig. 8, and at this time, the device (100) can unpack a group of sequence tokens into a sequence of vectors. Accordingly, in the second step of Fig. 12, sequence tokens (N vectors of length H) and memory tokens (M vectors of length H) can be output.

[0087] The third step of Fig. 12 corresponds to the concat (bottom) block of Fig. 8, and at this time, the device (100) can attach a memory token to the front of the sequence token. Accordingly, in the third step of Fig. 12, the memory + sequence token can be a vector of length H of (M + N).

[0088] The fourth step of FIG. 12 corresponds to the Add + LayerNorm (bottom) block of FIG. 8, and at this time, the device (100) can add the memory + sequence token (output 1) before embedding to the memory + sequence token and perform a layer normalization technique. Accordingly, in the fourth step of FIG. 12, output 3: memory + sequence token (M + N vectors of length H) can be output.

[0089] The fifth step of FIG. 12 corresponds to the FFN block of FIG. 8, and at this time, the device (100) can perform an operation by passing the sequence sequence + memory token through a depth 2 MLP neural network. Accordingly, the fifth step of FIG. 12 can output memory + sequence tokens (M + N vectors of length H).

[0090] The 6th step of Fig. 12 corresponds to the Add + LayerNorm (top) block of Fig. 8, and at this time, the device (100) can add the token (output 3) prior to passing the MLP to the memory + sequence token and perform the layer normalization technique. Accordingly, in the 6th step of Fig. 12, the memory + sequence token (M + N vectors of length H) can be output.

[0091] The first step of FIG. 13 corresponds to the split (upper) block of FIG. 8, and at this time, the device (100) can separate the memory token from the memory+sequence token. Accordingly, in the first step of FIG. 13, the sequence token (N vectors of length H) and the memory token (M vectors of length H) can be separated and output.

[0092] The second step of Fig. 13 is that the output of Layer n of Fig. 9 passes through a sigmoid function. This is the part that outputs the first equation of [Mathematical Equation 1] below. and the second equation It corresponds to .

[0093] [Mathematical Formula 1]

[0094]

[0095]

[0096] ( Is Input sequence tokens according to conditions, is the transformer layer, is a differentiable estimator, is the output token according to the ith recursion, , , : preset threshold, n: number of recursion)

[0097] At this time, the device (100) can obtain the incremental value of the stopping probability by applying a sigmoid function to the last element value of each vector of the sequence token. Accordingly, in the second step of Fig. 13, a sequence token (N vectors of length H), an incremental stopping probability (N real values ​​of length 1), and a memory token (M vectors of length H) can be output.

[0098] The third step of Fig. 13 is in Fig. 9 This corresponds to the part that determines whether to proceed to Layer n+1 or stop according to the condition, and the third equation of [Mathematical Formula 1] above . At this time, the device (100) adds the stop probability increment to the threshold value for each of the stop probability element values ​​of output 2. (ex. If the stopping probability exceeds 0.9, the stopping probability value becomes 1, and the stopping probability increment value becomes the value obtained by subtracting the previous stopping probability value from 1. If the value obtained by adding the stopping probability increment is the threshold, If it does not exceed , the stopping probability is the value obtained by adding the increment value to the stopping probability value. Accordingly, in the third step of Fig. 13, a sequence token (N vectors of length H), a stopping probability (N vectors of length 1), an increment of the stopping probability (N real numbers of length 1), and a memory token (M vectors of length H) can be output.

[0099] The fourth step of Fig. 13 is from the bottom of Fig. 9. to This corresponds to the added part, and the fourth equation of [Mathematical Formula 1] above . At this time, the device (100) can create a new output token by multiplying the stop probability increment value corresponding to each vector of the sequence token and adding it to the output token (output 2). Accordingly, in the fourth step of Fig. 13, an output token (N vectors of length H), a stop probability, and a memory token can be output.

[0100] According to the fifth step of Fig. 13, if the number of transformer repetitions is greater than the preset number of repetitions (Nmax) or all stop probabilities are If it exceeds , proceed to the 6th step of Fig. 13. Otherwise, return to step B of Fig. 11, update the output token, stopping probability, and memory token values, count the number of transformer iterations by +1, and recursively proceed again from the 4th step of Fig. 11.

[0101] The 6th step of FIG. 13 corresponds to the Mask Generation block of FIG. 4, and at this time, the device (100) can generate two voice masks using a depth 2 MLP neural network for each vector of the output token. At this time, each element value of the mask can have a value between [0 and 1]. Accordingly, in the 7th step of FIG. 13, a voice mask corresponding to the number of speakers identified (e.g., 2) and a vector mask of length H of 2*N can be obtained.

[0102] Since the 7th step of FIG. 13 is the end of the 4th step of FIG. 10, from the steps thereafter, according to the 5th to 7th steps of FIG. 10 (steps S1050 and S1060 of FIG. 3), the device (100) can finally generate an audio file that distinguishes each speaker.

[0103] When learning a neural network model according to the AWM of the embodiment according to FIGS. 8 to 13, learning can be performed by setting the AWM size to 16, the maximum depth of the transformer to 16, the chunk size to 150, the number of attention heads to 8, the size of the embedding token to 256, and the number of FFN hidden nodes to 1024. In addition, the kernel size of the encoder and decoder neural networks can be set to 16, and the stride can be set to 8. AWM can be trained through supervised learning, which is deep learning, and the training data used can be trained with WSJ0-2Mix dataset, 200 epochs, batch size 1, and 16-bit mixed-precision. The objective function of AWM can use utterance-level permutation-invariant SI-SNR, and can be trained based on the AdamW algorithm with a learning rate of 10^(-4), weight decay of 10^(-4), and exponential scheduling (decay coefficient 0.98). At this time, the maximum L2-norm of the gradient is clipped at 1. Training the entire deep learning model of AWM according to these settings took 220 hours on a GPU with Titan RTX performance.

[0104] Fig. 14 is a comparison table of the performance of a speech separation technique using a conventional transformer and the performance of a speech separation technique (Papez, Papez + DM) using AWM according to an embodiment of this document. Fig. 14 shows the results of measuring the accuracy of separated audio in SI-SNR from the WSJ0-2Mix dataset, which is a standard benchmark for speech separation technology, to verify the effectiveness of the present invention. Here, Papez + DM is the result of adding the conventional Dynamic Mixing technique (N. Zeghidour and D. Grangier, "Wavesplit: End-to-end speech separation by speaker clustering," IEEE / ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 2840-2849, 2021.) to the Papez technique according to the embodiment of this document.

[0105] Referring to FIG. 14, Papez, a technique according to an embodiment of this document, was shown to achieve speech separation accuracy close to or higher than that of existing techniques even with fewer computing resources. Papez can achieve SI-SNR that is 1.1 to 4.2 dB higher than existing resource-efficient speech separation techniques, and can achieve a model size that is 3.6 to 4.14 times smaller and an inference delay that is 1.67 to 2.44 times faster. In addition, Papez can reduce the model size by up to 17.7 times compared to Sepformer, a high-performance speech separation technique with minimal performance degradation. Auditory working memory technology, which streamlines bidirectional speech processing into a single direction, contributed to the reduction in model size and delay time, and adaptive token pruning technology, which adaptively performs token operations and computes only the necessary parts, contributed to the reduction in inference delay time.

[0106] The techniques on the top of the horizontal bar in the center of Fig. 14 are performance-oriented techniques that do not consider computational efficiency, and the techniques on the bottom of the horizontal bar in the center of Fig. 14 are speech separation techniques that consider computational efficiency. Compared to performance-oriented techniques, Papez of the present invention showed a delay time reduction of several times or more on both CPU and GPU (Titan XP), and showed a reduction in the number of parameters of more than ten times. Compared to existing techniques that considered computational efficiency, Papez showed a significant performance difference in the Libri2Mix and WSJ0-2Mix speech separation datasets.

[0107] According to the above-described embodiment, the present invention can extract global information of the context contained in a voice while resolving the inefficiency of the dual-path process by utilizing auditory working memory, and can increase the efficiency of the calculation by removing duplicate or unnecessary tokens according to the input content.

[0108] Through this, the present invention can shorten voice processing delay time and reduce computing resource usage, and thus can be expected to find wider use in embedded systems such as smartphones, AR devices, and AI robots in the future.

[0109] The various embodiments and terms used in this document are not intended to limit the technical features described in this document to specific embodiments, but should be understood to encompass various modifications, equivalents, or alternatives of the embodiments. In connection with the description of the drawings, similar reference numerals may be used to refer to similar or related components. The singular form of a noun corresponding to an item may include one or more items, unless the context clearly indicates otherwise.

[0110] In this document, the phrases "A or B", "at least one of A and B", "at least one of A or B," "A, B, or C," "at least one of A, B, and C," and "at least one of A, B, or C" can each include all possible combinations of the items listed together in that phrase. Terms such as "1", "2", or "first" or "second" may be used merely to distinguish the corresponding component from other corresponding components and do not limit the corresponding components in any other respect (e.g., importance or order). When a (e.g., a first) component is referred to as "coupled" or "connected" to another (e.g., a second) component, with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0111] The term "module" as used herein may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0112] Various embodiments of the present document may be implemented as software (e.g., a program) including one or more instructions stored in a storage medium (e.g., a memory) readable by a device (e.g., an electronic device). The storage medium may include random access memory (RAM), a memory buffer, a hard drive, a database, erasable programmable read-only memory (EPROM), electrically erasable read-only memory (EEPROM), read-only memory (ROM), and / or the like.

[0113] Additionally, the processor of the embodiments of the present document can call at least one command among one or more commands stored from a storage medium and execute it. This enables the device to operate to perform at least one function according to the at least one command called. These one or more commands may include code generated by a compiler or code executable by an interpreter. The processor may be a general-purpose processor, a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), and / or the like.

[0114] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory" simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0115] The methods according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store (e.g., Play Store) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated on a machine-readable storage medium, such as a manufacturer's server, an application store's server, or a server's memory.

[0116] According to various embodiments, each component (e.g., a module or a program) of the described components may include a single or multiple entities. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. A method for performing a voice separation device operated by a processor, An action to obtain an audio file containing the voices of multiple speakers; An action of encoding the above audio file into an audio vector of a predetermined format; An operation of dividing the above audio vector into sequence tokens of preset units to be used as inputs of a transformer model and embedding them; An operation of generating a global information token by connecting a memory token of a predetermined size allocated from a short-term memory to each segmented sequence token, and inputting the global information token into a transformer model that performs voice masking to perform masking on vector information of each speaker; An operation of generating audio vector information in which each speaker's voice is separated based on the masked vector information in the audio vector; and Including an operation of decoding audio vector information in which each speaker's voice is separated to generate an audio file in which each speaker's voice is separated. method.

2. In paragraph 1, The action of performing the above masking is An operation that includes generating a global information token by connecting a memory token of a predetermined size allocated from a short-term memory to each segmented sequence token, and inputting the global information token into a transformer model that performs masking for each speaker based on the global information token, thereby performing masking for vector information of each speaker. method.

3. In paragraph 2, The action of performing the above masking is An operation including performing masking on vector information of each speaker by performing an Intra-Chunk Transformer process for processing information within the token based on the structure of the Sepformer model and an Inter-Chunk Transformer process for processing information between the tokens. method.

4. In paragraph 3, The above memory token is Characterized in that information about the global context included in the audio file is stored by the Intra-Chunk Transform process according to the Sepformer model. method.

5. In paragraph 4, The Inter-Chunk Transform operation for the above memory token is performed by short-term memory. method.

6. In paragraph 3, The above Intra-Chunk Transformer process and the above Inter-Chunk Transformer process It is performed recursively a preset number of times, and in each recursive process, it is characterized by stopping information processing in the layer for tokens determined to be pruned among the global information tokens in the transformer layer included in the Intra-Chunk Transformer process and the Inter-Chunk Transformer process. method.

7. In paragraph 6, The above Intra-Chunk Transformer process and the above Inter-Chunk Transformer process It is performed recursively a preset number of times, and in each recursive process, the transformer layer included in the Intra-Chunk Transformer process and the Inter-Chunk Transformer process determines whether to stop or continue information processing for each global information token based on the probability calculated based on the following [Mathematical Formula 1] for the global information token. ( Is Input sequence tokens according to conditions, is the transformer layer, 는 differentiable estimator, is the output token according to the ith recursion, , , : preset threshold, n: number of recursion) method.

8. In paragraph 7, When the operation processing for a token determined to have been pruned according to the above [Mathematical Formula 1] is stopped It is characterized in that the number of tokens and the layer depth of the transformer layer included in the above Intra-Chunk Transform process and the above Inter-Chunk Transform process are dynamically reduced. method.

9. In paragraph 1, The above audio vector is Contains a two-dimensional spectrogram containing intensity values ​​according to frequency band. method.

10. Memory containing instructions; and Includes a processor that performs a predetermined operation based on the above command, The operation of the above processor is as follows: An action to obtain an audio file containing the voices of multiple speakers; An action of encoding the above audio file into an audio vector of a predetermined format; An operation of dividing the above audio vector into sequence tokens of preset units to be used as inputs of a transformer model and embedding them; An operation of generating a global information token by connecting a memory token of a predetermined size allocated from a short-term memory to each segmented sequence token, and inputting the global information token into a transformer model that performs voice masking to perform masking on vector information of each speaker; An operation of generating audio vector information in which each speaker's voice is separated based on the masked vector information in the audio vector; and Including an operation of decoding audio vector information in which each speaker's voice is separated to generate an audio file in which each speaker's voice is separated. Voice separation device.

Images