Electronic device and method for identifying utterance signal of speaker, and non-transitory computer-readable storage medium

Abstract

According to one embodiment, a non-transitory computer-readable storage medium can store instructions. When executed by at least one processor of an electronic device including a memory, the instructions can instruct the electronic device to: input voice data of a speaker into an embedder included in an embedding model; acquire embedding information related to the speaker on the basis of an output of the embedder; input the embedding information into a speaker-centroid prediction model included in the embedding model; acquire speaker-centroid information related to the speaker on the basis of an output of the speaker-centroid prediction model; after the speaker-centroid information is stored in the memory, receive an audio signal including a target signal and an interference signal related to an utterance of the speaker; input the audio signal and the speaker-centroid information into a separation model; and identify an utterance signal related to the target signal from the audio signal on the basis of an output of the separation model.

Description

Electronic device, method and non-transitory computer-readable storage medium for identifying a speaker's speech signal

[0001] The present disclosure relates to an electronic device, a method and a non-transitory computer-readable storage medium for identifying a speaker's speech signal.

[0002] An electronic device is being developed to identify the speech of a registered user. The electronic device can receive audio data regarding the speech of the registered user. Based on the audio data, the electronic device can identify the content of the user's speech.

[0003] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0004] According to one embodiment, a non-transitory computer-readable storage medium may store instructions. The instructions, when executed by at least one processor of an electronic device including a memory, may cause the electronic device to input speech data of a speaker into an embedder included in an embedding model, obtain embedding information about the speaker based on an output of the embedder, input the embedding information into a speaker-centric prediction model included in the embedding model, obtain speaker-centric information about the speaker based on an output of the speaker-centric prediction model, and after the speaker-centric information is stored in the memory, receive an audio signal including a target signal and an interference signal about an utterance of the speaker, input the audio signal and the speaker-centric information into a separation model, and identify a speech signal about the target signal among the audio signal based on an output of the separation model.

[0005] According to one embodiment, a method performed by an electronic device may include an operation of inputting speech data of a speaker into an embedder included in an embedding model, an operation of obtaining embedding information about the speaker based on an output of the embedding model, an operation of inputting the embedding information into a speaker-centroid prediction model included in the embedding model, an operation of obtaining speaker-centred information about the speaker based on an output of the speaker-centred prediction model, an operation of receiving an audio signal including a target signal and an interference signal about an utterance of the speaker after the speaker-centred information is stored in a memory of the electronic device, an operation of inputting the audio signal and the speaker-centred information into a separation model, and an operation of identifying a speech signal about the target signal among the audio signals based on an output of the separation model.

[0006] According to one embodiment, an electronic device may include a microphone, a memory storing instructions and including one or more storage media, and at least one processor including a processing circuit. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to input speech data of a speaker into an embedder included in an embedding model, obtain embedding information about the speaker based on an output of the embedding model, input the embedding information into a speaker-centroid prediction model included in the embedding model, obtain speaker-centred information about the speaker based on an output of the speaker-centred prediction model, and after the speaker-centred information is stored in the memory, receive an audio signal including a target signal and an interference signal about an utterance of the speaker through the microphone, input the audio signal and the speaker-centred information into a separation model, and identify a speech signal about the target signal among the audio signals based on an output of the separation model.

[0007] FIG. 1 is a block diagram of an electronic device according to one embodiment.

[0008] FIG. 2 illustrates an example of the operation of an electronic device for identifying a speech signal through an embedding model and a separation model, according to one embodiment.

[0009] FIG. 3 illustrates a flowchart of the operation of an electronic device for obtaining speaker-centric information according to one embodiment.

[0010] FIG. 4 illustrates a flowchart of the operation of an electronic device for identifying a speech signal from an audio signal, according to one embodiment.

[0011] FIG. 5 illustrates a flowchart of the operation of an electronic device for training an embedding model, according to one embodiment.

[0012] FIG. 6 illustrates a flowchart of the operation of an electronic device for training a separation model, according to one embodiment.

[0013] FIG. 7 illustrates a flowchart of the operation of an electronic device for training an embedding model and a separation model, according to one embodiment.

[0014] Hereinafter, various embodiments of this document are described with reference to the attached drawings.

[0015] The various embodiments of this document and the terminology used therein are not intended to limit the technology described in this document to a specific embodiment, but should be understood to include various modifications, equivalents, and / or substitutes of the embodiment. In connection with the description of the drawings, similar reference numerals may be used for similar components. The singular expression may include plural expressions unless the context clearly indicates otherwise. In this document, expressions such as "A or B", "at least one of A and / or B", "A, B, or C", or "at least one of A, B, and / or C" may include all possible combinations of the items listed together. Expressions such as "first", "second", "first", or "second" may modify the corresponding components regardless of order or importance, and are only used to distinguish one component from another, but do not limit the corresponding components. When it is said that a component (e.g., a first component) is “(functionally or communicatively) connected” or “connected” to another component (e.g., a second component), the component may be directly connected to the other component, or may be connected via another component (e.g., a third component).

[0016] The term "module" as used in this document includes a unit composed of hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimal unit or portion thereof that performs one or more functions. For example, a module may be composed of an application-specific integrated circuit (ASIC).

[0017] FIG. 1 is a block diagram of an electronic device according to one embodiment.

[0018] Referring to FIG. 1, the electronic device (101) may be a server for receiving data from an external electronic device and storing or processing the received data. For example, the electronic device (101) may be a terminal owned by a user. The terminal may include, for example, a personal computer (PC) such as a laptop or desktop, a smartphone, a smartpad, or a tablet PC. The terminal may include a portable gaming device and / or a gaming console. The terminal may include a smart accessory such as a smartwatch, a smart ring, and / or a head-mounted device (HMD). The terminal may be referred to as a mobile device, a user terminal, a user equipment (UE), a multi-function device, a portable communication device, or a portable device.

[0019] Referring to FIG. 1, the electronic device (101) may include at least one of a processor (110), a memory (115), a communication circuit (120), and a microphone (126). The processor (110), the memory (115), the communication circuit (120), and the microphone (126) may be electrically and / or operably coupled with each other by an electronic component (or electrical component), such as a communication bus (102). Hereinafter, the electronic components being operably coupled may mean that a direct connection or an indirect connection is established between the electronic components, either wired or wireless, so that a second electronic component is controlled by a first electronic component among the electronic components. Although illustrated based on different blocks, the embodiment is not limited thereto, and some of the electronic components of FIG. 1 (e.g., at least a portion of the processor (110), the memory (115), the communication circuit (120), and the microphone (126)) may be included in a single integrated circuit such as a system on a chip (SoC). The type and / or number of electronic components included in the electronic device (101) is not limited to that illustrated in FIG. 1. For example, the electronic device (101) may include only some of the electronic components illustrated in FIG. 1. One or more clusters in which a plurality of electronic devices including the electronic device (101) are grouped may be referred to as servers.

[0020] In one embodiment, the processor (110) may include circuitry (e.g., processing circuitry) for processing data based on one or more instructions. The circuitry for processing data may include, for example, an arithmetic and logic unit (ALU), a floating point unit (FPU), a field programmable gate array (FPGA), a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), and / or an application processor (AP). For example, the number of processors may be one or more. The processing circuitry of the processor that loads (or fetches) instructions and performs calculations corresponding to the loaded instructions may be referred to as or referred to as a core circuit (or core). For example, the processor may have a multi-core processor architecture including multiple core circuits, such as a dual core, a quad core, a hexa core, or an octa core. The functions and / or operations described with reference to the present disclosure may be collectively performed by one or more processing circuits included in a processor.

[0021] In one embodiment, the memory (115) may include circuitry for storing data and / or instructions input to or output from the processor (110). The memory may include, for example, volatile memory such as random-access memory (RAM) and / or non-volatile memory such as read-only memory (ROM). The volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disc, solid state drive (SSD), and embedded multimedia card (eMMC). The processor (110) of the electronic device (101) can execute instructions of the memory (115) within the electronic device (101) to perform functions and / or operations indicated by the instructions.

[0022] In one embodiment, the communication circuit (120) may include hardware for supporting transmission and / or reception of data between the electronic device (101) and an external electronic device. The communication circuit may include, for example, at least one of a modem (MODEM), an antenna, and an optical / electronic (O / E) converter. The communication circuit may support transmission and / or reception of electrical signals based on various types of protocols, such as Ethernet, a local area network (LAN), a wide area network (WAN), wireless fidelity (Wi-Fi), near field communication (NFC), Bluetooth, Bluetooth low energy (BLE), ZigBee, long term evolution (LTE), fifth generation (5G), new radio (NR), sixth generation (6G), and / or above-6G.

[0023] In one embodiment, the processor (110) of the electronic device (101) may obtain an electrical signal representing vibrations of the atmosphere from the microphone (126). The electronic device (101) may include one or more microphones.

[0024] Referring to FIG. 1, one or more instructions (or commands) indicating operations and / or actions to be performed on data by the processor (110) may be stored in the memory (115). A set of one or more instructions may be referred to as firmware, an operating system, a process, a routine, a sub-routine, a program, and / or a software application (hereinafter, an application). Hereinafter, installation of an application may mean that one or more instructions provided in the form of an application are stored in the memory, and that the one or more applications are stored in a format executable by the processor (e.g., a file having an extension designated by the operating system of the electronic device (101).

[0025] Referring to FIG. 1, an exemplary program (e.g., an embedding model (210) or a separation model (220)) installed in an electronic device (101) is illustrated. For example, a set of parameters related to the embedding model (210) or the separation model (220) may be stored in the memory (115) of the electronic device (101). Each of the embedding model (210) or the separation model (220) may be configured based on a neural network.

[0026] For example, a neural network can be a cognitive model implemented in software or hardware that mimics the computational capabilities of biological systems using a large number of artificial neurons (or nodes). Parameters associated with a neural network can represent the weights assigned to multiple nodes and / or connections between multiple nodes within the neural network.

[0027] For example, at least a portion of a neural network may include multiple layers. For example, the neural network may include an input layer, one or more hidden layers, and an output layer. The input layer may receive a vector representing input data (e.g., a vector having elements corresponding to the number of nodes included in the input layer). Signals generated from each of the nodes in the input layer, which are generated by the input data, may be transmitted from the input layer to the hidden layers. The output layer may generate output data of the neural network based on one or more signals received from the hidden layers. The output data may include, for example, a vector having elements corresponding to the number of nodes included in the output layer.

[0028] For example, the one or more hidden layers are not limited to a feedforward-based topology, and may be, for example, convolutional filters or fully connected layers in a convolutional neural network (CNN), or various types of filters or layers grouped based on specific functions or characteristics. In one embodiment, the one or more hidden layers may be layers based on a recurrent neural network (RNN) whose output values ​​are fed back to the hidden layer of the current time. For example, the input layer, the one or more hidden layers, and / or the output layer may be some layers of a transformer model. According to one embodiment, a neural network may include numerous hidden layers to form a deep neural network. Training a deep neural network is called deep learning. Among the nodes of a neural network, a node included in a hidden layer refers to a hidden node.

[0029] According to one embodiment, the electronic device (101) can perform calculations regarding an embedding model (210) or a separation model (220).

[0030] For example, the embedding model (210) may be used to obtain speaker-centric information based on the speaker's speech data. The speaker-centric information may be related to the characteristics of the speaker's speech based on the speaker's speech data. For example, the electronic device (101) may obtain speaker-centric information indicating the speaker's speech pattern, method, characteristics, tone, intonation, etc. using the speech data to identify the speaker. According to an embodiment, the embedding model (210) may be configured based on a transformer-based artificial intelligence model, but is not limited thereto.

[0031] For example, the separation model (220) can be used to identify a speech signal related to a target signal from an audio signal that includes a target signal and an interference signal related to a speaker's speech. For example, the speech signal may correspond to the target signal. The target signal may be a signal representing an actual speaker's speech. The speech signal may be a signal representing a speaker's speech predicted from the audio signal.

[0032] According to one embodiment, an audio signal may include a target signal and an interference signal related to a speaker's speech. For example, the interference signal may be generated according to the speech of another speaker, ambient noise, reverberation, and characteristics of a microphone. The electronic device (101) may use the above-described embedding model (210) and separation model (220) to identify (or extract) a speech signal (or target signal) representing the speaker's speech from the audio signal. In order to identify the speech signal representing the speaker's speech through the audio signal, speaker-centric information representing the speaker's speech characteristics may be obtained through the embedding model (210) using the speaker's voice data that is distinct from the audio signal. The electronic device (101) may identify the speech signal representing the speaker's speech from the audio signal based on inputting the audio signal and the speaker-centric information into the separation model (220).

[0033] The specific configuration of the embedding model (210) and separation model (220) for the above-described embodiment will be described below.

[0034] FIG. 2 illustrates an example of the operation of an electronic device for identifying a speech signal through an embedding model and a separation model, according to one embodiment.

[0035] Referring to FIG. 2, the processor (110) of the electronic device (101) can identify (or extract) a speech signal from an audio signal using an embedding model (210) and a separation model (220).

[0036] According to one embodiment, the processor (110) may input the speaker's speech data into the embedding model (210). The processor (110) may obtain speaker-centric information based on the output of the embedding model (210). The processor (110) may input an audio signal and speaker-centric information into the separation model (220). For example, the audio signal may include a target signal and an interference signal related to the speaker's speech. The processor (110) may identify (or obtain) a speech signal related to the target signal based on the output of the separation model (220).

[0037] For example, the ignition signal can be expressed by a mathematical formula as follows.

[0038]

[0039] Referring to mathematical formula 1, indicates a firing signal. represents a function for the embedding model (210) and the separation model (220). represents an audio signal. represents the speaker's voice data. represents the parameters of the embedding model (210). represents the parameters of the separation model (220). represents a function for the separation model (220). represents speaker-centered information, which is the output of the embedding model (210).

[0040] For example, the embedding model (210) may include an embedder (211) and a speaker-centric prediction model (230). The embedder (211) may be referred to as a pretrained embedder. The speaker-centric prediction model (230) may include a multi-head attention (MHA) model (231), a regularization model (232), and an attentive statistics pooling (ASP) model (233).

[0041] For example, the processor (110) may input speech data into the embedding device (211). The processor (110) may obtain embedding information based on the output of the embedding device (211). For example, the embedding device (211) may be trained using a generalized end-to-end (GE2E) loss function referred to as a d-vector. The embedding information may be expressed as the following mathematical formula.

[0042]

[0043] Referring to mathematical formula 2, represents the embedding information for each frame (f). represents a function for the embedding (211).

[0044] For example, the processor (110) can input embedding information into a speaker-centered prediction model (230). The processor (110) can obtain speaker-centered information based on the output of the speaker-centered prediction model (230).

[0045] For example, the processor (110) may input embedding information into the MHA model (231) of the speaker-centered prediction model (230). The processor (110) may obtain center vector information based on the output of the MHA model (231). By inputting the frame-by-frame d-vector into the MHA model (231), the processor (110) may obtain a center vector that is separated from various negative factors that affect speaker information. The center vector information may be expressed as the following mathematical formula.

[0046]

[0047] Referring to mathematical formula 3, is the center vector information for each frame (f). represents a function for the MHA model (231).

[0048] For example, the processor (110) may input center vector information into the normalization model (232). The processor (110) may obtain normalized center vector information based on the output of the normalization model (232). The normalization model (232) may be configured based on L2 normalization. The center vector may be normalized to a unit center vector.

[0049] For example, the processor (110) can input normalized center vector information into the ASP model (233). The processor (110) can obtain speaker-centered information based on the output of the ASP model (233). The ASP model (233) can obtain speaker-centered information by generating a single vector using an attention layer that focuses on an important frame among a plurality of frames.

[0050] According to one embodiment, the processor (110) can identify frame-by-frame d-vectors (or embedding information) for all utterances of speech data using the embedding device (211) and identify speaker-centered information. The speaker-centered information can be expressed as the following mathematical formulas.

[0051]

[0052]

[0053] Referring to Equations 4 and 5, i is the index for the speaker. is the embedding information of the utterance level. The embedding information of the utterance level can be obtained by averaging and normalizing the embedding information for each frame. is speaker-centric information for speaker i. Speaker-centric information for speaker i can be obtained by averaging utterance-level embedding information. For example, speaker-centric information can be identified based on the average of embedding information obtained from various utterances. Therefore, speaker-centric information can be separated from negative factors other than speaker information.

[0054] According to the above-described embodiment, the processor (110) may store speaker-centered information in the memory (115) based on acquiring speaker-centered information using the embedding model (210). For example, the processor (110) may acquire speaker-centered information by training the embedding model (210) (or the separation model (220)) using the speaker's voice data, and store the acquired speaker-centered information in the memory (115).

[0055] According to one embodiment, the processor (110) may identify a speech signal among the audio signal based on the audio signal and speaker-centric information. For example, the processor (110) may input the audio signal and speaker-centric information into a separation model (220). The processor (110) may identify (or obtain) the speech signal based on the output of the separation model (220).

[0056] For example, the separation model (220) may include an encoder (221), a mask extractor (222), and a decoder (223).

[0057] For example, the processor (110) may input an audio signal into an encoder (221) of a separation model (220). The processor (110) may input the output of the encoder (221) and speaker-centered information into a mask extractor (222). For example, the mask extractor (222) may be configured based on a Temporal Convolutional Network (TCN). The processor (110) may combine the output of the encoder (221) and the output of the mask extractor (222). The processor (110) may input the combined outputs into a decoder (223). The processor (110) may identify a speech signal corresponding to a target signal among the audio signals based on the output of the decoder (223).

[0058] According to one embodiment, the processor (110) of the electronic device (101) may receive an audio signal including a target signal and an interference signal related to a speaker's speech using a microphone (126). The processor (110) may input the audio signal and speaker-centered information into a separation model (220). Based on the output of the separation model (220), the processor (110) may identify a speech signal related to the target signal among the audio signals.

[0059] According to one embodiment, the processor (110) of the electronic device (101) may receive an audio signal including a target signal and an interference signal related to a speaker's speech from an external electronic device. The processor (110) may input the audio signal and speaker-centric information into a separation model (220). Based on the output of the separation model (220), the processor (110) may identify a speech signal related to the target signal among the audio signals. The processor (110) may transmit information about the speech signal to the external electronic device.

[0060] Figure 3 illustrates a flowchart of the operations of an electronic device for acquiring speaker-centric information, according to one embodiment. In the following embodiments, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0061] Referring to FIGS. 2 and 3, operations 310 to 340 may be related to operations of the processor (110) for obtaining speaker-centric information based on the speaker's voice data. The processor (110) may train the embedding model (210) (or separation model (220)) using the speaker's voice data according to operations 310 to 340.

[0062] In operation 310, the processor (110) of the electronic device (101) may input the speaker's voice data into the embedding model (211) included in the embedding model (210). For example, the speaker's voice data may be data for learning the speaker's speech.

[0063] In operation 320, the processor (110) may obtain embedding information based on inputting the speaker's voice data into the embedding device (211). For example, the processor (110) may obtain an embedding vector based on inputting the speaker's voice data into the embedding device (211).

[0064] In operation 330, the processor (110) may input embedding information into a speaker-centric prediction model (230) included in an embedding model (210). For example, the speaker-centric prediction model (230) may include an MHA model (231), a normalization model (232), and an ASP model (233).

[0065] In operation 340, the processor (110) may obtain speaker-centric information about the speaker. For example, the processor (110) may obtain speaker-centric information about the speaker based on the output of the speaker-centric prediction model (230).

[0066] According to one embodiment, the processor (110) may store the acquired speaker-centered information in the memory (115). The processor (110) may store the speaker-centered information in the memory (115) to identify a speech signal corresponding to a target signal regarding the speaker's speech from a received audio signal.

[0067] According to the above-described embodiment, the embedding model (210) can be configured to provide information about an accurate target speaker.

[0068] Figure 4 illustrates a flowchart of the operation of an electronic device for identifying a speech signal from an audio signal, according to one embodiment. In the following embodiments, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0069] Referring to FIG. 4, operations 410 to 430 may be related to operations of a processor (110) for identifying a speaker's speech signal from an audio signal. The processor (110) may identify a speech signal related to a target signal from an audio signal including a target signal and an interference signal related to the speaker's speech according to operations 410 to 430. Operations 410 to 430 may be performed after operations 310 to 340 of FIG. 3 are performed. For example, operations 410 to 430 may be performed after speaker-centered information obtained according to operations 310 to 340 is stored in a memory (115).

[0070] Referring to operation 410, the processor (110) of the electronic device (101) may receive an audio signal including a target signal and an interference signal related to the speaker's speech. For example, the processor (110) may receive the audio signal using a microphone (126). For example, the processor (110) may receive the audio signal from an external electronic device.

[0071] For example, an audio signal may contain not only a target signal related to a speaker's speech, but also interference signals. Interference signals may be generated by signals related to other speakers' speech, noise (or external noise), reverberation, or microphone characteristics.

[0072] Referring to operation 420, the processor (110) may input an audio signal and speaker-centered information into a separation model (220). The processor (110) may input the audio signal and speaker-centered information into the separation model (220) to identify a speaker's speech signal from the audio signal.

[0073] Referring to operation 430, the processor (110) may identify a speech signal related to a target signal among the audio signal. For example, the processor (110) may identify a speech signal related to a target signal among the audio signal based on the output of the separation model (220). The target signal may represent a speaker's speech in a real environment. The speech signal may represent an estimated speaker's speech within the audio signal.

[0074] According to one embodiment, the processor (110) may provide a speech signal identified from an audio signal to a user of the electronic device (101). According to one embodiment, the processor (110) may provide a speech signal identified from an audio signal to an external electronic device that transmitted the audio signal.

[0075] According to the above-described embodiment, the separation model (220) can be configured to identify a speech signal with high accuracy (or quality). For example, since the speaker-centric information acquired according to operations 310 to 340 accurately represents the characteristics of the speaker, a speech signal with high accuracy (or quality) can be identified through the separation model (220).

[0076] Figure 5 illustrates a flowchart of the operations of an electronic device for training an embedding model, according to one embodiment. In the following embodiments, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0077] Referring to FIG. 5, in operation 510, the processor (110) of the electronic device (101) may input the speaker's voice data into the embedding model (210). Operation 510 may correspond to operation 310 of FIG. 3.

[0078] In operation 520, the processor (110) may obtain speaker-centric information. For example, the processor (110) may obtain speaker-centric information based on the output of the embedding model (210). Operation 520 may correspond to operation 340 of FIG. 3 .

[0079] In operation 530, the processor (110) may input other speech data of the speaker into the embedding model (210). For example, the processor (110) may input other speech data that is distinct from the speech data input in operation 510 into the embedding model (210). The other speech data may be related to an utterance of the speaker. The speaker of the other speech data may correspond to the speaker of the speech data of operation 510. The speaker of the other speech data may be the same as the speaker of the speech data of operation 510. Operation 530 may correspond to operation 310 or operation 510 of FIG. 3.

[0080] At operation 540, the processor (110) may obtain other speaker-centric information. For example, the processor (110) may obtain other speaker-centric information based on the output of the embedding model (210). The speaker-centric information according to operation 520 and the other speaker-centric information according to operation 540 may be associated with the same speaker. Operation 540 may correspond to operation 340 or operation 520 of FIG. 3 .

[0081] In operation 550, the processor (110) may obtain cosine similarity information between speaker-centric information and other speaker-centric information. For example, the cosine similarity information between speaker-centric information and other speaker-centric information may be referred to as first loss information. For example, the processor (110) may identify angular prototypical loss (AP loss) based on the speaker-centric information and other speaker-centric information. The AP loss may be identified based on the cosine similarity between the speaker-centric information and other speaker-centric information.

[0082] In operation 560, the processor (110) may train the embedding model (210) based on the cosine similarity information. For example, the processor (110) may train the embedding model (210) until the cosine similarity information falls within a reference range.

[0083] Figure 6 illustrates a flowchart of the operations of an electronic device for training a separation model, according to one embodiment. In the following embodiments, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0084] Referring to FIG. 6, in operation 610, the processor (110) of the electronic device (101) may obtain SNR information between the target signal and the speech signal. For example, the processor (110) may obtain SNR information between the target signal and the speech signal based on identifying the speech signal through the separation model (220). For example, the SNR information may include scale-invariant signal-to-noise ratio loss (SI-SNR loss) information. For example, the SNR information between the target signal and the speech signal may indicate whether the target signal and the speech signal are similar. For example, the SNR information between the target signal and the speech signal may be referred to as second loss information.

[0085] In operation 620, the processor (110) may train the separation model (220) based on the SNR information. For example, the processor (110) may train the separation model (220) until the SNR information falls within a reference range.

[0086] Figure 7 illustrates a flowchart of the operations of an electronic device for training an embedding model and a separation model, according to one embodiment. In the following embodiments, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0087] Referring to FIG. 7, in operation 710, the processor (110) of the electronic device (101) may obtain the overall loss information of the embedding model (210) and the separation model (220) based on the cosine similarity information (or first loss information) between the speaker-centric information and other speaker-centric information and the SNR information (or second loss information) between the target signal and the speech signal. For example, the cosine similarity information (or first loss information) may be obtained based on operations 510 to 560 of FIG. 5. For example, the SNR information (or second loss information) may be obtained based on operations 610 and 620 of FIG. 6.

[0088] According to one embodiment, the processor (110) can identify the overall loss information of the embedding model (210) and the separation model (220) based on the cosine similarity information (or first loss information) and the SNR information (or second loss information) according to the following mathematical equations.

[0089]

[0090]

[0091] Referring to Equations 6 and 7, is the total loss information of the embedding model (210) and the separation model (220). is the cosine similarity information (or first loss information). is the SNR information (or second loss information). w is the weight. is a function for RDT (role division training). is the threshold for RDT.

[0092] For example, the processor (110) can identify the overall loss information of the embedding model (210) and the separation model (220) according to Equation 6. The processor (110) can reflect the SNR information into the overall loss information only when the SNR information (or the second loss information) is within a reference range (e.g., a threshold for RDT).

[0093] At operation 720, the processor (110) may train each of the embedding model (210) and the separation model (220) until the overall loss information is identified as being below a reference loss level.

[0094] For example, the processor (110) can independently train the embedding model (210) and the separation model (220). As an example, the processor (110) can train the embedding model (210) based on cosine similarity information (or first loss information) between speaker-centered information and other speaker-centered information obtained using speech data and other speech data. As an example, the processor (110) can train the separation model (220) based on SNR information between a speech signal and a target signal obtained using an audio signal and speaker-centered information.

[0095] For example, the processor (110) may have total loss information Each of the embedding model (210) and the separation model (220) can be trained until the threshold loss level is identified as being below the reference loss level. According to an embodiment, the processor (110) sets a threshold for the RDT can be reduced each time training is performed.

[0096] In one embodiment, SNR information is a threshold for RDT. If the SNR information exceeds the threshold for RDT, the processor (110) may determine that the utterance of another speaker included in the interference signal has been identified. Based on the determination that the utterance of another speaker included in the interference signal has been identified, the processor (110) may determine that the speaker-centered information has not been accurately identified. Accordingly, the processor (110) may determine that the SNR information is a threshold for RDT. If it exceeds , the learning of the separation model (220) can be bypassed based on the SNR information.

[0097] According to one embodiment, a non-transitory computer-readable storage medium may store instructions. The instructions, when executed by at least one processor of an electronic device including a memory, may cause the electronic device to input speech data of a speaker into an embedder included in an embedding model, obtain embedding information about the speaker based on an output of the embedder, input the embedding information into a speaker-centric prediction model included in the embedding model, obtain speaker-centric information about the speaker based on an output of the speaker-centric prediction model, and after the speaker-centric information is stored in the memory, receive an audio signal including a target signal and an interference signal about an utterance of the speaker, input the audio signal and the speaker-centric information into a separation model, and identify a speech signal about the target signal among the audio signal based on an output of the separation model.

[0098] For example, the speaker-centric prediction model may include a multi-head attention (MHA) model, a normalization model, and an attentive statistics pooling (ASP) model.

[0099] For example, the instructions, when executed by the at least one processor, may cause the electronic device to obtain the speaker center information based on inputting the embedding information into the MHA model, obtaining center vector information based on an output of the MHA model, inputting the center vector information into a normalization model, obtaining normalized center vector information based on an output of the normalization model, and inputting the normalized center vector information into the ASP model.

[0100] For example, the instructions, when executed by the at least one processor, may cause the electronic device to input other speech data of the speaker into the embedding model, obtain other speaker-centric information based on an output of the embedding model, and train the embedding model based on the speaker-centric information and the other speaker-centric information.

[0101] For example, the instructions, when executed by the at least one processor, may cause the electronic device to obtain cosine similarity information (Euclidean distance information) between the speaker-centric information and the other speaker-centric information, and to train the embedding model based on the cosine similarity information.

[0102] For example, the instructions, when executed by the at least one processor, may cause the electronic device to train the separation model based on the target signal and the utterance signal.

[0103] For example, the instructions, when executed by the at least one processor, may cause the electronic device to obtain signal to noise ratio (SNR) information between the target signal and the speech signal, and to train the separation model based on the SNR information.

[0104] For example, the instructions, when executed by the at least one processor, may cause the electronic device to obtain entire loss information of the embedding model and the separation model based on the cosine similarity information and the SNR information.

[0105] For example, the instructions, when executed by the at least one processor, may cause the electronic device to train each of the embedding model and the separation model until the overall loss information is identified as being below a reference loss level.

[0106] For example, the instructions, when executed by the at least one processor, may cause the electronic device to obtain the speech data of the speaker from an external electronic device, store the speaker-centric information obtained based on the speech data in the memory, and then receive the audio signal including the target signal and the interference signal regarding speech of the speaker from the external electronic device, and transmit the speech signal regarding the target signal among the audio signals to the external electronic device.

[0107] According to one embodiment, a method performed by an electronic device may include an operation of inputting speech data of a speaker into an embedder included in an embedding model, an operation of obtaining embedding information about the speaker based on an output of the embedding model, an operation of inputting the embedding information into a speaker-centroid prediction model included in the embedding model, an operation of obtaining speaker-centred information about the speaker based on an output of the speaker-centred prediction model, an operation of receiving an audio signal including a target signal and an interference signal about an utterance of the speaker after the speaker-centred information is stored in a memory of the electronic device, an operation of inputting the audio signal and the speaker-centred information into a separation model, and an operation of identifying a speech signal about the target signal among the audio signals based on an output of the separation model.

[0108] For example, the speaker-centric prediction model may include a multi-head attention (MHA) model, a normalization model, and an attentive statistics pooling (ASP) model.

[0109] For example, the method may include an operation of inputting the embedding information into the MHA model, an operation of obtaining center vector information based on an output of the MHA model, an operation of inputting the center vector information into a normalization model, an operation of obtaining normalized center vector information based on an output of the normalization model, and an operation of obtaining the speaker center information based on inputting the normalized center vector information into the ASP model.

[0110] For example, the method may include an operation of inputting other speech data of the speaker into the embedding model, an operation of obtaining other speaker-centric information based on an output of the embedding model, and an operation of training the embedding model based on the speaker-centric information and the other speaker-centric information.

[0111] For example, the method may include an operation of obtaining cosine similarity information (Euclidean distance information) between the speaker-centered information and the other speaker-centered information, and an operation of training the embedding model based on the cosine similarity information.

[0112] For example, the method may include an operation of training the separation model based on the target signal and the speech signal.

[0113] For example, the method may include an operation of obtaining signal to noise ratio (SNR) information between the target signal and the speech signal, and an operation of training the separation model based on the SNR information.

[0114] For example, the method may include an operation of obtaining entire loss information of the embedding model and the separation model based on the cosine similarity information and the SNR information.

[0115] For example, the method may include training each of the embedding model and the separation model until the overall loss information is identified as being below a reference loss level.

[0116] According to one embodiment, an electronic device may include a microphone, a memory storing instructions and including one or more storage media, and at least one processor including a processing circuit. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to input speech data of a speaker into an embedder included in an embedding model, obtain embedding information about the speaker based on an output of the embedding model, input the embedding information into a speaker-centroid prediction model included in the embedding model, obtain speaker-centred information about the speaker based on an output of the speaker-centred prediction model, and after the speaker-centred information is stored in the memory, receive an audio signal including a target signal and an interference signal about an utterance of the speaker through the microphone, input the audio signal and the speaker-centred information into a separation model, and identify a speech signal about the target signal among the audio signals based on an output of the separation model.

[0117] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0118] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0119] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may be one that continuously stores a computer-executable program or one that temporarily stores it for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program commands, including ROM, RAM, and flash memory. In addition, examples of other media may include recording media or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc.

[0120] Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0121] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. A non-transitory computer-readable storage medium storing instructions, wherein the instructions, when executed by at least one processor of an electronic device including a memory, cause the electronic device to: Input the speaker's voice data into the embedder included in the embedding model, Based on the output of the above embedding, embedding information about the speaker is obtained, The above embedding information is input into a speaker-centric prediction model included in the above embedding model, Based on the output of the speaker-centric prediction model, speaker-centric information about the speaker is obtained, After the speaker-centered information is stored in the memory, an audio signal including a target signal and an interference signal regarding the speaker's utterance is received, Input the above audio signal and the speaker-centered information into a separation model, Based on the output of the above separation model, causing the speech signal related to the target signal to be identified among the audio signals, Non-transitory computer-readable storage medium.

2. In the first paragraph, the speaker-centered prediction model, MHA (multi-head attention) model; normalization model; and Including the ASP (attentive statistics pooling) model, Non-transitory computer-readable storage medium.

3. In the second paragraph, when the instructions are executed by the at least one processor, the electronic device: Input the above embedding information into the MHA model, Based on the output of the above MHA model, center vector information is obtained, Input the above center vector information into the normalization model, Based on the output of the above normalization model, normalized center vector information is obtained, Based on inputting the normalized center vector information into the ASP model, causing the speaker center information to be obtained, Non-transitory computer-readable storage medium.

4. In the first paragraph, when the instructions are executed by the at least one processor, the electronic device: Input the other voice data of the above speaker into the above embedding model, Based on the output of the above embedding model, other speaker-centric information is obtained, causing the embedding model to be trained based on the speaker-centric information and the other speaker-centric information, Non-transitory computer-readable storage medium.

5. In the fourth paragraph, when the instructions are executed by the at least one processor, the electronic device: Obtain cosine similarity information (Euclidean distance information) between the above speaker-centered information and the other speaker-centered information, Based on the above cosine similarity information, causing the embedding model to be trained, Non-transitory computer-readable storage medium.

6. In the fifth paragraph, when the instructions are executed by the at least one processor, the electronic device: causing the separation model to be trained based on the target signal and the speech signal, Non-transitory computer-readable storage medium.

7. In the sixth paragraph, when the instructions are executed by the at least one processor, the electronic device: Obtaining SNR (signal to noise ratio) information between the target signal and the speech signal, Based on the above SNR information, causing the separation model to be trained, Non-transitory computer-readable storage medium.

8. In the seventh paragraph, when the instructions are executed by the at least one processor, the electronic device: Based on the cosine similarity information and the SNR information, causing the entire loss information of the embedding model and the separation model to be obtained. Non-transitory computer-readable storage medium.

9. In the 8th paragraph, when the instructions are executed by the at least one processor, the electronic device: Causing each of the embedding model and the separation model to be trained until the total loss information is identified as being below a reference loss level. Non-transitory computer-readable storage medium.

10. In the first paragraph, when the instructions are executed by the at least one processor, the electronic device: Obtaining the above-mentioned speaker's voice data from an external electronic device, After the speaker-centered information obtained based on the voice data is stored in the memory, the audio signal including the target signal and the interference signal regarding the speaker's speech is received from the external electronic device, Among the above audio signals, causing the speech signal regarding the target signal to be transmitted to the external electronic device, Non-transitory computer-readable storage medium.

11. In a method performed by an electronic device, The act of inputting a speaker's voice data into an embedder included in an embedding model; An operation of obtaining embedding information about the speaker based on the output of the embedding device; An action of inputting the above embedding information into a speaker centroid prediction model included in the above embedding model; An operation of obtaining speaker-centric information about the speaker based on the output of the speaker-centric prediction model; An operation of receiving an audio signal including a target signal and an interference signal regarding the speaker's utterance after the speaker-centered information is stored in the memory of the electronic device; An operation of inputting the above audio signal and the speaker-centered information into a separation model; and Based on the output of the above separation model, an operation for identifying a speech signal related to the target signal among the audio signals is included. method.

12. In the 11th paragraph, the speaker-centered prediction model, MHA (multi-head attention) model; normalization model; and Including the ASP (attentive statistics pooling) model, method.

13. In the 12th paragraph, the method, An action of inputting the above embedding information into the MHA model; An operation of obtaining center vector information based on the output of the above MHA model; An action of inputting the above center vector information into a normalization model; An operation of obtaining normalized center vector information based on the output of the above normalization model; and An operation of obtaining the speaker center information based on inputting the normalized center vector information into the ASP model, method.

14. In the 11th paragraph, the method, An action of inputting other voice data of the speaker into the embedding model; An operation of obtaining other speaker-centric information based on the output of the above embedding model; and An operation of training the embedding model based on the speaker-centric information and the other speaker-centric information, method.

15. In the 14th paragraph, the method, An operation of obtaining cosine similarity information (Euclidean distance information) between the speaker-centered information and the other speaker-centered information; and Based on the above cosine similarity information, including an operation of training the embedding model, method.

16. In the 15th paragraph, the method, An operation of training the separation model based on the target signal and the speech signal, method.

17. In the 16th paragraph, the method, An operation of obtaining SNR (signal to noise ratio) information between the target signal and the speech signal; and Based on the above SNR information, including an operation of training the separation model, method.

18. In the 17th paragraph, the method, An operation of obtaining entire loss information of the embedding model and the separation model based on the cosine similarity information and the SNR information is included. method.

19. In the 18th paragraph, the method, Comprising an operation of training each of the embedding model and the separation model until the total loss information is identified as being below a reference loss level. method.

20. In electronic devices, microphone; A memory storing instructions and including one or more storage media; and At least one processor comprising a processing circuit, The above instructions, when individually or collectively executed by the at least one processor, Input the speaker's voice data into the embedder included in the embedding model, Based on the output of the above embedding, embedding information about the speaker is obtained, The above embedding information is input into a speaker-centric prediction model included in the above embedding model, Based on the output of the speaker-centric prediction model, speaker-centric information about the speaker is obtained, After the speaker-centered information is stored in the memory, an audio signal including a target signal and an interference signal regarding the speaker's utterance is received through the microphone, Input the above audio signal and the speaker-centered information into a separation model, Causing the electronic device to identify a speech signal related to the target signal among the audio signals based on the output of the separation model. Electronic devices.

Images