Electronic device, method, and non-transitory computer-readable storage medium for identifying speech signal corresponding to designated text

Abstract

A processor of an electronic device according to an embodiment may be configured to acquire, from a first neural network to which a speech signal received via a microphone has been input, a first sequence of feature information about portions of the speech signal corresponding to designated frame units. The processor may be configured to acquire, from a second neural network to which the designated text has been input, a second sequence of one or more phonetic symbols for the designated text. The processor may be configured to acquire, from a third neural network to which the first sequence and the second sequence have been input, a first parameter indicating the degree of correspondence between the speech signal and the designated text. The processor may be configured to acquire one or more second parameters indicating the degree of correspondence between the portions of the speech signal and each of the one or more phonetic symbols included in the second sequence.

Description

Electronic device, method, and non-transitory computer-readable storage medium for identifying a speech signal corresponding to a specified text

[0001] The present disclosure relates to an electronic device, a method, and a non-transitory computer-readable storage medium for identifying a speech signal corresponding to a specified text.

[0002] Recently, the proliferation of various types of electronic devices, such as smartphones, tablet PCs, wireless earphones, and / or smartwatches, has increased. These electronic devices can provide functions for user interaction based on human-machine interfaces (HMIs). These electronic devices can provide these functions through speech recognition services.

[0003] An electronic device according to one embodiment can use a neural network to identify whether portions of a voice signal received through a microphone match specified text.

[0004] An electronic device according to one embodiment can identify, in a noisy environment, whether portions of a voice signal received by the electronic device through a microphone match specified text.

[0005] The technical problems to be achieved in this document are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention belongs from the description below.

[0006] An electronic device according to one embodiment may include a microphone, a memory, and a processor. The processor may be configured to obtain, from a first neural network into which a voice signal received through the microphone is input, a first sequence of portions of the voice signal corresponding to a specified frame unit. The processor may be configured to obtain, from a second neural network into which a specified text is input, a second sequence of one or more phonetic symbols for the specified text. The processor may be configured to obtain, from a third neural network into which the first sequence and the second sequence are input, one or more data sets indicating a relationship between the first sequence and the second sequence.

[0007] In one embodiment, a method performed by an electronic device may include an operation of obtaining, from a first neural network into which a voice signal received through a microphone is input, a first sequence of portions of the voice signal corresponding to a specified frame unit. The method may include an operation of obtaining, from a second neural network into which a specified text is input, a second sequence of one or more phonetic symbols for the specified text. The method may include an operation of obtaining, from a third neural network into which the first sequence and the second sequence are input, one or more data sets indicating a relationship between the first sequence and the second sequence.

[0008] A computer-readable storage medium storing one or more programs according to one embodiment, wherein the one or more programs may be configured to include instructions that, when executed by a processor of an electronic device, cause the electronic device to obtain, from a first neural network into which a voice signal received through a microphone has been input, a first sequence of portions of the voice signal corresponding to a specified frame unit. The one or more programs may be configured to include instructions that, when executed by the processor of the electronic device, cause the electronic device to obtain, from a second neural network into which a specified text has been input, a second sequence of one or more phonetic symbols for a specified text. The one or more programs may be configured to include instructions that cause the electronic device to obtain, from a third neural network into which the first sequence and the second sequence have been input, one or more data sets representing a relationship between the first sequence and the second sequence.

[0009] In one embodiment, an electronic device may include a microphone, a speaker, a processor, and a memory storing instructions. The instructions, when executed by the processor, may cause the electronic device to obtain a first sequence of a first voice signal received through the microphone from a first neural network into which the first voice signal is input. The instructions, when executed by the processor, may cause the electronic device to obtain a second sequence of a second voice signal output through the speaker from a second neural network into which the second voice signal is input. The instructions, when executed by the processor, may cause the electronic device to obtain a third sequence for a specified text from a third neural network into which the specified text is input. The instructions, when executed by the processor, may cause the electronic device to obtain a fourth sequence for a third speech signal with reduced echo corresponding to the second speech signal from the first speech signal based on inputting the first sequence and the second sequence to a fourth neural network. The instructions, when executed by the processor, may cause the electronic device to obtain data sets indicating a relationship between the third sequence and the fourth sequence from a fifth neural network into which the third sequence and the fourth sequence have been input.

[0010] A method is disclosed. The method can be performed in an electronic device including a microphone and a speaker. The method may include an operation of obtaining a first sequence of a first voice signal from a first neural network into which a first voice signal received through the microphone is input. The method may include an operation of obtaining a second sequence of a second voice signal from a second neural network into which a second voice signal output through the speaker is input. The method may include an operation of obtaining a third sequence for a specified text from a third neural network into which a specified text is input. The method may include an operation of obtaining a fourth sequence for a third voice signal in which echo corresponding to the second voice signal is reduced in the first voice signal based on inputting the first sequence and the second sequence to a fourth neural network. The method may include an operation of obtaining data sets indicating a relationship between the third sequence and the fourth sequence from a fifth neural network into which the third sequence and the fourth sequence are input.

[0011] A non-transitory computer-readable storage medium is disclosed. The non-transitory computer-readable storage medium may store one or more programs. The one or more programs, when executed by a processor of an electronic device including a microphone and a speaker, may cause the electronic device to obtain a first sequence of a first voice signal received through the microphone from a first neural network into which the first voice signal is input. The one or more programs, when executed by the processor, may cause the electronic device to obtain a second sequence of a second voice signal output through the speaker from a second neural network into which the second voice signal is input. The one or more programs, when executed by the processor, may cause the electronic device to obtain a third sequence for a specified text from a third neural network into which the specified text is input. The one or more programs, when executed by the processor, may cause the electronic device to obtain a fourth sequence for a third speech signal with reduced echo corresponding to the second speech signal from the first speech signal based on inputting the first sequence and the second sequence to a fourth neural network. The one or more programs, when executed by the processor, may cause the electronic device to obtain data sets representing a relationship between the third sequence and the fourth sequence from a fifth neural network into which the third sequence and the fourth sequence have been input.

[0012] An electronic device according to one embodiment can use a neural network to identify whether portions of a voice signal received through a microphone match specified text. The electronic device can identify whether each of the portions of the voice signal and the corresponding phonetic symbols correspond to the specified text.

[0013] An electronic device according to one embodiment may use a neural network to reduce (or remove) noise (or acoustic echo) from portions of a voice signal received through a microphone. The electronic device according to one embodiment may use a neural network to identify whether portions of the voice signal from which noise (or acoustic echo) has been reduced (or removed) match specified text. The electronic device may identify whether each of the portions of the voice signal from which noise (or acoustic echo) has been reduced (or removed) and the corresponding pronunciation symbols correspond to the specified text.

[0014] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned can be clearly understood by a person having ordinary skill in the art to which the present disclosure belongs from the description below.

[0015] FIG. 1 illustrates an example of a block diagram of an electronic device according to one embodiment.

[0016] FIG. 2 illustrates an example for describing a neural network obtained from a set of parameters stored in a memory by an electronic device according to one embodiment.

[0017] FIG. 3 illustrates an example of an operation of an electronic device according to one embodiment of the present invention to identify a speech signal corresponding to a specified text through a neural network.

[0018] FIG. 4 illustrates an example of an operation of an electronic device according to one embodiment of the present invention to train a neural network to identify a speech signal corresponding to a specified text.

[0019] FIG. 5 illustrates an example of an operation of an electronic device according to one embodiment of the present invention to infer a speech signal corresponding to a specified text.

[0020] FIG. 6 illustrates an example of a user interface for setting a specified text in an electronic device according to one embodiment.

[0021] FIG. 7 illustrates an example of a user interface for indicating whether a speech signal corresponds to a specified text, according to one embodiment.

[0022] FIG. 8 illustrates an example of a state in which an electronic device according to one embodiment receives a voice signal for performing a specified function.

[0023] FIG. 9 illustrates an example of a flowchart illustrating the operation of an electronic device according to one embodiment.

[0024] FIG. 10 illustrates an example of a block diagram of an electronic device according to one embodiment.

[0025] FIG. 11A illustrates an example of an operation of an electronic device according to one embodiment of the present invention to identify a speech signal corresponding to a specified text through a neural network.

[0026] FIG. 11B illustrates an example of an operation of an electronic device according to an embodiment of the present invention to identify a mask for removing acoustic echo from a speech signal through a neural network.

[0027] FIG. 12 illustrates an example of an operation of an electronic device according to one embodiment of the present invention to train a neural network to identify a speech signal corresponding to a specified text.

[0028] FIG. 13 illustrates an example of an operation of an electronic device according to one embodiment of the present invention to infer a speech signal corresponding to a specified text.

[0029] FIG. 14 illustrates an example of a user interface for setting a specified text in an electronic device according to one embodiment.

[0030] FIG. 15 illustrates an example of a user interface for indicating whether a speech signal corresponds to a specified text, according to one embodiment.

[0031] FIG. 16 illustrates an example of a state in which an electronic device according to one embodiment receives a voice signal for performing a specified function.

[0032] FIG. 17 illustrates an example of a flowchart illustrating the operation of an electronic device according to one embodiment.

[0033] FIG. 18 illustrates an example of a flowchart illustrating the operation of an electronic device according to one embodiment.

[0034] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, electronic devices, or home appliances. Electronic devices according to the embodiments disclosed in this document are not limited to the aforementioned devices.

[0035] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0036] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0037] Various embodiments of the present document may be implemented as software (e.g., a program) including one or more instructions stored in a storage medium (e.g., internal memory or external memory) readable by a machine. For example, a processor of the machine may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily in the storage medium.

[0038] According to one embodiment, the method according to the various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0039] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0040] FIG. 1 illustrates an example block diagram of an electronic device according to one embodiment. Referring to FIG. 1, an electronic device (101) according to one embodiment may include a terminal owned by a user. The terminal may include a personal computer (PC) such as a laptop or desktop, a smartphone, a smartpad, a tablet PC, a smartwatch, and a smart accessory such as a head-mounted device (HMD).

[0041] Referring to FIG. 1, an electronic device (101) according to one embodiment may include at least one of a processor (110), a memory (120), a microphone (160), and a display (170). The processor (110), the memory (120), the microphone (160), and the display (170) may be electrically and / or operably coupled with each other by an electronic component such as a communication bus. The type and / or number of hardware components included in the electronic device (101) are not limited to those illustrated in FIG. 1. For example, the electronic device (101) may include only some of the hardware components illustrated in FIG. 1. Elements within the memory described below (e.g., layers and / or multiple neural networks (130)) may be logically separated. However, the present invention is not limited thereto.

[0042] A processor (110) of an electronic device (101) according to one embodiment may include a hardware component for processing data based on one or more instructions. The hardware component for processing data may include, for example, an arithmetic and logic unit (ALU), a field programmable gate array (FPGA), and / or a central processing unit (CPU). The number of processors (110) may be one or more. For example, the processor (110) may have a multi-core processor structure such as a dual core, a quad core, or a hexa core.

[0043] The memory (120) of the electronic device (101) according to one embodiment may include a hardware component for storing data and / or instructions input and / or output to the processor (110). The memory (120) may include, for example, a volatile memory such as a random-access memory (RAM) and / or a non-volatile memory such as a read-only memory (ROM). The volatile memory may include, for example, at least one of a dynamic RAM (DRAM), a static RAM (SRAM), a cache RAM, and a pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a flash memory, a hard disk, a compact disc, and an embedded multi media card (eMMC).

[0044] According to one embodiment, a microphone (160) of an electronic device (101) can receive a voice signal (e.g., a user's voice). The microphone (160) may be disposed in a portion of a housing of the electronic device (101). The microphone (160) may be referred to as a feedback microphone on the side disposed adjacent to a speaker (not shown). The microphone (160) may be disposed in a portion of the housing that includes a sensor (not shown) of the electronic device (101). The microphone (240) may be referred to as a feedforward microphone on the side disposed toward the outside of the electronic device (101). However, the present invention is not limited thereto.

[0045] In one embodiment, a display (170) of an electronic device (101) can output visualized information to a user of the electronic device (101). For example, the display (170) can be controlled by a processor (110) including a circuit such as a graphic processing unit (GPU) to output visualized information to the user. The display (170) can include a flat panel display (FPD) and / or electronic paper. The FPD can include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LEDs can include organic LEDs (OLEDs).

[0046] According to one embodiment, one or more instructions (or commands) representing operations and / or actions to be performed on data by a processor (110) of the electronic device (101) may be stored in the memory (120) of the electronic device (101). A set of one or more instructions may be referred to as firmware, an operating system, a process, a routine, a sub-routine, and / or an application. For example, the electronic device (101) and / or the processor (110) may perform at least one of the operations of FIG. 9 when a set of a plurality of instructions distributed in the form of an operating system, firmware, a driver, and / or an application is executed. Hereinafter, the fact that an application is installed in an electronic device (101) may mean that one or more instructions provided in the form of an application are stored in a memory (120), and that the one or more applications are stored in a format executable by the processor (110) (e.g., a file having an extension specified by the operating system of the electronic device (101). As an example, an application may include a program and / or a library related to a service provided to a user.

[0047] In one embodiment, a set of parameters related to a plurality of neural networks (130) may be stored in the memory (120) of the electronic device (101). The plurality of neural networks (130) is a recognition model implemented in software or hardware that mimics the computational capabilities of a biological system by using a large number of artificial neurons (or nodes). The plurality of neural networks (130) may perform human cognitive functions or learning processes through the artificial neurons. The parameters related to the plurality of neural networks (130) may represent, for example, weights assigned to a plurality of nodes included in the plurality of neural networks (130) and / or connections between the plurality of nodes. The structure of the plurality of neural networks (130) represented by the set of parameters stored in the memory (120) of the electronic device (101) according to one embodiment will be described later with reference to FIG. 2. The number of multiple neural networks (130) stored in the memory (120) is not limited to that shown in FIG. 1, and sets of parameters corresponding to each of the multiple neural networks may be stored in the memory (120).

[0048] For example, the electronic device (101) may segment a voice signal received through a microphone based on a specified period using the first neural network (131). The electronic device (101) may obtain a first sequence of feature information for portions of the voice signal corresponding to a specified frame unit from the first neural network into which the voice signal is input. The first neural network (131) may be an example of a pre-trained neural network for obtaining portions of the voice signal from the voice signal based on the specified period.

[0049] For example, the electronic device (101) may obtain a second sequence of one or more phonetic symbols for a specified text included in the text information (150) using a second neural network (132). The second neural network (132) may be an example of a pre-trained neural network for isolating a specified text using phonetic symbols corresponding to the specified text.

[0050] For example, the electronic device (101) can obtain data sets representing information corresponding to a voice signal and a specified text from a third neural network (133) into which the first sequence and the second sequence are input. The electronic device (101) can obtain data sets representing information corresponding to a voice signal for each of one or more pronunciation symbols included in the second sequence from the third neural network into which the first sequence and the second sequence are input.

[0051] For example, the electronic device (101) may obtain a first parameter indicating whether a voice signal corresponds to a specified text from a fourth neural network (134) into which one or more data sets representing a relationship between a first sequence and a second sequence are input. The fourth neural network (134) may be referred to as a discriminator and / or a discriminative model from the perspective of determining whether a voice signal corresponds to a specified text.

[0052] For example, the electronic device (101) may obtain one or more second parameters for identifying (or determining) whether each portion of a speech signal matches each of one or more pronunciation symbols, based on the number of one or more pronunciation symbols corresponding to a specified text, from a fifth neural network (135) into which one or more data sets have been input. The fifth neural network (135) may be referred to as a discriminator and / or a discriminative model from the perspective of determining whether each portion of a speech signal matches each of one or more pronunciation symbols.

[0053] For example, the electronic device (101) can train the first neural network (131), the second neural network (132), and / or the third neural network (133) using the first parameter and one or more second parameters. The electronic device (101) can tune at least a portion of the first neural network (131), at least a portion of the second neural network (132), and / or the third neural network (133) using the first parameter and one or more second parameters. The electronic device (101) can train the first neural network (131), the second neural network (132), and / or the third neural network (133) using the first parameter and one or more second parameters to identify whether a speech signal different from a predetermined speech signal corresponds to a specified text based on the reception of the speech signal. The operation of the electronic device (101) to train at least a portion of the plurality of neural networks (130) is described later in FIGS. 3 and 4.

[0054] A call software application (140) according to one embodiment may be an example of a software application for indicating to a user whether a voice signal corresponding to a specified text has been received. The call software application (140) may be used to execute at least one function using a plurality of neural networks (130). The electronic device (101) may perform at least one operation based on the reception of a voice signal that calls a specified text referring to the electronic device (101) using the call software application (140). An example of an operation in which the electronic device (101) executes at least one function based on the execution of the call software application (140) is described below with reference to FIGS. 6 to 8.

[0055] For example, the text information (150) may include a learning data set for training a plurality of neural networks (130). The text information (150) may include information about one or more phonetic symbols. A phonetic symbol may refer to a written symbol that represents a voice in a form that can be seen by the user. For example, the text information (150) may include information about a call word for a user of the electronic device (101) to use a function of the electronic device (101) based on the execution of a call software application (140). After identifying a voice signal corresponding to the call word, the electronic device (101) may wait to receive another voice signal indicating the initiation of a function of the electronic device (101). However, the present invention is not limited thereto. The electronic device (101) may provide the user with a user interface (e.g., the user interface (610) of FIG. 6) for changing the call word.

[0056] According to an embodiment as described above, the electronic device (101) can identify a call word that is not included in a training data set for training by using the plurality of trained neural networks (130) after training the plurality of neural networks (130). The electronic device (101) can provide functions related to human-machine interaction (HMI) to a user by using the plurality of neural networks (130) that can identify a call word that is not included in the training data set.

[0057] FIG. 2 illustrates an example for explaining a neural network that an electronic device according to one embodiment acquires from a set of parameters stored in a memory. Referring to FIG. 2 , at least a portion of a plurality of neural networks (130) may include a plurality of layers. For example, the plurality of neural networks (130) may include an input layer (210), one or more hidden layers (220), and an output layer (230). The input layer (210) may receive a vector representing input data (e.g., a vector having elements corresponding to the number of nodes included in the input layer (210). Signals generated from each of the nodes in the input layer (210) generated by the input data may be transmitted from the input layer (210) to the hidden layers (220). The output layer (230) may generate output data of the plurality of neural networks (130) based on one or more signals received from the hidden layers (220). The above output data may include, for example, a vector having elements corresponding to the number of nodes included in the output layer (230).

[0058] Referring to FIG. 2, one or more hidden layers (220) may be positioned between the input layer (210) and the output layer (230), and may convert input data transmitted through the input layer (210) into values ​​that are easy to predict. The input layer (210), one or more hidden layers (220), and the output layer (230) may include a plurality of nodes. The one or more hidden layers (220) are not limited to the illustrated feedforward-based topology, and may be, for example, a convolution filter or a fully connected layer in a convolutional neural network (CNN), or various types of filters or layers grouped based on special functions or characteristics. In one embodiment, the one or more hidden layers (220) may be a layer based on a recurrent neural network (RNN) in which the output value is input again to the hidden layer of the current time. For example, the input layer (210), one or more hidden layers (220), and / or the output layer (230) may be some layers of a transformer model. According to one embodiment, a plurality of neural networks (130) may include a number of hidden layers (220) to form a deep neural network. Training a deep neural network is called deep learning. Among the nodes of the plurality of neural networks (130), a node included in the hidden layers (220) is referred to as a hidden node.

[0059] Nodes included in the input layer (210) and one or more hidden layers (220) may be connected to each other through connecting lines having connecting weights, and nodes included in the hidden layer and the output layer may also be connected to each other through connecting lines having connecting weights. Tuning and / or training the plurality of neural networks (130) may mean changing the connecting weights between nodes included in each of the layers included in the plurality of neural networks (130) (e.g., the input layer (210), one or more hidden layers (220), and the output layer (230)). Tuning the plurality of neural networks (130) may be performed based on, for example, supervised learning and / or unsupervised learning.

[0060] An electronic device according to one embodiment may tune a plurality of neural networks (130) based on reinforcement learning in unsupervised learning. For example, the electronic device may change policy information used by the plurality of neural networks (130) to control an agent based on interactions between the agent and the environment. The policy information is a rule by which the electronic device determines the actions of the agent within the environment using the neural network, and the electronic device may change the policy information of the neural network by training the neural network based on interactions between the agent and the environment. For example, the policy information may be changed to determine an optimal action and / or sequence of actions for the agent to achieve an obtainable reward and / or goal. In one embodiment, the electronic device may cause a change in the policy information by the plurality of neural networks (130) in order to maximize the goal and / or reward of the agent through the interaction.

[0061] FIG. 3 illustrates an example of an operation of an electronic device according to one embodiment of the present invention to identify a speech signal corresponding to a specified text through a neural network. The electronic device (101) of FIG. 1 may include the electronic device (101) of FIG.

[0062] Referring to FIG. 3, an electronic device (101) according to one embodiment may receive a voice signal (310) from a user using a microphone (160). For example, the electronic device (101) may phonemicize the voice signal (310) using a first neural network (131). The electronic device (101) may segment the voice signal (310) based on a specified period. The electronic device (101) may obtain portions (310-1, 310-2, 310-3, 310-4) of the voice signal (310) based on a specified frame unit. For example, the specified frame unit may be identified based on phonemic symbols corresponding to the voice signal (310). For example, the specified frame unit may be changed based on the length of a specified text. However, the present invention is not limited thereto.

[0063] For example, the electronic device (101) can change the voice signal (310) into first voice data (311) based on frequency. The electronic device (101) can change the voice signal (310) into second voice data (312) based on time. The electronic device (101) can input the first voice data (311) and the second voice data (312) into the first neural network (131) to obtain a first sequence (320). The order and / or number of the first sequence may vary depending on the embodiment, based on the portions (310-1, 310-2, 310-3, 310-4) of the voice signal (310) distinguished based on a specified frame unit. The first sequence (320) may be an example of a matrix based on N dimensions. The first sequence (320) may include feature information based on multiple dimensions. The first sequence (320) may include vector parameters embedded from the speech signal (310).

[0064] For example, the electronic device (101) may process the first voice data (311) using the first operator (131-1) and / or the second operator (131-2) included in the first neural network (131). The first operator (131-1) may be an example of an operator used to identify changes in the first voice data (311) based on frequency over time. The first operator (131-1) may include a short time Fourier transform (STFT). The second operator (131-2) may include a one-dimensional based convolution operator (e.g., Conv1D). The second operator (131-2) may include an activation function. The activation function may include a non-linear function such as a rectified linear unit (ReLu) function or a sigmoid operation. The number of second operators (131-2) may be one or more.

[0065] For example, the electronic device (101) may process the second voice data (312) using the third operator (131-3) and / or the fourth operator (131-4) included in the first neural network (131). The third operator (131-3) may include a pre-trained embedder to vectorize the second voice data (312). The fourth operator (131-4) may include a time-based one-dimensional convolution operator (e.g., TConv1D). The fourth operator (131-4) may include an activation function. The fourth operator (131-4) may be used to identify one-dimensional sequence data, but is not limited thereto.

[0066] For example, the electronic device (101) can obtain first sequence data by processing first voice data (311) using a first operator (131-1) and a second operator (131-2). The electronic device (101) can obtain second sequence data by processing second voice data (312) using a third operator (131-3) and a fourth operator (131-4). The electronic device (101) can obtain a first sequence (320) by using the first sequence data and the second sequence data. The electronic device (101) can obtain the first sequence (320) based on calculating the first sequence data and the second sequence data using a designated operator.

[0067] According to one embodiment, the electronic device (101) may obtain a second sequence (330) from a second neural network (132) into which a specified text (315) included in text information (e.g., text information (150) of FIG. 1) is input. For example, the electronic device (101) may obtain the second sequence (330) using one or more phonetic symbols corresponding to the specified text (315). The electronic device (101) may isolate the specified text (315) based on the one or more phonetic symbols. For example, if the specified text (315) is 'Hello', the one or more phonetic symbols corresponding to the specified text (315) may include 'HH', 'EH1', 'L', and 'OW0'. However, the present invention is not limited thereto.

[0068] For example, the electronic device (101) can change the specified text (315) into the second sequence (330) using the fifth operator (132-1) and / or the sixth operator (132-2) included in the second neural network (132). The fifth operator (132-1) can include a pre-trained embedding to vectorize (or data-ize) the specified text (315). The data of the specified text (315) processed through the fifth operator (132-1) can be an example of vector data based on multiple dimensions (e.g., two dimensions). The sixth operator (132-2) can include fully connected layers. The sixth operator (132-2) can include an activation function. However, the present invention is not limited thereto.

[0069] According to one embodiment, the electronic device (101) may input a first sequence (320) and a second sequence (330) into a third neural network (133) to obtain one or more data sets (340). For example, the electronic device (101) may obtain one or more data sets (340) representing a relationship between the first sequence (320) and the second sequence (330) from the third neural network (133) into which the first sequence (320) and the second sequence (330) are input. The one or more data sets (340) may include information representing a connection (or relationship) between each of the first sequence data sets included in the first sequence (320) and each of the second sequence data sets included in the second sequence (330). The one or more data sets (340) may include pattern information between the first sequence (320) and the second sequence (330). From the perspective of obtaining pattern information between the first sequence (320) and the second sequence (330), the third neural network (133) may be referred to as a pattern extractor.

[0070] According to one embodiment, a first data set (340-1) among one or more data sets (340) may indicate a degree of association of a second sequence (330) with a first sequence (320). For example, the first data set (340-1) may include information indicating a degree to which each of one or more pronunciation symbols corresponds to each of the portions (310-1, 310-2, 310-3, 310-4) of the speech signal (310). As an example, the first data set (340-1) may include information indicating a degree to which the entire specified text (315) corresponds to the entire speech signal (310). For example, a second data set (340-2) among one or more data sets (340) may indicate a degree of association of the first sequence (320) with a second sequence (330). The second data set (340-1) may include information indicating a relationship between one or more pronunciation symbols and portions (310-1, 310-2, 310-3, 310-4) of the speech signal (310). For example, the second data set (340-2) may include information indicating the degree to which each portion (310-1, 310-2, 310-3, 310-4) of the speech signal (310) corresponds to each of one or more pronunciation symbols. However, the present invention is not limited thereto.

[0071] An electronic device (101) according to one embodiment may obtain a first parameter (350) from a fourth neural network (134) into which at least a portion of one or more data sets (340) is input. The first parameter (350) may indicate the degree to which a speech signal (310) and a specified text (315) correspond. The first parameter (350) may indicate whether the speech signal (310) matches the specified text (315). The fourth neural network (134) may include a seventh operator (134-1). The seventh operator (134-1) may further include one or more operators. The seventh operator (134-1) may include a deep learning framework (e.g., a long short term memory (LSTM) or a gated recurrent unit (GRU)), a fully connected layer, and / or an activation function. The electronic device (101) can use the first parameter (350) to identify whether the voice signal (310) matches the specified text (315).

[0072] An electronic device (101) according to one embodiment may obtain one or more second parameters (360) from a fifth neural network (135) into which at least a portion (e.g., a second data set (340-2)) of one or more data sets (340) is input. The fifth neural network (135) may include an eighth operator (135-1). The eighth operator (135-1) may include a fully connected layer and / or an activation function (e.g., a sigmoid function). The number of the one or more second parameters (360) may correspond to the number of one or more phonetic symbols corresponding to a specified text (315). The one or more second parameters (360) may indicate the degree to which portions (310-1, 310-2, 310-3, 310-4) of the speech signal (310) correspond to each of the one or more phonetic symbols. Referring to FIG. 3, the first sub-parameter (360-1) may indicate that the first part (310-1) of the speech signal (310) matches (e.g., true) the first pronunciation symbol (e.g., 'HH'). The second sub-parameter (360-2) may indicate that the second part (310-2) of the speech signal (310) does not match (e.g., false) the second pronunciation symbol (e.g., 'EH1'). The third sub-parameter (360-3) may indicate that the third part (310-3) of the speech signal (310) matches the first pronunciation symbol (e.g., 'L'). The fourth sub-parameter (360-4) may indicate that the fourth part (310-4) of the speech signal (310) matches the fourth pronunciation symbol (e.g., 'OW0'). However, the present invention is not limited thereto. Parameters (e.g., the first parameter (350) and / or one or more second parameters (360)) may be identified based on dummy variables. A dummy variable may be a variable that can indicate whether a specified feature is present (e.g., whether a speech signal (310) and a specified text (315) match) using '0' or '1'.A dummy variable may be referred to as a binary indicator (or a boolean indicator), a binary variable, an indicator variable, and / or a design variable. For example, when the first sub-parameter (360-1) is '1', it may indicate that the first part (310-1) of the speech signal (310) matches (e.g., true) the first pronunciation symbol (e.g., 'HH'). When the first sub-parameter (360-1) is '0', it may indicate that the first part (310-1) of the speech signal (310) does not match the first pronunciation symbol (e.g., 'HH').

[0073] Hereinafter, with reference to FIG. 4, an example of an operation of an electronic device (101) according to one embodiment to train a plurality of neural networks (130) using a first parameter (350) and / or one or more second parameters (360) is described.

[0074] FIG. 4 illustrates an example of an operation of an electronic device according to an embodiment of the present invention to train a neural network to identify a speech signal corresponding to a specified text. The electronic device (101) of FIG. 4 may include the electronic device (101) of FIG. 1 . The plurality of neural networks (130) of FIG. 4 may include the first neural network (131), the second neural network (132), the third neural network (133), the fourth neural network (134), and / or the fifth neural network (135) of FIG. 3 .

[0075] An electronic device (101) according to one embodiment can receive a voice signal (310) (e.g., NCYA) using a microphone in a state (400). The electronic device (101) can identify a designated text (415) (e.g., NCYA) included in the text information. The designated text (415) can correspond to a call word for calling the electronic device (101).

[0076] For example, the electronic device (101) can obtain predicted values ​​(440) from a plurality of neural networks (130) into which a speech signal (310) and a designated text (415) are input. The predicted values ​​(440) can indicate the degree to which the speech signal (310) and the designated text (415) correspond. The predicted values ​​(440) can indicate the degree to which each of the portions of the speech signal (310) (e.g., portions (310-1, 310-2, 310-3, 310-4) of FIG. 3) corresponds to each of one or more phonetic symbols corresponding to the designated text (415). The predicted values ​​(440) can have at least one value between 0 and 1. The predicted values ​​(440) can correspond to the first parameter (350) and / or one or more second parameters (360) of FIG. 3.

[0077] An electronic device (101) according to one embodiment can set labeling values ​​(450) for learning of a plurality of neural networks (130) based on supervised learning. In a state (400), since a text corresponding to a voice signal (310) matches a designated text (415), the electronic device (101) can set the labeling values ​​(450) to a value (e.g., 1) indicating that the voice signal (310) and the designated text (415) match.

[0078] For example, the electronic device (101) can train a plurality of neural networks (130) so that the predicted values ​​(440) and the labeling values ​​(450) match. For example, the electronic device (101) can train a plurality of neural networks (130) so that the predicted values ​​(440) converge to the labeling values ​​(450). For example, the electronic device (101) can train a plurality of neural networks (130) so that a first predicted value (440-1) (e.g., 0.95) corresponding to a first parameter (e.g., the first parameter (350) of FIG. 3) matches a first labeling value (450-1). For example, the electronic device (101) may train a plurality of neural networks (130) based on supervised learning so that second prediction values ​​(440-2) corresponding to one or more second parameters (e.g., one or more second parameters (360) of FIG. 3) are matched to second labeling values ​​(450-2). As an example, the electronic device (101) may obtain parameters for training the plurality of neural networks (130) based on binary cross entropy error (BCEE).

[0079] For example, the electronic device (101) can train a plurality of neural networks (130) using a loss value (470) (or a loss parameter representing an error rate) representing the difference between predicted values ​​(440) and labeled values ​​(450). The loss value (470) can be used to identify a similarity in speech level between a speech signal (310) and a specified text (415).

[0080] For example, referring to FIG. 3, at least a part of the first neural network (131) (e.g., the second operator (131-2)), at least a part of the second neural network (132) (e.g., the sixth operator (132-2)), the third neural network (133), the fourth neural network (134), and / or the fifth neural network (135) can be trained based on the loss value (470).

[0081] According to one embodiment, an electronic device (101) may receive a voice signal (310) using a microphone in a state (405). A text corresponding to the voice signal (310) (e.g., NCYA) and a designated text (416) (e.g., ENVYA) may be different. The electronic device (101) may obtain prediction values ​​(441) from a plurality of neural networks (130) into which the voice signal (310) and the designated text (416) are input. Since the text corresponding to the voice signal (310) (e.g., NCYA) and the designated text (416) (e.g., ENVYA) are different, the electronic device (101) may set a first label value (460-1) among the labeling values ​​(460) to 0. For example, at least one of the second labeling values ​​(460-2) may be set to 0 because at least one of the portions of the speech signal (310) (e.g., '씨') is different from a portion of the specified text (416) (e.g., '비'). The electronic device (101) may train the plurality of neural networks (130) so that the predicted values ​​(441) obtained from the plurality of neural networks (130) and the labeling values ​​(460) are matched. The electronic device (101) may train the plurality of neural networks (130) based on supervised learning using a loss value (471) representing an error rate between the predicted values ​​(441) and the labeling values ​​(460). The loss value (471) may be used to identify phoneme-level similarities between portions of a speech signal (310) and one or more phonetic symbols corresponding to a specified text (415), but is not limited thereto.

[0082] An electronic device (101) according to one embodiment can train a plurality of neural networks (130) using speech data used in everyday life, which is distinct from a learning data set for training a neural network. The electronic device (101) can accumulate more data by training the plurality of neural networks (130) using the speech data. The electronic device (101) can improve the performance of the plurality of neural networks (130) based on training the plurality of neural networks (130) using the speech data. The electronic device (101) can learn to be more suitable for the user based on training the plurality of neural networks (130) using the speech data.

[0083] Below, an example of an electronic device (101) that infers a call word using multiple learned neural networks (130) is described with reference to FIG. 5.

[0084] FIG. 5 illustrates an example of an operation of an electronic device according to an embodiment of the present invention to infer a voice signal corresponding to a specified text. The electronic device (101) of FIG. 5 may include the electronic device (101) of FIGS. 1 to 4 . Referring to FIG. 5 , a plurality of neural networks (130) according to an embodiment of the present invention may be examples of neural networks trained to identify whether a voice signal received through a microphone matches a specified text.

[0085] An electronic device (101) according to one embodiment may receive a voice signal (510) using a microphone. The voice signal (510) may correspond to information representing a word, a sentence, and / or an utterance. For example, based on receiving the voice signal (510), the electronic device (101) may infer (or identify) whether at least a portion of the voice signal (510) corresponds to a designated text (515) using a plurality of neural networks (130). For example, the electronic device (101) may set the designated text (515). The electronic device (101) may receive an input from a user to change the designated text (515) using a user interface for setting the designated text (515). Based on receiving the input, the electronic device (101) may set the designated text (515). The electronic device (101) can, at least temporarily, refrain from training the plurality of neural networks (130) to identify a speech signal corresponding to the specified text (515) by changing the specified text (515). Since the electronic device (101) has trained the plurality of neural networks (130) using a speech-level data set, the electronic device (101) can, independently of performing training of the plurality of neural networks (130) corresponding to the changed specified text (515), infer whether the received speech signal (510) corresponds to the specified text (515).

[0086] For example, the electronic device (101) can input a specified text (515) and a voice signal (510) into a plurality of neural networks (130) to obtain a first parameter (520) and one or more second parameters (530). The electronic device (101) can obtain portions of the voice signal (510) based on a specified frame unit. The electronic device (101) can identify one or more phonetic symbols for the specified text (515). The electronic device (101) can obtain a first parameter (520) indicating whether the entire voice signal (510) corresponds to the entire specified text (515). The electronic device (101) can obtain one or more second parameters (530) indicating whether each of the portions of the voice signal (510) corresponds to each of one or more phonetic symbols.

[0087] For example, a first parameter (520) and one or more second parameters (530) can be obtained based on a binary indicator (e.g., a value having 0 or 1). The electronic device (101) can infer the value of the first parameter (520) based on a specified probability distribution for identifying the first parameter (520). The electronic device (101) can obtain the first parameter (520) having a value closer to 1 as the degree to which the entire speech signal (510) and the entire specified text (515) corresponds is relatively high.

[0088] For example, the electronic device (101) may infer the value of one or more second parameters (530) based on a specified probability distribution for identifying one or more second parameters (530). The electronic device (101) may obtain one or more second parameters (530) having a value relatively close to 1 as the degree to which each of the portions of the voice signal (510) corresponds to each of the one or more phonetic symbols is relatively high. At least one of the one or more second parameters (530) may match a value of 0. For example, if at least one of the portions of the voice signal (510) does not match at least one of the one or more phonetic symbols, the electronic device (101) may obtain at least one of the one or more second parameters (530) having a value relatively close to 0. However, the present invention is not limited thereto.

[0089] As described above, the electronic device (101) according to one embodiment can infer whether it has identified a voice signal corresponding to at least some randomly set call words by using a plurality of learned neural networks (130).

[0090] Referring to FIG. 6 below, an example of a user interface for setting a call word according to one embodiment is described below.

[0091] FIG. 6 illustrates an example of a user interface for setting a designated text in an electronic device according to an embodiment. The electronic device (101) of FIG. 6 may include the electronic device (101) of FIGS. 1 to 5. Referring to FIG. 6, in a state (600), the electronic device (101) according to an embodiment may display a user interface (610) on the display (170) for setting a call word for calling the electronic device (101) based on the execution of the call software application (140) of FIG. 1. For example, the user interface (610) may include a visual object (620) for obtaining a call word for initiating execution of the call software application (140) from a user. The electronic device (101) may receive a text object (e.g., Envya) representing the call word from the user using the visual object (620). For example, the electronic device (101) may set the text object as a call word based on receiving an input indicating selection of the icon (630). The electronic device (101) may initiate execution of the call software application (140) based on identifying a voice signal corresponding to the call word through a plurality of neural networks (130) using a microphone. The electronic device (101) may perform at least one function based on the execution of the call software application (140). The electronic device (101) may receive another voice signal indicating the name of at least one function based on the execution of the call software application (140), and may initiate execution of at least one function. However, the present invention is not limited thereto.

[0092] FIG. 7 illustrates an example of a user interface for indicating whether a voice signal corresponds to a specified text, according to one embodiment. The electronic device (101) of FIG. 7 may include the electronic device (101) of FIGS. 1 to 6 . Based on the execution of the calling software application (140), the electronic device (101) may, at a state (700), provide a user with an indication of whether a voice signal corresponding to a specified text (720) has been received.

[0093] According to one embodiment, an electronic device (101) may display a user interface (710) on a display to indicate whether a voice signal corresponding to a specified text (720) has been received. For example, the electronic device (101) may guide a user using a visual object (750) as to whether a voice signal received using a microphone corresponds to the specified text (720). For example, the electronic device (101) may guide a user using visual objects (760) as to whether each portion of the voice signal corresponds to each of one or more phonetic symbols (730). For example, the visual object (750) may correspond to a first parameter (e.g., the first parameter (520) of FIG. 5). For example, when the first parameter is 1, the electronic device (101) may display a visual object (750) indicating that the voice signal and the specified text are matched. For example, the visual objects (760) may correspond to one or more second parameters (e.g., one or more second parameters (530) of FIG. 5). As an example, the electronic device (101) may display a visual object indicating that at least one of the portions of the speech signal corresponding to the at least one second parameter and at least one of the phonetic symbols (730) corresponding to the at least one second parameter do not match, if at least one of the second parameters is 0. However, the present invention is not limited thereto.

[0094] An electronic device (101) according to one embodiment may receive an input indicating the selection of a visual object (740). Based on the input received, the electronic device (101) may provide a notification requesting the user to speak using a microphone, although this is not limited thereto. The specified text (720) may be included in the text information (150) of FIG. 1.

[0095] As described above, the electronic device (101) according to one embodiment can display a user interface (710) on the display for correcting the user's pronunciation. The electronic device (101) can guide the user on the accuracy of the user's speech in a second language received from the user based on the first language through a plurality of neural networks (130).

[0096] FIG. 8 illustrates an example of a state in which an electronic device according to one embodiment receives a voice signal for performing a specified function. The electronic device (101) of FIG. 8 may include the electronic device (101) of FIGS. 1 to 7 . For example, the electronic device (101) may display a user interface (810) representing a game on the display in state (800).

[0097] An electronic device (101) according to one embodiment can identify text information included in a user interface (810). For example, the electronic device (101) can identify a first nickname (813) of a first character (815) representing a user of the electronic device (101) (e.g., User A), and a second nickname (833) of a second character (835) representing another user (e.g., User B). For example, the electronic device (101) can receive a voice signal (805) using a microphone. The electronic device (101) can identify a text corresponding to the voice signal (805) through a plurality of neural networks (130) of FIG. 1. If the text matches the second nickname (833), the electronic device (101) can transmit data representing the voice signal (805) to an external electronic device corresponding to the second character (835).

[0098] For example, the electronic device (101) can control the first character (815) to identify text representing an available skill (820) within the user interface (810). If a voice signal (805) received using a microphone matches the text representing the skill (820), the electronic device (101) can control the first character (815) to use the skill (820). However, the present invention is not limited thereto. Based on receiving a voice signal for controlling the first character (815) within the user interface (810), the electronic device (101) can perform a function corresponding to the voice signal using the first character (815).

[0099] For example, the electronic device (101) can receive a voice signal (805) representing a sentence through a microphone. The electronic device (101) can identify whether the voice signal (805) representing the sentence corresponds to a specified text through a plurality of neural networks (130). For example, the electronic device (101) can control the first character (815) to use the skill (820) on the second character (835) based on identifying the voice signal (805) representing a sentence (e.g., attacking the second character (835) using the skill (820). However, the present invention is not limited thereto.

[0100] As described above, the electronic device (101) according to one embodiment can acquire a voice signal (805) using a microphone while displaying a user interface (810) for providing a game service. The voice signal (805) can correspond to a command for controlling a first character (815) corresponding to a user of the electronic device (101) within the user interface (810). Based on acquiring the voice signal (805), the electronic device (101) can identify a designated text matching the voice signal (805) and control the first character (815) to perform a function corresponding to the designated text. The electronic device (101) can provide a service capable of controlling the first character (815) based on the user's voice signal. The electronic device (101) can provide a service capable of controlling the first character (815) based on a hands-free operation.

[0101] FIG. 9 illustrates an example of a flowchart illustrating the operation of an electronic device according to one embodiment. The electronic device of FIG. 9 may include the electronic device (101) of FIG. 1. At least one of the operations of FIG. 9 may be performed by the electronic device (101) of FIG. 1. At least one of the operations of FIG. 9 may be controlled by the processor (110) of FIG. 1. Each of the operations of FIG. 9 may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each of the operations may be changed, and at least two operations may be performed in parallel.

[0102] Referring to FIG. 9, in operation 910, a processor according to an embodiment may obtain a first sequence of portions of a speech signal corresponding to a specified frame unit from a first neural network into which a speech signal is input. The processor may obtain portions of the speech signal based on a specified time from the speech signal. The processor may segment the speech signal based on the specified time to obtain portions of the speech signal. The processor may change the portions of the speech signal into first speech data (311) of FIG. 3 based on frequency. The processor may change the portions of the speech signal into second speech data (312) of FIG. 3 based on time. The processor may obtain a first sequence (e.g., the first sequence (320) of FIG. 3) through a first neural network into which the first speech data and the second speech data are input (e.g., the first neural network (131) of FIG. 1).

[0103] Referring to FIG. 9, in operation 920, a processor according to one embodiment may obtain a second sequence of one or more phonetic symbols for a specified text from a second neural network into which the specified text has been input. The processor may obtain the second sequence (e.g., the second sequence (330) of FIG. 3) using one or more phonetic symbols corresponding to the specified text. The processor may obtain the second sequence based on the number of one or more phonetic symbols.

[0104] Referring to FIG. 9, in operation 930, a processor according to one embodiment may obtain one or more data sets representing a relationship between the first sequence and the second sequence from a third neural network into which the first sequence and the second sequence are input, and from the third neural network into which the first sequence and the second sequence are input.

[0105] For example, the processor may identify a first data set that indicates the degree to which each of one or more phonetic symbols corresponds to each of the portions of the speech signal among one or more data sets. The processor may input the first data set to a fourth neural network to obtain a first parameter (e.g., the first parameter (350) of FIG. 3) indicating whether the speech signal corresponds to a specified text.

[0106] For example, the processor may identify a second data set that indicates the degree to which each of the portions of the speech signal corresponds to each of one or more phonetic symbols among the one or more data sets. The processor may input the second data set to a fifth neural network to obtain one or more second parameters (e.g., one or more second parameters (350) of FIG. 3) having a number equal to the number of the one or more phonetic symbols. The one or more second parameters may indicate whether the portions of the speech signal correspond to each of the one or more phonetic symbols. For example, the processor may further input the second data set to a fourth neural network to obtain the first parameter. The processor may further input the second data set, as well as the first data set, to the fourth neural network to improve the accuracy of the first parameter. However, the present invention is not limited thereto.

[0107] Fig. 10 illustrates an example of a block diagram of an electronic device according to an embodiment. Referring to Fig. 10, an electronic device (101) according to an embodiment may correspond to the electronic device (101) of Fig. 1.

[0108] Referring to FIG. 10, an electronic device (1001) according to an embodiment may include at least one of a processor (1010), a memory (1020), a microphone (1060), a display (1070), or a speaker (1080). The processor (1010), the memory (1020), the microphone (1060), and the display (1070) of FIG. 10 may correspond to the processor (1010), the memory (120), the microphone (160), and the display (170) of FIG. 1, respectively.

[0109] In the description of the electronic device (1001) of FIG. 10, any content that overlaps with the description of the electronic device (101) of FIG. 1 may not be repeated.

[0110] The elements within the memory described below (e.g., layers and / or multiple neural networks (1030)) may be logically separated, but are not limited thereto.

[0111] In one embodiment, the speaker (1080) of the electronic device (1001) may output an audio signal. The speaker (1080) may be disposed in a portion of the housing of the electronic device (1001). In one embodiment, the audio signal may include a voice signal generated by the electronic device (1001) (or an application running on the electronic device (1001). However, the present invention is not limited thereto.

[0112] According to one embodiment, one or more instructions (or commands) representing operations and / or actions to be performed on data by a processor (1010) of the electronic device (1001) may be stored in a memory (1020) of the electronic device (1001). A set of one or more instructions may be referred to as firmware, an operating system, a process, a routine, a sub-routine, and / or an application. For example, the electronic device (1001) and / or the processor (1010) may perform at least one of the operations of FIGS. 17 and 18 when a set of multiple instructions distributed in the form of an operating system, firmware, driver, and / or application is executed.

[0113] In one embodiment, a set of parameters related to a plurality of neural networks (1030) may be stored in a memory (1020) of an electronic device (1001).

[0114] The plurality of neural networks (1030) may include the plurality of neural networks (130) of FIG. 1. For example, the third neural network (1033) of FIG. 10 may correspond to the second neural network (132) of FIG. 1. For example, the fifth neural network (1035) of FIG. 10 may correspond to the third neural network (133) of FIG. 1. For example, the sixth neural network (1036) of FIG. 10 may correspond to the fourth neural network (134) of FIG. 1. For example, the seventh neural network (1037) of FIG. 10 may correspond to the fifth neural network (135) of FIG. 1.

[0115] The plurality of neural networks (1030) may include neural networks different from the plurality of neural networks (130) of FIG. 1. For example, the second neural network (1032) of FIG. 10 may be a neural network different from the plurality of neural networks (130) of FIG. 1. For example, the fourth neural network (1034) of FIG. 10 may be a neural network different from the plurality of neural networks (130) of FIG. 1.

[0116] For example, the connection structure of the plurality of neural networks (1030) of FIG. 10 may be different from the connection structure of the plurality of neural networks (130) of FIG. 1. The connection structure of the plurality of neural networks (1030) of FIG. 10 may be described below with reference to FIG. 11a.

[0117] The structure of a plurality of neural networks (1030) represented by a set of parameters stored in a memory (1020) of an electronic device (1001) according to one embodiment may have a structure described through FIG. 2. The number of the plurality of neural networks (1030) stored in the memory (1020) is not limited to that illustrated in FIG. 10, and sets of parameters corresponding to each of the plurality of neural networks may be stored in the memory (1020).

[0118] For example, the electronic device (1001) can receive a first audio signal through a microphone (1060).

[0119] For example, the electronic device (1001) may segment a first voice signal received through a microphone (1060) based on a specified period using a first neural network (1031). The electronic device (1001) may generate a first sequence of feature information (e.g., E) for portions of the first voice signal corresponding to a specified frame unit from the first neural network (1031) into which the first voice signal is input. m ∈R Ta X 128 ) can be obtained. The first neural network (1031) may be an example of a pre-trained neural network to obtain portions of the first speech signal from the first speech signal based on a specified period. In one embodiment, the first sequence (E m ) may be referred to as a first feature vector, or a first embedding. In one embodiment, the first sequence, the first feature vector, or the first embedding has a specified dimension (e.g., R Ta X 128 ) can be a vector. Here, T a can correspond to the number of parts of the first speech signal. 128 can represent the size of the dimension that each part of the first speech signal has by being embedded. Accordingly, the first sequence is T, which is a vector having a size of 128. a It can be a vector that is combined as many as a dog. In one embodiment, 128 is only an example, and each embedding included in the first sequence can be represented by a value of a different dimension. In one embodiment, the first speech signal can include the user's speech and one or more noises. In one embodiment, the one or more noises can include an echo (or acoustic reverberation) corresponding to the audio signal output through the speaker (1080).

[0120] For example, the electronic device (1001) can obtain an audio signal (or a second audio signal) output through the speaker (1080). For example, the electronic device (1001) can obtain an audio signal (or a second audio signal) output through the speaker (1080) while receiving a first audio signal through the microphone (1060).

[0121] For example, the electronic device (1001) may segment an audio signal (or a second voice signal) output through a speaker (1080) based on the specified period using the second neural network (1032). The electronic device (1001) may generate a second sequence of feature information (e.g., E) for portions of the second voice signal corresponding to the specified frame unit from the second neural network (1032) into which the second voice signal is input. p ∈R Ta X 128 ) can be obtained. The second neural network (1032) may be an example of a pre-trained neural network to obtain portions of the second speech signal from the second speech signal based on a specified period. In one embodiment, the second sequence (E p ) may be referred to as a second feature vector, or a second embedding. In one embodiment, the second sequence, the second feature vector, or the second embedding may be a specified dimension (e.g., R Ta X 128 ) may be a vector of. In one embodiment, the second sequence, the second feature vector, or the second embedding corresponds to the first sequence, the first feature vector, or the first embedding in a specified dimension (e.g., R Ta X 128 ) may have. In one embodiment, when the second voice signal is output through the speaker (1080) and then received through the microphone (1060), it may be considered as one component (e.g., noise or echo) of the first voice signal.

[0122] For example, the electronic device (1001) uses a third neural network (1033) to generate a third sequence of one or more phonetic symbols for a specified text included in text information (1050), e.g., E t ∈R Tt X 128 ) can be obtained. The third neural network (1033) may be an example of a pre-trained neural network to isolate a specified text using a phonetic symbol corresponding to the specified text. In one embodiment, the third sequence (E t ) may be referred to as a third feature vector, or a third embedding. In one embodiment, the third sequence, the third feature vector, or the third embedding may be a specified dimension (e.g., R Tt X 128 ) can be a vector. Here, T t can correspond to the number of pronunciation symbols corresponding to the specified text. T t is T a may be the same as or different from. In one embodiment, the third sequence (E t ) can be referred to as a text sequence, in that it is a sequence converted from text.

[0123] For example, the electronic device (1001) may, from a fourth neural network (1034) into which the first sequence and the second sequence are input, generate a fourth sequence (e.g., E) for the first voice signal with reduced (or removed) echo (or second voice signal). a ∈R Ta X 128 ) can be obtained. In one embodiment, the fourth sequence (E a ) may be referred to as a fourth feature vector, or a fourth embedding. In one embodiment, the fourth sequence, the fourth feature vector, or the fourth embedding may be a fourth sequence, a fourth feature vector, or a fourth embedding, which may be a fourth sequence, a fourth feature vector, or a fourth embedding. Ta X 128) may be a vector of. In one embodiment, the first speech signal with the second speech signal reduced (or removed) may be referred to as a third speech signal. In one embodiment, the fourth neural network (1034) may be referred to as a mask sub-network in terms of masking the second speech signal from the first speech signal. In one embodiment, the fourth sequence (E a ) may be referred to as an echo cancellation sequence, in that it is a sequence of a third voice signal from which a second voice signal is removed from a first voice signal.

[0124] For example, the electronic device (1001) outputs a fifth sequence (e.g., E) representing the degree of correspondence (or relationship) between a third speech signal and a specified text from a fifth neural network (1035) into which a third sequence and a fourth sequence are input. j ∈R (Ta + Tt) X 128 ) can be obtained. The electronic device (1001) can obtain a fifth sequence (e.g., E) indicating the degree to which each of one or more pronunciation symbols included in the fourth sequence and each of the parts included in the third speech signal correspond to each other from the fifth neural network (1035) into which the third sequence and the fourth sequence are input. j ∈R (Ta + Tt) X 128 ) can be obtained. In one embodiment, the fifth sequence (E j ) may be referred to as a fifth feature vector, or a fifth embedding. In one embodiment, the fifth sequence, the fifth feature vector, or the fifth embedding may be a fifth sequence, a fifth feature vector, or a fifth embedding, which may be a fifth sequence, a fifth feature vector, or a fifth embedding. (Ta + Tt) X 128 ) can be a vector.

[0125] For example, the electronic device (1001) may obtain a first parameter indicating whether the fifth sequence is a third voice signal corresponding to a specified text from the sixth neural network (1036) into which the fifth sequence is input. The sixth neural network (1036) may be referred to as a discriminator and / or a discriminating model from the perspective of determining whether the sixth voice signal is a third voice signal corresponding to the specified text. According to an embodiment, the electronic device (1001) may obtain a first parameter indicating whether the first partial sequence of the fifth sequence is a third voice signal corresponding to the specified text from the sixth neural network (1036) into which the first partial sequence of the fifth sequence is input. Here, the first partial sequence may be a sequence corresponding to the third voice input (or the fourth sequence) among the fifth sequences. Here, the first partial sequence may indicate the degree to which each of the parts included in the third voice signal corresponds to each of one or more pronunciation symbols.

[0126] For example, the electronic device (1001) may obtain one or more second parameters for identifying (or determining) whether each of the portions of the third speech signal matches each of one or more pronunciation symbols from the seventh neural network (1037) into which the second partial sequence of the fifth sequence is input. The seventh neural network (1037) may be referred to as a discriminator and / or a discriminant model in terms of determining whether each of the portions of the speech signal matches each of one or more pronunciation symbols. Here, the second partial sequence may be a sequence corresponding to a specified text (or the third sequence) among the fifth sequences. Here, the second partial sequence may indicate the degree to which each of the one or more pronunciation symbols corresponds to each of the portions included in the third speech signal.

[0127] For example, the electronic device (1001) can train a first neural network (1031), a second neural network (1032), a third neural network (1033), a fourth neural network (1034), a fifth neural network (1035), a sixth neural network (1036), and / or a seventh neural network (1037) using a first parameter and one or more second parameters. The electronic device (1001) can tune at least a portion of the first neural network (1031), at least a portion of the second neural network (1032), at least a portion of the third neural network (1033), at least a portion of the fourth neural network (1034), a fifth neural network (1035), a sixth neural network (1036), and / or a seventh neural network (1037) using the first parameter and one or more second parameters. The electronic device (1001) may train a plurality of neural networks (1030) to identify whether a third speech signal (or a first speech signal from which the second speech signal has been reduced (or removed)) corresponds to a specified text using a first parameter and one or more second parameters. An operation of the electronic device (1001) training at least a portion of the plurality of neural networks (1030) is described below with reference to FIGS. 11A, 11B, and 12.

[0128] A call software application (1040) according to one embodiment may be an example of a software application for indicating to a user whether a voice signal corresponding to a specified text has been received. The call software application (1040) may be used to execute at least one function using a plurality of neural networks (1030). The electronic device (1001) may perform at least one operation based on the reception of a voice signal that calls a specified text referring to the electronic device (1001) using the call software application (1040). An example of an operation in which the electronic device (1001) executes at least one function based on the execution of the call software application (1040) is described below with reference to FIGS. 14 to 16.

[0129] For example, the text information (1050) may include a learning data set for training a plurality of neural networks (1030). The text information (1050) may include information about one or more phonetic symbols. A phonetic symbol may refer to a written symbol that represents a voice in a form that can be seen by the user. For example, the text information (1050) may include information about a call word for the user of the electronic device (1001) to use a function of the electronic device (1001) based on the execution of the call software application (1040). After identifying a voice signal corresponding to the call word, the electronic device (1001) may wait to receive another voice signal indicating the initiation of a function of the electronic device (1001). However, the present invention is not limited thereto. The electronic device (1001) may provide the user with a user interface (e.g., the user interface (1410) of FIG. 14) for changing the call word.

[0130] The electronic device (1001) according to one embodiment described above can identify a wake word acquired through a microphone in an environment where noise generated through a speaker exists. The electronic device (1001) according to one embodiment described above can identify a wake word that is not included in a training data set for training by using the plurality of trained neural networks (1030) after training a plurality of neural networks (1030). The electronic device (1001) can provide functions related to human-machine interaction (HMI) to a user by using the plurality of neural networks (1030) that can identify a wake word that is not included in the training data set.

[0131] FIG. 11A illustrates an example of an operation of an electronic device according to an embodiment of the present invention to identify a speech signal corresponding to a specified text through a neural network. FIG. 11B illustrates an example of an operation of an electronic device according to an embodiment of the present invention to identify a mask for removing acoustic echo from a speech signal through a neural network. FIGS. 11A and 11B may be described with reference to the electronic device (1001) of FIG. 10 and / or components of the electronic device (1001) of FIG. 10.

[0132] Referring to FIG. 11A, an electronic device (1001) according to an embodiment may receive a first voice signal (1110) from a user using a microphone (1060). In an embodiment, the first voice signal (1110) may include a user's speech and one or more noises. In an embodiment, the one or more noises may include an echo corresponding to an audio signal (or a second voice signal) output through a speaker (1080).

[0133] For example, the electronic device (1001) can phonemicize the first speech signal (1110) using the first neural network (1031). The electronic device (1001) can segment the first speech signal (1110) based on a specified period. The electronic device (1001) can obtain portions (1110-1, 1110-2, 1110-3, 1110-4) of the first speech signal (1110) based on a specified frame unit. For example, the specified frame unit can be identified based on phonemic symbols corresponding to the first speech signal (1110). For example, the specified frame unit can be changed based on the length of a specified text. However, the present invention is not limited thereto.

[0134] For example, the electronic device (1001) can change the first voice signal (1110) into first voice data (1111) based on frequency. The electronic device (1001) can change the first voice signal (1110) into second voice data (1112) based on time. The electronic device (1001) inputs the first voice data (1111) and the second voice data (1112) into the first neural network (1031) to generate a first sequence (1120) (or, E m ) can be obtained. The order and / or number of the first sequence (1120) may vary depending on the embodiment, based on the portions (1110-1, 1110-2, 1110-3, 1110-4) of the first speech signal (1110) distinguished based on a specified frame unit. The first sequence (1120) may be R Ta X 128 An example of a dimension-based matrix may be a first sequence (1120). The first sequence (1120) may include feature information based on multiple dimensions. The first sequence (1120) may include vector parameters embedded from the first speech signal (1110). The first sequence (1120) may include R Ta X 128The first feature vector of the dimension may be referred to as the first embedding.

[0135] For example, the electronic device (1001) may process the first voice data (1111) using the first operator (1031-1) and / or the second operator (1031-2) included in the first neural network (1031). The first operator (1031-1) may be an example of an operator used to identify changes in the first voice data (1111) based on frequency over time. The first operator (1031-1) may include a short time Fourier transform (STFT). The second operator (1031-2) may include a one-dimensional based convolution operator (e.g., Conv1D). The second operator (1031-2) may include an activation function. The activation function may include a non-linear function such as the ReLu (rectified linear unit) function or the sigmoid operation. The number of second operators (1031-2) may be one or more.

[0136] For example, the electronic device (1001) may process the second voice data (1112) using the third operator (1031-3) and / or the fourth operator (1031-4) included in the first neural network (1031). The third operator (1031-3) may include a pre-trained embedder to vectorize the second voice data (1112). The fourth operator (1031-4) may include a time-based one-dimensional convolution operator (e.g., TConv1D). The fourth operator (1031-4) may include an activation function. The fourth operator (1031-4) may be used to identify one-dimensional sequence data, but is not limited thereto.

[0137] For example, the electronic device (1001) can obtain first sequence data by processing first voice data (1111) using a first operator (1031-1) and a second operator (1031-2). The electronic device (1001) can obtain second sequence data by processing second voice data (1112) using a third operator (1031-3) and a fourth operator (1031-4). The electronic device (1001) can obtain a first sequence (1120) by using the first sequence data and the second sequence data. The electronic device (1001) can obtain the first sequence (1120) based on calculating the first sequence data and the second sequence data using a designated operator.

[0138] According to an embodiment, an electronic device (1001) may identify a second voice signal (1113) output using a speaker (1080). For example, the electronic device (1001) may phonetize the second voice signal (1113) using a second neural network (1032). The electronic device (1001) may segment the second voice signal (1113) based on a specified period. The electronic device (1001) may obtain portions (1113-1, 1113-2, 1113-3, 1113-4) of the second voice signal (1113) based on a specified frame unit. For example, the specified frame unit may be identified based on phoneme symbols corresponding to the second voice signal (1113). For example, the specified frame unit may be changed based on the length of a specified text. However, the present invention is not limited thereto. For example, when the second voice signal (1113) is output through the speaker (1080), it may be acquired as noise (or echo) while the electronic device (1001) receives the sound signal using the microphone (1060).

[0139] For example, the electronic device (1001) can change the second voice signal (1113) into third voice data (1114) based on frequency. The electronic device (1001) can change the second voice signal (1113) into fourth voice data (1115) based on time. The electronic device (1001) inputs the third voice data (1114) and the fourth voice data (1115) into the second neural network (1032) to generate a second sequence (1130) (or, E p ) can be obtained. The order and / or number of the second sequence (1130) may vary depending on the embodiment, based on the portions (1113-1, 1113-2, 1113-3, 1113-4) of the second voice signal (1113) distinguished based on the specified frame unit. The second sequence (1130) may be R Ta X 128 It may be an example of a matrix based on a dimension. The second sequence (1130) may include feature information based on multiple dimensions. The second sequence (1130) may include vector parameters embedded from the second speech signal (1113). The second sequence (1130) may include R Ta X 128 The second feature vector of the dimension may be referred to as the second embedding. In one embodiment, portions (1113-1, 1113-2, 1113-3, 1113-4) of the second speech signal (1113) may correspond to portions (1110-1, 1110-2, 1110-3, 1110-4) of the first speech signal (1110), respectively. For example, the timing at which each of the portions (1113-1, 1113-2, 1113-3, 1113-4) is output through the speaker (1080) may be the same as the timing at which each of the portions (1110-1, 1110-2, 1110-3, 1110-4) is output through the microphone (1060). However, the present invention is not limited thereto.

[0140] For example, the electronic device (1001) may process the third voice data (1114) using the fifth operator (1032-1) and / or the sixth operator (1032-2) included in the second neural network (1032). The fifth operator (1032-1) may be an example of an operator used to identify changes in the third voice data (1114) based on frequency over time. The fifth operator (1032-1) may include STFT. The sixth operator (1032-2) may include a one-dimensional based convolution operator (e.g., Conv1D). The sixth operator (1032-2) may include an activation function. The activation function may include a non-linear function such as a ReLu function or a sigmoid operation. The number of sixth operators (1032-2) may be one or more.

[0141] For example, the electronic device (1001) may process the fourth speech data (1115) using the seventh operator (1032-3) and / or the eighth operator (1032-4) included in the second neural network (1032). The seventh operator (1032-3) may include a pre-trained embedding to vectorize the fourth speech data (1115). The eighth operator (1032-4) may include a time-based one-dimensional convolution operator (e.g., TConv1D). The eighth operator (1032-4) may include an activation function. The eighth operator (1032-4) may be used to identify one-dimensional sequence data, but is not limited thereto.

[0142] For example, the electronic device (1001) can obtain third sequence data by processing third voice data (1114) using the fifth operator (1032-1) and the sixth operator (1032-2). The electronic device (1001) can obtain fourth sequence data by processing fourth voice data (1115) using the seventh operator (1032-3) and the eighth operator (1032-4). The electronic device (1001) can obtain the second sequence (1130) by using the third sequence data and the fourth sequence data. The electronic device (1001) can obtain the second sequence (1130) based on calculating the third sequence data and the fourth sequence data through a designated operator.

[0143] In FIG. 11A, the functions and / or structures of the first neural network (1031) and the functions and / or structures of the second neural network (1032) may correspond to each other. For example, the functions and / or connection relationships of each of the operators (1031-1, 1031-2, 1031-3, 1031-4) included in the first neural network (1031) may correspond to the functions and / or connection relationships of each of the operators (1032-1, 1032-2, 1032-3, 1032-4) included in the second neural network (1032). The correspondence of the functions and / or connection relationships does not mean that the parameter values ​​correspond to each other. For example, the parameter values ​​of each of the operators (1031-1, 1031-2, 1031-3, 1031-4) may not be related to the parameter values ​​of each of the operators (1032-1, 1032-2, 1032-3, 1032-4) included in the second neural network (1032). However, this is not limited thereto. The first neural network (1031) may have a function and / or structure different from the function and / or structure of the second neural network (1032).

[0144] According to one embodiment, the electronic device (1001) outputs a third sequence (1140) (or, E) from a third neural network (1033) into which a specified text (1116) included in text information (e.g., text information (1050) of FIG. 1) is input. t ) can be obtained. The third sequence (1140) is R Tt X 128 This could be an example of a dimension-based matrix.

[0145] For example, the electronic device (1001) can obtain the third sequence (1140) using one or more phonetic symbols corresponding to the specified text (1116). The electronic device (1001) can isolate the specified text (1116) based on the one or more phonetic symbols. For example, if the specified text (1116) is 'Hello', the one or more phonetic symbols corresponding to the specified text (1116) can include 'HH', 'EH1', 'L', and 'OW0', but is not limited thereto.

[0146] For example, the electronic device (1001) can change the specified text (1116) into the third sequence (1140) using the ninth operator (1033-1) and / or the tenth operator (1033-2) included in the third neural network (1033). The ninth operator (1033-1) can include a pre-trained embedding to vectorize (or data-ize) the specified text (1116). The data of the specified text (1116) processed through the ninth operator (1033-1) can be an example of vector data based on multiple dimensions (e.g., two dimensions). The tenth operator (1033-2) can include fully connected layers. The tenth operator (1033-2) can include an activation function. However, the present invention is not limited thereto.

[0147] An electronic device (1001) according to one embodiment inputs a first sequence (1120) and a second sequence (1130) into a fourth neural network (1034), thereby generating a fourth sequence (1160) (or, E a ) can be obtained. For example, the electronic device (1001) can obtain a fourth sequence (1160) for a third audio signal in which the second audio signal (1113) is reduced (or removed) from the first audio signal (1110) from the fourth neural network (1034) into which the first sequence (1120) and the second sequence (1130) are input. The fourth sequence (1160) is R Ta X 128 This could be an example of a dimension-based matrix.

[0148] For example, the electronic device (1001) can process the first sequence (1120) and the second sequence (1130) using the eleventh operator (1034-1) included in the fourth neural network (1034). For example, the electronic device (1001) can obtain a mask (1150) from the eleventh operator (1034-1) into which the first sequence (1120) and the second sequence (1130) are input.

[0149] Referring to FIG. 11B, the eleventh operator (1034-1) may include an operator (e.g., CONCAT) for connecting the first sequence (1120) and the second sequence (1130). In one embodiment, the operator (e.g., CONCAT) may receive the first sequence (1120) and the second sequence (1130) as different input channels. However, the present invention is not limited thereto. In one embodiment, the operator (e.g., CONCAT) may connect (or concatenate) the first sequence (1120) and the second sequence (1130) along an embedding axis. For example, a sequence in which the first sequence (1120) and the second sequence (1130) are concatenated may be understood as having one channel.

[0150] For example, the eleventh operator (1034-1) may include a two-dimensional based convolution operator (e.g., Conv2D). The eleventh operator (1034-1) may include an activation function. The activation function may include a non-linear function such as a ReLU function or a sigmoid operation. The eleventh operator (1034-1) may have a structure in which a plurality of two-dimensional based convolution operators (e.g., Conv2D) and activation functions are connected. For example, the eleventh operator (1034-1) may have a structure in which, after CONCAT, Conv2D, ReLU, Conv2D, ReLU, Conv2D, and Sigmoid are connected sequentially. In one embodiment, a first convolution operator among the two-dimensional based convolution operators (e.g., Conv2D) may include filters (e.g., a 3 X 3 filter) (i.e., 8 filters) for expanding (or increasing) the output from the operator (e.g., CONCAT) to 8 channels. In one embodiment, a second convolution operator among the two-dimensional based convolution operators (e.g., Conv2D) may include filters (e.g., a 3 X 3 filter) (i.e., 8 filters) for maintaining the output of the first convolution operator and the activation function to 8 channels. In one embodiment, the third convolution operator among the two-dimensional based convolution operators (e.g., Conv2D) may include a filter (e.g., a 3 X 3 filter) (i.e., one filter) for reducing (or reducing) the output of the second convolution operator and the activation function to one channel.

[0151] The structure of the eleventh operator (1034-1) is not limited to that illustrated in FIG. 11B. For example, the eleventh operator (1034-1) may have a structure including an operator (e.g., CONCAT) for connecting the first sequence (1120) and the second sequence (1130) along an embedding axis, a fully connected layer (e.g., a dense layer), and an activation function.

[0152] Referring back to FIG. 11A, the electronic device (1001) can obtain the fourth sequence (1160) using the first sequence (1120) and the mask (1150). For example, the electronic device (1001) can obtain the fourth sequence (1160) by performing a multiplication operation (e.g., Hadamard product or element-wise product) on the first sequence (1120) and the mask (1150).

[0153] According to an embodiment, an electronic device (1001) may input a fourth sequence (1160) and a third sequence (1140) into a fifth neural network (1035) to obtain a fifth sequence (1170). For example, the electronic device (1001) may obtain a fifth sequence (1170) (or, E) representing a relationship between the fourth sequence (1160) and the third sequence (1140) from the fifth neural network (1035) into which the fourth sequence (1160) and the third sequence (1140) are input. j ) (or one or more data sets) can be obtained.

[0154] For example, the electronic device (1001) can obtain a concatenated sequence by concatenating (e.g., CONCAT) the fourth sequence (1160) and the third sequence (1140). For example, the electronic device (1001) can obtain a fifth sequence (1170) by performing a scaled-dot product operation on the concatenated sequence. For example, the electronic device (1001) can obtain the fifth sequence (1170) (e.g., E) by calculating self-attention of the concatenated sequence. j ∈R (Ta + Tt) X 128)) can be obtained. In one embodiment, the electronic device (1001) can obtain the fifth sequence (1170) by using a lower triangular matrix as an attention mask in order to maintain (or protect) causal information of each of the fourth sequence (1160) and the third sequence (1140). In one embodiment, using a lower triangular matrix as an attention mask can indicate that a part corresponding to the lower triangular matrix in a matrix representing a multiplication between a query (e.g., a concatenated sequence) and a key (e.g., a concatenated sequence) is replaced with a specific value (e.g., 0). However, the present invention is not limited thereto. The electronic device (1001) can obtain the fifth sequence (1170) by using an upper triangular matrix as an attention mask.

[0155] In one embodiment, the fifth sequence (1170) may include information indicating a connection (or relationship) between each of the first sequence data sets included in the fourth sequence (1160) and each of the second sequence data sets included in the third sequence (1140). The fifth sequence (1170) may include pattern information between the fourth sequence (1160) and the third sequence (1140). From the perspective of obtaining pattern information between the fourth sequence (1160) and the third sequence (1140), the fifth neural network (1035) may be referred to as a pattern extractor.

[0156] According to an embodiment, the fifth sequence (1170) may include a first subsequence (1170-1) (or a first data set) that may represent information indicating a correlation of the third sequence (1140) with the fourth sequence (1160). For example, the first subsequence (1170-1) may include information indicating a degree to which each of one or more pronunciation symbols corresponds to each of the portions (1110-1, 1110-2, 1110-3, 1110-4) of the speech signal (1110). As an example, the first subsequence (1170-1) may include information indicating a degree to which the entire specified text (1116) corresponds to the entire speech signal (1110).

[0157] For example, the fifth sequence (1170) may indicate the degree to which the second subsequence (1170-2) (or the second data set) of the fourth sequence (1160) relates to the third sequence (1140). The second subsequence (1170-1) may include information indicating the degree to which one or more pronunciation symbols and portions (1110-1, 1110-2, 1110-3, 1110-4) of the speech signal (1110) relate. For example, the second subsequence (1170-2) may include information indicating the degree to which each of the portions (1110-1, 1110-2, 1110-3, 1110-4) of the speech signal (1110) corresponds to each of the one or more pronunciation symbols. However, the present invention is not limited thereto.

[0158] According to an embodiment, the electronic device (1001) may obtain a first parameter (1180) from a sixth neural network (1036) into which a fifth sequence (1170) (or a part of the fifth sequence (1170) (e.g., a first subsequence (1170-1)) is input. The first parameter (1180) may indicate the degree to which a voice signal (1110) and a specified text (1116) correspond. The first parameter (1180) may indicate whether a third voice signal (or a first voice signal (1110) from which a second voice signal (1113) is reduced (or removed)) matches the specified text (1116). The sixth neural network (1036) may include a twelfth operator (1036-1). The twelfth operator (1036-1) may further include one or more operators. The operator (1036-1) may include a deep learning framework (e.g., a long short term memory (LSTM) or a single 128-dimensional gated recurrent unit (GRU)), a fully connected layer, and / or an activation function. The electronic device (1001) may use the first parameter (1180) to identify whether the third speech signal matches the specified text (1116).

[0159] According to an embodiment, an electronic device (1001) may obtain one or more second parameters (1190) from a seventh neural network (1037) into which at least a portion of a fifth sequence (1170) (e.g., a second subsequence (1170-2)) is input. The seventh neural network (1037) may include a thirteenth operator (1037-1). The thirteenth operator (1037-1) may include a fully connected layer and / or an activation function (e.g., a sigmoid function). The number of one or more second parameters (1190) may correspond to the number of one or more phonetic symbols corresponding to a specified text (1116). One or more second parameters (1190) may indicate the extent to which portions of the third speech signal (e.g., portions (1110-1, 1110-2, 1110-3, 1110-4) of the first speech signal (1110) from which the second speech signal (1113) is reduced (or removed)) correspond to each of one or more pronunciation symbols. Referring to FIG. 11A, the first sub-parameter (1190-1) may indicate that a first portion of the third speech signal (or a portion (1110-1) from which the second speech signal (1113) is reduced (or removed)) matches (e.g., true) the first pronunciation symbol (e.g., 'HH'). The second sub-parameter (1190-2) may indicate that the second part of the third speech signal (or the reduced (or removed) part (1110-2) of the second speech signal (1113)) does not match (e.g., false) with the second pronunciation symbol (e.g., 'EH1'). The third sub-parameter (1190-3) may indicate that the o'clock part of the third speech signal (or the reduced (or removed) part (1110-3) of the second speech signal (1113)) matches with the first pronunciation symbol (e.g., 'L'). The fourth sub-parameter (1190-4) may indicate that the fourth part of the third speech signal (or the reduced (or removed) part (1110-4) of the second speech signal (1113)) matches with the fourth pronunciation symbol (e.g., 'OW0').However, the present invention is not limited thereto. Parameters (e.g., the first parameter (1180) and / or one or more second parameters (1190)) may be identified based on dummy variables. A dummy variable may be a variable that can indicate whether a specified feature exists (e.g., whether the third speech signal and the specified text (1116) match) using '0' or '1'. A dummy variable may be referred to as a binary indicator (or a boolean indicator), a binary variable, an indicator variable, and / or a design variable. For example, when the first sub-parameter (1190-1) is '1', it may indicate that the first part of the third speech signal matches the first pronunciation symbol (e.g., 'HH') (e.g., true). If the first sub-parameter (1190-1) is '0', it may indicate that the first part of the third speech signal does not match the first pronunciation symbol (e.g., 'HH').

[0160] Hereinafter, with reference to FIG. 12, an example of an operation of an electronic device (1001) according to one embodiment to train a plurality of neural networks (1030) using a first parameter (1180) and / or one or more second parameters (1190) is described.

[0161] FIG. 12 illustrates an example of an operation of an electronic device according to an embodiment of the present invention to train a neural network to identify a speech signal corresponding to a specified text. The electronic device (1001) of FIG. 12 may include the electronic device (1001) of FIG. 10. The plurality of neural networks (1030) of FIG. 12 may include the first neural network (1031), the second neural network (1032), the third neural network (1033), the fourth neural network (1034), the fifth neural network (1035), the sixth neural network (1036), and / or the seventh neural network (1037) of FIG. 11A. In an embodiment, the first neural network (1031) and the second neural network (1032) may be referred to as a shared audio encoder (1220). For example, the third neural network (1033) may be referred to as a text encoder. In one embodiment, the fourth neural network (1034), which includes the eleventh operator (1034-1) and a multiplication operation (1230) (e.g., Hadamard product or element-wise product), may be referred to as a refiner. For example, the fifth neural network (1035) may be referred to as a pattern extractor. For example, the sixth neural network (1036) and the seventh neural network (1037) may be referred to as pattern discriminators.

[0162] According to one embodiment, an electronic device (1001) may receive a first voice signal (1210) (e.g., hello + NCYA) using a microphone in a state (1200). The first voice signal (1210) may include a call word (e.g., NCYA) for calling the electronic device (1001) and an echo (e.g., hello). In one embodiment, the echo (e.g., hello) may be an audio signal that is output using a speaker and then acquired using a microphone.

[0163] The electronic device (1001) can identify a second voice signal (1213) (e.g., “hello”) output using a speaker. In one embodiment, the second voice signal (1213) (e.g., “hello”) may be an audio signal acquired as an echo through a microphone when output using the speaker.

[0164] The electronic device (1001) can identify a designated text (1216) (e.g., NCYA) included in the text information. The designated text (1216) can correspond to a call word for calling the electronic device (1001).

[0165] For example, the electronic device (1001) can obtain prediction values ​​(1240) from a plurality of neural networks (1030) into which a first voice signal (1210), a second voice signal (1213), and a designated text (1216) are input. The prediction values ​​(1240) can indicate the degree to which the first voice signal (1210) (or the third voice signal) with the second voice signal (1213) reduced (or removed) corresponds to the designated text (1216). The prediction values ​​(1240) can indicate the degree to which each of the portions of the third voice signal (or the portions of the first voice signal (1210) with the second voice signal (1213) reduced (or removed)) corresponds to each of one or more phonetic symbols corresponding to the designated text (1216). The prediction values ​​(1240) can have at least one value between 0 and 1. The predicted values ​​(1240) may correspond to the first parameter (1180) and / or one or more second parameters (1190) of FIG. 11a.

[0166] An electronic device (1001) according to an embodiment may set labeling values ​​(1250) for training a plurality of neural networks (1030) based on supervised learning. In a state (1200), since a text corresponding to a third voice signal matches a designated text (1216), the electronic device (1001) may set the labeling values ​​(1250) to a value (e.g., 1) indicating that the third voice signal and the designated text (1216) match.

[0167] For example, the electronic device (1001) can train the plurality of neural networks (1030) so that the predicted values ​​(1240) and the labeling values ​​(1250) match. For example, the electronic device (1001) can train the plurality of neural networks (1030) so that the predicted values ​​(1240) converge to the labeling values ​​(1250). For example, the electronic device (1001) can train the plurality of neural networks (1030) so that the first predicted value (1240-1) (e.g., 0.9) corresponding to the first parameter (e.g., the first parameter (1180) of FIG. 11A) matches the first labeling value (1250-1) (e.g., 1). For example, the electronic device (1001) may train the plurality of neural networks (1030) based on supervised learning such that second prediction values ​​(1240-2) (e.g., 0.7, 0.2) corresponding to one or more second parameters (e.g., one or more second parameters (1190) of FIG. 11A) are matched to second labeling values ​​(1250-2) (e.g., 1, 0). As an example, the electronic device (1001) may obtain parameters for training the plurality of neural networks (1030) based on binary cross entropy error (BCEE).

[0168] For example, the electronic device (1001) can train multiple neural networks (1030) using a loss value (1270) (or a loss parameter representing an error rate) representing the difference (1260) between the predicted values ​​(1240) and the labeling values ​​(1250). The loss value (1270) can be used to identify a similarity in speech level between a third speech signal and a specified text (1216).

[0169] For example, referring to FIG. 11A, at least a portion of the first neural network (1031) (e.g., the second operator (1031-2)), at least a portion of the second neural network (1032) (e.g., the sixth operator (1032-2)), at least a portion of the third neural network (1033) (e.g., the tenth operator (1033-2)), at least a portion of the fourth neural network (1034) (e.g., the eleventh operator (1034-2)), the fifth neural network (1035), the sixth neural network (1036), and / or the seventh neural network (1037) can be trained based on the loss value (1270).

[0170] Below, an example of an electronic device (1001) that infers a call word using multiple learned neural networks (1030) is described with reference to FIG. 13.

[0171] FIG. 13 illustrates an example of an operation of an electronic device according to an embodiment of the present invention to infer a voice signal corresponding to a specified text. The electronic device (1001) of FIG. 13 may include the electronic device (1001) of FIGS. 10 to 12 . Referring to FIG. 13 , a plurality of neural networks (1030) according to an embodiment of the present invention may be examples of neural networks trained to identify whether a voice signal received through a microphone (1060) matches a specified text.

[0172] An electronic device (1001) according to one embodiment may receive a first voice signal (1310) using a microphone (1060). The first voice signal (1310) may correspond to information representing a word, sentence, and / or utterance.

[0173] An electronic device (1001) according to one embodiment can obtain a second audio signal (1313) output using a speaker. The second audio signal (1313) can correspond to an audio signal output using a speaker (1080) while receiving a first audio signal (1310).

[0174] According to an embodiment, an electronic device (1001) may reduce (or remove) a second voice signal (1313) from the first voice signal (1310) based on the second voice signal (1313) being output through a speaker (1080) while receiving a first voice signal (1310) using a microphone (1060). According to an embodiment, an electronic device (1001) may reduce (or remove) a second voice signal (1313) from the first voice signal (1310) using a plurality of neural networks (1030).

[0175] For example, based on obtaining a first voice signal (1310) (or a third voice signal) from which a second voice signal (1313) is reduced (or removed), the electronic device (1001) can infer (or identify) whether at least a portion of the third voice signal corresponds to a designated text (1316) using a plurality of neural networks (1030). For example, the electronic device (1001) can set the designated text (1316). The electronic device (1001) can receive an input from the user to change the designated text (1316) using a user interface for setting the designated text (1316). Based on receiving the input, the electronic device (1001) can set the designated text (1316). The electronic device (1001) can, at least temporarily, refrain from training the plurality of neural networks (1030) to identify a speech signal corresponding to the specified text (1316) by changing the specified text (1316). Since the electronic device (1001) has trained the plurality of neural networks (1030) using a speech-level data set, the electronic device (1001) can, independently of performing training of the plurality of neural networks (1030) corresponding to the changed specified text (1316), infer whether the third speech signal corresponds to the specified text (1316).

[0176] For example, the electronic device (1001) can input a specified text (1316) and a third voice signal into a plurality of neural networks (1030) to obtain a first parameter (1320) and one or more second parameters (1330). The electronic device (1001) can obtain portions of the third voice signal based on a specified frame unit. The electronic device (1001) can identify one or more phonetic symbols for the specified text (1316). The electronic device (1001) can obtain a first parameter (1320) indicating whether the entire third voice signal corresponds to the entire specified text (1316). The electronic device (1001) can obtain one or more second parameters (1330) indicating whether each of the portions of the third voice signal corresponds to each of one or more phonetic symbols.

[0177] For example, the first parameter (1320) and one or more second parameters (1330) can be obtained based on a binary indicator (e.g., a value having 0 or 1). The electronic device (1001) can infer the value of the first parameter (1320) based on a specified probability distribution for identifying the first parameter (1320). The electronic device (1001) can obtain the first parameter (1320) having a value closer to 1 as the degree to which the entire third speech signal corresponds to the entire specified text (1316) is relatively high.

[0178] For example, the electronic device (1001) can infer the value of one or more second parameters (1330) based on a specified probability distribution for identifying one or more second parameters (1330). The electronic device (1001) can obtain one or more second parameters (1330) having a value relatively closer to 1 as the degree to which each of the portions of the third voice signal corresponds to each of the one or more phonetic symbols is relatively high. At least one of the one or more second parameters (1330) can match a value of 0. For example, if at least one of the portions of the third voice signal does not match at least one of the one or more phonetic symbols, the electronic device (1001) can obtain at least one of the one or more second parameters (1330) having a value relatively close to 0. However, the present invention is not limited thereto.

[0179] As described above, the electronic device (1001) according to one embodiment can infer whether it has identified a voice signal corresponding to at least some randomly set call words by using a plurality of learned neural networks (1030).

[0180] Referring to FIG. 15 below, an example of a user interface for setting a call word according to one embodiment is described below.

[0181] FIG. 14 illustrates an example of a user interface for setting a designated text in an electronic device according to an embodiment. The electronic device (1001) of FIG. 14 may include the electronic devices (1001) of FIGS. 10 to 14. Referring to FIG. 14, in a state (1400), the electronic device (1001) according to an embodiment may display a user interface (1410) for setting a call word for calling the electronic device (1001) on the display (1070) based on the execution of the call software application (1040) of FIG. 10. For example, the user interface (1410) may include a visual object (1420) for obtaining a call word for initiating execution of the call software application (1040) from a user. The electronic device (1001) may receive a text object (e.g., NCYA) representing the call word from the user using the visual object (1420). For example, the electronic device (1001) may set the text object as a wake word based on receiving an input indicating selection of the icon (1430). The electronic device (1001) may initiate execution of the wake software application (1040) based on identifying a voice signal corresponding to the wake word through a plurality of neural networks (1030) using a microphone. The electronic device (1001) may perform at least one function based on the execution of the wake software application (1040). The electronic device (1001) may receive another voice signal indicating the name of the at least one function based on the execution of the wake software application (1040), and may initiate execution of the at least one function. However, the present invention is not limited thereto.

[0182] FIG. 15 illustrates an example of a user interface for indicating whether a voice signal corresponds to a specified text, according to an embodiment. The electronic device (1001) of FIG. 15 may include the electronic device (1001) of FIGS. 10 to 14 . The electronic device (1001) may, based on the execution of the calling software application (1040), provide a user with an indication of whether a voice signal corresponding to a specified text (1520) has been received at a state (1500).

[0183] According to an embodiment, an electronic device (1001) may display a user interface (1510) on a display to indicate whether a voice signal corresponding to a specified text (1520) has been received. For example, the electronic device (1001) may guide a user using a visual object (1550) as to whether a voice signal received using a microphone corresponds to the specified text (1520). For example, the electronic device (1001) may guide a user using visual objects (1560) as to whether each portion of the voice signal corresponds to each of one or more phonetic symbols (1530). For example, the visual object (1550) may correspond to a first parameter (e.g., the first parameter (1320) of FIG. 13). For example, when the first parameter is 1, the electronic device (1001) may display a visual object (1550) indicating that the voice signal and the specified text are matched. For example, the visual objects (1560) may correspond to one or more second parameters (e.g., one or more second parameters (1330) of FIG. 13). As an example, the electronic device (1001) may display a visual object indicating that at least one of the portions of the speech signal corresponding to the at least one second parameter and at least one of the phonetic symbols (1530) corresponding to the at least one second parameter do not match, if at least one of the second parameters is 0. However, the present invention is not limited thereto.

[0184] An electronic device (1001) according to one embodiment may receive an input indicating the selection of a visual object (1540). Based on the input received, the electronic device (1001) may provide a notification requesting the user to speak using a microphone, although this is not limited thereto. The specified text (1520) may be included in the text information (1050) of FIG. 10 .

[0185] As described above, the electronic device (1001) according to one embodiment may display a user interface (1510) on the display for correcting the user's pronunciation. The electronic device (1001) may guide the user on the accuracy of utterances in a second language received from the user based on a first language through a plurality of neural networks (1030).

[0186] FIG. 16 illustrates an example of a state in which an electronic device according to an embodiment receives a voice signal for performing a specified function. The electronic device (1001) of FIG. 16 may include the electronic device (1001) of FIGS. 10 to 15 . For example, the electronic device (1001) may display a user interface (1610) representing a game on the display in state (1600).

[0187] An electronic device (1001) according to an embodiment can identify text information included in a user interface (1610). For example, the electronic device (1001) can identify a first nickname (1613) of a first character (1615) representing a user of the electronic device (1001) (e.g., User A) and a second nickname (1633) of a second character (1635) representing another user (e.g., User B). For example, the electronic device (1001) can receive a voice signal (1605) using a microphone. The electronic device (1001) can obtain a voice signal from which noise has been removed from a voice signal (1605) including noise output through a speaker through a plurality of neural networks (1030) of FIG. 10, and identify text corresponding to the voice signal from which noise has been removed. The electronic device (1001) can transmit data representing a voice signal (1605) to an external electronic device corresponding to the second character (1635) if the text matches the second nickname (1633).

[0188] For example, the electronic device (1001) can control the first character (1615) to identify text representing an available skill (1620) within the user interface (1610). If a noise-removed voice signal output through a speaker from a voice signal (1605) received using a microphone matches the text representing the skill (1620), the electronic device (1001) can control the first character (1615) to use the skill (1620). However, the present invention is not limited thereto. Based on receiving a voice signal for controlling the first character (1615) within the user interface (1610), the electronic device (1001) can perform a function corresponding to the voice signal using the first character (1615).

[0189] For example, the electronic device (1001) may receive, through a microphone, a voice signal (1605) representing a sentence including noise output through a speaker. The electronic device (1001) may obtain a sentence from which noise has been removed from the voice signal (1605) representing the sentence including noise through a plurality of neural networks (1030), and identify whether the sentence from which noise has been removed corresponds to a specified text. For example, the electronic device (1001) may control the first character (1615) to use the skill (1620) on the second character (1635) based on identifying the voice signal (1605) representing the sentence from which noise has been removed (e.g., attacking the second character (1635) using the skill (1620). However, the present invention is not limited thereto.

[0190] As described above, the electronic device (1001) according to one embodiment may acquire a voice signal (1605) containing noise output through a speaker using a microphone while displaying a user interface (1610) for providing a game service. The voice signal (1605) containing noise output through the speaker may correspond to a command for controlling a first character (1615) corresponding to a user of the electronic device (1001) within the user interface (1610). Based on acquiring the voice signal (1605), the electronic device (1001) may identify a designated text matching the voice signal from which noise has been removed from the voice signal (1605), and control the first character (1615) to perform a function corresponding to the designated text. The electronic device (1001) may provide a service capable of controlling the first character (1615) based on the user's voice signal. The electronic device (1001) can provide a service that can control the first character (1615) based on hands-free.

[0191] FIG. 17 illustrates an example of a flowchart illustrating the operation of an electronic device according to an embodiment. The electronic device of FIG. 17 may include the electronic device (1001) of FIG. 10. At least one of the operations of FIG. 17 may be performed by the electronic device (1001) of FIG. 10. At least one of the operations of FIG. 17 may be controlled by the processor (1010) of FIG. 10. Each of the operations of FIG. 17 may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each of the operations may be changed, and at least two operations may be performed in parallel.

[0192] FIG. 17 may be described with reference to hardware components of the electronic device (1001) of FIGS. 10 to 16.

[0193] Referring to FIG. 17, in operation 1710, a processor (1010) according to an embodiment may obtain an echo removal sequence (1160) of a third voice signal in which a second voice signal (1113) output through a speaker (1080) is removed from a first voice signal (1110) received through a microphone (1060). The operation of obtaining the echo removal sequence (1160) may be described in detail below with reference to FIG. 18.

[0194] Referring to FIG. 17, in operation 1720, a processor (1010) according to an embodiment may obtain a text sequence (1140) for a specified text (1116). The processor (1010) may obtain a text sequence (1140) (or a second sequence (1140)) of one or more phonetic symbols for the specified text (1116) from a third neural network (1033) into which the specified text (1116) is input. The processor (1010) may obtain the text sequence (1140) (or the second sequence (1140)) based on the number of one or more phonetic symbols.

[0195] Referring to FIG. 17, in operation 1730, a processor (1010) according to an embodiment may obtain one or more data sets (1170) indicating a relationship between an echo removal sequence (1160) and a text sequence (1140). For example, the processor (1010) may obtain one or more data sets (1170) indicating a degree of correspondence (or relationship) between a third speech signal and a specified text (1116) from a fifth neural network (1035) into which an echo removal sequence (1160) and a text sequence (1140) are input. A first data set (1170-1) among the one or more data sets (1170) may indicate a degree of correspondence between one or more phonetic symbols for each of portions of a first speech signal (1110) from which a second speech signal (1113) has been removed. A second data set (1170-2) among one or more data sets (1170) may indicate the degree to which each of the portions of the first speech signal (1110) from which the second speech signal (1113) has been removed corresponds to each of one or more pronunciation symbols.

[0196] For example, the processor (1010) may input a first data set (1170-1) among one or more data sets (1170) into a sixth neural network (1036) to obtain a first parameter (e.g., the first parameter (1180) of FIG. 11A) indicating whether the third voice signal corresponds to a specified text (1116).

[0197] For example, the processor (1010) may input all of one or more data sets (1170) into the sixth neural network (1036) to obtain a first parameter (e.g., the first parameter (1180) of FIG. 11A) indicating whether the third speech signal corresponds to the specified text (1116). The processor (1010) may input all of one or more data sets (1170) other than the first data set (1170-1) into the sixth neural network (1036) to improve the accuracy of the first parameter (1180). However, the present invention is not limited thereto.

[0198] For example, the processor (1010) may identify a second data set (1170-2) among one or more data sets (1170). The processor (1010) may input the second data set (1170-2) into the seventh neural network (1037) to obtain one or more second parameters (e.g., one or more second parameters (1190) of FIG. 11A) having a number equal to the number of one or more phonetic symbols. The one or more second parameters (1190) may indicate whether the portions of the first speech signal (1110) from which the second speech signal (1113) is removed correspond to each of the one or more phonetic symbols.

[0199] FIG. 18 illustrates an example of a flowchart illustrating the operation of an electronic device according to an embodiment. The electronic device of FIG. 18 may include the electronic device (1001) of FIG. 10. At least one of the operations of FIG. 18 may be performed by the electronic device (1001) of FIG. 10. At least one of the operations of FIG. 18 may be controlled by the processor (1010) of FIG. 10. Each of the operations of FIG. 18 may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each of the operations may be changed, and at least two operations may be performed in parallel.

[0200] FIG. 18 may be described with reference to hardware components of the electronic device (1001) of FIGS. 10 to 17.

[0201] Referring to FIG. 18, in operation 1810, a processor (1010) according to an embodiment may obtain a first sequence (1120) from a first speech signal (1110). For example, the processor (1010) may obtain, from a first neural network (1031) into which the first speech signal (1110) is input, a first sequence (e.g., E) of feature information for portions (1110-1, 1110-2, 1110-3, 1110-4) of the first speech signal (1110) corresponding to a specified frame unit. m ∈R Ta X 128 ) can be obtained. The processor (1010) can obtain portions (1110-1, 1110-2, 1110-3, 1110-4) of the first voice signal (1110) based on a specified time from the first voice signal (1110). The processor (1010) can divide the first voice signal (1110) based on a specified time, and obtain portions (1110-1, 1110-2, 1110-3, 1110-4) of the first voice signal (1110). The processor (1010) can change the portions (1110-1, 1110-2, 1110-3, 1110-4) of the first voice signal (1110) into the first voice data (1111) of FIG. 11A based on a frequency. The processor (1010) can change portions (1110-1, 1110-2, 1110-3, 1110-4) of the first voice signal (1110) into second voice data (1112) of FIG. 11A based on time. The processor (1010) can obtain a first sequence (1120) through a first neural network (1031) into which the first voice data (1111) and the second voice data (1113) are input.

[0202] Referring to FIG. 18, in operation 1820, a processor (1010) according to an embodiment may obtain a second sequence (1130) from a second speech signal (1113). For example, the processor (1010) may obtain, from a second neural network (1032) into which the second speech signal (1113) is input, a second sequence (e.g., E) of feature information for portions (1113-1, 1113-2, 113-3, 1113-4) of the second speech signal (1113) corresponding to the specified frame unit. p ∈R Ta X 128 ) can be obtained. The processor (1010) can obtain portions (1113-1, 1113-2, 113-3, 1113-4) of the second voice signal (1113) based on a specified time from the second voice signal (1113). The processor (1010) can segment the second voice signal (1113) based on a specified time, and obtain portions (1113-1, 1113-2, 113-3, 1113-4) of the second voice signal (1113). The processor (1010) can change the portions of the second voice signal (1113) into the third voice data (1114) of FIG. 11A based on a frequency. The processor (1010) can change portions (1113-1, 1113-2, 113-3, 1113-4) of the second voice signal (1113) into the fourth voice data (1115) of FIG. 11A based on time. The processor (1010) can obtain the second sequence (1130) through the second neural network (1032) into which the third voice data (1114) and the fourth voice data (1115) are input.

[0203] Referring to FIG. 18, in operation 1030, the processor (1010) according to one embodiment may obtain a mask (1150) based on the first sequence (1120) and the second sequence (1130). For example, an echo cancellation sequence (1160) may be obtained. By inputting the first sequence (1120) and the second sequence (1130) into the eleventh operator (1034-1) included in the fourth neural network (1034), the mask (1150) may be obtained. For example, the eleventh operator (1034-1) may have a structure in which Conv2D, ReLU, Conv2D, ReLU, Conv2D, and Sigmoid are connected after CONCAT.

[0204] Referring to FIG. 18, in operation 1840, the processor (1010) according to one embodiment may obtain an echo cancellation sequence (1160) based on the first sequence (1120) and the mask (1150). For example, the processor (1010) may obtain the echo cancellation sequence (1160) by performing a multiplication operation (e.g., Hadamard product or element-wise product) on the first sequence (1120) and the mask (1150).

[0205] An electronic device (101) according to an embodiment as described above may include a microphone (160), a memory (120), and a processor (110). The processor (110) may be configured to obtain, from a first neural network (131) into which a voice signal (310) received through the microphone (160) is input, a first sequence (320) of portions (310-1, 310-2, 310-3, 310-4) of the voice signal (310) corresponding to a designated frame unit. The processor (110) may be configured to obtain, from a second neural network (132) into which a designated text (315) is input, a second sequence (330) of one or more phonetic symbols for the designated text (315). The processor (110) may be configured to obtain one or more data sets (340) representing a relationship between the first sequence (320) and the second sequence (330) from a third neural network (133) into which the first sequence (320) and the second sequence (330) are input.

[0206] For example, the processor (110) may be configured to obtain a first parameter (350) indicating whether the speech signal (310) corresponds to the specified text (315) from a fourth neural network (134) into which a first data set (340-1) indicating the degree to which each of the one or more pronunciation symbols corresponds to each of the portions (310-1, 310-2, 310-3, 310-4) of the speech signal (310) among the one or more data sets (340) is input. The processor (110) may be configured to obtain one or more second parameters (360) indicating whether the portions (310-1, 310-2, 310-3, 310-4) of the speech signal (310) correspond to each of the one or more pronunciation symbols from a fifth neural network (135) into which a second data set (340-2) distinct from the first data set (340-1) among the one or more data sets (340) is input.

[0207] For example, the number of the one or more second parameters (360) may be equal to the number of the one or more pronunciation symbols. The second data set (340-2) may indicate the degree to which each of the portions (310-1, 310-2, 310-3, 310-4) of the speech signal (310) corresponds to each of the one or more pronunciation symbols.

[0208] For example, the processor (110) may be configured to train at least a portion of the first neural network (131), at least a portion of the second neural network (132), the third neural network (133), the fourth neural network (134), and the fifth neural network (135) using at least one of the first parameter (350) and the one or more second parameters (360).

[0209] For example, the electronic device (101) may further include a display. The processor (110) may be configured to display a visual object on the display, using the one or more second parameters (360), indicating whether at least a portion of the second sequence (330) matches at least a portion of the first sequence (320).

[0210] For example, the third neural network (133) can be trained to identify whether at least a portion of the second sequence (330) and at least a portion of the first sequence (320) match based on supervised learning using the first parameter (350) and the one or more second parameters.

[0211] For example, the processor (110) may be configured to change the specified frame unit based on the length of the specified text (315).

[0212] For example, the processor (110) may identify the specified frame unit based on phonetic symbols corresponding to the voice signal (310) based on receiving the voice signal (310).

[0213] For example, the electronic device (101) may include a speaker (1080). The processor (110) may be configured to obtain a third sequence (1130) of another voice signal (1113) output through the speaker (1080) from a sixth neural network (1032) into which another voice signal (1113) is input. The processor (110) may be configured to obtain the first sequence (1160) with reduced echo from the voice signal (310, 1110) with reduced echo corresponding to the other voice signal (1113) based on inputting the first sequence (1120) and the third sequence (1130) to a seventh neural network (1034). The processor (110) may be configured to obtain one or more data sets (370) from the third neural network (133) into which the first sequence (1160) and the second sequence (330) with reduced echo are input.

[0214] The processor (110) may be configured to obtain a mask (1150) from a mask sub-network (1034) into which the first sequence (320) and the third sequence (1130) are input. The processor (110) may be configured to obtain the first sequence (1160) with reduced echo through an element-by-element product between the mask (1150) and the first sequence (320).

[0215] The above mask sub-network (1034) may include a first layer for obtaining the first sequence (320) and the third sequence (1130) as inputs of different channels.

[0216] The above mask sub-network (1034) may include a first layer for connecting the first sequence (320) and the second sequence (1130) in the direction of the embedding axis, and a dense layer for complete connection for the sequence in which the first sequence (320) and the third sequence (1130) are connected.

[0217] The above third neural network (333) can use a lower triangular matrix as an attention mask.

[0218] In a method performed by an electronic device (101) according to an embodiment as described above, the method may include an operation of obtaining, from a first neural network (131) into which a voice signal (310) received through a microphone (160) is input, a first sequence (320) of portions (310-1, 310-2, 310-3, 310-4) of the voice signal (310) corresponding to a specified frame unit. The method may include an operation of obtaining, from a second neural network (132) into which a specified text (315) is input, a second sequence (330) of one or more phonetic symbols for the specified text (315). The method may include an operation of obtaining one or more data sets (340) representing a relationship between the first sequence (320) and the second sequence (330) from a third neural network (133) into which the first sequence (320) and the second sequence (330) are input.

[0219] For example, the method may include an operation of obtaining a first parameter (350) indicating whether the speech signal (310) corresponds to the specified text (315) from a fourth neural network (134) into which a first data set (340-1) indicating the degree to which each of the one or more pronunciation symbols corresponds to each of the portions (310-1, 310-2, 310-3, 310-4) of the speech signal (310) among the one or more data sets (340) is input. The method may include an operation of obtaining one or more second parameters (360) indicating whether the portions (310-1, 310-2, 310-3, 310-4) of the speech signal (310) do not correspond to each of the one or more pronunciation symbols from a fifth neural network (135) into which a second data set (340-2) distinct from the first data set (340-1) among the one or more data sets (340) is input.

[0220] For example, the number of the one or more second parameters (360) may be equal to the number of the one or more pronunciation symbols. The second data set (340-2) may indicate the degree to which each of the portions (310-1, 310-2, 310-3, 310-4) of the speech signal (310) corresponds to each of the one or more pronunciation symbols.

[0221] For example, the method may include training at least a portion of the first neural network (131), at least a portion of the second neural network (132), the third neural network (133), the fourth neural network (134), and the fifth neural network (135) using at least one of the first parameter (350) and the one or more second parameters (360).

[0222] For example, the method may include displaying a visual object on a display that indicates whether at least a portion of the second sequence (330) matches at least a portion of the first sequence (320), using the one or more second parameters (360).

[0223] For example, the third neural network (133) can be trained to identify whether at least a portion of the second sequence (330) and at least a portion of the first sequence (320) match based on supervised learning using the first parameter (350) and the one or more second parameters.

[0224] For example, the method may include an operation of changing the specified frame unit based on the length of the specified text (315).

[0225] For example, the method may include an operation of obtaining a third sequence (1130) of another voice signal (1113) output through a speaker (1080) from a sixth neural network (1032) into which another voice signal (1113) is input. For example, the method may include an operation of obtaining the first sequence (1160) with reduced echo from the voice signal (310, 1110) with reduced echo corresponding to the other voice signal (1113) based on inputting the first sequence (1120) and the third sequence (1130) to a seventh neural network (1034). For example, the method may include an operation of obtaining the one or more data sets (370) from the third neural network (133) into which the first sequence (1160) with reduced echo and the second sequence (330) are input.

[0226] In a computer-readable storage medium storing one or more programs according to one embodiment as described above, the one or more programs may be configured to include instructions that, when executed by a processor (110) of an electronic device (101), cause the electronic device (101) to obtain, from a first neural network (131) into which a voice signal (310) received through a microphone (160) is input, a first sequence (320) of portions (310-1, 310-2, 310-3, 310-4) of the voice signal (310) corresponding to a designated frame unit. The one or more programs may be configured to include instructions that, when executed by the processor (110) of the electronic device (101), cause the electronic device (101) to obtain, from a second neural network (132) into which the specified text (315) has been input, a second sequence (330) of one or more phonetic symbols for the specified text (315). The one or more programs may be configured to include instructions that cause the electronic device (101) to obtain, from a third neural network (133) into which the first sequence (320) and the second sequence (330) have been input, one or more data sets (340) representing a relationship between the first sequence (320) and the second sequence (330).

[0227] For example, the one or more programs may be configured to include instructions that, when executed by the processor (110) of the electronic device (101), cause the electronic device (101) to obtain, from a fourth neural network (134) into which a first data set (340-1) indicating the degree to which each of the one or more pronunciation symbols corresponds, a first parameter (350) indicating whether the voice signal (310) corresponds to the specified text (315), for each of the portions (310-1, 310-2, 310-3, 310-4) of the voice signal (310) among the one or more data sets (340). The one or more programs may be configured to include instructions that cause the electronic device (101) to obtain, from a fifth neural network (135) into which a second data set (340-2) distinct from the first data set (340-1) among the one or more data sets (340) is input, one or more second parameters (360) indicating whether the portions (310-1, 310-2, 310-3, 310-4) of the speech signal (310) correspond to each of the one or more pronunciation symbols when executed by the processor (110) of the electronic device (101).

[0228] For example, the number of the one or more second parameters (360) may be equal to the number of the one or more pronunciation symbols. The second data set (340-2) may indicate the degree to which each of the portions (310-1, 310-2, 310-3, 310-4) of the speech signal (310) corresponds to each of the one or more pronunciation symbols.

[0229] For example, the one or more programs may be configured to include instructions that cause the electronic device (101) to train at least a portion of the first neural network (131), at least a portion of the second neural network (132), the third neural network (133), the fourth neural network (134), and the fifth neural network (135) using at least one of the first parameter (350) and the one or more second parameters (360) when executed by the processor (110) of the electronic device (101).

[0230] For example, the one or more programs may be configured to include instructions that, when executed by the processor (110) of the electronic device (101), cause the electronic device (101) to display a visual object on the display that indicates whether at least a portion of the second sequence (330) matches at least a portion of the first sequence (320), using the one or more second parameters (360).

[0231] For example, the one or more programs may be configured to include instructions that, when executed by the processor (110) of the electronic device (101), cause the electronic device (101) to obtain a third sequence (1130) of another voice signal (1113) from a sixth neural network (1032) into which another voice signal (1113) output through a speaker (1080) is input. The one or more programs may be configured to include instructions that cause the electronic device (101) to obtain the first sequence (1160) with reduced echo from the speech signal (310, 1110) with reduced echo corresponding to the other speech signal (1113) based on inputting the first sequence (1120) and the third sequence (1130) to the seventh neural network (1034) when executed by the processor (110) of the electronic device (101). The one or more programs may be configured to include instructions that cause the electronic device (101) to obtain the one or more data sets (370) from the third neural network (133) to which the first sequence (1160) with reduced echo and the second sequence (330) have been input when executed by the processor (110) of the electronic device (101).

[0232] As described above, the electronic device (1001) may include a microphone (1060), a speaker (1080), a processor (1010), and a memory (1020) that stores instructions. The instructions, when executed by the processor (1010), may cause the electronic device (1001) to obtain a first sequence (1120) of a first voice signal (1110) received through the microphone (1060) from a first neural network (1031) into which the first voice signal (1110) is input. The above instructions, when executed by the processor (1010), may cause the electronic device (1001) to obtain a second sequence (1130) of a second voice signal (1113) output through the speaker (1080) from a second neural network (1032) into which the second voice signal (1113) is input. The above instructions, when executed by the processor (1010), may cause the electronic device (1001) to obtain a third sequence (1140) for a specified text (1116) from a third neural network (1033) into which the specified text (1116) is input. The instructions, when executed by the processor (1010), may cause the electronic device (1001) to obtain a fourth sequence (1160) for a third voice signal with reduced echo corresponding to the second voice signal (1113) in the first voice signal (1110) based on inputting the first sequence (1120) and the second sequence (1130) to the fourth neural network (1034). The instructions, when executed by the processor (1010), may cause the electronic device (1001) to obtain data sets (1170) representing a relationship between the third sequence (1140) and the fourth sequence (1160) from a fifth neural network (1035) into which the third sequence (1140) and the fourth sequence (1160) have been input.

[0233] The above instructions, when executed by the processor (1010), may cause the electronic device (1001) to obtain a mask (1150) from a mask sub-network (1034) into which the first sequence (1120) and the second sequence (1130) are input. The above instructions, when executed by the processor (1010), may cause the electronic device (1001) to obtain the fourth sequence (1160) through an element-by-element product between the mask (1150) and the first sequence (1120).

[0234] The above mask sub-network (1034) may include a first layer for obtaining the first sequence (1120) and the second sequence (1130) as inputs of different channels.

[0235] The above mask sub-network (1034) may include a first layer for connecting the first sequence (1120) and the second sequence (1130) in the direction of the embedding axis, and a dense layer for complete connection for the sequence in which the first sequence (1120) and the second sequence (1130) are connected.

[0236] The connection structure of the operators (1031-1, 131-2, 131-3, 131-4) included in the first neural network (1031) and the other connection structures of the other operators (1032-1, 132-2, 132-3, 132-4) included in the second neural network (1032) may be identical to each other.

[0237] The above fifth neural network (1035) can use a lower triangular matrix as an attention mask.

[0238] The instructions, when executed by the processor (1010), may cause the electronic device (1001) to obtain a first parameter (1180) indicating whether the third speech signal corresponds to the designated text (1116) from a sixth neural network (1036) input with a first data set (1170-1) indicating the degree to which each of one or more pronunciation symbols of the designated text (1116) corresponds to each of portions of the third speech signal among the one or more data sets (1170). The above instructions, when executed by the processor (1010), may cause the electronic device (1001) to obtain, from a seventh neural network (1037) into which a second data set (1170-2) distinct from the first data set (1170-1) among the one or more data sets (1170) is input, one or more second parameters (1190) indicating whether the portions of the third speech signal correspond to each of the one or more pronunciation symbols.

[0239] The number of the one or more second parameters (1190) may be equal to the number of the one or more pronunciation symbols. The second data set (1170-2) may indicate the degree to which each of the portions of the third speech signal corresponds to each of the one or more pronunciation symbols.

[0240] The instructions, when executed by the processor (1010), may cause the electronic device (1001) to train at least a portion of the first neural network (1031), at least a portion of the second neural network (1032), at least a portion of the third neural network (1033), at least a portion of the fourth neural network (1034), the fifth neural network (1035), the sixth neural network (1036), and the seventh neural network (1037) using at least one of the first parameter (1180) and the one or more second parameters (1190).

[0241] The electronic device (1001) may include a display (1070). The instructions, when executed by the processor (1010), may cause the electronic device (1001) to display, on the display (1070), a visual object (710) indicating whether at least a portion of the third sequence (1140) matches at least a portion of the fourth sequence (1160), using the one or more second parameters (1190).

[0242] As described above, the method can be performed in an electronic device (1001) including a microphone (1060) and a speaker (1080). The method can include an operation of obtaining a first sequence (1120) of a first voice signal (1110) received through the microphone (1060) from a first neural network (1031) into which the first voice signal (1110) is input. The method can include an operation of obtaining a second sequence (1130) of a second voice signal (1113) output through the speaker (1080) from a second neural network (1032) into which the second voice signal (1113) is input. The method can include an operation of obtaining a third sequence (1140) for a specified text (1116) from a third neural network (1033) into which the specified text (1116) is input. The method may include an operation of obtaining a fourth sequence (1160) for a third speech signal with reduced echo corresponding to the second speech signal (1113) in the first speech signal (1110) based on inputting the first sequence (1120) and the second sequence (1130) into a fourth neural network (1034). The method may include an operation of obtaining data sets (1170) representing a relationship between the third sequence (1140) and the fourth sequence (1160) from a fifth neural network (1035) into which the third sequence (1140) and the fourth sequence (1160) are input.

[0243] The method may include an operation of obtaining a mask (1150) from a mask sub-network (1034) into which the first sequence (1120) and the second sequence (1130) are input. The method may include an operation of obtaining the fourth sequence (1160) through an element-by-element product between the mask (#50) and the first sequence (1120).

[0244] The method may include an operation of obtaining, from a sixth neural network (1036) into which a first data set (1170-1) indicating the degree to which each of one or more pronunciation symbols of the designated text (1116) corresponds to each of portions of the third speech signal among the one or more data sets (1170) is input, a first parameter (1180) indicating whether the third speech signal corresponds to the designated text (1116). The method may include an operation of obtaining, from a seventh neural network (1037) into which a second data set (1170-2) distinct from the first data set (1170-1) among the one or more data sets (1170) is input, one or more second parameters (1190) indicating whether the portions of the third speech signal correspond to each of the one or more pronunciation symbols.

[0245] The method may include an operation of training at least a portion of the first neural network (1031), at least a portion of the second neural network (1032), at least a portion of the third neural network (1033), at least a portion of the fourth neural network (1034), the fifth neural network (1035), the sixth neural network (1036), and the seventh neural network (1037) using at least one of the first parameter (1180) and the one or more second parameters (1190).

[0246] As described above, a non-transitory computer-readable storage medium can store one or more programs. When executed by a processor (1010) of an electronic device (1001) including a microphone (1060) and a speaker (1080), the one or more programs can cause the electronic device (1001) to obtain a first sequence (1120) of a first voice signal (1110) received through the microphone (1060) from a first neural network (1031) into which the first voice signal (1110) is input. The one or more programs, when executed by the processor (1010), may cause the electronic device (1001) to obtain a second sequence (1130) of the second voice signal (1113) output through the speaker (1080) from a second neural network (1032) into which the second voice signal (1113) is input. The one or more programs, when executed by the processor (1010), may cause the electronic device (1001) to obtain a third sequence (1140) for the specified text (1116) from a third neural network (1033) into which the specified text (1116) is input. The one or more programs, when executed by the processor (1010), may cause the electronic device (1001) to obtain a fourth sequence (1160) for a third voice signal with reduced echo corresponding to the second voice signal (1113) in the first voice signal (1110) based on inputting the first sequence (1120) and the second sequence (1130) to the fourth neural network (1034). The one or more programs, when executed by the processor (1010), may cause the electronic device (1001) to obtain data sets (1170) representing a relationship between the third sequence (1140) and the fourth sequence (1160) from the fifth neural network (1035) into which the third sequence (1140) and the fourth sequence (1160) are input.

[0247] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0248] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0249] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may be one that continuously stores a computer-executable program or one that temporarily stores it for execution or download. In addition, the medium may be various recording or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program commands, including ROM, RAM, and flash memory. In addition, examples of other media may include recording or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc.

[0250] Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0251] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. In electronic devices, mike; memory; and Contains a processor, The above processor, Obtaining a first sequence of portions of the voice signal corresponding to a designated frame unit from a first neural network into which a voice signal received through the microphone is input; Obtaining a second sequence of one or more phonetic symbols for the specified text from a second neural network into which the specified text has been input, and A third neural network configured to obtain one or more data sets representing a relationship between the first sequence and the second sequence from the third neural network into which the first sequence and the second sequence are input. Electronic devices.

2. In the first paragraph, the processor, For each of the portions of the speech signal among the one or more data sets, a first parameter indicating whether the speech signal corresponds to the specified text is obtained from a fourth neural network into which a first data set indicating the degree to which each of the one or more pronunciation symbols corresponds is input; and A fifth neural network configured to obtain one or more second parameters indicating whether the portions of the speech signal correspond to each of the one or more pronunciation symbols, wherein the second data set, which is distinct from the first data set among the one or more data sets, is input. Electronic devices.

3. In paragraph 2, The number of said one or more second parameters is equal to the number of said one or more pronunciation symbols, and The second data set indicates, for each of the one or more pronunciation symbols, the degree to which each of the portions of the speech signal corresponds, Electronic devices.

4. In the second paragraph, the processor, configured to train at least a portion of the first neural network, at least a portion of the second neural network, the third neural network, the fourth neural network, and the fifth neural network using at least one of the first parameter and the one or more second parameters; Electronic devices.

5. In the second paragraph, the electronic device further includes a display, The above processor, Using one or more of the second parameters, a visual object is displayed on the display, the visual object indicating whether at least a portion of the second sequence matches at least a portion of the first sequence. Electronic devices.

6. In paragraph 2, The third neural network is trained to identify whether at least a part of the second sequence matches at least a part of the first sequence, based on supervised learning, using the first parameter and the one or more second parameters. Electronic devices.

7. In paragraph 1, The above processor, configured to change the specified frame unit based on the length of the text specified above, Electronic devices.

8. In paragraph 1, The above processor, The above-mentioned designated frame unit is identified based on phonetic symbols corresponding to the speech signal, based on receiving the speech signal. Electronic devices.

9. In paragraph 1, Includes speakers, The above processor, Obtaining a third sequence of the other voice signal from a sixth neural network into which another voice signal output through the above speaker is input, Based on inputting the first sequence and the third sequence to the seventh neural network, the first sequence with reduced echo is obtained from the speech signal with reduced echo corresponding to the other speech signal, and configured to obtain one or more data sets from the third neural network into which the first sequence and the second sequence with reduced echo are input, Electronic devices.

10. In paragraph 9, The above processor, Obtaining a mask from the mask sub-network into which the first sequence and the third sequence are input, configured to obtain the first sequence with reduced echo through an element-by-element product between the mask and the first sequence, Electronic devices.

11. In paragraph 10, The above mask sub-network is, A first layer comprising the first sequence and the third sequence for obtaining inputs of different channels, Electronic devices.

12. In paragraph 10, The above mask sub-network is, A second layer for connecting the first sequence and the third sequence in the direction of the embedding axis, and A dense layer for complete connection of the sequence in which the first sequence and the third sequence are concatenated, Electronic devices.

13. In paragraph 9, The third neural network above uses a lower triangular matrix as an attention mask. Electronic devices.

14. In a method performed by an electronic device, the method comprises: An operation of obtaining a first sequence of portions of a voice signal corresponding to a specified frame unit from a first neural network into which a voice signal received through a microphone is input; An operation of obtaining a second sequence of one or more phonetic symbols for the specified text from a second neural network into which the specified text has been input; and An operation of obtaining one or more data sets representing a relationship between the first sequence and the second sequence from a third neural network into which the first sequence and the second sequence are input, method.

15. In paragraph 14, the method, An operation of obtaining a first parameter indicating whether the speech signal corresponds to the specified text from a fourth neural network input with a first data set indicating the degree to which each of the one or more pronunciation symbols corresponds, for each of the portions of the speech signal among the one or more data sets; and An operation of obtaining one or more second parameters indicating whether the portions of the speech signal correspond to each of the one or more pronunciation symbols from a fifth neural network into which a second data set distinct from the first data set among the one or more data sets is input. method.

16. In the 14th paragraph, the method, An operation for changing the specified frame unit based on the length of the specified text, method.

17. In paragraph 14, An operation of obtaining a third sequence of another voice signal from a sixth neural network into which another voice signal output through a speaker is input, An operation of obtaining the first sequence with reduced echo from the speech signal with reduced echo corresponding to the other speech signal based on inputting the first sequence and the third sequence to the seventh neural network, and An operation of obtaining one or more data sets from the third neural network into which the first sequence and the second sequence with reduced echo are input. method.

18. In a computer-readable storage medium storing one or more programs, when the one or more programs are executed by a processor of an electronic device, Obtaining a first sequence of portions of a voice signal corresponding to a specified frame unit from a first neural network into which a voice signal received through a microphone is input; Obtaining a second sequence of one or more phonetic symbols for the specified text from a second neural network into which the specified text has been input, and configured to include instructions that cause the electronic device to obtain one or more data sets representing a relationship between the first sequence and the second sequence from a third neural network into which the first sequence and the second sequence are input. Computer readable storage medium.

19. In paragraph 18, When said one or more programs are executed by said processor of said electronic device, For each of the portions of the speech signal among the one or more data sets, a first parameter indicating whether the speech signal corresponds to the specified text is obtained from a fourth neural network into which a first data set indicating the degree to which each of the one or more pronunciation symbols corresponds is input; and An electronic device configured to include instructions for causing the electronic device to obtain one or more second parameters indicating whether the portions of the speech signal correspond to each of the one or more pronunciation symbols from a fifth neural network into which a second data set distinct from the first data set among the one or more data sets has been input, Computer readable storage medium.

20. In paragraph 18, When said one or more programs are executed by said processor of said electronic device, Obtaining a third sequence of another voice signal from a sixth neural network into which another voice signal output through a speaker is input, Based on inputting the first sequence and the third sequence to the seventh neural network, the first sequence with reduced echo is obtained from the speech signal with reduced echo corresponding to the other speech signal, and configured to include instructions that cause the electronic device to obtain one or more data sets from the third neural network into which the first sequence and the second sequence with reduced echo are input, Computer readable storage medium.

Images