Audio separation method and electronic device for performing same

Abstract

Provided are a method for separating second audio from first audio by using one or more artificial neural networks, and an electronic device for performing same. The method comprises the steps of: extracting a plurality of pieces of first feature data from target information related to the second audio; generating second feature data by applying a plurality of feature separation vectors corresponding to a type of the target information to the plurality of pieces of first feature data; and separating the second audio from the first audio on the basis of the second feature data.

Description

Audio separation method and electronic device performing the same

[0001] The present disclosure relates to an audio processing method, and more particularly, to a method for separating target audio from mixed audio including a plurality of audios and an electronic device for performing the same.

[0002] Audio separation technology separates target audio from mixed audio containing multiple audio components. In related technologies, the audio separation method includes extracting audio features from the mixed audio using a convolutional neural network, generating a spectrogram corresponding to the target audio from the audio features using an up-convolutional neural network, and finally obtaining the target audio. At this time, target information corresponding to the target audio is required. For example, visual information (e.g., an image containing all or part of a speaker's face corresponding to the target audio), audio information (e.g., a pre-recorded voice of a speaker corresponding to the target audio), text information (e.g., a spoken text corresponding to the target audio), the gender or age of the target speaker, etc. can be used as target information.

[0003] Conventional audio separation techniques use separate audio separation models depending on the type of target information. For example, an audio separation model that uses the target speaker's mouth shape as target information cannot utilize the target speaker's pre-recorded voice as target information. Therefore, to separate target audio using multiple types of target information, multiple audio separation models capable of processing each type of target information are required. This increases memory usage proportionally to the number of target information types.

[0004] Furthermore, audio separation models that separate target audio using a single type of target information experience a sharp decline in audio separation performance when that target information is lost. For example, the performance of an audio separation model that separates target audio using the target speaker's mouth shape declines sharply in sections where the target speaker's mouth shape is obscured.

[0005] According to one aspect of the present disclosure, a method for processing first audio using one or more artificial neural networks is provided, the method comprising: extracting a plurality of first feature data from target information related to second audio; generating second feature data based on the plurality of first feature data and a plurality of feature separation vectors corresponding to a plurality of types of the target information; and separating the second audio from the first audio based on the second feature data.

[0006] According to one aspect of the present disclosure, there is provided an electronic device (1000) for processing first audio using one or more processors (1010) including processing circuitry; and one or more artificial neural networks (100) including a memory (1020) storing one or more instructions, wherein the one or more instructions, when individually or collectively executed by the one or more processors (1010), cause the electronic device (1000) to perform an operation including: extracting a plurality of first feature data from target information associated with second audio; generating second feature data based on the plurality of first feature data and a plurality of feature separation vectors corresponding to a plurality of types of the target information; and separating the second audio from the first audio based on the second feature data.

[0007] According to one aspect of the present disclosure, a computer-readable recording medium is provided, storing a program that, when executed by a computer, causes the computer to perform a method of processing first audio using one or more artificial neural networks, the method comprising: extracting a plurality of first feature data from target information related to second audio; generating second feature data based on the plurality of first feature data and a plurality of feature separation vectors corresponding to a plurality of types of the target information; and separating the second audio from the first audio based on the second feature data.

[0008] FIG. 1 illustrates an audio separation network according to one embodiment of the present disclosure.

[0009] FIG. 2 illustrates a feature projection result of a plurality of first feature data and second feature data according to one embodiment of the present disclosure.

[0010] FIG. 3 illustrates an audio separation module according to one embodiment of the present disclosure.

[0011] FIG. 4 is a flowchart of an audio separation method according to one embodiment of the present disclosure.

[0012] FIG. 5 is a conceptual diagram illustrating the operation of an audio separation network in an example where target information has disappeared, according to one embodiment of the present disclosure.

[0013] FIG. 6 is a flowchart illustrating an audio separation method in an example where target information has disappeared, according to one embodiment of the present disclosure.

[0014] FIG. 7 is a conceptual diagram illustrating an audio separation network selecting one target information from a plurality of target information according to one embodiment of the present disclosure.

[0015] FIG. 8 illustrates an audio separation network that separates target audio using multiple target information according to one embodiment of the present disclosure.

[0016] FIG. 9 is a flowchart illustrating an audio separation method according to one embodiment of the present disclosure.

[0017] FIG. 10 illustrates an electronic device executing an audio separation network according to one embodiment of the present disclosure.

[0018] Reference will now be made in detail to embodiments illustrated in the accompanying drawings, wherein like reference numerals designate like elements throughout. In this regard, the embodiments may have different forms and should not be construed as limited to the description set forth herein. Accordingly, embodiments are described below to illustrate aspects with reference to the drawings. The term "and / or" as used herein encompasses any combination of one or more associated listed items. The phrase "at least one" preceding a list of elements modifies the list as a whole and does not modify individual elements in the list.

[0019] Hereinafter, a display device and a method for manufacturing the display device will be described in detail with reference to the accompanying drawings. In the drawings below, like reference numbers represent like components, and the sizes of each component in the drawings may be exaggerated for clarity and convenience of explanation. The embodiments described below are merely examples, and various modifications are possible from these embodiments.

[0020] In this disclosure, the expression “at least one of a, b, and c” may refer to “a,” “b,” “c,” “a and b,” “a and c,” “b and c,” “all of a, b, and c,” or variations thereof.

[0021] The terms used in this disclosure are selected from widely used, current terms, taking into account the functions of the embodiments. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant description of the invention. Therefore, the terms used in this disclosure should not be understood simply as names, but rather based on their inherent meanings and the overall content of this disclosure.

[0022] While terms like "first" and "second" may be used to describe various components, these components should not be limited by these terms. These terms are used solely to distinguish one component from another. For example, without departing from the scope of the present invention, a first component could be referred to as a "second component," and similarly, a second component could also be referred to as a "first component."

[0023] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components in between. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components in between.

[0024] Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art described herein.

[0025] In this disclosure, it should be understood that terms such as “include” or “have” are intended to specify the presence of a feature, number, step, operation, component, part, or combination thereof described in this disclosure, but do not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0026] Embodiments of the present disclosure may be described and illustrated in terms of blocks that perform the described function or functions. These blocks, which may be referred to as managers, units, modules, hardware components, or the like, may be physically implemented by analog and / or digital circuits such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hardwired circuits, and the like, and may optionally be driven by firmware. The circuits may be implemented on, for example, one or more semiconductor chips, or a substrate support such as a printed circuit board. The circuits constituting the blocks may be implemented by dedicated hardware or processors (e.g., one or more programmed microprocessors and associated circuits), or a combination of dedicated hardware that performs some functions of the block and processors that perform other functions of the block. Each block of the embodiments may be physically separated into two or more interacting and separate blocks. Similarly, the blocks of the embodiments may be physically combined into more complex blocks without departing from the scope of the present disclosure. In this disclosure, components expressed as "units", "modules", etc. may be two or more components combined into one component, or one component may be divided into two or more components with more detailed functions. In addition, each component described below may additionally perform some or all of the functions performed by other components in addition to its own main function, and of course, some of the main functions performed by each component may be exclusively performed by other components.

[0027] According to one or more embodiments of the present disclosure, functions related to artificial intelligence may be operated via a processor and memory. The processor may be composed of one or more processors. In this case, one or more processors may be a general-purpose processor such as a CPU, an AP, a Digital Signal Processor (DSP), a graphics-only processor such as a GPU or a Vision Processing Unit (VPU), or an artificial intelligence-only processor such as an NPU. The one or more processors control the processing of input data according to predefined operation rules or artificial intelligence models stored in the memory. Alternatively, if one or more processors are artificial intelligence-only processors, the artificial intelligence-only processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0028] A processor may include various processing circuits and / or multiple processors. For example, the term “processor” as used herein, including in the claims, may include various processing circuits that include at least one processor. One or more of the at least one processor may be configured to perform the various functions described herein, individually and / or collectively in a distributed manner. As used herein, when “processor,” “at least one processor,” and / or “one or more processors” are described as being configured to perform multiple functions, these terms encompass, without limitation, situations where one processor performs some of the functions and other processor(s) perform other parts of the functions, and situations where a single processor may perform all of the functions. Furthermore, the at least one processor may include a combination of processors that perform the various functions described, e.g., in a distributed manner. The at least one processor may execute program instructions to achieve or perform the various functions.

[0029] The predefined operation rules or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that the basic artificial intelligence model is trained using a learning algorithm using a plurality of learning data, thereby creating a predefined operation rule or artificial intelligence model set to perform a specific desired task (or purpose). This learning may be performed on the device itself on which the artificial intelligence according to the present disclosure is performed, or may be performed through a separate server and / or system. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0030] An artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values, and performs neural network operations through operations between the operation results of the previous layer and the multiple weights. The multiple weights of the multiple neural network layers may be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be updated so that the loss value or cost value obtained from the artificial intelligence model is reduced or minimized during the learning process. The artificial neural network may include a deep neural network (DNN), and examples thereof include, but are not limited to, a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or deep Q-networks (DQN).

[0031] According to one or more embodiments of the present disclosure, a device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where data is stored temporarily. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0032] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0033] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein.

[0034] In the present disclosure, target information (10) may include information related to a speaker corresponding to audio to be separated (e.g., second audio (50)). For example, target information (10) may be information related to the target speaker. For example, target information (10) may be the mouth shape of a speaker in a video, the speaker's pre-recorded voice, text corresponding to the audio to be separated, or the speaker's gender, age, etc. Terms such as "target speaker information" may be used instead of "target information." However, the present disclosure is not limited thereto, and the target information may include other types of audio information.

[0035] In the present disclosure, a target speaker may include a speaker corresponding to an audio (or target audio) (e.g., second audio (50)) to be separated from a mixed audio (e.g., first audio (20)) of multiple audios. Instead of the term "target speaker," terms such as "target object" may be used. However, the present disclosure is not limited thereto.

[0036] FIG. 1 illustrates an audio separation network (100) according to one embodiment of the present disclosure.

[0037] Referring to FIG. 1, an audio separation network (100) according to one embodiment of the present disclosure can separate a second audio (50) from a first audio (20) based on target information (10). For example, the audio separation network (100) can use the target information (10) to separate a second audio (50) from a first audio (20).

[0038] In one embodiment, the first audio (20) may include multiple audios. The multiple audios may include the second audio (50).

[0039] In one embodiment, the second audio (50) may be a target audio to be separated from the first audio (20).

[0040] As an example, the first audio (20) may include the voices of multiple speakers, and the second audio (50) may be the voice of a target speaker among the multiple speakers. As another example, the first audio (20) may include the voices of one or more speakers and ambient noise (e.g., the sound of a car horn), and the second audio (50) may be the voice of a target speaker among the one or more speakers. As another example, the first audio (20) may include the vocals of a singer and the accompaniment of the song sung by the singer, and the second audio (50) may be the vocals of the singer.

[0041] In one embodiment, the target information (10) may be information related to the second audio (50) (i.e., the target audio). The audio separation network (100) may separate the audio corresponding to the target information (10) from the first audio (20) as the second audio (50).

[0042] In one embodiment, the target information (10) may be of various types. For example, the target information (10) may include information of various modalities. For example, the target information (10) may be of a visual type (or visual modality), an auditory type (or auditory modality), or a text type (or text modality). However, the present disclosure is not limited thereto, and the target information (10) may include other types of information or other modalities. According to one embodiment, the target information (10) of a visual type (or visual modality) may include, but is not limited to, an image (11) including the mouth of the target speaker. According to one embodiment, the target information (10) of an auditory type (or auditory modality) may include, but is not limited to, reference audio (12) in which the voice of the target speaker is recorded. According to one embodiment, the target information (10) of a text type (or text modality) may include, but is not limited to, a speech text (13) of the target speaker. For example, the spoken text may be a script or a subtitle. However, the type of target information (10) is not limited to the examples described above, and may be information such as the gender or age of the target speaker, for example.

[0043] In one embodiment, the audio separation network (100) may include a plurality of feature extraction modules, a feature separation module (120), and an audio separation module (130). For example, the plurality of feature extraction modules may include a first feature extraction module (111), a second feature extraction module (112), and a third feature extraction module (113). However, the present disclosure is not limited thereto, and the number of feature extraction modules may be different from three. The components (111, 112, 113, 120, 130) included in the audio separation network (100) of FIG. 1 may be software components implemented by a processor. For example, the processor may be the processor (1010) of the electronic device (1000) of FIG. 10 (described later) that executes a program stored in a memory (1020). However, the present disclosure is not limited thereto, and the configurations (111, 112, 113, 120, 130) may be virtual configurations in which no matching hardware device actually exists. For example, a hardware device (e.g., an AI processor) corresponding to at least one of the configurations (111, 112, 113, 120, 130) of the audio separation network (100) may be included in the electronic device (1000). Accordingly, the operations described as being performed by the configurations (111, 112, 113, 120, 130) included in the audio separation network (100) of FIG. 1 may be viewed as actually being performed by the processor (1010) of the electronic device (1000) by executing a program stored in the memory (1020).

[0044] In one embodiment, the plurality of feature extraction modules (111 to 113) may extract a plurality of first feature data from the target information (10). For example, the first feature extraction module (111) may extract a first piece of first feature data from the target information (10), the second feature extraction module (112) may extract a second piece of first feature data from the target information (10), and the third feature extraction module (113) may extract a third piece of first feature data from the target information (10). Each of the plurality of feature extraction modules (111 to 113) may be an artificial neural network model designed to extract first feature data from a corresponding type of target information (10).

[0045] For example, the first feature extraction module (111) may be an artificial neural network model that extracts a first piece (31) of first feature data from an image (11) including a mouth of a target speaker. In this case, the first feature extraction module (111) may be a model based on a 3D convolutional neural network, and the first piece (31) of first feature data extracted from visual type target information (10) may be referred to as a visual embedding.

[0046] For example, the second feature extraction module (112) may be an artificial neural network model that extracts a second piece (32) of first feature data from reference audio (12) in which the target speaker's voice is recorded. In this case, the second feature extraction module (112) may be a model based on a convolutional neural network (CNN) and a long short-term memory (LSTM), and the second piece (32) of first feature data extracted from auditory type target information (10) may be referred to as a voice embedding.

[0047] For example, the third feature extraction module (113) may be an artificial neural network model that extracts a third piece (33) of first feature data from the target speaker's spoken text (13). In this case, the third feature extraction module (113) may be a transformer-based model, and the third piece (33) of first feature data extracted from the text-type target information (10) may be referred to as a text embedding.

[0048] However, the types of feature extraction modules (111 to 113) are not limited to the examples described above, and the audio separation network (100) may include various feature extraction modules corresponding to various types of target information (10). In addition, the artificial neural network model is not limited to the examples described above, and other types of machine learning or artificial neural network models may be provided.

[0049] In one embodiment, the feature separation module (120) may apply a plurality of feature separation vectors (121 to 123) to a plurality of first feature data (31 to 33) to generate second feature data (40) based on the plurality of feature separation vectors (121 to 123) and the plurality of pieces (31 to 33) of the first feature data. For example, the feature separation module (120) may apply a first feature separation vector (121) to a first piece (31) of the first feature data, a second feature separation vector (122) to a second piece (32) of the first feature data, and a third feature separation vector (123) to a third piece (33) of the first feature data to generate second feature data (40). For example, the feature separation module (120) may include a plurality of feature separation vectors (121 to 123) corresponding to the type of target information (10). In one embodiment, the feature separation vectors (121 to 123) may be vectors learned according to the type of target information (10). The feature separation vectors (121 to 123) may be referred to as modality-encoding vectors. In FIG. 1, the feature separation module (120) is illustrated as including three feature separation vectors (121 to 123), but is not limited thereto, and the number of feature separation vectors may be different from three. For example, the feature separation module (120) may include the same number of feature separation vectors as the number of feature extraction modules.

[0050] In one embodiment, the feature separation module (120) can add the corresponding feature separation vectors (121 to 123) for each of the plurality of first feature data (31 to 33). For example, the feature separation module (120) can add the first feature separation vector (121) to the first feature data (31). The feature separation module (120) can add the second feature separation vector (122) to the first feature data (32). The feature separation module (120) can add the third feature separation vector (123) to the first feature data (33).

[0051] In one embodiment, the feature separation module (120) can generate second feature data (40) by combining the sum results of a plurality of first feature data (31 to 33) and a plurality of feature separation vectors (121 to 123).

[0052] According to one embodiment, each of the plurality of first feature data (31 to 33) is extracted from one target information (10), but the plurality of first feature data (31 to 33) are extracted by different feature extraction modules (111 to 113). For example, each of the different feature extraction modules (111 to 113) may extract a respective piece of first feature data from the same target information (10). In an example where the plurality of first feature data (31 to 33) are directly input to the audio separation module (130), the audio separation module (130) may not be able to identify the type of target information (10). In addition, in an example where various types of target information (10) are input using only one audio separation module (130), in order to separate the second audio (50) without performance degradation, it is necessary to provide the audio separation module (130) with information regarding the type of target information (10).

[0053] A feature separation module (120) according to one embodiment of the present disclosure can provide information on the type of target information (10) to an audio separation module (130) by separating a plurality of first feature data (31 to 33) within a feature space to generate second feature data (40). For example, the feature separation module (120) can position a plurality of first feature data (31 to 33) in different areas within the feature space according to the type of target information (10).

[0054] In one embodiment, the audio separation module (130) can separate the second audio (50) from the first audio (20) based on the second feature data (40). The audio separation module (130) can be an artificial neural network model that extracts feature data from the first audio (20) and restores the second audio (50) based on an attention operation on the feature data. The components and operation of the audio separation module (130) will be described below with reference to FIG. 3.

[0055] In one embodiment, the audio separation network (100) may be trained using a type of target information randomly selected from among a plurality of predetermined types of target information (10). For example, when training target information (10) of three types (e.g., visual, auditory, and text), the audio separation network (100) may be trained by inferring target audio from training audio using a type of target information (10) randomly selected from among the three types of target information (10), and calculating a loss function between the inferred target audio and the original target audio and performing backpropagation.

[0056] An audio separation network (100) according to one embodiment of the present disclosure extracts a plurality of first feature data (31 to 33) from target information (10) using a plurality of feature extraction modules (111 to 113), and generates second feature data (40) corresponding to the type of target information (10) using a feature separation module (120), thereby enabling the separation of second audio (50) using various types of target information (10) using only one audio separation module (130).

[0057] FIG. 2 illustrates a feature projection result of a plurality of first feature data and second feature data according to one embodiment of the present disclosure.

[0058] Figure 2 compares the feature space before and after the feature separation module (120) performs feature separation on two sets of first feature data. In Figure 2, the two sets of first feature data may be referred to as visual embedding and voice embedding, respectively.

[0059] In Fig. 2, the graph (210) represents the result of feature projection of visual embedding and voice embedding. In the graph (210), the visual embedding and voice embedding are randomly mixed within the feature space. Therefore, in the example where the visual embedding and voice embedding are directly input to the audio separation module (130), the audio separation module (130) may not be able to identify the type of target information (10).

[0060] In Fig. 2, the graph (220) represents the result of feature projection of the visual embedding and voice embedding to which the feature separation vector is applied by the feature separation module (120). In the graph (220), the visual embedding and the voice embedding are positioned separately within the feature space. For example, the feature separation vector may act as a bias for the visual embedding and the voice embedding to separate the visual embedding and the voice embedding within the feature space. Accordingly, the audio separation module (130) according to one embodiment of the present disclosure can identify the type of input target information (10). In addition, the audio separation network (100) can separate the second audio (50) using various types of target information (10) with only one audio separation module (130).

[0061] FIG. 3 illustrates an audio separation module (130) according to one embodiment of the present disclosure.

[0062] Referring to FIG. 3, in one embodiment, the audio separation module (130) can separate the second audio (50) from the first audio (20) based on the second feature data (40).

[0063] In one embodiment, the audio separation module (130) may include a frequency domain encoder (301), a first frequency and target information attention module (302), a first frequency domain self-attention module (303), a time domain encoder (304), a first time and target information attention module (305), a first time domain self-attention module (306), a first cross-domain self-attention module (307), a second frequency and target information attention module (308), a second frequency domain self-attention module (309), a second time and target information attention module (310), a second time domain self-attention module (311), a second cross-domain self-attention module (312), a third frequency and target information attention module (313), a third frequency domain self-attention module (314), and a frequency domain decoder (315). However, the components of the audio separation module (130) are not limited thereto. According to one embodiment, one or more components may be added to the audio separation module (130), or may be omitted from the audio separation module (130).

[0064] In one embodiment, the audio separation module (130) can convert the first audio (20) into a spectrogram (60). For example, the audio separation module (130) can apply a short-time Fourier transform (STFT) to the first audio (20) to generate the spectrogram (60). The spectrogram (60) is an image that visualizes audio data, and represents the difference in amplitude according to changes in time and frequency in the time-frequency domain as a difference in color or intensity of each pixel.

[0065] In one embodiment, the frequency domain encoder (301) can extract frequency domain feature data of the first audio (20) from the spectrogram (60). For example, the frequency domain encoder (301) can include one or more convolutional neural networks.

[0066] In one embodiment, the first frequency and target information attention module (302) may perform an operation on the frequency domain feature data of the first audio (20) and the second feature data (40). For example, the first frequency and target information attention module (302) may perform an operation based on mathematical expression 1.

[0067] [Mathematical Formula 1]

[0068]

[0069] Here, F is the frequency domain feature data of the first audio (20), and V is the second feature data (40). Through an operation based on mathematical expression 1, the first frequency and target information attention module (302) can designate which part of the feature data of the first audio (20) the audio separation module (130) should focus on.

[0070] In one embodiment, the first frequency domain self-attention module (303) may perform a self-attention operation on the output data of the first frequency and target information attention module (302). The self-attention operation may include an operation of obtaining query data, key data, and value data based on input data, calculating a weight corresponding to a correlation between the query (Q) data and the key (K) data, and applying the weight to the value (V) data. For example, the input data may be output data of the first frequency and target information attention module (302).

[0071] In one embodiment, the time domain encoder (304) can extract time domain feature data of the first audio (20) from the spectrogram (60). For example, the time domain encoder (304) can include one or more convolutional neural networks.

[0072] In one embodiment, the first time and target information attention module (305) can perform an operation based on the time domain feature data of the first audio (20) and the second feature data (40). For example, the first time and target information attention module (305) can perform an operation based on the aforementioned mathematical expression 1. In this case, F in mathematical expression 1 is the time domain feature data of the first audio (20). Through this operation based on mathematical expression 1, the first time and target information attention module (305) can designate which part of the time domain feature data of the first audio (20) the audio separation module (130) should focus on.

[0073] In one embodiment, the first time domain self-attention module (306) can perform a self-attention operation on the output data of the first time and target information attention module (305).

[0074] In one embodiment, the first cross-domain self-attention module (307) can perform one or more cross-attention operations on the output data of the first frequency-domain self-attention module (303) and the output data of the first time-domain self-attention module (306). The cross-attention operations can include operations for obtaining query data based on first input data, obtaining key data and value data based on second input data, calculating a weight corresponding to a correlation between the query (Q) data and the key (K) data, and applying the weight to the value (V) data.

[0075] In one embodiment, the first cross-domain self-attention module (307) may perform a first cross-attention operation that obtains query (Q) data based on output data of the first time-domain self-attention module (306), obtains key (K) data and value (V) data based on output data of the first frequency-domain self-attention module (303), calculates a weight corresponding to a correlation between the query (Q) data and the key (K) data, and applies the weight to the value (V) data.

[0076] In one embodiment, the first cross-domain self-attention module (307) may obtain query (Q) data based on output data of the first frequency domain self-attention module (303), obtain key (K) data and value (V) data based on output data of the first time domain self-attention module (306), calculate a weight corresponding to a correlation between the query (Q) data and the key (K) data, and perform a second cross-attention operation that applies the weight to the value (V) data.

[0077] In one embodiment, the second frequency and target information attention module (308) can perform a predetermined operation using the first cross-attention operation result of the first cross-domain self-attention module (307) and the second feature data (40). For example, the second frequency and target information attention module (308) can perform the operation of the aforementioned mathematical expression 1. In this case, F in mathematical expression 1 is the first cross-attention operation result of the first cross-domain self-attention module (307). Through this operation based on mathematical expression 1, the second frequency and target information attention module (308) can designate which part of the first cross-attention operation result the audio separation module (130) should focus on.

[0078] In one embodiment, the second frequency domain self-attention module (309) can perform a self-attention operation on the output data of the second frequency and target information attention module (308).

[0079] In one embodiment, the second time and target information attention module (310) can perform a predetermined operation using the second cross-attention operation result of the first cross-domain self-attention module (307) and the second feature data (40). For example, the second time and target information attention module (310) can perform the operation of the aforementioned mathematical expression 1. In this case, F in mathematical expression 1 is the second cross-attention operation result of the first cross-domain self-attention module (307). Through this operation based on mathematical expression 1, the second time and target information attention module (310) can designate which part of the second cross-attention operation result the audio separation module (130) should focus on.

[0080] In one embodiment, the second time domain self-attention module (311) can perform a self-attention operation on the output data of the second time and target information attention module (310).

[0081] In one embodiment, the second cross-domain self-attention module (312) can perform a cross-attention operation on the output data of the second frequency domain self-attention module (309) and the output data of the second time domain self-attention module (311).

[0082] In one embodiment, the second cross-domain self-attention module (312) may perform a cross-attention operation to obtain query (Q) data based on output data of the second time-domain self-attention module (311), obtain key (K) data and value (V) data based on output data of the second frequency-domain self-attention module (309), calculate a weight corresponding to a correlation between the query (Q) data and the key (K) data, and apply the weight to the value (V) data.

[0083] In one embodiment, the third frequency and target information attention module (313) can perform a predetermined operation using the cross-attention operation result of the second cross-domain self-attention module (312) and the second feature data (40). For example, the third frequency and target information attention module (313) can perform the operation of the aforementioned mathematical expression 1. In this case, F in mathematical expression 1 is the cross-attention operation result of the second cross-domain self-attention module (312). Through this operation based on mathematical expression 1, the third frequency and target information attention module (313) can designate which part of the cross-attention operation result of the second cross-domain self-attention module (312) the audio separation module (130) should focus on.

[0084] In one embodiment, the third frequency domain self-attention module (314) can perform a self-attention operation on the output data of the third frequency and target information attention module (313).

[0085] In one embodiment, the frequency domain decoder (315) can obtain a spectrogram (70) of the second audio (50) from the output data of the third frequency domain self-attention module (314). For example, the frequency domain decoder (315) can include one or more transpose convolutional neural networks.

[0086] In one embodiment, the audio separation module (130) can apply an inverse short-time Fourier transform (ISTF) to the spectrogram (70) to convert the spectrogram (70) into second audio (50).

[0087] FIG. 4 is a flowchart of an audio separation method according to one embodiment of the present disclosure.

[0088] In one embodiment, the audio separation method (400) may be performed by an audio separation network (100), or an electronic device (1000 of FIG. 10) executing the audio separation network (100).

[0089] Referring to FIG. 4, in operation 410, the method may include a step of extracting a plurality of first feature data from target information related to second audio. For example, the audio separation network (100) may extract a plurality of first feature data from target information related to second audio. In one embodiment, the second audio may be target audio to be separated, and the target information may be of various types, such as an image including the mouth of a target speaker, reference audio in which the voice of the target speaker is recorded, or speech text of the target speaker. Operation 410 may correspond to operations of a plurality of feature extraction modules (111 to 113).

[0090] At operation 420, the method may include generating second feature data based on a plurality of first feature data and a plurality of feature separation vectors corresponding to a plurality of types of target information. For example, the audio separation network (100) may generate the second feature data by applying a plurality of feature separation vectors corresponding to the types of target information to the plurality of first feature data. In one embodiment, the feature separation vectors may be vectors learned according to the types of target information. The feature separation vectors may act as biases for the plurality of first feature data to separate each of the plurality of first feature data within the feature space. Operation 420 may correspond to the operation of the feature separation module (120).

[0091] At operation 430, the method may include separating the second audio from the first audio based on the second feature data. For example, the audio separation network (100) may separate the second audio from the first audio based on the second feature data. The audio separation network (100) may extract feature data from the first audio and reconstruct the second audio based on an attention operation on the feature data. Operation 430 may correspond to the operation of the audio separation module (130).

[0092] FIG. 5 is a conceptual diagram illustrating the operation of an audio separation network (100) in an example where target information has disappeared, according to one embodiment of the present disclosure.

[0093] An audio separation network (100) according to one embodiment of the present disclosure can utilize various types of target information (10), so that in an example where target audio is separated based on one target information (10) and then that target information (10) disappears, the target audio can be separated based on other target information.

[0094] Figure 5 illustrates a case where the target speaker's mouth is covered while the audio separation network (100) separates the target audio using the target speaker's mouth movement.

[0095] Referring to FIG. 5, in the T1 section, the audio separation network (100) can separate target audio based on the mouth movements of the target speaker.

[0096] However, in the T2 segment, in the example where the target speaker's mouth is covered, the audio separation network (100) can separate the target audio based on the separated portion (501) in the T1 segment of the target audio. For example, the audio separation network (100) can utilize the separated portion (501) in the T1 segment of the target audio as target information instead of the target speaker's mouth movements. In this case, the separated portion (501) of the target audio can function as reference audio in which the target speaker's voice is recorded as described above.

[0097] In the T3 segment, in the example where the target speaker's mouth reappears, the audio separation network (100) can again separate the target audio based on the target speaker's mouth movements.

[0098] In this way, the audio separation network (100) according to one embodiment of the present disclosure can separate the target audio using a different type of target information without a decrease in audio separation performance, even in an example where the target information disappears while separating the target audio using one type of target information. However, the present disclosure is not limited thereto, and according to one embodiment, the audio separation network (100) can separate the target audio based on the initially separated portion (501) of the target audio in the T1 section even when the target speaker's mouth is not covered in the T2 section. For example, in the T2 section, the audio separation network (100) can separate the target audio based on the initially separated portion (501) of the target audio in the T1 section, the mouth movements of the target speaker, or a combination based on the initially separated portion (501) of the target audio in the T1 section and the mouth movements of the target speaker.

[0099] FIG. 6 is a flowchart illustrating an audio separation method in an example where target information has disappeared, according to one embodiment of the present disclosure.

[0100] According to one embodiment, the audio separation method (600) can be performed by an audio separation network (100), or an electronic device (1000 of FIG. 10) executing the audio separation network (100).

[0101] Referring to FIG. 6, at operation 610, the method may include a step of acquiring target information. For example, the audio separation network (100) may acquire target information. In one embodiment, the target information may be of various types, such as an image including the mouth of the target speaker, reference audio recording of the target speaker's voice, or text of the target speaker's speech.

[0102] At operation 620, the method may include a step of determining whether a first type of target information exists. For example, the audio separation network (100) may determine whether the first type of target information exists. In one embodiment, the audio separation network (100) may further include a module for determining whether the first type of target information exists. In an example where the first type of target information is an image including a mouth of a target speaker, the audio separation network (100) may include an artificial neural network model for detecting the mouth of the target speaker.

[0103] At operation 630, the method may include separating the second audio from the first audio based on the first type of target information. For example, if the first type of target information is present, the audio separation network (100) may separate the second audio from the first audio based on the first type of target information. In one embodiment, operation 630 may include operations 410, 420, and 430 of FIG. 4.

[0104] At operation 640, the method may include generating a plurality of third feature data from second type target information corresponding to the sorted portion of the second audio. For example, if the first type of target information does not exist, the audio separation network (100) may generate a plurality of third feature data from second type target information corresponding to the sorted portion of the second audio. Operation 640 may correspond to the operations of the plurality of feature extraction modules (111 to 113).

[0105] At operation 650, the method may include generating fourth feature data based on a plurality of feature separation vectors corresponding to the type of target information and a plurality of third feature data. For example, the audio separation network (100) may generate the fourth feature data by applying a plurality of feature separation vectors corresponding to the type of target information to the plurality of third feature data. In one embodiment, the feature separation vectors may be vectors learned according to the type of target information. The feature separation vectors may act as biases for the plurality of third feature data to separate each of the plurality of third feature data within the feature space. Operation 650 may correspond to the operation of the feature separation module (120).

[0106] At operation 660, the method may include separating the second audio from the first audio based on the fourth feature data. For example, the audio separation network (100) may separate the second audio from the first audio based on the fourth feature data. The audio separation network (100) may extract feature data from the first audio and reconstruct the second audio based on an attention operation on the feature data. Operation 660 may correspond to the operation of the audio separation module (130).

[0107] FIG. 7 is a conceptual diagram illustrating an audio separation network (100) selecting one target information from among a plurality of target information according to one embodiment of the present disclosure.

[0108] Referring to FIG. 7, in an example where multiple target information (10) is input, the audio separation network (100) can separate the second audio (50) from the first audio (20) based on the target information selected by the target information selection module (710).

[0109] In one embodiment, the target information selection module (710) can select one target information from among multiple target information (10). At this time, the selection criteria for the target information can be implemented in various ways.

[0110] In one embodiment, the target information selection module (710) can determine the reliability of each of the plurality of target information (10) and select the target information with the highest reliability. The criteria for determining reliability may vary depending on the type of target information (10).

[0111] As an example, the target information selection module (710) may determine the reliability of an image (11) containing the target speaker's mouth based on the ratio of the time interval in which the target speaker's mouth appears among the entire time interval. In an example where the entire time interval is 10 seconds and the target speaker's mouth appears for 6 seconds of that, the target information selection module (710) may determine the reliability of the image (11) containing the target speaker's mouth to be 60%.

[0112] As another example, the target information selection module (710) may determine the reliability of the reference audio (12) in which the target speaker's voice is recorded based on the extent to which the reference audio (12) in which the target speaker's voice is recorded contains background sounds. In this case, the target information selection module (710) may include an artificial neural network model that extracts background sounds.

[0113] In one embodiment, the target information selection module (710) receives a user input for selecting target information, and may select one target information from among a plurality of target information (10) based on the received user input. For example, the target information selection module (710) may receive a user input for selecting target information through an input / output interface (1030 of FIG. 10).

[0114] FIG. 8 illustrates an audio separation network (800) that separates target audio using multiple target information according to one embodiment of the present disclosure.

[0115] Referring to FIG. 8, an audio separation network (800) according to one embodiment of the present disclosure can separate a second audio (50) from a first audio (20) using a plurality of target information (10). The types of each of the plurality of target information (10) may be different from each other. For example, as illustrated in FIG. 8, the audio separation network (800) can separate a second audio (50) using both an image (11) including a target speaker's mouth and reference audio (12) in which the target speaker's voice is recorded.

[0116] FIG. 8 illustrates an image (11) including a target speaker's mouth and reference audio (12) in which the target speaker's voice is recorded as examples of multiple target information (10), but the multiple target information (10) may further include various types of target information described above.

[0117] In one embodiment, the audio separation network (800) may include a feature extraction module (810), a feature separation module (820), and an audio separation module (830).

[0118] In one embodiment, the feature extraction module (810) may extract first feature data (80) from a plurality of target information (10). The first feature data (80) may include combined feature data of the plurality of target information (10). The feature extraction module (810) may be an artificial neural network model designed to extract the first feature data (80) from the plurality of target information (10). For example, the feature extraction module (810) may include one or more convolutional neural networks (CNNs).

[0119] In one embodiment, the feature separation module (820) may apply a feature separation vector (821) to the first feature data (80) to generate the second feature data (90). The feature separation module (820) may include a feature separation vector (821) corresponding to the number of target information (10). In one embodiment, the feature separation vector (821) may be a vector learned according to the number of target information (10). In one embodiment, the feature separation module (820) may add the feature separation vector (821) to the first feature data (80).

[0120] According to one embodiment, since the first feature data (80) corresponds to a combined feature of a plurality of target information (10), in an example where the first feature data (80) is directly input to the audio separation module (830), the audio separation module (830) cannot identify the number of target information (10). In order to separate the second audio (50) based on an arbitrary number of target information (10) using only one audio separation module (830), it is necessary to provide information regarding the number of target information (10) to the audio separation module (830).

[0121] A feature separation module (820) according to one embodiment of the present disclosure can provide information on the number of target information (10) to an audio separation module (830) by generating second feature data (90) including information on the number of target information (10) using a feature separation vector (821).

[0122] In one embodiment, the audio separation module (830) may separate the second audio (50) from the first audio (20) based on the second feature data (90). The audio separation module (830) may be an artificial neural network model that extracts feature data from the first audio (20) and restores the second audio (50) based on an attention operation on the feature data. For example, the audio separation module (830) may correspond to the audio separation module (130) of FIGS. 1 and 3.

[0123] FIG. 9 is a flowchart illustrating an audio separation method (900) according to one embodiment of the present disclosure.

[0124] In one embodiment, the audio separation method (900) may be performed by an audio separation network (800), or an electronic device (1000 of FIG. 10) executing the audio separation network (800).

[0125] Referring to FIG. 9 , at operation 910, the method may include extracting first feature data from a plurality of target information associated with second audio. For example, the audio separation network (800) may extract first feature data from a plurality of target information associated with second audio. In one embodiment, the second audio may be target audio to be separated, and the target information may be of various types, such as an image including the mouth of a target speaker, reference audio in which the voice of the target speaker is recorded, or speech text of the target speaker. Operation 910 may correspond to the operation of the feature extraction module (810).

[0126] At operation 920, the method may include generating second feature data based on the feature separation vector corresponding to the number of target information and the first feature data. For example, the audio separation network (800) may generate the second feature data by applying the feature separation vector corresponding to the number of target information to the first feature data. In one embodiment, the feature separation vector may be a vector learned according to the number of target information. Operation 920 may correspond to the operation of the feature separation module (820).

[0127] At operation 930, the method may include separating the second audio from the first audio based on the second feature data. For example, the audio separation network (800) may separate the second audio from the first audio based on the second feature data. The audio separation network (800) may extract feature data from the first audio and reconstruct the second audio based on an attention operation on the feature data. Operation 930 may correspond to the operation of the audio separation module (830).

[0128] FIG. 10 illustrates an electronic device (1000) executing an audio separation network (100, 800) according to one embodiment of the present disclosure.

[0129] An electronic device (1000) according to one embodiment of the present disclosure can separate a second audio (50) from a first audio (20) using target information (10) by executing an audio separation network (100 as shown in FIG. 1 or 800 as shown in FIG. 8).

[0130] Referring to FIG. 10, an electronic device (1000) according to one embodiment of the present disclosure may include a processor (1010), a memory (1020), and an input / output interface (1030). However, the components of the electronic device (1000) are not limited to those illustrated in FIG. 10, and the electronic device (1000) may include more or fewer components than the components illustrated in FIG. 10.

[0131] In one embodiment, the processor (1010) may perform one or more operations of the electronic device (1000). For example, the processor (1010) may be a configuration that controls a series of processes so that the electronic device (1000) operates according to embodiments described in the present disclosure. According to one embodiment, the processor (1010) may be composed of one or more processors. The one or more processors may be circuitry such as a System on Chip (SoC), an Integrated Circuit (IC), or the like. The one or more processors included in the processor (1010) may be a general-purpose processor such as a Central Processing Unit (CPU), a Micro Processor Unit (MPU), an Application Processor (AP), a Digital Signal Processor (DSP), a graphics-only processor such as a Graphics Processing Unit (GPU), a Vision Processing Unit (VPU), an artificial intelligence-only processor such as a Neural Processing Unit (NPU), or a communication-only processor such as a Communication Processor (CP). If one or more processors included in the processor (1010) are artificial intelligence-dedicated processors, the artificial intelligence-dedicated processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0132] In one embodiment, the processor (1010) can write data to the memory (1020), read data stored in the memory (1020), and process data according to predefined operating rules or artificial intelligence models, particularly by executing a program or at least one instruction stored in the memory (1020). Accordingly, the processor (1010) can perform the operations described in the present disclosure, and operations described as being performed by the electronic device (1000) in the present disclosure can be viewed as being performed by the processor (1010) unless otherwise specifically described.

[0133] In one embodiment, the memory (1020) is configured to store various programs or data, and may be configured as a storage medium or a combination of storage media such as a ROM, a RAM, a hard disk, a CD-ROM, and a DVD. The memory (1020) may not exist separately and may be configured to be included in the processor (1010). The memory (1020) may be configured as a volatile memory, a non-volatile memory, or a combination of volatile memory and non-volatile memory. A program or at least one instruction for performing operations according to the embodiments described in the present disclosure may be stored in the memory (1020). The memory (1020) may also provide stored data to the processor (1010) upon request of the processor (1010).

[0134] In one embodiment, the input / output interface (1030) may include an input interface (e.g., a touch screen, a hard button, a microphone, etc.) for receiving control commands or information from a user, and an output interface (e.g., a display panel, a speaker, etc.) for displaying the results of execution of an operation according to the user's control or the status of the electronic device (1000). In one embodiment, the input / output interface (1030) may play a second audio (50) separated from the first audio (20), and may receive a user input for selecting target information from the user.

[0135] According to one embodiment of the present disclosure, a method for processing first audio using one or more artificial neural networks (100) may include a step of extracting a plurality of first feature data from target information related to second audio, a step of generating second feature data based on the plurality of first feature data and a plurality of feature separation vectors corresponding to a plurality of types of the target information, and a step of separating the second audio from the first audio based on the second feature data.

[0136] In one embodiment, the plurality of types of target information may include at least one of a visual type, an auditory type, or a text type.

[0137] In one embodiment, the step of generating the second feature data may include the step of applying the plurality of feature separation vectors to the plurality of first feature data to separate each of the plurality of first feature data within a feature space.

[0138] In one embodiment, the plurality of feature separation vectors may each be a vector learned according to a corresponding one of the plurality of types of the target information.

[0139] In one embodiment, the step of generating the second feature data may include the step of adding a corresponding feature separation vector among the plurality of feature separation vectors for each of the plurality of first feature data.

[0140] In one embodiment, the step of separating the second audio from the first audio based on the second feature data may include the steps of: applying a short-time Fourier transform to the first audio to convert the first audio into a spectrogram; extracting first domain feature data of the first audio from the spectrogram; performing a first operation based on the first domain feature data of the first audio and the second feature data; performing a first self-attention operation on a result of the first operation; extracting second domain feature data of the first audio from the spectrogram; performing a second operation based on the second domain feature data of the first audio and the second feature data; performing a second self-attention operation on a result of the second operation; and performing one or more cross-attention operations on a result of the first self-attention operation and a result of the second self-attention operation.

[0141] In one embodiment, the method may further include: determining whether a first type of target information exists; generating a plurality of third feature data from the second type of target information corresponding to the separated portion of the second audio based on a determination that the first type of target information does not exist; generating a fourth feature data based on the plurality of feature separation vectors and the plurality of third feature data; and separating the second audio from the first audio based on the fourth feature data.

[0142] In one embodiment, the target information may include a plurality of pieces of target information, and the step of extracting a plurality of pieces of first feature data from the target information related to the second audio may include a step of determining a reliability of each piece of the plurality of pieces of target information, a step of selecting target information having the highest reliability among the plurality of pieces of target information, and a step of extracting the plurality of pieces of first feature data from the target information having the highest reliability.

[0143] In one embodiment, the target information may include a plurality of target information, and the step of extracting a plurality of first feature data from the target information related to the second audio may include the steps of receiving a user input for selecting target information, and extracting the plurality of first feature data from target information corresponding to the user input among the plurality of target information.

[0144] In one embodiment, the one or more artificial neural networks (100) can be trained using a type of target information randomly selected from among the plurality of types of target information.

[0145] According to one embodiment of the present disclosure, an electronic device (1000) for processing first audio using one or more artificial neural networks (100) may include one or more processors (1010) including processing circuitry and a memory (1020) for storing one or more instructions, wherein the one or more instructions, when individually or collectively executed by the one or more processors (1010), cause the electronic device (1000) to perform an operation including: extracting a plurality of first feature data from target information related to second audio, generating second feature data based on the plurality of first feature data and a plurality of feature separation vectors corresponding to a plurality of types of the target information, and separating the second audio from the first audio based on the second feature data.

[0146] In one embodiment, the plurality of types of target information may include at least one of a visual type, an auditory type, or a text type.

[0147] In one embodiment, generating the second feature data may include applying the plurality of feature separation vectors to the plurality of first feature data to separate each of the plurality of first feature data within a feature space.

[0148] In one embodiment, the plurality of feature separation vectors may each be a vector learned according to a corresponding one of the plurality of types of the target information.

[0149] In one embodiment, generating the second feature data may include summing a corresponding feature separation vector among the plurality of feature separation vectors for each of the plurality of first feature data.

[0150] In one embodiment, separating the second audio from the first audio based on the second feature data may include applying a short-time Fourier transform to the first audio to convert the first audio into a spectrogram, extracting first domain feature data of the first audio from the spectrogram, performing a first operation based on the first domain feature data of the first audio and the second feature data, performing a first self-attention operation on a result of the first operation, extracting second domain feature data of the first audio from the spectrogram, performing a second operation based on the second domain feature data of the first audio and the second feature data, performing a second self-attention operation on a result of the second operation, and performing one or more cross-attention operations on a result of the first self-attention operation and a result of the second self-attention operation.

[0151] In one embodiment, the operation may further include determining whether a first type of target information exists, generating a plurality of third feature data from a second type of target information corresponding to a separated portion of the second audio based on a determination that the first type of target information does not exist, generating a fourth feature data based on the plurality of feature separation vectors and the plurality of third feature data, and separating the second audio from the first audio based on the fourth feature data.

[0152] In one embodiment, the target information may include a plurality of pieces of target information, and extracting a plurality of pieces of first feature data from the target information related to the second audio may include determining a reliability of each piece of the plurality of pieces of target information, selecting a piece of target information having a highest reliability among the pieces of target information, and extracting the plurality of pieces of first feature data from the piece of target information having the highest reliability.

[0153] In one embodiment, the target information may include a plurality of target information, and extracting a plurality of first feature data from the target information related to the second audio may include receiving a user input for selecting target information, and extracting the plurality of first feature data from target information corresponding to the user input among the plurality of target information.

[0154] According to one embodiment of the present disclosure, a computer-readable recording medium may store a program that, when executed by a computer, causes the computer to perform a method including the steps of extracting a plurality of first feature data from target information related to second audio, generating second feature data based on the plurality of first feature data and a plurality of feature separation vectors corresponding to a plurality of types of the target information, and separating the second audio from the first audio based on the second feature data.

Claims

1. A method for processing first audio using one or more artificial neural networks (100), A step of extracting a plurality of first feature data from target information related to second audio; A step of generating second feature data based on a plurality of first feature data and a plurality of feature separation vectors corresponding to a plurality of types of target information; and A step of separating the second audio from the first audio based on the second feature data; A method comprising:

2. In paragraph 1, A method wherein the plurality of types of said target information include at least one of a visual type, an auditory type, or a text type.

3. In paragraph 1 or 2, The step of generating the above second feature data is A method comprising the step of applying the plurality of feature separation vectors to the plurality of first feature data to separate each of the plurality of first feature data within a feature space.

4. In any one of paragraphs 1 to 3, A method wherein the above plurality of feature separation vectors are vectors each learned according to a corresponding one of the plurality of types of the target information.

5. In any one of paragraphs 1 to 4, The step of generating the above second feature data is: A method comprising the step of summing corresponding feature separation vectors among the plurality of feature separation vectors for each of the plurality of first feature data.

6. In any one of paragraphs 1 to 5, Based on the second feature data, the step of separating the second audio from the first audio is: A step of converting the first audio into a spectrogram by applying a short-time Fourier transform to the first audio, A step of extracting first domain feature data of the first audio from the spectrogram; A step of performing a first operation based on the first domain feature data of the first audio and the second feature data, A step of performing a first self-attention operation on the result of the first operation, A step of extracting second domain feature data of the first audio from the spectrogram; A step of performing a second operation based on the second domain feature data of the first audio and the second feature data, A step of performing a second self-attention operation on the result of the second operation, and A step of performing one or more cross-attention operations on the result of the first self-attention operation and the result of the second self-attention operation; A method comprising:

7. In any one of paragraphs 1 to 6, A step of determining whether a first type of target information exists; A step of generating a plurality of third feature data from the second type of target information corresponding to the selected portion of the second audio based on a determination that the first type of target information does not exist; A step of generating fourth feature data based on the plurality of feature separation vectors and the plurality of third feature data; A step of separating the second audio from the first audio based on the fourth feature data; A method further comprising:

8. In any one of paragraphs 1 to 7, The above target information includes multiple target information, The step of extracting a plurality of first feature data from target information related to the second audio is as follows: A step of determining the reliability of each of the above multiple target information, A step of selecting target information with the highest reliability among the above multiple target information, and A step of extracting the plurality of first feature data from the target information having the highest reliability, A method comprising:

9. In any one of paragraphs 1 to 8, The above target information includes multiple target information, The step of extracting a plurality of first feature data from target information related to the second audio is as follows: A step of receiving user input for selecting target information, and A step of extracting the plurality of first feature data from target information corresponding to the user input among the plurality of target information; A method comprising:

10. In any one of paragraphs 1 to 9, A method in which the above one or more artificial neural networks (100) are trained using a type of target information randomly selected from the plurality of types of target information.

11. In an electronic device (1000) that processes first audio using one or more artificial neural networks (100), One or more processors (1010) including processing circuitry; and Contains a memory (1020) storing one or more instructions, The one or more instructions, when individually or collectively executed by the one or more processors (1010), cause the electronic device (1000) to use the one or more artificial neural networks to: Extracting multiple first feature data from target information related to second audio; Generating second feature data based on the plurality of first feature data and the plurality of feature separation vectors corresponding to the plurality of types of target information, and Separating the second audio from the first audio based on the second feature data; An electronic device that causes an action to be performed, including:

12. In paragraph 11, An electronic device, wherein the plurality of types of said target information include at least one of a visual type, an auditory type, or a text type.

13. In any one of paragraphs 11 to 12, An electronic device wherein the above plurality of feature separation vectors are vectors each learned according to a corresponding one of the plurality of types of the target information.

14. In any one of paragraphs 11 to 13, The above actions are, Determining the presence of type 1 target information; Generating a plurality of third feature data from the second type of target information corresponding to the selected portion of the second audio based on a determination that the first type of target information does not exist; Generating fourth feature data based on the plurality of feature separation vectors and the plurality of third feature data; and Separating the second audio from the first audio based on the fourth feature data; An electronic device further comprising:

15. A computer-readable recording medium storing a program that, when executed by a computer, causes the computer to perform any one of the methods of claims 1 to 10.

Images