Electronic device and control method thereof

Abstract

The electronic device of the present disclosure comprises: a memory for storing at least one instruction; and at least one processor that segments the audio data into a plurality of portions to include an overlapping region, obtains an audio feature from each of the plurality of segmented portions, identifies a first audio source and a second audio source in each of the plurality of segmented portions based on the audio feature, and outputs the identified first audio source and second audio source. And obtaining first audio data corresponding to the first audio source and second audio data corresponding to the second audio source from the audio data.

Description

Technical Field

[0001] The present disclosure relates to an apparatus and method for processing an audio signal, and more particularly, to an electronic device and method for separating an audio signal into audio sources. Background Art

[0002] In audio processing technology, an audio separation model can be used to separate the audio signal for each speaker from a mixed audio signal. For example, in a scenario where there is a 10-second audio signal with a male speaker and a female speaker speaking at the same time, the audio separation model can separate the 10-second audio signal into a 10-second audio signal emitted by a male and a 10-second audio signal emitted by a female.

[0003] However, there may be several speakers in the audio signal to be analyzed, and in this case, the quality of the output results from the audio separation model may be low. For example, audio signals of different speakers may be combined with each other. In addition, the audio signal may include audio from more than two speakers.

[0004] Furthermore, even in a scenario with only two speakers, when the audio signal is segmented into short time segments and output by an audio separation model, the order of the speakers in the output result may be irregular, and thus audio signals of different speakers may be combined with each other.

[0005] Furthermore, in the event that the audio separation model separates an audio signal that includes more speakers than the number of speakers that can be separated is permissible, the quality of the output results from the audio separation model may be low. Summary of the invention

[0006] Technical Solution

[0007] The present disclosure provides an electronic device and a control method thereof which can divide audio information into a plurality of audio information including an overlapping area and acquire audio data for each audio source from the divided audio information.

[0008] According to one aspect of the present disclosure, an electronic device is provided, comprising: a memory configured to store at least one instruction; and at least one processor configured to execute the at least one instruction to perform the following operations: segmenting audio data into a plurality of parts of audio data based on a plurality of time periods having a plurality of overlapping areas, acquiring an audio feature from each of the plurality of parts, identifying a first audio source and a second audio source in each of the plurality of parts based on the audio feature, and acquiring first audio data corresponding to the first audio source and second audio data corresponding to the second audio source from the audio data.

[0009] At least one processor may also be configured to segment the audio data to perform the following operations: obtain a first part corresponding to a first time period among multiple time periods, and obtain a second part corresponding to a second time period among the multiple time periods, wherein the second time period is different from the first time period, and wherein the first part and the second part include overlapping areas among multiple overlapping areas.

[0010] The at least one processor may also be configured to identify a plurality of audio blocks based on the first audio source or the second audio source in each of a plurality of time periods; obtain first audio data by combining two or more first audio blocks corresponding to the first audio source from the plurality of audio blocks with each other; and obtain second audio data by combining two or more second audio blocks corresponding to the second audio source from the plurality of audio blocks with each other.

[0011] The at least one processor may be further configured to identify a plurality of overlapping regions in the plurality of audio blocks, and to obtain the first audio data and the second audio data by comparing the plurality of overlapping regions with each other.

[0012] At least one processor may also be configured to: obtain similarities between multiple overlapping areas in multiple audio blocks, obtain first audio data by combining two or more first audio blocks whose similarities are greater than or equal to a first threshold; and obtain second audio data by combining two or more second audio blocks whose similarities are greater than or equal to the first threshold.

[0013] The at least one processor may be further configured to expand at least one overlapping region among the plurality of overlapping regions when the similarity is less than a first threshold and greater than or equal to a second threshold, and segment the audio data based on the expanded overlapping region.

[0014] The at least one processor may be further configured to obtain a peak value of an audio signal in a first audio block among the multiple audio blocks, obtain an average value of an audio signal included in an overlapping region of the first audio block, and identify whether the overlapping region is silent based on a difference between the peak value and the average value.

[0015] The at least one processor may be further configured to identify a number of audio sources included in the audio data, and segment the audio data when the number of identified audio sources is less than a threshold value.

[0016] The at least one processor may be further configured to segment the audio data when a duration of the audio data is greater than or equal to a threshold.

[0017] At least one processor may also be configured to identify a number of audio sources in the audio data based on a first artificial intelligence model stored in the memory, and to obtain the first audio data and the second audio data based on a second artificial intelligence model stored in the memory, the first artificial intelligence model being different from the first artificial intelligence model.

[0018] According to another aspect of the present disclosure, a control method of an electronic device is provided, the method comprising: dividing audio data into a plurality of parts of audio data based on a plurality of time periods having a plurality of overlapping areas; acquiring an audio feature from each of the plurality of parts; identifying a first audio source and a second audio source in each of the plurality of parts based on the audio feature; and acquiring first audio data corresponding to the first audio source and second audio data corresponding to the second audio source from the audio data.

[0019] Segmenting the audio data may include: obtaining a first part corresponding to a first time period among multiple time periods, and obtaining a second part corresponding to a second time period among the multiple time periods, wherein the second time period is different from the first time period, and wherein the first part and the second part include an overlapping area among multiple overlapping areas.

[0020] The method may also include identifying multiple audio blocks based on the first audio source or the second audio source in each of the multiple time periods, wherein obtaining the first audio data and the second audio data includes: obtaining the first audio data by combining two or more first audio blocks corresponding to the first audio source from the multiple audio blocks with each other; and obtaining the second audio data by combining two or more second audio blocks corresponding to the second audio source from the multiple audio blocks with each other.

[0021] Acquiring the first audio data and the second audio data may include identifying a plurality of overlapping regions in the plurality of audio blocks, and acquiring the first audio data and the second audio data by comparing the plurality of overlapping regions with each other.

[0022] The method may also include: obtaining similarities between multiple overlapping areas in multiple audio blocks, wherein obtaining the first audio data and the second audio data includes: obtaining the first audio data by combining two or more first audio blocks whose similarities are greater than or equal to a first threshold; and obtaining the second audio data by combining two or more second audio blocks whose similarities are greater than or equal to the first threshold. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a block diagram illustrating an electronic device according to an example embodiment of the present disclosure.

[0024] Figure 2 It is shown Figure 1A block diagram of the specific configuration of an electronic device.

[0025] Figure 3 It is a view for explaining an operation of identifying a plurality of audio sources from one audio information.

[0026] Figure 4 is a flow chart illustrating an operation of acquiring audio data corresponding to each of a plurality of audio sources.

[0027] Figure 5 is a flowchart illustrating an operation of analyzing an audio source by segmenting audio information.

[0028] Figure 6 is a flowchart illustrating an operation of acquiring audio data corresponding to an audio source by combining audio blocks with each other.

[0029] Figure 7 is a view for illustrating an operation of dividing audio information to separate a plurality of audio sources from each other.

[0030] Figure 8 is a view for showing segmented audio blocks.

[0031] Fig. 9 is a view for illustrating an operation of dividing audio information until a predetermined number of audio sources are identified.

[0032] Fig.10 is a flow chart illustrating an operation of acquiring audio data corresponding to each of a plurality of audio sources.

[0033] Fig.11 is a flow chart illustrating an operation of dividing audio information by identifying the number of audio sources.

[0034] Fig.12 is a flowchart for illustrating an operation of considering a minimum time of audio information to be segmented.

[0035] Fig.13 is a flow chart illustrating the operation of not performing analysis when the number of audio sources exceeds an acceptable range.

[0036] Fig.14 is a flowchart illustrating an operation of acquiring audio data corresponding to an audio source by combining audio blocks with each other.

[0037] Fig.15 is a flowchart showing an operation of adding a silent block when acquiring audio data.

[0038] Fig.16 is a view for illustrating an operation of acquiring similarity between audio blocks.

[0039] Fig.17is a view for illustrating an operation of acquiring audio data corresponding to each audio source by dividing audio information.

[0040] Fig.18 is a view for illustrating an overlapping area in an operation of acquiring similarity.

[0041] Fig.19 is a view for illustrating an operation of combining audio blocks with each other based on similarity.

[0042] Fig. 20 is a flowchart for illustrating an operation of acquiring similarities between audio blocks.

[0043] Fig.21 is a flowchart for illustrating a specific example of an operation of acquiring similarities between audio blocks.

[0044] Fig. 22 is a view for illustrating an operation of analyzing an audio block in various examples.

[0045] Fig.23 is a flow chart illustrating an operation of analyzing audio features based on whether an overlapping area is silent.

[0046] Fig.24 is a flowchart illustrating a specific example of an operation of analyzing audio features based on whether an overlapping area is silent.

[0047] Fig.25 is a flow chart illustrating an operation of analyzing audio features based on whether an overlapping area of ​​another time period is silent.

[0048] Fig.26 is a flowchart showing a specific example of an operation of analyzing audio features based on whether an overlapping area of ​​another period is silent.

[0049] Fig. 27 is a flow chart illustrating the operation of increasing the overlap area.

[0050] Fig.28 is a flow chart illustrating the operation of analyzing audio features.

[0051] Fig.29 is a view for illustrating an operation of determining whether an overlapping area is silent in various examples.

[0052] Fig.30 is a view for illustrating an operation of determining whether an overlapping area is silent in various examples.

[0053] Fig.31 is a view for illustrating an operation of acquiring similarity.

[0054] Fig.32is a view for illustrating a user interface (UI) related to an audio extraction operation in various examples.

[0055] Fig.33 is a view for illustrating an operation of displaying a UI for changing a volume of separated audio data at a position corresponding to a specific object.

[0056] Fig.34 is a flowchart illustrating an operation of displaying a UI for changing the volume of separated audio data at a position corresponding to a specific object.

[0057] Fig.35 is a flowchart for illustrating a control method of an electronic device in various embodiments. DETAILED DESCRIPTION

[0058] Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings.

[0059] Considering the functions in the present disclosure, currently widely used general terms are selected as the terms used in the embodiments of the present disclosure, and may be changed based on the intention of those skilled in the art or the emergence of judicial precedents, new technologies, etc. In addition, in certain cases, there may be terms arbitrarily selected by the applicant. In this case, the meanings of these terms will be mentioned in detail in the corresponding description parts of the present disclosure. Therefore, the terms used in the present disclosure need to be defined based on the meanings of the terms and the overall content of the present disclosure, rather than the simple names of the terms.

[0060] In the present disclosure, expressions such as “having”, “may have”, “including”, “may include”, etc. indicate the existence of corresponding features (eg, values, functions, operations, or components (eg, parts)), and do not exclude the existence of additional features.

[0061] The expression “at least one of A or / and B” may mean “A or B” or “both A and B”.

[0062] The expressions "first", "second", etc. used in the present disclosure may modify various components regardless of the order and / or importance of the components. These expressions are only used to separate one component from other components, and do not limit the corresponding components.

[0063] In the case where any component (e.g., a first component) is mentioned as being "coupled (operably or communicatively) with" / "coupled to" or "connected to" another component (e.g., a second component), it will be understood that the arbitrary component may be directly coupled to the other component or may also be coupled to the other component through yet another component (e.g., a third component).

[0064] Unless the context clearly indicates otherwise, terms in the singular may include their plural forms. It will be understood that the terms "comprising" or "formed by" used in this specification specify the presence of the features, numbers, steps, operations, components, parts, or combinations thereof mentioned in this specification, and do not exclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0065] In the present disclosure, a "module" or "device" may perform at least one function or operation and may be implemented by hardware or software, or by a combination of hardware and software. In addition, multiple "modules" or multiple "devices" may be integrated into at least one module and implemented by at least one processor (not shown), except for "modules" or "devices" that need to be implemented by specific hardware.

[0066] In the specification, terms such as "user" may refer to a person using an electronic device or a device (e.g., an artificial intelligence electronic device) using the electronic device.

[0067] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0068] Figure 1 is a block diagram illustrating an electronic device 100 according to an embodiment of the present disclosure.

[0069] Reference Figure 1 , the electronic device 100 may include a memory 110 and at least one processor 120. For example, the electronic device 100 may include one or more processors configured to perform one or more operations of the electronic device 100.

[0070] The electronic device 100 may be a device that analyzes audio data. For example, the electronic device 100 may be a device that includes an artificial intelligence model that receives audio data and separates an audio signal for each audio source.

[0071] The memory 110 may store one or more instructions or audio data. According to an example embodiment, the memory 110 may store one or more instructions and audio data. According to another example embodiment, the memory 110 may store other data in addition to the one or more instructions and audio data.

[0072] The at least one processor 120 may perform overall control operations of the electronic device 100. In detail, the at least one processor 120 may be used to control overall operations of the electronic device 100. The at least one processor 120 may perform the following operations by executing one or more instructions stored in the memory 110.

[0073] At least one processor 120 may segment the audio data into a plurality of time periods to include overlapping areas, obtain audio features from each of the segmented time periods, identify a first audio source and a second audio source in each of the plurality of segmented time periods based on the audio features, and obtain first audio data corresponding to the first audio source and second audio data corresponding to the second audio source from the audio data.

[0074] Here, audio data may be referred to as audio information or audio files. The segmented audio data may be referred to as segmented data (e.g., first segmented data and second segmented data). Therefore, a plurality of time periods may indicate time periods in which audio data is segmented based on a predetermined criterion. In addition, segmented data may indicate audio data corresponding to the segmented time periods.

[0075] Here, the overlapping area can indicate Fig.18 Overlapping periods or time periods shown.

[0076] According to an example embodiment, the at least one processor 120 may segment the audio data stored in the memory 110 at a predetermined time based on the overlapping area, obtain audio features based on the segmented audio data, identify a first audio source and a second audio source from the segmented audio data based on the audio features, and obtain first audio data corresponding to the first audio source and second audio data corresponding to the second audio source.

[0077] According to another example embodiment, at least one processor 120 may segment the audio data into multiple segments of audio data based on multiple overlapping time periods, obtain audio features based on the multiple segments of audio data, identify a first audio source and a second audio source from the multiple segments of audio data based on the audio features, and obtain first audio data corresponding to the first audio source and second audio data corresponding to the second audio source.

[0078] The operation of acquiring the audio feature based on the segmented audio data may indicate an operation of acquiring a standard square error of an overlapping period based on the segmented audio data.

[0079] The operation of identifying the first audio source and the second audio source from the segmented audio data based on the audio features may include identifying the first audio source and the second audio source from the segmented audio data based on the standard square error for each time period. The at least one processor 120 may acquire audio data (or original information). The audio data may include human speech and sounds produced by non-human objects. Humans or objects that make sounds may be referred to as audio sources. The audio data may indicate sound information.

[0080] According to an example embodiment, at least one processor 120 may segment the received audio data. For example, at least one processor 120 segments the audio data based on time. According to an example embodiment, the time may be a predetermined time that may indicate a time corresponding to a threshold ratio of the total time or a unit time. For example, the predetermined time may indicate a time corresponding to a threshold ratio of the total time of the audio data or a unit time.

[0081] According to an example embodiment, the threshold ratio of the total time may be 50%, and the at least one processor 120 may segment the audio data based on 50% of the total time. For example, in the case where the audio data includes an 8-second audio signal, the at least one processor 120 may segment the 8-second audio signal into two audio signals based on 4 seconds.

[0082] The unit time may indicate a unit time predetermined by a user or a predetermined unit time applied to an artificial intelligence model. The at least one processor 120 may segment the audio data based on the unit time. For example, the unit time may be 2 seconds, and the at least one processor 120 may segment an 8-second audio signal into four 2-second audio signals. However, the unit time of 2 seconds is only an example, and therefore, according to another exemplary embodiment, the unit time may be different from 2 seconds, and / or the unit time may be changed by a user setting or an artificial intelligence model to be applied.

[0083] Reference Figure 7 , Figure 8 and Fig. 9 Describes a detailed description related to the classification operation.

[0084] According to an example embodiment, at least one processor 120 may acquire audio features based on the segmented audio signal (or segmented audio data). The audio features may include waveforms, frequencies, amplitudes, etc. of the audio signal. At least one processor 120 may acquire (or extract) audio features based on the segmented audio signal (or segmented audio data).

[0085] According to an example embodiment, at least one processor 120 may input the audio data as input data to the artificial intelligence model. According to an example embodiment, the artificial intelligence model may be a deep learning model. Then, at least one processor 120 may obtain audio features corresponding to the audio data as output data through the artificial intelligence model. Here, the artificial intelligence model may indicate an audio recognition model that analyzes an audio signal.

[0086] In detail, at least one processor 120 may receive audio data (or an audio file). For example, the audio file may include an audio signal of one second.

[0087] In the case where the audio data is analog data, at least one processor 120 may acquire digital data corresponding to the analog data. The digital data may indicate the original data. For example, the digital data may be data representing the sound every 0.001 seconds in digital form.

[0088] According to an example embodiment, at least one processor 120 may segment digital data in units of a predetermined time. For example, one second of digital data may be segmented in units of 25 ms. One second of data may be segmented into 40 time periods. If there are overlapping time periods, 1 second of data may be segmented into more than 40 time periods.

[0089] According to an example embodiment, the at least one processor 120 may acquire a Mel-frequency cepstral coefficient (MFCC) based on a plurality of segmented data (or period data) segmented by a predetermined time unit.

[0090] According to an example embodiment, at least one processor 120 may normalize the MFCC values ​​of the input data for the artificial intelligence model. The artificial intelligence model may have a predetermined range of the input data, and at least one processor 120 may normalize the MFCC values ​​for the predetermined range of the input data.

[0091] According to an example embodiment, at least one processor 120 may input (or apply) the normalized MFCC value as input data to the artificial intelligence model. In addition, at least one processor 120 may obtain audio features (or output data) corresponding to the normalized MFCC value (or input data) from the artificial intelligence model.

[0092] According to an example embodiment, the split audio signal may be mixed with an audio signal output (or generated) from at least one audio source. According to an example embodiment, at least one processor 120 may separate the mixed audio data for each audio source. The separated first audio data and second audio data may be referred to as a first audio signal and a second audio signal.

[0093] According to an example embodiment, at least one processor 120 may identify an audio source based on an audio feature (or extracted audio feature). According to an example embodiment, at least one processor 120 may identify an audio source based on an audio feature extracted or output from an artificial intelligence model. In the case where a plurality of audio sources are included in the segmented audio data, at least one processor 120 may determine a first audio source and a second audio source based on the audio feature.

[0094] According to an exemplary embodiment, the at least one processor 120 may acquire first audio data corresponding to a first audio source from among all audio data (or original information). In addition, the at least one processor 120 may acquire second audio data corresponding to a second audio source from among the audio data (or original information). Thus, the at least one processor 120 may separate an audio signal output from each audio source from the audio data (or original information) in which audio signals output from the first audio source and the second audio source are mixed.

[0095] In addition, the at least one processor 120 may acquire first segmented data of a first period and second segmented data of a second period different from the first period by segmenting the audio data, and the first segmented data and the second segmented data may include an overlapping area.

[0096] For example, at least one processor 120 may segment the audio data based on a predetermined time (e.g., 4 seconds). At least one processor 120 may segment the audio data of the entire time period (e.g., from 0 seconds to 8 seconds) to obtain first segmented data of the first time period (e.g., from 0 seconds to 4 seconds) and second segmented data of the second time period (e.g., from 3.9 seconds to 8 seconds). The overlapping area may be 3.9 seconds to 4 seconds.

[0097] In another example, at least one processor 120 may segment the audio data of the entire period (e.g., from 0 seconds to 8 seconds) to obtain first segmented data of the first period (e.g., from 0 seconds to 4.1 seconds) and second segmented data of the second period (e.g., from 3.9 seconds to 8 seconds). The overlapping area may be 3.9 seconds to 4.1 seconds.

[0098] Reference Fig.18 and Fig.19 Describes a detailed description related to the overlapping area.

[0099] In addition, the at least one processor 120 may identify a plurality of audio blocks segmented based on audio sources in the segmented audio data, obtain first audio data by combining audio blocks corresponding to a first audio source with each other, and obtain second audio data by combining audio blocks corresponding to a second audio source with each other.

[0100] According to an example embodiment, the at least one processor 120 may identify a plurality of audio blocks for each audio source in the segmented audio data. The at least one processor 120 may separate audio signals corresponding to different audio sources from each other without specifying the audio source. The at least one processor 120 may separate audio signals of various audio sources mixed in the audio data.

[0101] According to an example embodiment, the at least one processor 120 may acquire a first audio block and a second audio block having different audio sources from the first segmented data. In addition, the at least one processor 120 may acquire a third audio block and a fourth audio block having different audio sources from the second segmented data. The at least one processor 120 may acquire only a plurality of audio blocks having different audio sources without determining which audio source (e.g., an adult male or car noise) each audio block corresponds to.

[0102] According to an exemplary embodiment, the at least one processor 120 may acquire one audio data by combining highly correlated blocks among the acquired audio blocks with each other. Fig.17 Describe the specific operations related to this.

[0103] In addition, the at least one processor 120 may identify an overlap region included in the plurality of audio blocks, and acquire the first audio data and the second audio data by comparing the identified overlap region with each other.

[0104] According to an exemplary embodiment, the at least one processor 120 may identify an audio signal included in an overlapping region among audio signals included in the audio blocks. The at least one processor 120 may combine audio blocks corresponding to specific audio sources with each other by comparing the audio signals included in the overlapping region. Fig.19 Describe the specific operations related to this.

[0105] In addition, the at least one processor 120 may acquire similarities between overlapping regions included in the plurality of audio blocks, and acquire the first audio data and the second audio data by combining audio blocks respectively having similarities greater than or equal to a first threshold with each other.

[0106] According to an example embodiment, at least one processor 120 may determine whether to combine the audio block acquired in the first time period and the audio block acquired in the second time period with each other based on the similarity between the overlapping regions. According to an example embodiment, at least one processor 120 may compare the similarity between the overlapping regions included in the plurality of audio blocks. The operation of acquiring and comparing the similarity between the overlapping regions may be performed on audio blocks having adjacent time periods.

[0107] For example, when the first audio block and the second audio block are acquired within a period of 0 seconds to 2 seconds and the third audio block and the fourth audio block are acquired within a period of 2 seconds to 4 seconds, the at least one processor 120 may determine whether to combine the first audio block with the third audio block or the fourth audio block. In addition, the at least one processor 120 may determine whether to combine the second audio block with the third audio block or the fourth audio block.

[0108] According to an exemplary embodiment, when the similarity is greater than or equal to the first threshold, the at least one processor 120 may combine the audio blocks used to obtain the similarity with each other. In addition, the at least one processor 120 may obtain audio data of the entire time period by repeating the operation of combining the audio blocks with each other.

[0109] Furthermore, when the similarity is less than the first threshold and greater than or equal to the second threshold, the at least one processor 120 may expand the overlapping area and segment the audio data based on the expanded overlapping area.

[0110] The first threshold may be greater than the second threshold.

[0111] According to an example embodiment, when the similarity is greater than or equal to the first threshold, the electronic device 100 may combine all audio blocks compared for acquiring the similarity with each other.

[0112] According to an exemplary embodiment, when the similarity is less than the first threshold and greater than or equal to the second threshold, the electronic device 100 may re-segment the audio data by expanding (or increasing) the overlapping area. This operation is to re-evaluate the similarity comparison by increasing the overlapping area. Fig. 27 A detailed description thereof is described. In the case where the overlapping area is extended, more accurate comparison can be performed even if the processing time increases.

[0113] According to an exemplary embodiment, when the similarity is less than the second threshold, the electronic device 100 may acquire an audio feature for the target audio block. Here, the at least one processor 120 determines that no more audio blocks are to be connected.

[0114] In addition, the at least one processor 120 may obtain a peak value of an audio signal included in a first audio block among the audio blocks, obtain an average value of an audio signal included in an overlapping region of the first audio block, and identify whether the overlapping region is silent based on a difference between the peak value and the average value.

[0115] According to an example embodiment, at least one processor 120 may determine whether the overlapping area included in the audio block is silent. Even if the audio signal is separated for each audio source, noise may exist and errors may occur during the separation process. Therefore, at least one processor 120 may determine whether the overlapping area is silent by a specific method.

[0116] According to an example embodiment, the at least one processor 120 may determine whether the overlapping area is silent based on the absolute value of the audio signal. In the case where the average value of the audio signal included in the overlapping area is less than or equal to the threshold, the at least one processor 120 may determine that the overlapping area is silent.

[0117] According to an example embodiment, at least one processor 120 may obtain a peak value of an audio signal within the entire time period of an audio block. At least one processor 120 may obtain an average value of an audio signal included in an overlapping region. At least one processor 120 may determine whether the overlapping region is silent by comparing the peak value and the average value with each other. In the case where the difference between the peak value and the average value is greater than or equal to a threshold value, at least one processor 120 may determine that the overlapping region is silent. The average value may be less than the peak value, and the difference between the peak value and the average value being greater than the threshold value may indicate a situation where the average value is relatively significantly lower than the peak value. Compared with a method of only determining whether the average value is lower than a threshold value, a method of determining whether an overlapping region is silent using a peak value may have higher accuracy.

[0118] Reference Fig.29 and Fig.30 Describes the detailed description of using peak and average values.

[0119] In addition, the at least one processor 120 may identify the number of audio sources included in the audio data, and split the audio data if the number of identified audio sources is less than a threshold value. Fig.11 A detailed description describing this operation.

[0120] In addition, the at least one processor 120 may split the audio data if the time of the audio data is greater than or equal to the threshold time. Fig.12 and Fig.13 A detailed description describing this operation.

[0121] In addition, at least one processor 120 may identify the number of audio sources included in the audio data based on a first artificial intelligence model stored in the memory 110, and obtain the first audio data and the second audio data based on a second artificial intelligence model stored in the memory 110 and different from the first artificial intelligence model.

[0122] The first artificial intelligence model may be a model that receives audio data as input data and obtains the number of audio sources in the audio data as output data. The at least one processor 120 may input the segmented audio data to the first artificial intelligence model.

[0123] According to an example embodiment, the at least one processor 120 may identify only the number of audio sources in the audio data through the first artificial intelligence model. However, the at least one processor 120 may not be able to specify the audio source. In this case, an additional operation for extracting and analyzing audio features may be implemented to specify or identify each of the audio sources.

[0124] When the number of audio sources is greater than or equal to the threshold, the accuracy of the audio analysis operation may be lower. Therefore, the first artificial intelligence model may determine whether the audio data can be used as input data for the audio analysis operation.

[0125] According to an example embodiment, a second artificial intelligence model may be provided. The second artificial intelligence model may be a model that receives audio data as input data and separates an audio signal included in the audio data for each audio source. The at least one processor 120 may input the segmented audio data to the second artificial intelligence model.

[0126] According to an example embodiment, at least one processor 120 may separate the audio signal for each audio source through a second artificial intelligence model. At least one processor 120 may obtain a plurality of audio blocks from the segmented audio data, and combine the plurality of audio blocks for each audio source based on the similarity between overlapping regions. In addition, at least one processor 120 may extract audio features of the combined audio data (or a plurality of audio blocks) for each audio source. In addition, at least one processor 120 may specify the audio source of the combined audio data based on the audio features.

[0127] The first artificial intelligence model may perform operations that determine input data to be applied (or input) to the second artificial intelligence model.

[0128] The at least one processor 120 may separate the audio signal for each audio source through the second artificial intelligence model.

[0129] In addition, the above description describes that the first artificial intelligence model and the second artificial intelligence model are different from each other according to the example embodiment. However, the present disclosure is not limited thereto, and according to another example embodiment, one artificial intelligence model may perform all of the above operations. According to the example embodiment, the first software module and the second software module may be included in one artificial intelligence model. The first software module may identify the number of audio sources included in the audio data, and the second software module may separate the audio signal for each audio source.

[0130] According to various exemplary embodiments, the electronic device 100 may acquire audio data including audio signals output from various audio sources. In addition, the electronic device 100 may extract an audio signal for each audio source from the audio data. When extracting the audio signal for each audio source, the divided audio blocks may be combined with each other using an overlapping area. When the overlapping area is used, the audio combination process may have higher accuracy.

[0131] Furthermore, when determining the similarity between the overlapping regions, errors occurring in the combining process may be additionally corrected by expanding the overlapping regions. Thus, the result of separating the audio signals may have improved quality.

[0132] Furthermore, even when the quality of audio data or an audio block is low, by segmenting the audio data or an audio block in consideration of the number of audio sources, a threshold time, or other parameters, an audio source may be identified with high accuracy.

[0133] In addition, the above description only shows and describes a simple configuration of the electronic device 100, and various configurations may be provided when it is implemented. Figure 2 Describe the situation.

[0134] Figure 2 It is used to show Figure 1 A block diagram of a specific configuration of an electronic device 100.

[0135] Reference Figure 2 , the electronic device 100 may include at least one of a memory 110 , at least one processor 120 , a communication interface 130 , a display 140 , an operation interface 150 , an input / output interface 160 , a speaker 170 , or a microphone 180 .

[0136] Furthermore, this description omits redundant descriptions of operations that are the same as those described above, among operations of the electronic device 100 .

[0137] In the case where the electronic device 100 is implemented as a server, the electronic device 100 may receive audio data from an external device through the communication interface 130 .

[0138] In the case where the electronic device 100 is implemented as a television (TV), the electronic device 100 may include a display 140 and an input / output interface 160 .

[0139] In case that the electronic device 100 corresponds to an artificial intelligence (AI) speaker, the electronic device 100 may include the speaker 170 .

[0140] In the case where the electronic device 100 analyzes audio data collected in real time, the electronic device 100 may include the microphone 180 .

[0141] The electronic device 100 in various examples of the present specification may include, for example, a smartphone, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader, a desktop PC, a laptop PC, a notebook computer, a workstation, a server, and a personal digital assistant (PDA), a portable multimedia player (PMP), an MP3 player, a medical device, a camera, or at least one of a wearable device. The wearable device may include at least one of an accessory type (e.g., a watch, a ring, a bracelet, an anklet, a necklace, glasses, contact lenses, or a head-mounted device (HMD), a textile or clothing integral type (e.g., an electronic garment), a body attachment type (e.g., a skin pad or a tattoo), or a living implant type circuit. In some examples, the electronic device may include, for example, a television, a digital video disc (DVD) player, an audio player, a refrigerator, an air conditioner, a vacuum cleaner, an oven, a microwave oven, a washing machine, an air purifier, a set-top box, a home automation control panel, a security control panel, a media box (e.g., Samsung HomeSync TM , Apple TV TM or Google TV TM ), game consoles (e.g., Xbox TM or PlayStation TM ), at least one of an electronic dictionary, an electronic key, a camera, or an electronic photo frame.

[0142] The memory 110 may be implemented as an internal memory included in at least one processor 120, such as a read-only memory (ROM) (e.g., an electrically erasable programmable read-only memory (EEPROM)) or a random access memory (RAM), or may be implemented as a memory separate from the at least one processor 120. In this case, based on the data storage purpose, the memory 110 may be implemented in the form of a memory embedded in the electronic device 100, or may be implemented in the form of a memory detachable from the electronic device 100. For example, data for driving the electronic device 100 may be stored in the memory embedded in the electronic device 100, and data for extended functions of the electronic device 100 may be stored in the memory detachable from the electronic device 100.

[0143] In addition, the memory embedded in the electronic device 100 can be implemented as a volatile memory (for example, dynamic RAM (DRAM), static RAM (SRAM) or synchronous dynamic RAM (SDRAM)), a non-volatile memory (for example, a one-time programmable ROM (OTPROM), a programmable ROM (PROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM)), a mask ROM, a flash ROM, a flash memory (for example, NAND flash memory, NOR flash memory, etc.), a hard disk drive or a solid-state drive (SSD); and the memory that can be removed from the electronic device 100 can be implemented as a memory card (for example, compact flash (CF), secure digital (SD), micro secure digital (Micro-SD), mini secure digital (mini-SD), extreme digital (xD), multimedia card (MMC)), etc.), an external memory that can be connected to a universal serial bus (USB) port (for example, a USB memory), etc.

[0144] At least one processor 120 may be implemented as a digital signal processor (DSP), a microprocessor, or a time controller (TCON) for processing digital signals. However, the processor 120 is not limited thereto, and may include one or more of a central processing unit (CPU), a microcontroller unit (MCU), a microprocessing unit (MPU), a controller, an application processor (AP), a graphics processing unit (GPU), a communication processor (CP), or an advanced reduced instruction set computer (RISC) machine (ARM) processor, or may be defined by these terms. In addition, at least one processor 120 may be implemented as a system on chip (SoC) or a large-scale integrated circuit (LSI) in which a processing algorithm is embedded, or may be implemented as a field programmable gate array (FPGA). In addition, at least one processor 120 may perform various functions by running computer executable instructions stored in the memory 110.

[0145] The communicator 130 is a component that communicates with various types of external servers in various types of communication modes. The communication interface 130 may include a wireless communication module or a wired communication module. Here, each communication module may be implemented in the form of at least one hardware chip.

[0146] The wireless communication module may be a module for wirelessly communicating with an external device. For example, the wireless communication module may include at least one of a wireless fidelity (Wi-Fi) module, a Bluetooth module, an infrared communication module, or other communication modules.

[0147] The Wi-Fi module and the Bluetooth module can perform communication in a Wi-Fi manner and a Bluetooth manner, respectively. When using the Wi-Fi module or the Bluetooth module, various connection information (such as a service set identifier (SSID)) or a session key may be first sent and received, the communication may be connected using this connection information, and then various information may be sent and received.

[0148] The infrared communication module may perform communication based on an Infrared Data Association (IrDA) technology that wirelessly transmits data within a short distance using infrared rays between visible waves and millimeter waves.

[0149] In addition to the above-mentioned communication methods, other communication modules may include at least one communication chip for performing communication based on various wireless communication standards (such as Zigbee, third generation (3G), third generation partnership project (3GPP), long term evolution (LTE), advanced LTE (LTE-A), fourth generation (4G) and fifth generation (5G)).

[0150] The wired communication module may be a module for communicating with an external device through a wire. For example, the wired communication module may include at least one of a local area network (LAN) module, an Ethernet module, a twisted pair cable, a coaxial cable, an optical fiber cable, or an ultra-wideband (UWB) module.

[0151] The display 140 may be implemented in various types of displays such as a liquid crystal display (LCD), an organic light emitting diode (OLED) display, and a plasma display (PDP). The display 140 may include a driving circuit, a backlight unit, and the like, which may be implemented in the form of, for example, an amorphous silicon thin film transistor (a-si TFT), a low temperature polycrystalline silicon (LTPS) TFT, or an organic TFT (OTFT). In addition, the display 140 may be implemented as a touch screen combined with a touch sensor, a flexible display, a three-dimensional (3D) display, and the like. In addition, according to an exemplary embodiment of the present disclosure, the display 140 may include not only a display panel that outputs an image, but also a frame that accommodates the display panel. In particular, according to an exemplary embodiment of the present disclosure, the frame may include a touch sensor (not shown) that detects user interaction.

[0152] The operation interface (150) may be implemented as a device such as a button, a touch panel, a mouse or a keyboard, or may be implemented as a touch screen that can perform an operation input function in addition to the above-mentioned display function. Here, the button may be any one of various types of buttons (such as a mechanical button, a touch panel or a scroll wheel), which is located in any area (such as the front surface part, the side surface part or the rear surface part of the main body appearance of the electronic device 100).

[0153] The input / output interface 160 may be any one of a high-definition multimedia interface (HDMI), a mobile high-definition link (MHL), a universal serial bus (USB), a display port (DP), a thunderbolt interface, a video graphics array (VGA) port, a red, green, and blue (RGB) port, a D-SUB, or a digital video interface (DVI). The input / output interface 160 may input / output at least one of an audio signal or a video signal. In some examples, the input / output interface 160 may include a port for inputting and outputting only an audio signal and a port for inputting and outputting only a video signal as separate ports thereof, or may be implemented as a single port for inputting and outputting both an audio signal and a video signal. In addition, the electronic device 100 may send at least one of an audio signal and a video signal to an external device (e.g., an external display device or an external speaker) through the input / output interface 160. In detail, an output interface included in the input / output interface 160 may be connected to an external device, and the electronic device 100 may send at least one of an audio signal and a video signal to an external device through the output port.

[0154] Here, the input / output interface 160 may be connected to the communication interface. The input / output interface 160 may transmit information received from an external device to the communication interface or transmit information received through the communication interface to the external device.

[0155] The speaker 170 may be a component that outputs not only various audio data but also various notification sounds or voice messages.

[0156] The microphone 180 may be a component that receives a user's voice or other sounds and converts the user's voice or other sounds into audio data. The microphone 180 may receive the user's voice when activated. For example, the microphone 180 may be formed integrally with the electronic device 100 above, in front of, or to the side of the electronic device 100. The microphone 180 may include various components, such as a microphone that collects the user's voice in analog form, an amplifier circuit that amplifies the collected user's voice, an analog-to-digital (A / D) conversion circuit that samples the amplified user's voice and converts it into a digital signal, and a filter circuit that removes noise components from the converted digital signal.

[0157] Figure 3 It is a view used to explain the operation of identifying multiple audio sources from one audio data.

[0158] Reference Figure 3Item 310, the audio data of aaa.mp3 may include audio data corresponding to two audio sources. For example, it can be assumed that the audio sources are all humans. Here, aaa.mp3 may be an audio file in which two voices are mixed with each other. However, the present disclosure is not limited to this, and therefore, an audio file with more than two voices or more than two sounds may be provided. The electronic device 100 may obtain audio data corresponding to each audio source by analyzing one audio data. The electronic device 100 may obtain audio data 311 including only the voice of the first speaker (or the first channel) and audio data 312 including only the voice of the second speaker (or the second channel).

[0159] Reference Figure 3 Item 320 of the electronic device 100 may analyze the audio data of bbb.mp3 to obtain audio data 321 including only the voice of the first speaker (or the first channel) and audio data 322 including only the voice of the second speaker (or the second channel). Figure 3 As shown, the characteristics of the audio data of aaa.mp3 and bbb.mp3 may be different.

[0160] Figure 4 is a flow chart illustrating an operation of acquiring audio data corresponding to each of a plurality of audio sources.

[0161] Reference Figure 4 , the electronic device 100 may acquire audio data (S410). The audio data may indicate information including audio or sound. The audio data may be referred to as audio content, audio file, etc.

[0162] The electronic device 100 may identify a plurality of audio sources included in the audio data (S420). An audio source may indicate a source of an audio signal. For example, an audio source of a user's voice may be a user. In addition, an audio source of an exhaust sound of a car may be a car. An audio source may be referred to as a sound source, an audio object, an audio channel, etc.

[0163] The electronic device 100 may acquire audio data corresponding to each of the plurality of audio sources (S430). The electronic device 100 may acquire audio data corresponding to the plurality of audio sources by analyzing an audio signal included in the audio data. For example, the electronic device 100 may acquire first audio data emitted by a first speaker (or a first channel) and second audio data emitted by a second speaker (or a second channel) based on one audio data.

[0164] Figure 5 is a flowchart illustrating an operation of analyzing an audio source by segmenting audio data.

[0165] Reference Figure 5, the electronic device 100 may acquire the audio data (S510). Then, the electronic device 100 may segment the audio data (S520). In the case where multiple audio sources are combined with each other in one audio data, the analysis accuracy may be low. Therefore, the electronic device 100 may segment the audio data into multiple pieces to analyze the audio sources in detail.

[0166] The electronic device 100 may analyze (or identify) an audio source based on the segmented audio data (S530). According to an example embodiment, the electronic device 100 may analyze each of the segmented audio data and identify the audio source based on the analysis. According to an example embodiment, identifying the audio source may include determining the source of the audio, but is not limited thereto. For example, the electronic device 100 may identify a person or object that generates the audio or sound.

[0167] After completing the analyzing operation, the electronic device 100 may combine the divided audio data into one audio data (S540). The electronic device 100 may determine the audio source based on the divided audio data and perform the combining operation to convert the audio data into one data format.

[0168] Figure 6 is a flowchart illustrating an operation of acquiring audio data corresponding to an audio source by combining audio blocks with each other.

[0169] Reference Figure 6 , the electronic device 100 may acquire audio data (S610). The electronic device 100 may segment the audio data (S620). The electronic device 100 may identify an audio source based on the segmented audio data (S630). The electronic device 100 may analyze an audio signal by analyzing each of the segmented audio data, and identify at least one audio source based on the analysis result.

[0170] The electronic device 100 may obtain an audio block corresponding to an audio source based on each of the segmented audio data (S640). For example, the electronic device 100 may segment 10 seconds of audio data into 10 audio data in units of 1 second. However, the present disclosure is not limited thereto, and therefore, according to another exemplary embodiment, the audio data may have a duration different from 10 seconds and / or the audio data may be segmented in units different from 1 second. The electronic device 100 may obtain an audio block for each audio source from the first segmented data in units of 1 second. The electronic device 100 may determine which audio source each of the plurality of audio blocks corresponds to.

[0171] The electronic device 100 may combine the audio blocks within the entire period with each other based on the audio source (S650). For example, the electronic device 100 may combine a plurality of audio blocks obtained from each audio data in units of 1 second within a period of 10 seconds (S650).

[0172] The electronic device 100 may obtain audio data corresponding to the audio source (S660). In a scenario where there are two audio sources, the electronic device 100 may obtain two audio data. For example, the electronic device 100 may obtain 10 seconds of first audio data emitted by the first speaker (or the first channel) and 10 seconds of second audio data emitted by the second speaker (or the second channel).

[0173] Figure 7 is a view for illustrating an operation of dividing audio data to separate a plurality of audio sources from each other.

[0174] According to an example embodiment, Figure 7 , item 710 shows an audio block corresponding to each of a plurality of audio sources (e.g., audio source 1, audio source 2, audio source 3, and audio source 4). However, the present disclosure is not limited thereto, and therefore, a different number of audio sources may be provided. For example, item 710 shows that the first audio source 1 outputs audio from 0 seconds to 3 seconds, the second audio source 2 outputs audio from 2 seconds to 5 seconds, the third audio source 3 outputs audio from 4 seconds to 8 seconds, and the fourth audio source 4 outputs audio from 0 seconds to 2 seconds and from 6.5 seconds to 8 seconds.

[0175] exist Figure 7 , table 720 shows the process of segmenting the audio data in item 710. It can be assumed that the original audio data is 8 seconds, but the present disclosure is not limited thereto. The electronic device 100 can identify four audio sources in the entire period from 0 seconds to 8 seconds. In the case where it is difficult to segment the four audio sources in one analysis, the electronic device 100 can perform a classification operation.

[0176] Therefore, the electronic device 100 may divide the 8-second audio data into 4-second units (a first classification step (or operation)). The electronic device 100 may obtain audio data corresponding to the first group from 0 seconds to 4 seconds and audio data corresponding to the second group from 4 seconds to 8 seconds. The electronic device 100 may identify three audio sources within the period from 0 seconds to 4 seconds, and identify three audio sources within the period from 4 seconds to 8 seconds. In the case where it is difficult to divide the three audio sources in one analysis, the electronic device 100 may perform additional classification operations.

[0177] Therefore, the electronic device 100 may divide the 4-second audio data into 2-second units (second classification (or operation)). The electronic device 100 may obtain audio data corresponding to the 1-1 group from 0 seconds to 2 seconds, audio data corresponding to the 1-2 group from 2 seconds to 4 seconds, audio data corresponding to the 2-1 group from 4 seconds to 6 seconds, and audio data corresponding to the 2-2 group from 6 seconds to 8 seconds. The electronic device 100 may identify two audio sources within a period from 0 seconds to 2 seconds, two audio sources within a period from 2 seconds to 4 seconds, two audio sources within a period from 4 seconds to 6 seconds, and two audio sources within a period from 6 seconds to 8 seconds. In the case where two audio sources can be divided in one analysis, the electronic device 100 may not perform any additional classification operations.

[0178] Figure 8 is a view for showing segmented audio blocks.

[0179] Reference Figure 8 In item 810, the electronic device 100 may obtain audio blocks segmented based on the audio source from the segmented audio data.

[0180] The electronic device 100 may acquire an audio block 811 - 1 corresponding to the first audio source 1 within a period of 0 seconds to 2 seconds, and acquire an audio block 811 - 2 corresponding to the first audio source 1 within a period of 2 seconds to 4 seconds.

[0181] The electronic device 100 may obtain an audio block 812-1 corresponding to the second audio source 2 within a period of 2 seconds to 4 seconds, and obtain an audio block 812-2 corresponding to the second audio source 2 within a period of 4 seconds to 6 seconds.

[0182] The electronic device 100 may acquire the audio block 813 - 1 corresponding to the third audio source 3 within a period of 4 seconds to 6 seconds, and acquire the audio block 813 - 2 corresponding to the third audio source 3 within a period of 6 seconds to 8 seconds.

[0183] The electronic device 100 may obtain an audio block 814 - 1 corresponding to the fourth audio source 4 within a period of 0 seconds to 2 seconds, and obtain an audio block 814 - 2 corresponding to the fourth audio source 4 within a period of 6 seconds to 8 seconds.

[0184] Fig. 9 is a view for illustrating an operation of dividing audio data until a predetermined number of audio sources are identified.

[0185] exist Fig. 9, item 910 shows an audio block corresponding to each of a plurality of audio sources (e.g., audio source 1, audio source 2, audio source 3, and audio source 4). According to an exemplary embodiment, item 910 shows that the first audio source 1 outputs audio from 0 seconds to 3 seconds, the second audio source 2 outputs audio from 3 seconds to 6.2 seconds, the third audio source 3 outputs audio from 4 seconds to 8 seconds, and the fourth audio source 4 outputs audio from 0 seconds to 3 seconds and from 6.5 seconds to 8 seconds.

[0186] According to an example embodiment, Fig. 9 Table 920 shows the process of segmenting the audio data in item 910. It can be assumed that the original audio data is 8 seconds. The electronic device 100 can identify four audio sources in the entire period from 0 seconds to 8 seconds. In the case where it is difficult to segment the four audio sources in one analysis, the electronic device 100 can perform a classification operation.

[0187] Therefore, the electronic device 100 can divide the 8-second audio data into 4-second units (first classification step). The electronic device 100 can obtain audio data corresponding to the first group from 0 seconds to 4 seconds and audio data corresponding to the second group from 4 seconds to 8 seconds. The electronic device 100 can identify three audio sources in the period from 0 seconds to 4 seconds, and identify three audio sources in the period from 4 seconds to 8 seconds. In the case where it is difficult to divide the three audio sources in one analysis, the electronic device 100 can perform additional classification operations.

[0188] Therefore, the electronic device 100 may divide the 4-second audio data into 2-second units (second classification step). The electronic device 100 may obtain audio data corresponding to the 1-1 group from 0 seconds to 2 seconds, audio data corresponding to the 1-2 group from 2 seconds to 4 seconds, audio data corresponding to the 2-1 group from 4 seconds to 6 seconds, and audio data corresponding to the 2-2 group from 6 seconds to 8 seconds. The electronic device 100 may identify two audio sources within a period of 0 seconds to 2 seconds, three audio sources within a period of 2 seconds to 4 seconds, two audio sources within a period of 4 seconds to 6 seconds, and three audio sources within a period of 6 seconds to 8 seconds. In the case where two audio sources can be divided in one analysis, the electronic device 100 may not perform any additional classification operations. In the case where it is difficult to divide three audio sources in one analysis, the electronic device 100 may perform additional classification operations.

[0189] Therefore, the electronic device 100 may divide the 2-second audio data into 1-second units (third classification step). The electronic device 100 may obtain audio data corresponding to the 1-2-1 group from 2 seconds to 3 seconds, audio data corresponding to the 1-2-2 group from 3 seconds to 4 seconds, audio data corresponding to the 2-2-1 group from 6 seconds to 7 seconds, and audio data corresponding to the 2-2-2 group from 7 seconds to 8 seconds. The electronic device 100 may identify two audio sources within a period of 2 seconds to 3 seconds, two audio sources within a period of 3 seconds to 4 seconds, two audio sources within a period of 6 seconds to 7 seconds, and two audio sources within a period of 7 seconds to 8 seconds. In the case where two audio sources can be divided in one analysis, the electronic device 100 may not perform any additional classification operations.

[0190] Fig.10 is a flow chart illustrating an operation of acquiring audio data corresponding to each of a plurality of audio sources.

[0191] Reference Fig.10 , the electronic device 100 can obtain audio data (S1010). Figure 4 A detailed description of the audio data is provided, and thus a redundant description thereof is omitted.

[0192] The electronic device 100 may divide the audio data into a plurality of time periods (S1020). The electronic device 100 may divide the entire audio data into a plurality of time periods based on a predetermined unit time. The divided audio data may be referred to as divided data. The predetermined unit time may be changed based on user settings. However, the present disclosure is not limited thereto, and therefore, the unit time may be determined in another manner.

[0193] The electronic device 100 may acquire audio features based on the segmented audio data (S1030). The electronic device 100 may extract features of an audio signal included in the audio data. The audio features may include a waveform, a frequency, an amplitude, etc. of the audio signal.

[0194] The electronic device 100 may identify the first audio source and the second audio source based on the audio feature (S1040). It may be assumed that the audio data to be analyzed (or the segmented audio data) includes audio signals output by a plurality of audio sources. The electronic device 100 may identify the first audio source and the second audio source based on the audio feature.

[0195] The electronic device 100 may acquire first audio data corresponding to the first audio source and second audio data corresponding to the second audio source (S1050). The electronic device 100 may acquire audio data corresponding to each audio source in one audio data in which audio signals output from a plurality of audio sources are combined with each other. The electronic device 100 may acquire one audio data by combining the first audio data corresponding to the first audio source and the second audio data corresponding to the second audio source with each other. The acquired audio data may be the same / similar to the audio data acquired in the first operation S1010.

[0196] For example, the electronic device 100 may separate first audio data emitted by a first speaker and second audio data emitted by a second speaker from one audio signal. In the case where the first audio data and the second audio data are combined with each other, the first audio data may be acquired.

[0197] Fig.11 is a flow chart illustrating an operation of dividing audio data by identifying the number of audio sources.

[0198] Fig.11 Operations S1110, S1130, S1140, and S1135 may correspond to Fig.10 Therefore, the redundant description thereof is omitted in this specification.

[0199] After acquiring the audio data, the electronic device 100 may identify the number of audio sources (S1121). The electronic device 100 may identify the number of audio sources by using the first artificial intelligence model. The electronic device 100 may identify whether the number of audio sources is less than a threshold number (S1122). The threshold number may indicate a limit value required to accurately analyze the audio source. It may be difficult for the electronic device 100 to analyze audio data including the threshold number or more of audio sources.

[0200] When the number of audio sources is not less than the threshold number (S1122-N), the electronic device 100 may divide the audio data into a plurality of time periods (S1123). When the number of audio sources is greater than or equal to the threshold number, the electronic device 100 may determine that it is difficult to analyze the audio data. Therefore, the electronic device 100 may divide the audio data. For example, the classification criterion may be 50% of the total time period. The electronic device 100 may divide 8 seconds of audio data into 4-second units. Then, the electronic device 100 may perform operations S1121 to S1122 again.

[0201] When the number of audio sources is less than the threshold number (S1122-Y), the electronic device 100 may perform operations S1130 to S1150. When the number of audio sources is less than the threshold number, the electronic device 100 may determine that the audio source can be accurately identified. The electronic device 100 may analyze the audio data by using the second artificial intelligence model. In detail, the electronic device 100 may input the segmented audio data into the second artificial intelligence model. Then, the electronic device 100 may obtain audio data for each audio source from the second artificial intelligence model.

[0202] Fig.12 is a flowchart for illustrating an operation of considering a minimum time of audio data to be divided.

[0203] Fig.12 Operation S1210, operation S1221, operation S1222, operation S1223, operation S1230, operation S1240, and operation S1250 may correspond to Fig.11 Therefore, the redundant description thereof is omitted in this specification.

[0204] After acquiring the audio data, the electronic device 100 may identify whether the time of the audio data is less than a threshold time (S1220). According to an example embodiment, the time of the audio data may correspond to the duration of the audio data. The threshold time may indicate a limited time required to analyze the audio feature. It may be difficult for the electronic device 100 to analyze the audio data including an audio signal less than the threshold time. According to an example embodiment, the threshold time may only indicate a time for prohibiting classification, and the electronic device 100 may analyze an audio signal less than the threshold time.

[0205] When the time of the audio data (or the segmented audio data) is less than the threshold time (S1220-Y), the electronic device 100 may perform operations S1230 to S1250. When the time of the audio data (or the segmented audio data) is less than the threshold time, the electronic device 100 may immediately acquire the audio features without performing any additional classification operations.

[0206] In a case where the time of the audio data (or the divided audio data) is not less than the threshold time (S1220-N), the electronic device 100 may perform operations S1221 to S1223 and operations S1230 to S1250.

[0207] Fig.13 is a flow chart illustrating the operation of not performing analysis when the number of audio sources exceeds an acceptable range.

[0208] Fig.13 Operations S1310, S1321, S1322, S1323, S1330, S1340, and S1350 may correspond to Fig.12 Therefore, the redundant description thereof is omitted in this specification.

[0209] After acquiring the audio data, the electronic device 100 may identify whether the time of the audio data is less than a threshold time ( S1320 ).

[0210] In the case where the time of the audio data is not less than the threshold time (S1320-N), the electronic device 100 may identify the number of audio sources (S1321). The electronic device 100 may identify whether the number of audio sources is less than the first threshold number (S1322). In the case where the number of audio sources is less than the first threshold number (S1322-Y), the electronic device 100 may perform operations S1330 to 1350. In the case where the number of audio sources is not less than the first threshold number (S1322-N), the electronic device 100 may divide the audio data into a plurality of time periods (1323), and then repeat operations S1320 to 1322.

[0211] Fig.12 An example of immediately acquiring audio features when the time of audio data is less than a threshold time is shown.

[0212] In various examples, when the time of the audio data is less than the threshold time (S1320-Y), the electronic device 100 may identify the number of audio sources (S1320-2). The electronic device 100 may identify whether the number of audio sources is less than a second threshold number (S1320-3).

[0213] In the case where the number of audio sources is less than the second threshold number (S1320-3-Y), the electronic device 100 may perform operations S1330 to 1350. In the case where the number of audio sources is not less than the second threshold number (S1320-3-N), the electronic device 100 may not identify the audio source. The reason is that in the case where the audio data includes more audio sources than the allowable number, the accuracy of the analysis may be low. Therefore, the electronic device 100 may not perform the audio source recognition operation on the corresponding part.

[0214] In various examples, the second threshold number may be greater than the first threshold number. The first threshold number may be a value used to determine whether to analyze an audio source in audio data of a suitable (or relatively long) time period. The second threshold number may be a value used to determine whether to analyze an audio source in audio data of an unsuitable (or relatively short) time period. Therefore, the first threshold number may be greater than the second threshold number. In the case where a precise audio analysis operation is required, the electronic device 100 may set the first threshold number to be greater than the second threshold number.

[0215] In various examples, the first threshold quantity may be less than the second threshold quantity. The audio data determined by the second threshold quantity may be audio data having a relatively short time period. For audio data having a shorter time period, no additional classification operations may be performed. Therefore, the electronic device 100 may obtain the analysis result by increasing the allowable threshold. Therefore, the second threshold quantity may be greater than the first threshold quantity. In the case of an audio analysis operation where it is important to obtain result data, the electronic device 100 may set the first threshold quantity to be less than the second threshold quantity.

[0216] In various examples, the first threshold number and the second threshold number may be applied equally. In general, the electronic device 100 may consistently apply the threshold number used to analyze the audio source. Regardless of the time period of the audio data, the number of audio sources may be an important factor when obtaining audio features. Therefore, the electronic device 100 may equally apply the first threshold number and the second threshold number. In the case of an audio analysis operation that requires consistent performance when identifying an audio source, the electronic device 100 may equally set the first threshold number and the second threshold number.

[0217] Fig.14 is a flowchart illustrating an operation of acquiring audio data corresponding to an audio source by combining audio blocks with each other.

[0218] Fig.14 Operation S1410, operation S1420, operation S1430, and operation S1440 may correspond to Fig.10 The electronic device 100 may perform operations S1010, S1020, S1030, and S1040 of the present invention. Therefore, the redundant description thereof is omitted in the present specification. After identifying the first audio source and the second audio source based on the audio features, the electronic device 100 may obtain an audio block corresponding to the first audio source and an audio block corresponding to the second audio source (S1451).

[0219] The electronic device 100 may acquire an audio block for each audio source based on each of the divided audio data.

[0220] The electronic device 100 may obtain the first audio data by combining the audio blocks corresponding to the first audio source in the entire period (or the entire time period) of the audio data (S1452). The electronic device 100 may combine the audio blocks related to the first audio source among the plurality of audio blocks obtained from each of the divided audio data into one data (or the first audio data).

[0221] The electronic device 100 may obtain the second audio data by combining the audio blocks corresponding to the second audio source in the entire period (or the entire time period) of the audio data (S1453). The electronic device 100 may combine the audio blocks related to the second audio source among the plurality of audio blocks obtained from each of the divided audio data into one data (or the second audio data).

[0222] Fig.15 is a flowchart showing an operation of adding a silent block when acquiring audio data.

[0223] According to an example embodiment, Fig.15 , item 1510 shows the Figure 8 The electronic device 100 may obtain a plurality of audio blocks from the segmented audio data, and may obtain an audio feature based on each audio block. In addition, the electronic device 100 may identify an audio source based on the audio feature.

[0224] The electronic device 100 may identify four audio sources in item 810. Then, the electronic device 100 may combine the audio blocks corresponding to each of the four audio sources into one data. For example, the electronic device 100 may combine the audio blocks corresponding to the first audio source in the plurality of audio blocks into one data 1511. The electronic device 100 may combine the audio blocks corresponding to the second audio source in the plurality of audio blocks into one data 1512. The electronic device 100 may combine the audio blocks corresponding to the third audio source in the plurality of audio blocks into one data 1513. The electronic device 100 may combine the audio blocks corresponding to the fourth audio source in the plurality of audio blocks into one data 1514. The data 1514 may be one data whose time is divided into 0 seconds to 2 seconds and 6.5 seconds to 8 seconds. The data 1514 may be data in which no audio signal is recognized between 2 seconds and 6.5 seconds.

[0225] exist Fig.15, item 1520 shows the operation of the electronic device 100 according to an example embodiment, wherein the electronic device 100 may add a silent block (or a silent signal 1521, a silent signal 1522, a silent signal 1523, a silent signal 1524, or a silent signal 1525) to a time period in which an audio signal is not recognized. For example, the electronic device 100 may add a silent block to a time period of 2.5 seconds to 8 seconds in the first audio data. In addition, the electronic device 100 may add a silent block to a time period of 0 seconds to 2 seconds and 4.5 seconds to 8 seconds in the second audio data. In addition, the electronic device 100 may add a silent block to a time period of 0 seconds to 4 seconds in the third audio data. In addition, the electronic device 100 may add a silent block to a time period of 2 seconds to 6.5 seconds in the fourth audio data.

[0226] In case of adding a silent block, the first to fourth audio data may include audio signals corresponding to 0 seconds to 8 seconds.

[0227] Fig.16 is a flowchart for illustrating an operation of acquiring similarities between audio blocks.

[0228] Reference Fig.16 , it may be assumed that the electronic device 100 obtains the audio block 1611 and the audio block 1621 from the first divided data, and obtains the audio block 1612 and the audio block 1622 from the second divided data. The electronic device 100 may combine blocks having the same audio source among the plurality of audio blocks with each other.

[0229] The electronic device 100 may determine whether to combine the audio blocks with each other based on the similarity. The electronic device 100 may identify a similarity greater than or equal to a threshold value among the similarities to be compared, and combine the audio blocks corresponding to the identified similarities with each other. In the case where there are multiple similarities greater than or equal to the threshold value, the electronic device 100 may combine the audio blocks corresponding to the highest similarity with each other.

[0230] The electronic device 100 determines whether the audio block 1611 obtained from the first segmented data can be combined with one of the audio block 1612 and the audio block 1622 obtained from the second segmented data. The electronic device 100 may obtain the similarity between the audio block 1611 and the audio block 1612. The electronic device 100 may obtain the similarity between the audio block 1611 and the audio block 1622. When the similarity is greater than or equal to the threshold, the electronic device 100 may identify a block corresponding to the same audio source. The electronic device 100 may determine that the audio block 1611 and the audio block 1612 correspond to the same audio source.

[0231] The electronic device 100 may determine whether the audio block 1621 obtained from the first segmented data may be combined with one of the audio block 1612 and the audio block 1622 obtained from the second segmented data. The electronic device 100 may obtain the similarity between the audio block 1621 and the audio block 1612. The electronic device 100 may obtain the similarity between the audio block 1621 and the audio block 1622. When the similarity is greater than or equal to the threshold value, the electronic device 100 may identify a block corresponding to the same audio source. The electronic device 100 may determine that the audio block 1621 and the audio block 1622 correspond to the same audio source.

[0232] In various examples, in the case where the audio block 1611 and the audio block 1612 are identified as corresponding to the same audio source, the electronic device 100 may not determine the similarity between the audio block 1621 and the audio block 1612. Through this operation, the electronic device 100 may shorten the processing time.

[0233] Fig.17 is a view for illustrating an operation of acquiring audio data corresponding to each audio source by dividing the audio data.

[0234] exist Fig.17 , item 1710 shows audio data 1711 including all audio signals output from a plurality of audio sources. The electronic device 100 may acquire the audio data 1711. For example, it may be assumed that the audio data includes an audio signal for 8 seconds.

[0235] exist Fig.17 , item 1720 shows the audio data to be segmented. It can be assumed that the classification time point is 6 seconds. The reason why the classification time point is 6 seconds instead of 4 seconds (i.e., half of 8 seconds) is that the audio signal output from the new audio source can be identified after 8 seconds. The audio data between 0 seconds and 12 seconds can be segmented into a first time period of 0 seconds to 6 seconds and a second time period of 6 seconds to 12 seconds. Here, when the new audio source is output after 8 seconds, the data of the second time period can be further segmented into a third time period of 6 seconds to 8 seconds and a fourth time period of 8 seconds to 10 seconds. In item 1720, the electronic device 100 can obtain first segmented data 1721 segmented based on the time period of 0 seconds to 6 seconds and second segmented data 1722 segmented based on the time period of 6 seconds to 8 seconds. Fig.17 In the illustration of , the classification time point may be 4 seconds instead of 6 seconds. However, the classification time point may be changed based on user settings and is not limited to a specific method.

[0236] exist Fig.17, item 1730 shows an operation of acquiring multiple audio blocks from the segmented audio data and combining them. The electronic device 100 may acquire a first audio block 1731-1 and a second audio block 1731-2 having different audio sources from the first segmented data 1721. The electronic device 100 may acquire a third audio block 1732-1 and a fourth audio block 1732-2 having different audio sources from the second segmented data 1722. The electronic device 100 may acquire similarities between the acquired multiple audio blocks. The electronic device 100 may combine audio blocks corresponding to the same audio source based on similarities. The electronic device 100 may acquire a first audio data 1731-3 by combining the audio block 1731-1 and the audio block 1731-2 corresponding to the first audio source. The electronic device 100 may acquire a second audio data 1732-3 by combining the audio block 1732-1 and the audio block 1732-2 corresponding to the second audio source.

[0237] exist Fig.17 , item 1740 shows one audio data 1741 including first audio data 1731-3 and second audio data 1732-3. The electronic device 100 may obtain the audio data 1741 by combining the first audio data 1731-3 and the second audio data 1732-3. The audio data 1741 and the audio data 1711 may be identical or similar to each other. Some calculation errors may occur during the classification and synthesis process, and therefore, the audio data 1741 and the audio data 1711 may not be completely identical to each other.

[0238] Fig.18 is a view for illustrating an overlapping area in an operation of acquiring similarity.

[0239] The electronic device 100 may segment the audio data based on a predetermined time. In addition, the electronic device 100 may set an overlapping area based on a threshold time in a predetermined time. The overlapping area may indicate that among the multiple segmented audio data, the audio data at a first time point and the audio data at a second time point after the first time point include an area (or time period) of the same audio signal.

[0240] According to an example embodiment, Fig.18, item 1810 shows that the overlapping area can be a period before a threshold time (e.g., 0.1 seconds) starting from a predetermined time (e.g., in units of 2 seconds). In the case of dividing the period from 0 seconds to 8 seconds in units of 2 seconds, the electronic device 100 can obtain four audio data. When segmenting the audio data, the electronic device 100 can set the overlapping area to be from the predetermined time to the previous period (or range) according to the threshold time. For example, the electronic device 100 can obtain the first segmented data 1811 based on the period from 0 seconds to 2 seconds, the second segmented data 1812 based on the period from 1.9 seconds to 4 seconds, the third audio data 1813 based on the period from 3.9 seconds to 6 seconds, and the fourth audio data 1814 based on the period from 5.9 seconds to 8 seconds.

[0241] The overlapping area of ​​the first segmented data and the second segmented data may have a period of 1.9 seconds to 2 seconds. The overlapping area of ​​the second segmented data and the third audio data may be a period of 3.9 seconds to 4 seconds. The overlapping area of ​​the third audio data and the fourth audio data may be a period of 5.9 seconds to 6 seconds. The overlapping area may be 0.1 seconds.

[0242] According to an example embodiment, Fig.18 , item 1820 shows that the overlapping area can be a period before a threshold time (e.g., 0.1 second) and after a threshold time (e.g., 0.1 second) starting from a predetermined time (e.g., in units of 2 seconds). In the case of dividing the period from 0 seconds to 8 seconds in units of 2 seconds, the electronic device 100 can obtain four audio data. When segmenting the audio data, the electronic device 100 can set the overlapping area to a period (or range) from a predetermined time to before the threshold time and a period (or range) from a predetermined time to after the threshold time. For example, the electronic device 100 can obtain the first segmented data 1821 based on the period from 0 seconds to 2.1 seconds, the second segmented data 1822 based on the period from 1.9 seconds to 4.1 seconds, the third segmented data 1823 based on the period from 3.9 seconds to 6.1 seconds, and the fourth audio data 1824 based on the period from 5.9 seconds to 8 seconds.

[0243] The overlapping area of ​​the first segmented data and the second audio data may be a period of 1.9 seconds to 2.1 seconds. The overlapping area of ​​the second segmented data and the third audio data may be a period of 3.9 seconds to 4.1 seconds. The overlapping area of ​​the third audio data and the fourth audio data may be a period of 5.9 seconds to 6.1 seconds. The overlapping area may be 0.2 seconds.

[0244] Fig.19 is a view for illustrating an operation of combining audio blocks with each other based on similarity.

[0245] exist Fig.19, item 1910 shows audio data 1911 including all audio signals output from a plurality of audio sources. The electronic device 100 may acquire the audio data 1911. For example, it may be assumed that the audio data includes an audio signal for 8 seconds.

[0246] exist Fig.19 , item 1920 shows the audio data to be segmented. It can be assumed that the classification time point is 6 seconds. Fig.17 The classification time point is described, so its detailed description is omitted. The overlapping area can be based on the time period before the threshold time (e.g., 0.1 seconds) and after the threshold time (e.g., 0.1 seconds) of the classification time point. The electronic device 100 can obtain the first segmented data 1921 segmented based on the time period of 0 seconds to 6.1 seconds and the second segmented data 1922 segmented based on 5.9 seconds to 8 seconds.

[0247] The electronic device 100 may set an overlapping area 1923 in the acquired multiple audio blocks. The overlapping area 1923 may indicate an area including all audio data to be segmented into two parts. The electronic device 100 may perform a classification operation so that both audio data to be segmented include the overlapping area. Therefore, the first segmented data 1921 and the second segmented data 1922 may include the same overlapping area 1923.

[0248] exist Fig.19 , item 1930 shows an operation of obtaining a plurality of audio blocks from the segmented audio data and combining them. The electronic device 100 may obtain a first audio block 1931-1 and a second audio block 1931-2 having different audio sources from the first segmented data 1921. The electronic device 100 may obtain a third audio block 1932-1 and a fourth audio block 1932-2 having different audio sources from the second segmented data 1922.

[0249] The electronic device 100 may acquire similarity based on the acquired overlapping regions 1923 of the plurality of audio blocks. The first audio block 1931-1 may include an overlapping region 1933-1. The second audio block 1931-2 may include an overlapping region 1933-2. The third audio block 1932-1 may include an overlapping region 1935-1. The fourth audio block 1932-2 may include an overlapping region 1935-2.

[0250] The electronic device 100 may compare similarities between the overlapped region included in the first segmented data and the overlapped region included in the second segmented data.

[0251] The electronic device 100 may acquire the similarity between the overlapping region 1933-1 and the overlapping region 1933-2. The electronic device 100 may acquire the similarity between the overlapping region 1933-1 and the overlapping region 1935-2. In addition, the electronic device 100 may determine an audio block to be combined with the first audio block 1931-1 based on the two acquired similarities.

[0252] The electronic device 100 may acquire the similarity between the overlapping region 1935-1 and the overlapping region 1933-2. The electronic device 100 may acquire the similarity between the overlapping region 1935-1 and the overlapping region 1935-2. In addition, the electronic device 100 may determine an audio block to be combined with the second audio block 1932-1 based on the two acquired similarities.

[0253] The electronic device 100 may combine audio blocks corresponding to the same audio source based on similarity. The electronic device 100 may obtain a first audio data 1931-3 by combining the audio block 1931-1 and the audio block 1931-2 corresponding to the first audio source. The electronic device 100 may obtain a second audio data 1932-3 by combining the audio block 1932-1 and the audio block 1932-2 corresponding to the second audio source.

[0254] exist Fig.19 , item 1940 shows one audio data 1941 including first audio data 1931-3 and second audio data 1932-3. The electronic device 100 can obtain the audio data 1941 by combining the first audio data 1931-3 and the second audio data 1932-3. The audio data 1941 and the audio data 1911 may be identical or similar to each other. Some calculation errors may occur during the classification and synthesis process, and therefore, the audio data 1941 and the audio data 1911 may not be completely identical to each other.

[0255] Fig. 20 is a flowchart for illustrating an operation of acquiring similarities between audio blocks.

[0256] Fig. 20 Operation S2010, operation S2040, and operation S2050 may correspond to Fig.10 Therefore, the redundant description thereof is omitted in this specification.

[0257] After acquiring the audio data, the electronic device 100 may divide the audio data into a plurality of audio data including overlapping areas (S2021). Fig.19As shown in item 1920, the two audio data to be segmented may include the same overlapping area. All audio data to be segmented may not include the same overlapping area, and the two audio data to be segmented based on a specific time point may include the same overlapping area.

[0258] The electronic device 100 may obtain a plurality of audio blocks from the segmented audio data (S2031). The electronic device 100 may segment the audio blocks determined to have different audio sources. In this operation, the electronic device 100 may only segment the audio signals determined to have different audio sources into separate blocks without specifying their audio sources. For example, Fig.19 As shown in item 1930 , the first segmented data 1921 may be segmented into a first audio block 1931 - 1 and a second audio block 1932 - 1 .

[0259] The electronic device 100 may obtain similarities between the multiple audio blocks (S2032). The similarity may indicate similarities between overlapping regions. Fig.19 As shown in item 1930, the electronic device 100 may obtain the similarity between the overlapping region 1933-1 and the overlapping region 1933-2. In addition, the electronic device 100 may obtain the similarity between the overlapping region 1933-1 and the overlapping region 1935-2.

[0260] The electronic device 100 may acquire audio data by combining the audio blocks based on the similarity acquisition result (S2033). The electronic device 100 may determine which blocks are to be combined with each other based on the acquired similarity. The electronic device 100 may combine blocks determined to have the same audio source.

[0261] The electronic device 100 may obtain an audio feature based on the audio data (S2034). The electronic device 100 may obtain an audio feature based on the audio data combined with at least one audio block. For example, Fig.19 As shown in item 1930, the electronic device 100 may obtain audio features based on first audio data 1931-3 in which audio block 1931-1 and audio block 1931-2 are combined with each other, and obtain audio features based on second audio data 1932-3 in which audio block 1932-1 and audio block 1932-2 are combined with each other.

[0262] Then, the electronic device 100 may perform operations S2040 and S2050. The electronic device 100 may identify the audio source corresponding to each audio data. The electronic device 100 may only obtain the result that the first audio data 1931-3 and the second audio data 1932-3 have different audio sources. In addition, the electronic device 100 may obtain the result that the first audio data 1931-3 is output from a specific audio source (e.g., male) and the second audio data 1932-3 is output from a specific audio source (e.g., female).

[0263] Fig.21 is a flowchart for illustrating a specific example of an operation of acquiring similarities between audio blocks.

[0264] Reference Fig.21 , the electronic device 100 may acquire the audio data (S2110). Then, the electronic device 100 may segment the audio data into first segmented data and second segmented data including the overlapping area (S2120).

[0265] The electronic device 100 may acquire an audio feature based on the first segmented data (S2130-1). The electronic device 100 may identify a first audio source and a second audio source based on the audio feature (S2140-1). The electronic device 100 may acquire a first audio block corresponding to the first audio source and a second audio block corresponding to the second audio source (S2141-1).

[0266] The electronic device 100 may acquire audio features based on the second segmented data (S2130-2). The electronic device 100 may identify the first audio source and the second audio source based on the audio features (S2140-2). The electronic device 100 may acquire a third audio block corresponding to the first audio source and a fourth audio block corresponding to the second audio source (S2141-2).

[0267] The electronic device 100 may obtain similarities between audio blocks obtained from the first segmented data (e.g., the first audio block and the second audio block), and obtain similarities between audio blocks obtained from the second segmented data (e.g., the third audio block and the fourth audio block) (S2151).

[0268] The electronic device 100 may obtain first audio data by combining the first audio block and the third audio block based on the similarity acquisition result (S2152). The electronic device 100 may obtain second audio data by combining the second audio block and the fourth audio block based on the similarity acquisition result (S2153).

[0269] Fig. 22 is a view for illustrating an operation of analyzing an audio block in various examples.

[0270] exist Fig. 22 , item 2210 shows audio signals output from different audio sources. For example, diagram 2211 shows an audio signal output from a first audio source, and diagram 2212 shows an audio signal output from a second audio source. In addition, diagram 2211 shows that the first audio source outputs an audio signal from 2 seconds to 8 seconds, and diagram 2212 shows that the second audio source outputs an audio signal from 0 seconds to 8 seconds.

[0271] exist Fig. 22, item 2220 shows audio signals output from different audio sources. For example, diagram 2221 shows an audio signal output from a first audio source, and diagram 2222 shows an audio signal output from a second audio source. In addition, diagram 2221 shows that the first audio source outputs an audio signal from 0 seconds to 8 seconds, and diagram 2222 shows that the second audio source outputs an audio signal from 0 seconds to 8 seconds.

[0272] Fig.23 is a flow chart illustrating an operation of analyzing audio features based on whether an overlapping area is silent.

[0273] Reference Fig.23 , the electronic device 100 may acquire audio data (S2310). The electronic device 100 may segment the audio data into a plurality of audio data including an overlapping area (S2320). The electronic device 100 may acquire a plurality of audio blocks from the segmented audio data (S2330). The electronic device 100 may analyze the audio blocks in a specific period of time (S2340).

[0274] The electronic device 100 may determine whether the analysis of the audio data is completed (S2341). The electronic device 100 may determine whether the analysis of the entire time period of the audio data before segmentation is completed. If the analysis is completed (S2341-Y), the electronic device 100 may obtain the audio data corresponding to the audio source (S2350).

[0275] In a case where the analysis is not completed (S2341-N), the electronic device 100 may determine whether an overlapping area of ​​the divided audio data is silent (S2342).

[0276] In the case where the overlapping area is silent (S2342-Y), the electronic device 100 may identify whether the entire area of ​​the segmented audio data is silent (S2343). In the case where the entire area of ​​the segmented audio data is not silent (S2343-N), the electronic device 100 may obtain audio features based on the segmented audio data (S2344). Operation S2344 may be an operation performed when it is determined that there is no connected audio signal because the overlapping area is silent. In the case where the overlapping area is silent and the area other than the overlapping area is not silent, the electronic device 100 may determine that the segmented audio data is independent audio data. In addition, the electronic device 100 may analyze the audio blocks in the next time period (S2340). Then, the electronic device 100 may repeat operations S2341 to S2350.

[0277] In the case that the entire area of ​​the segmented audio data is silent (S2343-Y), the electronic device 100 may analyze the audio block in the next period (S2340). Then, the electronic device 100 may repeat operations S2341 to S2350.

[0278] When the overlapping area is not silent (S2342-N), the electronic device 100 may obtain the similarity between the multiple audio blocks (S2345). The electronic device 100 may identify whether there are audio blocks with a similarity greater than or equal to a threshold value (S2346). In operation S2346, a similarity comparison operation may be performed on all audio blocks that may be combined with a specific audio block.

[0279] In the absence of an audio block whose similarity is greater than or equal to the threshold value (S2346-N), the electronic device 100 may obtain an audio feature based on the current target audio block (S2347). Operation S2347 may be performed when it is determined that no audio block is connected to the current target audio block. In the case where the overlapping area is not silent and there is no audio block whose similarity is greater than or equal to the threshold value, the electronic device 100 may determine that the target audio block is an independent audio block. In addition, the electronic device 100 may analyze the audio block in the next time period (S2340). Then, the electronic device 100 may repeat operations S2341 to S2350.

[0280] In the case where there are audio blocks whose similarity is greater than or equal to the threshold value (S2346-Y), the electronic device 100 may combine the audio blocks based on the similarity acquisition result (S2348). In addition, the electronic device 100 may analyze the audio blocks in the next period (S2340). Then, the electronic device 100 may repeat operations S2341 to S2350.

[0281] exist Fig.23 In the example of , audio blocks having a similarity greater than or equal to a threshold value may be continuously combined with each other. In the case where similar audio blocks are repeatedly combined with each other over the entire time period and there are no more audio blocks to be combined with each other, the electronic device 100 may obtain audio features to perform an analysis operation. The method of analyzing audio data in which audio blocks are combined with each other may be more accurate than the method of analyzing each audio block separately.

[0282] Fig.24 is a flowchart illustrating a specific example of an operation of analyzing audio features based on whether an overlapping area is silent.

[0283] Fig.24 An example of Fig.23 Therefore, this specification omits its redundant description.

[0284] The electronic device 100 may acquire audio data (S2410). The electronic device 100 may segment the audio data into first segmented data and second segmented data including an overlapping area (S2420). The electronic device 100 may acquire a first audio block and a second audio block from the first segmented data, and acquire a third audio block and a fourth audio block from the second segmented data (S2430). The electronic device 100 may analyze the audio blocks in a specific period (S2440).

[0285] The electronic equipment 100 may determine whether the analysis of the audio data is completed (S2441). The electronic device 100 may determine whether the analysis of the entire time period of the audio data before segmentation is completed. If the analysis is completed (S2441-Y), the electronic device 100 may obtain the first audio data corresponding to the first audio source and the second audio data corresponding to the second audio source (S2450).

[0286] In a case where the analysis is not completed ( S2441 -N), the electronic device 100 may determine whether the overlapping area of ​​the first audio block is silent ( S2442 ).

[0287] If the overlapping area of ​​the first audio block is silent (S2442-Y), the electronic device 100 may identify whether the entire area of ​​the first segmented data is silent (S2443). If the entire area of ​​the first segmented data is not silent (S2443-N), the electronic device 100 may obtain audio features based on the first segmented data (S2444).

[0288] In the case that the entire area of ​​the segmented audio data is silent (S2443-Y), the electronic device 100 may analyze the audio block in the next period (S2440). Then, the electronic device 100 may repeat operations S2441 to S2450.

[0289] In the case where the overlapping area is not silent (S2442-N), the electronic device 100 may obtain a first similarity between the first audio block and the third audio block, and obtain a second similarity between the second audio block and the fourth audio block (S2445). The electronic device 100 may identify whether the first similarity or the second similarity is greater than or equal to a threshold value (S2446). In operation S2446, a similarity comparison operation may be performed on all audio blocks that may be combined with a specific audio block.

[0290] In the case where neither the first similarity nor the second similarity is greater than or equal to the threshold value (S2446-N), the electronic device 100 may obtain an audio feature based on the first audio block (S2447). Operation S2447 may be performed when it is determined that no audio block is connected to the current target audio block. In the case where the overlapping area is not silent and there is no audio block whose similarity is greater than or equal to the threshold value, the electronic device 100 may determine that the target audio block is an independent audio block. In addition, the electronic device 100 may analyze the audio block in the next time period (S2440). Then, the electronic device 100 may repeat operations S2441 to S2450.

[0291] When the first similarity or the second similarity is greater than or equal to the threshold value (S2446-Y), the electronic device 100 may combine the audio block (e.g., the third audio block or the fourth audio block) whose similarity is greater than or equal to the threshold value with the first audio block (S2448). In addition, the electronic device 100 may analyze the audio block in the next period (S2440). Then, the electronic device 100 may repeat operations S2441 to S2450.

[0292] Fig.25 is a flow chart illustrating an operation of analyzing audio features based on whether an overlapping area of ​​another time period is silent.

[0293] Fig.25 Operations S2510, S2525, S2530, S2540, S2541, S2542, S2543, S2544, S2545, S2546, S2547, and S2548 may correspond to Fig.23 Operation S2310, operation S2323, operation S2330, operation S2340, operation S2341, operation S2342, operation S2343, operation S2344, operation S2345, operation S2346, operation S2347, and operation S2348 are described in detail. Therefore, the redundant description thereof is omitted in this specification.

[0294] If the overlapping area of ​​the audio data is not silent (S2542-N), the electronic device 100 may identify whether all overlapping areas of another period are silent (S2542-2). The overlapping area of ​​another period may indicate the overlapping area of ​​the next period that may be combined with the current target audio block.

[0295] In a case where all overlapping areas of another period are silent (S2542-2-Y), the electronic device 100 may perform operation S2547.

[0296] In the case that all overlapping areas of another period are not silent (S2542-2-N), the electronic device 100 may perform operations S2545 to S2548.

[0297] Fig.26 is a flowchart showing a specific example of an operation of analyzing audio features based on whether an overlapping area of ​​another period is silent.

[0298] Fig.26 Operations S2610, S2626, S2630, S2640, S2641, S2642, S2643, S2644, S2645, S2646, S2647, and S2648 may correspond to Fig.24 Therefore, the redundant description thereof is omitted in this specification.

[0299] When the overlapping area of ​​the first audio block is not silent (S2642-N), the electronic device 100 can identify whether the overlapping area of ​​the third audio block and the overlapping area of ​​the fourth audio block are both silent (S2642-2). The overlapping area of ​​another time period may indicate the overlapping area of ​​the next time period that may be combined with the current target audio block.

[0300] For example, refer to Fig.19 According to item 1930, when the overlapping area 1933-1 of the target audio block 1931-1 is not silent, the electronic device 100 can identify whether the overlapping area 1933-2 of the audio block 1931-2 and the overlapping area 1932-2 of the audio block 1932-2 of the next time period are both silent.

[0301] In a case where both the overlapping region of the third audio block and the overlapping region of the fourth audio block are silent (S2642-2-Y), the electronic device 100 may perform operation S2647.

[0302] In a case where the overlapping region of the third audio block and the overlapping region of the fourth audio block are not both silent (S2642-2-N), the electronic device 100 may perform operations S2645 to S2648.

[0303] Fig. 27 is a flow chart illustrating the operation of increasing the overlap area.

[0304] Fig. 27Operations S2710, S2727, S2730, S2740, S2741, S2742, S2743, S2744, S2745, S2747, and S2748 may correspond to Fig.23 Therefore, the redundant description thereof is omitted in this specification.

[0305] After acquiring the similarities between the plurality of audio blocks, the electronic device 100 may identify whether the similarities are greater than or equal to a first threshold ( S2746 - 1 ).

[0306] In the case where the similarity is greater than or equal to the first threshold (S2746-1-Y), the electronic device 100 may combine the audio blocks with each other based on the similarity acquisition result (S2748)

[0307] When the similarity is not greater than or equal to the first threshold (S2746-1), the electronic device 100 may identify whether the similarity is greater than or equal to the second threshold (S2746-2-Y). When the similarity is not greater than or equal to the second threshold (S2746-2-N), the electronic device 100 may obtain audio features based on the audio block (S2747). When the similarity is greater than or equal to the second threshold (S2746-2-Y), the electronic device 100 may increase the overlapping area (S2746-3). In addition, the electronic device 100 may perform the classification operation (S2720) again based on the increased overlapping area. Then the electronic device 100 may repeat operations S2730 to S2750.

[0308] The first threshold may be greater than the second threshold.

[0309] In a case where the similarity is greater than or equal to the first threshold, the electronic device 100 may combine the audio blocks that are compared with each other to obtain the similarity.

[0310] When the similarity is less than the first threshold and greater than or equal to the second threshold, the electronic device 100 may re-segment the audio data by increasing the overlapping area. This operation is to re-evaluate the similarity comparison by increasing the overlapping area.

[0311] When the similarity is less than the second threshold, the electronic device 100 may obtain the audio feature for the target audio block because it is determined that there are no more audio blocks to be connected.

[0312] Fig.28 is a flow chart for illustrating the operation of analyzing audio features.

[0313] Reference Fig.28 , the electronic device 100 may acquire the audio feature (S2810). In addition, the electronic device 100 may identify whether the audio feature is a feature of a pre-stored audio source (S2820). The electronic device 100 may store audio features related to at least one audio source. The electronic device 100 may compare whether the audio feature acquired in operation S2810 is similar to the feature of a pre-stored audio source.

[0314] If the acquired audio feature is a feature of a pre-stored audio source (S2820-Y), the electronic device 100 may identify the pre-stored audio source based on the audio feature (S2830). The electronic device 100 may determine that the audio block (or audio data) having the acquired audio feature is output by the pre-stored audio source.

[0315] In the case where the acquired audio feature is not a feature of a pre-stored audio source (S2820-N), the electronic device 100 may identify a new audio source (S2840). The electronic device 100 may determine that the audio block having the acquired audio feature is output by the new audio source. The electronic device 100 may store information about the new audio source.

[0316] Fig.29 is a view for illustrating an operation of determining whether an overlapping area is silent in various examples.

[0317] exist Fig.29 Item 2910 shows the audio data of the period from 0 seconds to 8 seconds. The electronic device 100 can obtain the peak value in the entire period of the audio data.

[0318] exist Fig.29 , item 2920 shows the segmented audio data. The electronic device 100 can obtain the first segmented data of the time period from 0 seconds to 6 seconds and the second segmented data of the time period from 6 seconds to 8 seconds by segmenting the audio data of the time period from 0 seconds to 8 seconds. The electronic device 100 can obtain the first average value aver1 of the audio signal included in the first segmented data. The electronic device 100 can obtain the second average value aver2 of the audio signal included in the second segmented data.

[0319] The electronic device 100 may compare the peak value obtained in item 2910 with the first average value aver1 or the second average value aver2. In detail, the electronic device 100 may obtain the difference between the peak value and the average value. In addition, the electronic device 100 may determine whether the difference is greater than or equal to a threshold value. In the case where the difference is greater than or equal to the threshold value, the electronic device 100 may determine that the corresponding audio data is silent. There may be an acquired audio signal, and the audio signal may include ambient noise. In the case where the average value is significantly lower than the peak value, the electronic device 100 may determine that the corresponding audio signal is silent.

[0320] exist Fig.29 , item 2930 shows that the first segmented data is segmented again into four audio data. The electronic device 100 may obtain an average value aver1, an average value aver2, an average value aver3, or an average value aver4 based on each of the segmented audio data 2931, audio data 2932, audio data 2933, or audio data 2934.

[0321] The electronic device 100 may compare the obtained peak value with the average values ​​aver1, aver2, aver3, and aver4. In addition, the electronic device 100 may obtain the difference between the peak value and the average values ​​aver1, aver2, aver3, and aver4. In addition, the electronic device 100 may determine whether the difference is greater than or equal to a threshold value. In the case where the difference is greater than or equal to the threshold value, the electronic device 100 may determine that the corresponding audio data is silent.

[0322] Fig.30 is a view for illustrating an operation of determining whether an overlapping area is silent in various examples.

[0323] The electronic device 100 may compare overlapping areas commonly included in different audio data to determine whether audio blocks are combined with each other. The electronic device 100 may determine whether the overlapping area is silent. The actual audio signal may include noise, and therefore, the audio signal may not be fully measured. Therefore, the electronic device 100 may apply the standard for determining silence to the audio analysis operation.

[0324] exist Fig.30 , item 3010 shows audio data of a period from 0 seconds to 8 seconds. The electronic device 100 may obtain a peak value in the entire period of the audio data. The electronic device 100 may divide the audio data based on 6 seconds.

[0325] exist Fig.30 Item 3020 shows the divided audio data. The electronic device 100 can obtain first divided data 3021 of a period of 0 to 6 seconds and second divided data 3022 of a period of 6 to 8 seconds by dividing the audio data of a period of 0 to 8 seconds.

[0326] exist Fig.30 Item 3030 shows an overlap region extracted from the segmented audio data. The electronic device 100 may obtain an overlap region 3031 from the first segmented data, and obtain an overlap region 3032 from the second segmented data.

[0327] The electronic device 100 may obtain an average value aver1 of the audio signal included in the overlapping area 3031. The electronic device 100 may obtain an average value aver2 of the audio signal included in the overlapping area 3032. In addition, the electronic device 100 may compare the peak value in the item 3010 with the average value aver1 and the average value aver2. In detail, when the difference between the peak value and the average value aver1 is greater than or equal to the threshold value, the electronic device 100 may determine that the overlapping area 3031 is silent. When the difference between the peak value and the average value aver2 is greater than or equal to the threshold value, the electronic device 100 may determine that the overlapping area 3032 is silent.

[0328] Reference Fig.29 and Fig.30 The described method of using the difference between the peak value and the average value may indicate variance calculation, standard deviation calculation, mean square error calculation, and the like.

[0329] Fig.31 is a view for illustrating an operation of acquiring similarity.

[0330] exist Fig.31 Item 3110 shows the divided audio blocks. The electronic device 100 may obtain a first audio block 3111 and a second audio block 3112.

[0331] exist Fig.31 Item 3120 shows the extracted overlapping area. The electronic device 100 may obtain the overlapping area 3121 from the first audio block 3111 and obtain the overlapping area 3122 from the second audio block 3112.

[0332] exist Fig.32 , item 3130 shows a normalized overlapping area. The electronic device 100 may perform a normalization operation to compare on the same scale and perform accurate and fast calculations. The electronic device 100 may obtain first normalized data 3131 by normalizing the overlapping area 3121. The electronic device 100 may obtain second normalized data 3132 by normalizing the overlapping area 3122. The electronic device 100 may obtain the similarity between the first normalized data 3131 and the second normalized data 3132. In theory, the first normalized data 3131 and the second normalized data 3132 need to be identical to each other because the operation is a comparison operation of the data in the overlapping area. However, considering that noise and errors may occur during the actual calculation process, the electronic device 100 may determine whether the similarity is greater than or equal to a threshold value. In the case where the similarity between the normalized data is greater than or equal to the threshold value, the electronic device 100 may combine the first audio block 3111 with the second audio block 3112.

[0333] Fig.32is a view for illustrating a user interface (UI) related to an audio extraction operation in various examples.

[0334] Reference Fig.32 , the electronic device 100 may provide a user with a UI related to audio extraction. It may be assumed that the electronic device 100 runs an audio extraction application. In addition, it may be assumed that the electronic device 100 performs an audio extraction operation on content including audio.

[0335] In various examples, the content including audio may be audio data including only audio. In various examples, the content including audio may be content including both audio and images.

[0336] It may be assumed that the user inputs a command for extracting an audio signal for specific content to the electronic device 100. The electronic device 100 may perform an audio signal extraction operation on the specific content selected by the user by using an application related to the audio signal.

[0337] The electronic device 100 may provide (or display) a screen 3210 related to audio signal extraction. The screen 3210 may include: a UI 3211 including text information 3211 indicating that an audio extraction operation is performed, a UI 3212 including an image of content as an object of the audio extraction operation, a UI 3213 for selecting an audio extraction period, a UI 3214 including result information of the audio extraction operation, a UI 3215 for performing an audio extraction operation on another content, or at least one of a UI 3216 for temporarily storing all result information of the audio extraction operation.

[0338] In addition, UI 3214 may include information about at least one audio signal extracted from specific content. The information may include at least one of identification information indicating the extracted (or separated) audio source (e.g., the first speaker) or feature information indicating the audio source (e.g., male and 15 years old). In addition, the information may include a signal waveform of the extracted (or separated) audio source. UI 3214 may include a UI for separately storing (or sharing) each extracted (or separated) audio source.

[0339] In addition, the screen 3210 may further include a UI for user input for performing an audio extraction operation. The execution UI may be included in the UI 3211 or the UI 3213.

[0340] In addition, the electronic device 100 may individually store (or share) each audio signal separated based on the received user input through the UI 3214 .

[0341] Fig.33 is a view for illustrating an operation of displaying a UI for changing a volume of separated audio data at a position corresponding to a specific object.

[0342] Reference Fig.33 In item 3310, the electronic device 100 may receive content including image data and audio data. The image data in the received content may include the first speaker 10 and the second speaker 20. The audio data in the received content may include the voice of the first speaker 10 and the voice of the second speaker 20.

[0343] The audio data in the received content may be an audio signal including together the voice of the first speaker 10 and the voice of the second speaker 20. The electronic device 100 may separate (or segment) the voice of the first speaker 10 and the voice of the second speaker 20 from each other.

[0344] Referring to item 3320, the electronic device 100 may obtain audio data from the received content. Here, the audio data may be an original sound source. Therefore, the audio data included in the content may be referred to as original audio data. The electronic device 100 may segment the first audio data 3321 and the second audio data 3322 separated from the original audio data based on the audio source.

[0345] The electronic device 100 may segment the original audio data for each audio source. The electronic device 100 may identify whether multiple audio sources exist in the original audio data in the received content based on the audio features. In the case where multiple audio sources exist in the original audio data, the electronic device 100 may extract audio data corresponding to each of the multiple audio sources from the original audio data. The electronic device 100 may specify the audio source based on the audio features. For example, the audio source may be specified based on representative feature information (e.g., gender or age).

[0346] Referring to item 3330, the electronic device 100 may identify a predetermined object from the image data in the received content. The predetermined object may indicate an object that may be an audio source. The object may indicate a human object, an animal object, a speaking object, etc. that may output audio. The electronic device 100 may identify the predetermined object based on the image data.

[0347] The electronic device 100 may map (or match) the recognized object and the extracted audio data. A standard for mapping the recognized object and the extracted audio data may be required. The electronic device 100 may map the recognized object and the extracted audio data based on representative feature information.

[0348] The electronic device 100 may acquire representative feature information of the audio source based on the extracted audio data. That is, the electronic device 100 may analyze the audio data to acquire representative feature information corresponding to the audio data, and may specify the audio data (or audio source) based on the acquired representative feature information.

[0349] The electronic device 100 may acquire representative feature information of the object based on the recognized object. The recognized object may be an object that may be an audio source, and therefore, the representative feature information of the object may be referred to as representative feature information of the audio source. That is, the electronic device 100 may acquire representative feature information of the recognized object by analyzing the image data, and may specify the object (or audio source) based on the acquired representative feature information.

[0350] The electronic device 100 may compare the representative feature information obtained based on the audio data and the representative feature information obtained based on the image data. The electronic device 100 may perform a mapping operation based on the comparison result. In detail, the electronic device 100 may map the audio data and the object based on whether the representative feature information matches each other. In the case where the representative feature information of the first audio data matches the representative feature information of the first object, the electronic device 100 may map the first audio data and the first object to each other.

[0351] In various examples, similarity may be determined as a criterion for matching. If the similarity between the representative feature information of the first audio data and the representative characteristic information of the first object is greater than or equal to a threshold, the electronic device 100 may map the first audio data and the first object to each other.

[0352] For example, the electronic device 100 may acquire representative feature information (male and 30 to 39 years old) of the first audio data 3321 acquired in the item 3320 , and acquire representative feature information (male and 0 to 9 years old) of the second audio data 3322 .

[0353] For example, the electronic device 100 may identify a first object 3331 corresponding to the first speaker 10 and a second object 3332 corresponding to the second speaker 20 from the image described in the item 3310. The electronic device 100 may acquire representative characteristic information (male and 30 to 39 years old) of the first object 3331, and acquire representative characteristic information (male and 0 to 9 years old) of the second object 3332.

[0354] For example, the representative characteristic information (male and 30 to 39 years old) of the first audio data 3321 may match the representative characteristic information (male and 30 to 39 years old) of the first object 3331, and therefore, the electronic device 100 may map the first audio data 3321 and the first object 3331 to each other. In addition, the representative characteristic information (male and 0 to 9 years old) of the second audio data 3322 may match the representative characteristic information (male and 0 to 9 years old) of the second object 3332, and therefore, the electronic device 100 may map the second audio data 3322 and the second object 3332 to each other.

[0355] Referring to item 3340, the electronic device 100 may display a UI for changing the volume of the audio data based on the mapping result. In detail, the electronic device 100 may display a UI for changing the volume of the audio data at a position corresponding to the mapped object.

[0356] For example, it may be assumed that the first audio data 3321 and the first object 3331 are mapped to each other. It may also be assumed that the second audio data 3322 and the second object 3332 are mapped to each other. The electronic device 100 may display a UI 3341 for changing the volume of the first audio data 3321 at a position corresponding to the first object 3331. In addition, the electronic device 100 may display a UI 3342 for changing the volume of the second audio data 3322 at a position corresponding to the second object 3332.

[0357] In case a user input for changing the volume is received through the UI 3341 or the UI 3342, the electronic device 100 may change the volume of audio data corresponding to the position where the user input is received.

[0358] For example, when receiving user input through UI 3341, the electronic device 100 may change the volume of the first audio data 3321 based on the user input. In addition, when receiving user input through UI 3342, the electronic device 100 may change the volume of the second audio data 3322 based on the user input.

[0359] The user input may be a touch input or a drag input.

[0360] It may be assumed that the user input is a touch input. In the case of receiving a user input of touching the left side or the bottom, the electronic device 100 may change the volume of the audio data corresponding to the position where the user input is received to a value less than the currently set value. On the other hand, in the case of receiving a user input of touching the right side or the top, the electronic device 100 may change the volume of the audio data corresponding to the position where the user input is received to a value greater than the currently set value.

[0361] It may be assumed that the user input is a drag input. In the case of receiving a user input of dragging to the left or bottom, the electronic device 100 may change the volume of the audio data corresponding to the position where the user input is received to a value less than the currently set value. On the other hand, in the case of receiving a user input of dragging to the right or top, the electronic device 100 may change the volume of the audio data corresponding to the position where the user input is received to a value greater than the currently set value.

[0362] also, Fig.33 A first speaker 10 and a second speaker 20 are shown. However, the first speaker 10 and the second speaker 20 may be described as a first audio source and a second audio source. That is, the first speaker 10 and the second speaker 20 may be described as a first audio source and a second audio source. Fig.33 The operation may not necessarily be applied only to human voice, and the volume control function may be applied to each audio source.

[0363] Fig.34 is a flowchart illustrating an operation of displaying a UI for changing the volume of separated audio data at a position corresponding to a specific object.

[0364] Reference Fig.34 , the electronic device 100 may receive content including image data and audio data (S3405). The electronic device 100 may identify an audio source from the content. In addition, the electronic device 100 may identify the number of audio sources identified from the content.

[0365] The electronic device 100 may determine whether there are multiple audio sources (S3410). In the case where there are no multiple audio sources (S3410-N), the electronic device 100 may repeat operation S3410 until multiple audio sources are identified. It may be assumed that the electronic device 100 finally analyzes two audio sources.

[0366] In the case where there are multiple audio sources (S3410-Y), the electronic device 100 may obtain first audio data and second audio data (S3415) by dividing (or separating) the audio data. The first audio data may include an audio signal corresponding to the first audio source, and the second audio data may include an audio signal corresponding to the second audio source.

[0367] In addition, the electronic device 100 may recognize a first object corresponding to the first audio data and a second object corresponding to the second audio data from the image data (S3420).

[0368] The electronic device 100 may identify a first object corresponding to the first audio data and a second object corresponding to the second audio data using the representative feature information.

[0369] In various examples, the electronic device 100 may identify an object corresponding to the audio data by comparing representative feature information of the audio data and representative characteristic information of the object.

[0370] For example, the electronic device 100 may obtain representative feature information (male and 30 to 39 years old) of the first audio data, and obtain representative feature information of each of the multiple objects included in the image data. It may be assumed that "male and 30 to 39 years old" is the representative feature information of the first object among the multiple objects. The representative feature information (male and 30 to 39 years old) of the first audio data may match the representative feature information (male and 30 to 39 years old) of the first object, and therefore, the electronic device 100 may recognize that the first object is the object corresponding to the first audio data.

[0371] In addition, the electronic device 100 may acquire only representative feature information of a predetermined object among a plurality of objects included in the image data. Fig.33 It is described.

[0372] Furthermore, in various examples, the electronic device 100 may recognize an object corresponding to the audio data from the image data based on representative feature information of the audio data.

[0373] For example, the electronic device 100 may identify an object representing representative feature information (male and 30 to 39 years old) among a plurality of objects included in the image data as a first object based on the representative feature information (male, 30-39 years old) of the first audio data.

[0374] After identifying the object corresponding to each audio data, the electronic device 100 may display a first UI for changing the volume of the first audio data at a first position corresponding to the first object, and display a second UI for changing the volume of the second audio data at a second position corresponding to the second object (S3425).

[0375] The electronic device 100 may determine whether a user input is received through the first UI or the second UI (S3430). In the case where a user input is not received through the first UI or the second UI (S3430-N), the electronic device 100 may repeat operation S3430.

[0376] In the case of receiving user input through the first UI or the second UI (S3430-Y), the electronic device 100 may change the volume based on the user input (S3435). For example, in the case of receiving user input through the first UI, the electronic device 100 may change the volume of the first audio data. In addition, in the case of receiving user input through the second UI, the electronic device 100 may change the volume of the second audio data.

[0377] Fig.35 is a flowchart for illustrating a control method of the electronic device 100 in various embodiments.

[0378] Reference Fig.35 , a control method of an electronic device 100 may include: dividing audio data into a plurality of time periods to include an overlapping area (S3505); acquiring an audio feature from each of the plurality of divided time periods (S3510); identifying a first audio source and a second audio source in each of the plurality of divided time periods based on the audio feature (S3515); and acquiring first audio data corresponding to the first audio source and second audio data corresponding to the second audio source from the audio data (S3520).

[0379] Furthermore, the segmenting ( S3505 ) may include acquiring first segmented data of a first period and second segmented data of a second period different from the first period by segmenting the audio data, and the first segmented data and the second segmented data may include an overlapping area.

[0380] In addition, the control method may further include identifying a plurality of audio blocks to be segmented based on an audio source in each of a plurality of segmented time periods, wherein, when acquiring the first audio data and the second audio data (S3520), the first audio data may be acquired by combining the audio blocks corresponding to the first audio source with each other, and the second audio data may be acquired by combining the audio blocks corresponding to the second audio source with each other.

[0381] Furthermore, when acquiring the first audio data and the second audio data (S3520), an overlapping region included in a plurality of audio blocks may be identified, and the first audio data and the second audio data may be acquired by comparing the identified overlapping regions with each other.

[0382] In addition, the control method may further include obtaining the similarity between overlapping areas included in multiple audio blocks, wherein, when obtaining the first audio data and the second audio data (S3520), the first audio data and the second audio data can be obtained by combining audio blocks having similarities greater than or equal to a first threshold value with each other.

[0383] In addition, the control method may further include expanding the overlapping area when the similarity is less than the first threshold and greater than or equal to the second threshold, and segmenting the audio data based on the expanded overlapping area.

[0384] In addition, the control method may further include obtaining a peak value of an audio signal in a first audio block included in the audio block, obtaining an average value of an audio signal included in an overlapping area of ​​the first audio block, and identifying whether the overlapping area is silent based on a difference between the peak value and the average value.

[0385] Furthermore, in the segmentation (S3505), the number of audio sources included in the audio data may be identified, and in a case where the number of identified audio sources is less than a threshold number, the audio data may be segmented.

[0386] Furthermore, in the segmentation (S3505), when the time of the audio data is greater than or equal to the threshold time, the audio data may be segmented.

[0387] In addition, in segmentation (S3505), the number of audio sources included in the audio data can be identified based on the first artificial intelligence model, and when obtaining the first audio data and the second audio data (S3520), the first audio data and the second audio data can be obtained based on a second artificial intelligence model different from the first artificial intelligence model.

[0388] In addition, it can be Figure 1 or Figure 2 The electronic device configured to perform as Fig.35 The control method of the electronic device shown in the figure can also be performed by an electronic device with another configuration as shown in the figure. Fig.35 The control method of the electronic device shown.

[0389] Furthermore, the methods in the various examples of the present disclosure described above may be implemented in the form of an application that can be installed on an existing electronic device.

[0390] Furthermore, the methods in the various examples of the present disclosure described above may be implemented only by software upgrade or hardware upgrade of an existing electronic device.

[0391] Furthermore, the various embodiments of the present disclosure described above may be performed by an embedded server included in the electronic device, or may be performed by an external server of at least one of the electronic device or the display apparatus.

[0392] In addition, according to an example embodiment of the present disclosure, the various examples described above may be implemented by software including instructions stored in a machine-readable storage medium (e.g., a computer-readable storage medium). A machine may be a device that calls stored instructions from a storage medium and operates based on the called instructions, and may include an electronic device in the example of the present disclosure. In the case where the instructions are executed by a processor, the processor may directly execute the functions corresponding to the instructions, or other components may execute the functions corresponding to the instructions under the control of the processor. The instructions may include code provided or run by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory" means that the storage medium is tangible, does not include signals, and does not distinguish whether the data is semi-permanently stored in the storage medium or temporarily stored in the storage medium.

[0393] In addition, according to various embodiments of the present disclosure, the methods in the various examples described above may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a CD-ROM) or distributed through an application store (e.g., PlayStore). TM ) Online distribution. In the case of online distribution, at least part of the computer program product may be at least temporarily stored in a storage medium, such as a memory of a manufacturer's server, a server of an application store or a relay server, or provided temporarily.

[0394] In addition, according to various embodiments, each of the components described above (e.g., modules or programs) may include a single entity or multiple entities, and in various embodiments, some of the corresponding subcomponents described above may be omitted, or other subcomponents may also be included. Alternatively or additionally, some of the components (e.g., modules or programs) may be integrated into one entity, and the functions performed by each corresponding component before integration may be performed in the same or similar manner. According to various embodiments, the operations performed by a module, program, or other component may be performed in a sequential manner, in a parallel manner, in an iterative manner, or in a heuristic manner, and at least some of the operations may be performed or omitted in a different order, or other operations may be added.

[0395] Although the embodiments of the present disclosure are shown and described above, the present disclosure is not limited to the above specific embodiments, and those skilled in the art to which the present disclosure belongs may make various modifications without departing from the scope and spirit of the present disclosure disclosed in the appended claims. These modifications are also to be understood as falling within the scope of the present disclosure.

Claims

1. An electronic device, comprising: a memory configured to store at least one instruction; as well as At least one processor is configured to execute the at least one instruction to perform the following operations: audio data by segmenting the audio data into a plurality of portions based on a plurality of time periods having a plurality of overlapping regions, obtaining audio features from each of the plurality of parts, identifying a first audio source and a second audio source in each of the plurality of portions based on the audio features, and First audio data corresponding to the first audio source and second audio data corresponding to the second audio source are acquired from the audio data.

2. The electronic device according to claim 1, wherein: The at least one processor is further configured to segment the audio data to perform the following operations: obtaining a first portion corresponding to a first period of the plurality of time periods, and Get a second time period corresponding to a second time period in the plurality of time periods, wherein the second period is different from the first period, and The first portion and the second portion include an overlapping area among the multiple overlapping areas.

3. The electronic device according to claim 1, wherein: The at least one processor is configured to: identifying a plurality of audio blocks in each of the plurality of time periods based on the first audio source or the second audio source; acquiring the first audio data by combining two or more first audio blocks corresponding to the first audio source among the plurality of audio blocks with each other; as well as The second audio data is acquired by combining two or more second audio blocks corresponding to the second audio source among the plurality of audio blocks with each other.

4. The electronic device according to claim 3, wherein: The at least one processor is further configured to: identifying the plurality of overlapping regions in the plurality of audio blocks, and The first audio data and the second audio data are acquired by comparing the plurality of overlapping areas with each other.

5. The electronic device according to claim 3, wherein: The at least one processor is further configured to: obtaining similarities between the multiple overlapping regions in the multiple audio blocks, Acquire the first audio data by combining the two or more first audio blocks whose similarity is greater than or equal to a first threshold; as well as The second audio data is acquired by combining the two or more second audio blocks whose similarity is greater than or equal to the first threshold.

6. The electronic device according to claim 5, wherein: The at least one processor is further configured to: When the similarity is less than the first threshold and greater than or equal to a second threshold, at least one overlapping region among the plurality of overlapping regions is expanded, and The audio data is segmented based on the extended overlapping area.

7. The electronic device according to claim 3, wherein: The at least one processor is further configured to: obtaining a peak value of an audio signal in a first audio block among the plurality of audio blocks, obtaining an average value of the audio signal included in the overlapping area of ​​the first audio block, and Whether the overlapping area is silent is identified based on the difference between the peak value and the average value.

8. The electronic device according to claim 1, wherein: The at least one processor is further configured to: identifying a number of audio sources included in the audio data, and The audio data is segmented when the number of identified audio sources is less than a threshold number.

9. The electronic device according to claim 8, wherein: The at least one processor is further configured to segment the audio data when a duration of the audio data is greater than or equal to a threshold time.

10. The electronic device according to claim 8, wherein: The at least one processor is configured to: identifying a number of audio sources in the audio data based on a first artificial intelligence model stored in the memory, and The first audio data and the second audio data are obtained based on a second artificial intelligence model stored in the memory, the first artificial intelligence model being different from the first artificial intelligence model.

11. A method for controlling an electronic device, the method comprising: audio data that divides the audio data into a plurality of portions based on a plurality of time periods having a plurality of overlapping regions; obtaining audio features from each of the plurality of parts; identifying a first audio source and a second audio source in each of the plurality of portions based on the audio features; as well as First audio data corresponding to the first audio source and second audio data corresponding to the second audio source are acquired from the audio data.

12. The method of claim 11, wherein: Segmenting the audio data includes: obtaining a first portion corresponding to a first period of the plurality of time periods, and obtaining a second portion corresponding to a second period of the plurality of time periods, wherein the second period is different from the first period, and The first portion and the second portion include an overlapping area among the multiple overlapping areas.

13. The method of claim 11, further comprising: identifying a plurality of audio blocks in each of the plurality of time periods based on the first audio source or the second audio source, Wherein, acquiring the first audio data and the second audio data includes: acquiring the first audio data by combining two or more first audio blocks corresponding to the first audio source among the plurality of audio blocks with each other; and The second audio data is acquired by combining two or more second audio blocks corresponding to the second audio source among the plurality of audio blocks with each other.

14. The method of claim 13, wherein: Acquiring the first audio data and the second audio data includes: identifying the plurality of overlapping regions in the plurality of audio blocks, and The first audio data and the second audio data are acquired by comparing the plurality of overlapping areas with each other.

15. The method of claim 14, further comprising: Obtaining similarities between the multiple overlapping regions in the multiple audio blocks, wherein obtaining the first audio data and the second audio data includes: acquiring the first audio data by combining the two or more first audio blocks whose similarity is greater than or equal to a first threshold; and The second audio data is acquired by combining the two or more second audio blocks whose similarity is greater than or equal to the first threshold.

Images