Information processing device, information processing method, and program

Abstract

An apparatus and method for generating and outputting display data corresponding to sound sources separated by sound source separation processing and performing correction of each sound source data according to user input are provided. The apparatus includes: a sound source separation processing execution unit that executes sound source separation processing; and a voice data editing processing unit that generates display data corresponding to the sound sources separated by the sound source separation processing and outputs the display data to a display unit. A voice data editing processing unit generates display data enabling recognition of a timing offset, a pitch offset, or the like of different sound source data, outputs the display data together with pitch change data associated with a change over time of the sound source data, and performs processing of correcting the sound source data in accordance with a user input.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and a program. More specifically, the present disclosure relates to an information processing apparatus, an information processing method, and a program that perform editing processing of each of a plurality of sound sources included in audio data and high-precision sound source separation processing. BACKGROUND

[0002] Recently, the use of applications that perform data editing processing (for example, editing processing such as noise reduction processing, pitch adjustment processing, and volume adjustment processing on various types of audio data) on recorded data of a user's voice or an instrument performance has increased.

[0003] Specifically, for example, processing of editing audio data recorded in a smartphone using an audio editing application installed in the smartphone is often performed.

[0004] Note that, for example, Patent Literature 1 (WO 2022 / 209171 A) discloses a conventional technology that discloses editing processing of recorded data.

[0005] Patent Literature 1 discloses processing that improves the sound quality of recorded data by removing reverberation sound and noise from the recorded data.

[0006] However, generally, editing processing of recorded data is performed as processing of the entire audio data included in the recorded data. For example, various types of processing such as noise reduction, pitch adjustment, and volume adjustment are performed on the entire recorded audio data.

[0007] Therefore, in the case of audio data including a plurality of different sound sources in recorded data, for example, in the case of including a plurality of sound source data such as a human voice (vocal sound) and a guitar sound as a background sound, editing processing and correction processing are collectively performed on the entire plurality of sound source data.

[0008] In such an editing processing mode, for example, editing processing in units of sound source data such as processing of correcting only the human voice (vocal sound) or the guitar sound in recorded data including the human voice (vocal sound) and the guitar sound cannot be performed.

[0009] LIST OF CITATIONS

[0010] PATENT LITERATURE

[0011] Patent Literature 1: WO 2022 / 209171 A1 SUMMARY

[0012] PROBLEMS TO BE SOLVED BY THE INVENTION

[0013] The present disclosure was completed, for example, in view of the above problems, and an object of the present disclosure is to provide an information processing apparatus, an information processing method, and a program capable of independently editing audio data of each of a plurality of sound sources included in recorded data, for example, in a case where the recorded data includes a plurality of sound source data.

[0014] Further, the embodiments of the present disclosure provide an information processing apparatus, an information processing method, and a program capable of performing high-precision sound source separation processing according to a sound source distance by changing an execution mode of sound source separation processing according to a sound source distance.

[0015] Solution to the problem

[0016] A first aspect of the present disclosure is:

[0017] An information processing apparatus comprising:

[0018] a sound source separation processing execution unit that executes sound source separation processing; and

[0019] an audio data editing processing unit that generates display data corresponding to sound sources separated by the sound source separation processing, and outputs the display data to a display unit.

[0020] Further, a second aspect of the present disclosure is:

[0021] An information processing method executed by an information processing apparatus, the information processing method comprising:

[0022] a sound source separation processing execution step of executing sound source separation processing using a sound source separation processing execution unit; and

[0023] a step of generating display data corresponding to sound sources separated by the sound source separation processing using an audio data editing processing unit and outputting the display data to a display unit.

[0024] Further, a third aspect of the present disclosure is:

[0025] A program that causes an information processing apparatus to execute processing including:

[0026] a sound source separation processing execution step of executing sound source separation processing using a sound source separation processing execution unit; and

[0027] a step of generating display data corresponding to sound sources separated by the sound source separation processing using an audio data editing processing unit and outputting the display data to a display unit.

[0028] Note that the program in the present disclosure is, for example, a program that can be provided by a storage medium or a communication medium, which provides the program in a computer-readable format for an information processing apparatus or a computer system capable of executing various program codes. By providing such a program in a computer-readable format, a process corresponding to the program is implemented on the information processing apparatus or the computer system.

[0029] Other objects, features and advantages of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, which illustrates, by way of example, the principles of the present disclosure. Note that in this specification, a system is a logical collection configuration of a plurality of apparatuses, and is not limited to a system in which apparatuses having various configurations are located in the same housing.

[0030] With the configurations according to the embodiments of the present disclosure, a device and a method for generating and outputting display data corresponding to sound sources separated by sound source separation processing and performing correction on each sound source data according to user input are realized.

[0031] Specifically, for example, a sound source separation processing execution unit that executes sound source separation processing, and an audio data editing processing unit that generates display data corresponding to sound sources separated by sound source separation processing and outputs the display data to a display unit. The audio data editing processing unit generates display data that can recognize timing offset, pitch offset, and the like between different sound source data and outputs it together with pitch change data of the sound source data over time, and performs correction processing on each sound source data according to user input.

[0032] With these configurations, a device and a method for generating and outputting display data corresponding to sound sources separated by sound source separation processing and performing correction on each sound source data according to user input are realized.

[0033] Note that the effects described in this specification are merely examples and are not limiting, and additional effects can be provided. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 is a diagram illustrating an outline of a process executed by an information processing apparatus in the present disclosure.

[0035] Figure 2 is a diagram illustrating an outline of a process executed by an information processing apparatus in the present disclosure.

[0036] Figure 3 is a diagram illustrating an outline of a process executed by an information processing apparatus in the present disclosure.

[0037] Figure 4 is a diagram illustrating an outline of a process executed by an information processing apparatus in the present disclosure.

[0038] Figure 5 is a block diagram illustrating a configuration example of an information processing apparatus in the present disclosure.

[0039] Figure 6 is a diagram illustrating a specific configuration example of a sound source separation processing execution unit and a specific processing executed by the sound source separation processing execution unit.

[0040] Figure 7 is a diagram illustrating a detailed configuration of an audio data editing processing unit of an information processing apparatus in the present disclosure.

[0041] Figure 8 is a diagram illustrating a specific example of display data (UI) displayed by an information processing apparatus in the present disclosure, a user operation on the UI, and a correction processing.

[0042] Figure 9 is a diagram illustrating a specific example of display data (UI) displayed by an information processing apparatus in the present disclosure, a user operation on the UI, and a correction processing.

[0043] Figure 10 is a diagram illustrating a specific example of display data (UI) displayed by an information processing apparatus in the present disclosure, a user operation on the UI, and a correction processing.

[0044] Figure 11 is a diagram illustrating a specific example of display data (UI) displayed by an information processing apparatus in the present disclosure, a user operation on the UI, and a correction processing.

[0045] Figure 12 is a diagram illustrating a specific example of display data (UI) displayed by an information processing apparatus in the present disclosure, a user operation on the UI, and a correction processing.

[0046] Figure 13 is a diagram illustrating a specific example of display data (UI) displayed by an information processing apparatus in the present disclosure, a user operation on the UI, and a correction processing.

[0047] Figure 14 is a diagram illustrating a specific example of display data (UI) displayed by an information processing apparatus in the present disclosure, a user operation on the UI, and a correction processing.

[0048] Figure 15 is a diagram illustrating a specific example of display data (UI) displayed by an information processing apparatus in the present disclosure, a user operation on the UI, and a correction processing.

[0049] Figure 16is a diagram illustrating a specific example of display data (UI) displayed by the information processing apparatus in the present disclosure, a user operation on the UI, and a correction process.

[0050] Figure 17 is a diagram illustrating a specific example of display data (UI) displayed by the information processing apparatus in the present disclosure, a user operation on the UI, and a correction process.

[0051] Figure 18 is a diagram illustrating a specific example of display data (UI) displayed by the information processing apparatus in the present disclosure, a user operation on the UI, and a correction process.

[0052] Figure 19 is a diagram illustrating a specific example of display data (UI) displayed by the information processing apparatus in the present disclosure, a user operation on the UI, and a correction process.

[0053] Figure 20 is a diagram illustrating a specific example of display data (UI) displayed by the information processing apparatus in the present disclosure, a user operation on the UI, and a correction process.

[0054] Figure 21 is a diagram illustrating a specific example of display data (UI) displayed by the information processing apparatus in the present disclosure, a user operation on the UI, and a correction process.

[0055] Figure 22 is a diagram illustrating a specific example of display data (UI) displayed by the information processing apparatus in the present disclosure, a user operation on the UI, and a correction process.

[0056] Figure 23 is a diagram illustrating a specific example of display data (UI) displayed by the information processing apparatus in the present disclosure, a user operation on the UI, and a correction process.

[0057] Figure 24 is a diagram illustrating a specific example of display data (UI) displayed by the information processing apparatus in the present disclosure, a user operation on the UI, and a correction process.

[0058] Figure 25 is a diagram illustrating a specific example of display data (UI) displayed by the information processing apparatus in the present disclosure, a user operation on the UI, and a correction process.

[0059] Figure 26 is a diagram illustrating a specific example of display data (UI) displayed by the information processing apparatus in the present disclosure, a user operation on the UI, and a correction process.

[0060] Figure 27is a diagram illustrating a specific example of display data (UI) displayed by the information processing apparatus in the present disclosure, a user operation on the UI, and a correction process.

[0061] Figure 28 is a diagram illustrating a specific example of display data (UI) displayed by the information processing apparatus in the present disclosure, a user operation on the UI, and a correction process.

[0062] Figure 29 is a diagram illustrating a configuration example of the information processing apparatus that calculates a sound source distance and performs a sound source separation process according to the sound source distance.

[0063] Figure 30 is a diagram illustrating an example of data stored in a trained model (DNN).

[0064] Figure 31 is a diagram illustrating an example of a training process for generating a trained model specific to a sound source type and a sound source distance.

[0065] Figure 32 is a diagram illustrating an example of training data used for a training process for generating a trained model specific to a sound source type and a sound source distance.

[0066] Figure 33 is a diagram illustrating another example of a training process for generating a trained model specific to a sound source type and a sound source distance.

[0067] Figure 34 is a diagram illustrating another example of training data used for a training process for generating a trained model specific to a sound source type and a sound source distance.

[0068] Figure 35 is a diagram illustrating a specific configuration example of a sound source distance corresponding sound source separation process execution unit and a process executed by the sound source distance corresponding sound source separation process execution unit.

[0069] Figure 36 is a diagram illustrating a specific process example of a sound source distance calculation process executed by a sound source distance calculation unit.

[0070] Figure 37 is a diagram illustrating a specific process example of a sound source distance calculation process executed by a sound source distance calculation unit.

[0071] Figure 38 is a diagram illustrating a specific process example of a sound source distance calculation process executed by a sound source distance calculation unit.

[0072] Figure 39is a diagram illustrating a flowchart of a specific sequence of sound source distance calculation processing performed by the sound source distance calculation unit.

[0073] Figure 40 is a diagram illustrating a flowchart of a specific sequence of sound source distance calculation processing performed by the sound source distance calculation unit.

[0074] Figure 41 is a diagram illustrating another flowchart of a specific sequence of sound source distance calculation processing performed by the sound source distance calculation unit.

[0075] Figure 42 is a diagram illustrating another flowchart of a specific sequence of sound source distance calculation processing performed by the sound source distance calculation unit.

[0076] Figure 43 is a diagram illustrating another flowchart of a specific sequence of sound source distance calculation processing performed by the sound source distance calculation unit.

[0077] Figure 44 is a diagram illustrating a flowchart of a sequence of sound source separation processing performed by the information processing apparatus in the present disclosure.

[0078] Figure 45 is a diagram illustrating an example of a system that performs sound source separation processing using an external device.

[0079] Figure 46 is a diagram illustrating a configuration example of each apparatus in the system that performs sound source separation processing using an external device.

[0080] Figure 47 is a diagram illustrating a hardware configuration example of the information processing apparatus in the present disclosure. DETAILED DESCRIPTION

[0081] Hereinafter, the information processing apparatus, the information processing method, and the program according to the present disclosure will be described in detail with reference to the drawings. Note that the description will be given in accordance with the following items.

[0082] 1. Overview of processing performed by the information processing apparatus in the present disclosure

[0083] 2. Configuration example of the information processing apparatus in the present disclosure

[0084] 3. Detailed configuration of the sound source separation processing execution unit of the information processing apparatus in the present disclosure and details of the sound source separation processing

[0085] 4. Detailed configuration of the audio data editing processing unit of the information processing apparatus in the present disclosure and details of the audio data editing processing performed by the information processing apparatus in the present disclosure

[0086] 5. Specific examples of display data (UI) displayed by the information processing apparatus in the present disclosure, user operations on the UI, and correction processing

[0087] 5-1. (Processing example 1) A processing example in which a UI that can identify a timing offset region between individual sound source data is displayed and a timing offset between individual sound source data is corrected in accordance with a user operation on the UI

[0088] 5-2. (Processing example 2) A processing example in which a UI that can identify a pitch offset region in sound source data is displayed and a pitch offset in sound source data is corrected in accordance with a user operation on the UI

[0089] 6. Embodiments in which a sound source distance is calculated and sound source separation processing is performed in accordance with the sound source distance

[0090] 7. Sequence of sound source separation processing performed by the information processing apparatus in the present disclosure

[0091] 8. Configuration example in which sound source separation processing is performed in an external device

[0092] 9. Hardware configuration example of the information processing apparatus in the present disclosure

[0093] 10. Summary of the configuration in the present disclosure

[0094] [1. Overview of processing performed by the information processing apparatus in the present disclosure]

[0095] First, 1. An overview of processing performed by the information processing apparatus according to the present disclosure will be described.

[0096] Figure 1 An example of a smartphone that will be described as an example of the information processing apparatus 100 in the present disclosure is illustrated.

[0097] The smartphone is an example of the information processing apparatus 100 in the present disclosure, and the information processing apparatus 100 in the present disclosure is not limited to the smartphone, and can be one of various apparatuses including, for example, a PC, a tablet terminal, and the like.

[0098] Figure 1 The user 1 illustrated in FIG. 1 sings a song with a guitar played by the user. A vocal sound that is the user 1's singing voice and a guitar sound that is a performance sound are input to Figure 1 a microphone 101 of the information processing apparatus (smartphone) 100 illustrated in FIG. 1, and are stored (recorded) in a memory (storage unit) 102 inside the information processing apparatus 100.

[0099] When the recording is completed, next, the user 1 can start sound source separation processing on the audio data stored (recorded) in the memory (storage unit) 102 inside the information processing apparatus 100.

[0100] For example, the display unit of the information processing device 100 displays something like... Figure 2 The diagram shows the user interface (UI), which the user uses to request the start of audio source separation processing.

[0101] Figure 2 The UI shown in the middle diagram is used to select and specify the type of sound source included in the recorded audio data.

[0102] (a) Vocals and guitar

[0103] (b) Vocals and piano

[0104] Figure 2 The UI shown in the image allows you to choose one of these two options.

[0105] If the recorded audio data includes both vocals and guitar sounds, select "(a) Vocals and Guitar".

[0106] If the recorded audio data includes human voices and piano sounds, select "(b) Human voices and piano".

[0107] When the user uses Figure 2 When the user selects the type of audio source included in the recorded audio data using the UI shown in the figure and requests to start audio source separation, the data processing unit (audio source separation processing execution unit) in the information processing device 100 starts audio source separation processing.

[0108] As a result of the audio source separation process, an audio data file for each audio source is generated and stored in a memory (storage unit).

[0109] Once the audio source separation process is complete, the user can perform editing on the audio data of each audio source.

[0110] For example, the display unit of the information processing device 100 displays such as Figure 3 The editing screen shown in the image.

[0111] exist Figure 3 In the example shown in the diagram, the audio data of the human voice and the audio data of the guitar sound, generated through sound source separation processing, are displayed next to each other.

[0112] The upper part illustrates the pitch change data of human voice audio data over time. The “C8, B7, A#7, A7, G#7, G7” shown on the left are pitch identification codes 104, indicating the pitch.

[0113] The data shown from left to right illustrates the changes in pitch in a human voice over time.

[0114] When the user touches the display area change button 105L at the left end, the display area changes, and the pitch change data of the earlier time is displayed. When the user touches the display area change button 105R at the right end, the display area changes, and the pitch change data of the later time is displayed.

[0115] Note that the processing of changing the display area can also be performed by a swipe processing of sliding a finger on the screen.

[0116] The lower portion illustrates the pitch change data of the audio data of the guitar sound over time.

[0117] Similar to the vocal sound, the pitch change data is based on "C8, B7, A#7, A7, G#7, G7" illustrated at the left end.

[0118] In the top portion of the display unit, two individual sound source playback request buttons 106 and one all sound source playback request button 107 are displayed.

[0119] If the user touches the sound source playback request button 106 on the left side among the two individual sound source playback request buttons 106, the vocal sound is played back. If the user touches the individual sound source playback request button 106 on the right side, the guitar sound is played back.

[0120] If the user touches the all sound source playback request button 107, the processing of playing back the audio data in which the vocal sound and the guitar sound are combined is performed.

[0121] Note that the example of selecting the sound source to be played back using the playback request buttons illustrated in Figure 3 is one example. The following configuration is also possible: for example, without providing the playback request buttons, a check box corresponding to each of the sound tracks of the vocal sound and the guitar sound is provided, and the sound source (sound track) selected by the user is played back.

[0122] Note that the audio data of each sound source is generated by the sound source separation processing described above with reference to Figure 2 , and is stored in the storage (storage unit). The audio data in which the vocal sound and the guitar sound are combined is an audio data file generated by first recording, and is also stored in the storage (storage unit).

[0123] Note that the audio data stored in the storage (storage unit) can be edited, that is, changed by the user using the editing UI as illustrated in Figure 3 , and the changed audio data file can be stored in the storage (storage unit).

[0124] Note that the Figure 3The audio data editing screen shown in the image (that is, the editing UI) is an example, and various editing buttons, icons, messages, etc. are displayed depending on the specific editing process.

[0125] Specific examples of these UIs will be described later.

[0126] Furthermore, although as Figure 3 The UI of the editing screen shown in the figure displays the time transformation data of pitch changes in each sound source on a set of pitch identifiers for each individual sound source, but alternatively, the time transformation data of pitch changes in multiple sound sources can be displayed together on a set of pitch identifiers.

[0127] Note that the description will follow later. Figure 4 The illustrations in the middle illustrate examples of display data (UI) processing for audio data editing, that is, display examples of various types of display data (UI) and examples of using display data (UI).

[0128] [2. Example of the configuration of the information processing device in this disclosure]

[0129] Next, an example of the configuration of the information processing apparatus in this disclosure will be described.

[0130] Figure 5 This is a block diagram illustrating an example configuration of the information processing apparatus 100 in this disclosure.

[0131] like Figure 5 As illustrated in the figure, the information processing device 100 of this disclosure includes an audio input unit (microphone) 111, a storage unit 112, an audio source separation processing execution unit 113, a trained digital neural network (DNN) 114, an audio data editing and processing unit 115, a playback control unit 116, a display unit 117, and an audio output unit (speaker) 118.

[0132] It should be noted that, for Figure 5 The components (processing blocks) illustrated in the information processing apparatus 100 are the main components necessary for performing the processes described in this disclosure. In addition to these components, the information processing apparatus 100 also includes various other components. For example, there are various components such as communication units, input units, control units, and cameras.

[0133] The following will describe Figure 5 Details of the components of the information processing device 100 shown in the figure.

[0134] The audio input unit (microphone) 111 includes a microphone, an amplifier, an A / D converter, etc., receives the user's voice and the sound of a guitar, etc., input through the microphone, performs digital processing, and stores the digital audio data in the storage unit 112.

[0135] The storage unit 112 records audio data input via the audio input unit (microphone) 111.

[0136] The sound source separation processing execution unit 113 executes the audio data recorded in the storage unit 112 and separates the audio data into data corresponding to each sound source included in the audio data.

[0137] Details of the sound source separation processing will be described later.

[0138] Note that the audio data file of each sound source separated by the sound source separation processing execution unit 113 is stored in the storage unit 112.

[0139] The trained model (DNN) 114 is a trained model used in the sound source separation processing executed by the sound source separation processing execution unit 113, for example, a neural network such as a deep neural network (DNN).

[0140] Details of the trained model (DNN) 114 used by the information processing apparatus in the present disclosure will be described later.

[0141] The audio data editing processing unit 115 is, for example, a data processing unit that displays a UI as described above with reference to Figure 3 and Figure 4 and executes processing of editing the audio data of each sound source according to an instruction of the user.

[0142] The audio data editing processing unit 115 reads the audio data file of each sound source separated by the sound source separation processing execution unit 113 from the storage unit 112, displays the time transform data of the pitch change in each sound source described with reference to Figure 3 and Figure 4 for example, on the display unit 117, edits the audio data of each sound source according to an instruction of the user, and stores the result in the storage unit 112.

[0143] The playback control unit 116 executes processing of reading the audio data stored in the storage unit 112 from the storage unit 112 (for example, the audio data at the time of recording or the audio data file of each sound source separated by the processing execution unit 113) and playing back the audio data.

[0144] The display unit 117 displays a UI described with reference to Figure 3 and Figure 4 for example. Note that the display unit 117 is a touch panel display unit and has a configuration capable of executing various types of input processing and instructions by user operation.

[0145] For example, the audio output unit (speaker) 118 is a speaker that outputs audio data and the like that is played back by processing performed by the playback control unit 116.

[0146] [3. Detailed configuration of the sound source separation processing execution unit of the information processing apparatus in the present disclosure and details of the sound source separation processing]

[0147] Next, the detailed configuration of the sound source separation processing execution unit 113 of the information processing apparatus 100 in the present disclosure and the details of the sound source separation processing illustrated in the upper left portion will be described. Figure 5

[0148] A specific configuration example of the sound source separation processing execution unit 113 and a specific processing performed by the sound source separation processing execution unit 113 will be described with reference to Figure 6

[0149] The sound source separation processing execution unit 113 receives input data illustrated in the upper left portion from the storage unit 112. Figure 6

[0150] The input data is audio data in which a human voice and a guitar sound emitted by the user 1 are mixed, and is audio data that changes along a time axis.

[0151] The sound source separation processing execution unit 113 receives audio data in which a human voice and a guitar sound are mixed, and performs processing of extracting only the human voice and processing of extracting only the guitar sound.

[0152] The sound source separation processing execution unit 113 performs the sound source separation processing using a trained model including a neural network (for example, a neural network such as a deep neural network (DNN)).

[0153] As illustrated in the upper left portion of Figure 6 , the sound source separation processing execution unit 113 includes:

[0154] a series of data processing sections including a preprocessing section (STFT) 113a1, a human voice separation section (human voice separation DNN) 113a2, and a post-processing section (ISTFT) 113a3; and

[0155] a series of data processing sections including a preprocessing section (STFT) 113b1, a guitar sound separation section (guitar sound separation DNN) 113b2, and a post-processing section (ISTFT) 113b3.

[0156] The human voice included in the input data is extracted by the components illustrated in the upper portion, that is, the preprocessing section (STFT) 113a1, the human voice separation section (human voice separation DNN) 113a2, and the post-processing section (ISTFT) 113a3.

[0157] ​​​On the other hand, the guitar sound included in the input data is extracted by the components illustrated in the lower portion, that is, a pre-processing section (STFT) 113b1, a guitar sound separation section (guitar sound separation DNN) 113b2, and a post-processing section (ISTFT) 113b3.

[0158] The pre-processing section (STFT) 113a1 illustrated in the upper portion performs a short-time Fourier transform (STFT) on the input data.

[0159] With the STFT processing, the input data, which is data in a time domain in which an audio signal changes along a time axis, is converted into data in a frequency domain (spectrum).

[0160] This converted data is input to a vocal sound separation section (vocal sound separation DNN) 113a2.

[0161] The vocal sound separation section (vocal sound separation DNN) 113a2 is a sound source separation section configured using a trained model such as a neural network or, more specifically, for example, a deep neural network (DNN).

[0162] The vocal sound separation section (vocal sound separation DNN) 113a2 is a neural network (trained model) that extracts sound data having a feature (frequency or the like) of a vocal sound corresponding to a human voice from audio data of various sound sources included in the input data.

[0163] This neural network (trained model) is a neural network (trained model) generated through a training process performed in advance and provided with parameters for extracting sound data having a feature of a vocal sound corresponding to a human voice.

[0164] The vocal sound separation section (vocal sound separation DNN) 113a2 extracts audio data corresponding to a vocal sound corresponding to a human voice from audio data of various sound sources included in the input data.

[0165] The audio data extracted by the vocal sound separation section (vocal sound separation DNN) 113a2 is input to a post-processing section (ISTFT) 113a3.

[0166] The post-processing section (ISTFT) 113a3 performs inverse processing of the processing performed by the aforementioned pre-processing section (STFT) 113a1. That is, a short-time inverse Fourier transform (ISTFT) is performed.

[0167] With the ISTFT processing, the data in the frequency domain (spectrum) is converted into data in a time domain in which an audio signal changes along an original time axis.

[0168] The output of the post-processing section (ISTFT) 113a3 is the audio data extracted by the vocal sound separation section (vocal sound separation DNN) 113a2, that is, the audio data of a vocal sound.

[0169] On the other hand, the pre-processing section (STFT) 113b1 illustrated in the lower portion also performs a short-time Fourier transform (STFT) on the input data to convert the input data, which is data in the time domain, into data in the frequency domain (spectrum).

[0170] This converted data is input to the guitar sound separation section (guitar sound separation DNN) 113b2.

[0171] The guitar sound separation section (guitar sound separation DNN) 113b2 is also a sound source separation section configured using a trained model such as a neural network or, more specifically, for example, a deep neural network (DNN).

[0172] The guitar sound separation section (guitar sound separation DNN) 113b2 is a neural network (trained model) that extracts sound data having a feature (frequency, etc.) of a guitar sound from audio data of various sound sources included in the input data.

[0173] This neural network (trained model) is also a neural network (trained model) generated by a training process performed in advance and provided with parameters for extracting sound data having a feature of a guitar sound.

[0174] The guitar sound separation section (guitar sound separation DNN) 113b2 extracts audio data corresponding to a guitar sound from audio data of various sound sources included in the input data.

[0175] The audio data extracted by the guitar sound separation section (guitar sound separation DNN) 113b2 is input to the post-processing section (ISTFT) 113b3.

[0176] The post-processing section (ISTFT) 113b3 performs reverse processing of the pre-processing section (STFT) 113b1. That is, a short-time inverse Fourier transform (ISTFT) is performed.

[0177] With the ISTFT processing, data in the frequency domain (spectrum) is converted into data in the time domain in which the audio signal changes along the original time axis.

[0178] The output of the post-processing section (ISTFT) 113b3 is the audio data extracted by the guitar sound separation section (guitar sound separation DNN) 113b2, that is, the audio data of the guitar sound.

[0179] As described above, in a configuration in which a sound source separation process is performed using a neural network such as a DNN, a sound source-specific neural network corresponding to the type of sound source to be separated (that is, for example, the type of sound source to be separated such as a human voice, a guitar, a piano, or a violin) is used.

[0180] In other words, a neural network with parameters set specifically for the sound source to be separated is used.

[0181] [4. Detailed configuration of the audio data editing and processing unit of the information processing apparatus of this disclosure and details of the audio data editing and processing performed by the information processing apparatus of this disclosure]

[0182] Next, we will describe Figure 5 The figure shows the detailed configuration of the audio data editing and processing unit 115 of the information processing apparatus 100 in this disclosure, as well as the details of the audio data editing and processing performed by the information processing apparatus in this disclosure.

[0183] First, refer to Figure 7 The detailed configuration of the audio data editing and processing unit 115 of the information processing apparatus 100 in this disclosure is described.

[0184] As referenced above Figure 5 As described, the audio data editing and processing unit 115 displays as shown Figure 3 and Figure 4 The UI shown in the diagram is a data processing unit that performs editing of audio data for each sound source based on user instructions.

[0185] In other words, the audio data editing and processing unit 115 reads the audio data file of each audio source separated by the audio source separation processing execution unit 113 from the storage unit 112, and, for example, refers to... Figure 3 and Figure 4 The time-varying data of pitch changes in each sound source described are displayed on the display unit 117. The audio data of each sound source is edited according to the user's instructions, and the results are stored in the storage unit 112.

[0186] Figure 7 This is a block diagram illustrating the detailed configuration of the audio data editing and processing unit 115.

[0187] like Figure 7 As shown in the figure, the audio data editing and processing unit 115 includes a sound source data analysis unit 121, a display data generation unit 122, an instruction analysis unit (correction / playback, etc.) 123, a sound source data correction unit 124, and a sound source synthesis unit 125.

[0188] Note that the audio source data analysis unit 121 includes the first audio source data analysis section 131-1 to the nth audio source data analysis section 131-n and the comprehensive analysis section 132.

[0189] In addition, the audio source data correction unit 124 includes a first audio source data correction unit 141-1 to an nth audio source data correction unit 141-n.

[0190] The sound source data analysis section 121 analyzes the audio data of each sound source separated by the sound source separation processing execution section 113. That is, for example, an audio data analysis process of each sound source included in the recorded data such as a human voice (vocal) or a guitar sound is executed. For example, a transformation of a pitch of each sound source, a shift of the pitch, a transformation of a volume, and the like are analyzed.

[0191] For example, the first sound source data analysis section 131-1 executes a process of analyzing a human voice (vocal), and the second sound source data analysis section 131-2 executes a process of analyzing a guitar sound.

[0192] Note that, in the example described above with reference to Figures 1 to 4 the sound input to the information processing apparatus 100 and recorded is a sound including two sound sources of a human voice and a guitar sound, but Figure 7 The sound source data analysis section 121 of the sound source data analysis section 121 illustrated in

[0193] The comprehensive analysis section 132 receives the analysis result of each sound source data from the first sound source data analysis section 131-1 to the n-th sound source data analysis section 131-n, and analyzes a pitch shift, a timing shift, and the like between the sound sources.

[0194] The display data generation section 122 receives the analysis result of each sound source and the comprehensive analysis result from the sound source data analysis section 121, generates display data for presenting these analysis results to the user, and displays the display data on the display section 117.

[0195] For example, the pitch transformation data of each sound source to be displayed in the display data (UI) described above with reference to Figure 3 and Figure 4 is generated and displayed.

[0196] Note that, based on the analysis result of each sound source and the comprehensive analysis result from the sound source data analysis section 121, the display data generation section 122 also generates, for example, a message asking the user whether to correct the sound source data as a correction candidate, and outputs the message or the like to the display section 117.

[0197] Specifically, for example, a region in which a timing shift occurs between the respective sound source data is presented, and a message asking the user whether to execute the correction is generated and displayed.

[0198] Note that, later, specific examples of the display data generated by the display data generation section 122 will be described.

[0199] The instruction analysis section (correction / playback, etc.) 123 analyzes a correction instruction and a playback instruction from the user with respect to the analysis result of each sound source data generated by the display data generation section 122 and displayed on the display unit 117.

[0200] Note that the display unit 117 is configured as, for example, a touch panel display unit, and is configured so that the user can make various instruction inputs with a picture touch process.

[0201] For example, in a case where the instruction analysis section (correction / playback, etc.) 123 receives a correction instruction via the display unit (input unit) 117, the input correction instruction is input to the sound source data correction section 124, and various types of correction processing according to the user instruction are executed in the sound source data correction section 124.

[0202] Further, in a case where the instruction analysis section (correction / playback, etc.) 123 receives a playback instruction via the display unit (input unit) 117, the playback instruction is input to the playback control unit 116, and playback processing is executed.

[0203] Note that the playback instruction can be given for either of playback processing of each sound source data and processing of playing back a plurality of sound source data all at once, and playback processing according to the user's designation is executed.

[0204] As Figure 7 As illustrated in FIG. 1, the sound source data correction section 124 includes a first sound source data correction section 141-1 to an n-th sound source data correction section 141-n. Each of the sound source data correction sections 141-1 to 141-n of the sound source data correction section 124 executes correction of each sound source data, for example, correction of pitch shift, correction of timing shift, and the like, based on a user instruction input via the instruction analysis section (correction / playback, etc.) 123.

[0205] Each sound source data corrected by the sound source data correction section 124 is stored in the storage unit 112 and input to the sound source data synthesis unit 125.

[0206] The sound source data synthesis unit 125 synthesizes the sound source data corrected by the sound source data correction section 124, generates synthesized audio data of a plurality of corrected sound source data, and stores the synthesized audio data in the storage unit 112.

[0207] Note that, when each sound source data corrected by the sound source data correction section 124 and the synthesized audio data of a plurality of corrected sound source data generated by the sound source data synthesis unit 125 are stored in the storage unit 112, this data can be recorded as an overwriting data file of a data file before correction, or as a new data file different from the data file before correction.

[0208] [5. Specific examples of display data (UI) displayed by the information processing apparatus in the present disclosure, user operations on the UI, and correction processing]

[0209] Next, specific examples of display data (UI) displayed by the information processing apparatus 100 in the present disclosure, user operations on the UI, and correction processing will be described.

[0210] The following two processing examples will be described sequentially.

[0211] (Processing Example 1) Processing example of displaying a UI that can identify a timing offset region between individual sound source data and correcting a timing offset between individual sound source data according to a user operation on the UI

[0212] (Processing Example 2) Processing example of displaying a UI that can identify a pitch offset region in sound source data and correcting a pitch offset in sound source data according to a user operation on the UI

[0213] (5-1. (Processing Example 1) Processing example of displaying a UI that can identify a timing offset region between individual sound source data and correcting a timing offset between individual sound source data according to a user operation on the UI)

[0214] First, the following processing example will be described as (Processing Example 1): a UI that can identify a timing offset region between multiple sound source data is displayed, and a timing offset between the multiple sound source data is corrected according to a user operation on the UI.

[0215] The following three processing examples will be described sequentially with reference to Figure 8 and the subsequent drawings.

[0216] (Processing Example 1a) Processing example of displaying a sound source data correction example for resolving a timing offset in sound source data

[0217] (Processing Example 1b) Processing example of clearly indicating a timing offset region in sound source data and allowing a user to select sound source data to be corrected

[0218] (Processing Example 1c) Processing example of displaying a timing offset check line for multiple sound source data

[0219] (5-1-1. (Processing Example 1a) Processing example of displaying a sound source data correction example for resolving a timing offset in sound source data)

[0220] First, the following processing example will be described as (Processing Example 1a): an example of a sound source data correction for resolving a timing offset in sound source data is displayed.

[0221] Figure 8is a diagram illustrating an example of display data (UI) displayed on the display unit of the information processing apparatus 100 in the present disclosure.

[0222] Note that the display data (UI) is generated by the display data generation section 122 of the audio data editing processing unit 115 illustrated in Figure 7

[0223] Figure 8 The display data (UI) illustrated in is an UI similar to the UI described above with reference to Figure 4 and shows the change in pitch over time in the two sound source data on the pitch identification codes "C8, B7, A#7, A7, G#7, G7" indicating the pitches.

[0224] That is, the pitch change data of the audio data of the human voice over time and the pitch change data of the audio data of the guitar sound over time are illustrated adjacent to each other.

[0225] The two sound source data (human voice and guitar sound) are the two sound source data generated by the sound source separation processing performed by the sound source separation processing execution unit 113 of the information processing apparatus 100 illustrated in Figure 5 The pitch change data of the two sound source data (human voice and guitar sound) over time is generated by the sound source data analysis processing of the sound source data analysis section 121 described above with reference to

[0226] Figure 7

[0227] Further, Figure 7 The comprehensive analysis unit 132 of the sound source data analysis section 121 illustrated in analyzes the timing offset between the two sound source data (human voice and guitar sound) and detects that the audio output timing tl of the human voice and the audio output timing t2 of the guitar sound do not collide with each other. Figure 8

[0228] That is, it is detected that the timing of outputting the two sound source data (human voice and guitar sound) are offset from each other.

[0229] The comprehensive analysis unit 132 in the sound source data analysis section 121 outputs the detection result to the display data generation section 122.

[0230] The display data generation section 122 receives the detection result from the comprehensive analysis unit 132, and displays the display data (UI) as illustrated in Figure 8 ​​​​The display data (UI) illustrated in FIG. 13 is display data in which the correction example for correcting the offset is presented, display data (UI) in which a message asking whether or not to perform correction according to the correction example is generated, and the display data is output to the display unit 117.

[0231] Note that, in the display data (UI) illustrated in FIG. 13, the time information (t1 and t2) shown is not display data, but time information shown for the purpose of description. However, the time information (t1, t2) can also be displayed on the display data (UI). Figure 8

[0232] As illustrated in FIG. 13, the message asking whether or not to perform correction according to the correction example is the following message illustrated in the pitch display section of the audio output timing t2 of the guitar sound which is delayed from the audio output timing t1 of the vocal sound. Figure 8

[0233] "Has a delay. Correct the delay? (Yes / No)"

[0234] If the user views the message and determines to perform correction according to the correction example, the user inputs "Yes" as illustrated in FIG. 14. Figure 9 That is, the user touches the "Yes" button shown at the bottom.

[0235] This user input information is input to the instruction analysis section 123 in the audio data editing processing unit 115 of the information processing apparatus 100 illustrated in FIG. 12, and the instruction analysis section 123 determines that the user has requested correction according to the correction example, and outputs a correction according to the correction example execution request to the sound source data correction section 124. Figure 7

[0236] For example, the sound source data correction section 124 performs the following guitar sound correction processing: correcting the audio output timing t2 of the guitar sound so that it matches the audio output timing t1 of the vocal sound in the second sound source data correction section 141-2 illustrated in FIG. 15. Figure 7

[0237] When the correction processing of the sound source data correction section 124 is completed, the sound source data correction section 124 notifies the display data generation section 122 that the correction has been completed.

[0238] Upon receiving the correction completion notification from the sound source data correction section 124, the display data generation section 122 generates the display data (UI) illustrated in FIG. 16, and displays the display data on the display unit 117. Figure 10

[0239] The display data generation section 122 generates​​​​​Figure 10 The display data (UI) illustrated in FIG. 12 is generated, and the display data is displayed on the display unit 117.

[0240] That is, the display data (UI) indicating that the correction processing for matching the audio output timing of the guitar sound with the audio output timing tl of the vocal sound has been completed, and further including a message inquiring whether or not the correction is to be cancelled, is generated and output to the display unit 117.

[0241] As Figure 10 The inquiry message as to whether or not the correction is to be cancelled is the following message illustrated in the pitch display portion of the corrected audio output timing tl of the guitar sound in FIG. 12.

[0242] "Corrected. Cancel correction? (Yes / No)"

[0243] If the user has viewed the message and determined that the correction is to be cancelled, the user inputs "Yes" as illustrated in FIG. 13. That is, the user touches the "Yes" button shown at the bottom. Figure 11

[0244] This user input information is input to the instruction analysis section 123 in the audio data editing processing unit 115 of the information processing apparatus 100 illustrated in FIG. 11, and the instruction analysis section 123 determines that the user has requested cancellation of the correction, and outputs a request to perform re-correction for the cancelled correction to the sound source data correction section 124. Figure 7 For example, the sound source data correction section 124 performs the following guitar sound correction processing: in the second sound source data correction section 141-2 that performs the guitar sound source data correction processing illustrated in FIG. 10, the audio output timing of the guitar sound is reset to a timing t2 different from the audio output timing tl of the vocal sound.

[0245] Figure 7 When the correction processing of the sound source data correction section 124 is completed, the sound source data correction section 124 notifies the display data generation section 122 that the correction has been completed.

[0246] Upon receiving the correction completion notification from the sound source data correction section 124, the display data generation section 122 generates the display data (UI) illustrated in FIG. 14, and displays the display data on the display unit 117.

[0247] The display data generation section 122 generates the display data (UI) illustrated in FIG. 12, and displays the display data on the display unit 117. Figure 12 The display data generation section 122 generates the display data (UI) illustrated in FIG. 12, and displays the display data on the display unit 117.

[0248] Figure 12 The display data generation section 122 generates the display data (UI) illustrated in FIG. 12, and displays the display data on the display unit 117.

[0249] ​​​That is, display data (UI) is generated and output to the display unit 117, the display data indicating that the audio output timing of the guitar sound has been reset to a timing t2 different from the audio output timing tl of the vocal sound, and the display data further including a message "Undo" for notifying that the undo processing is complete.

[0250] On the other hand, as described above with reference to Figure 10 and Figure 11 if the user views the inquiry message about whether to undo the correction, that is,

[0251] "Corrected. Undo the correction? (Yes / No)"

[0252] and determines not to undo the correction, the user inputs "No", as illustrated in Figure 13 . That is, the user touches the "No" button shown at the bottom.

[0253] This user input information is input to the instruction analysis section 123 in the audio data editing processing unit 115 of the information processing apparatus 100 illustrated in Figure 7 , the instruction analysis section 123 determining that the user has requested not to undo the correction, confirming the correction processing for matching the audio output timing of the sound source data of the guitar sound with the audio output timing tl of the vocal sound, and the sound source data of the guitar sound being stored in the storage unit 112 as new sound source data that has undergone the correction processing.

[0254] Further, in a case where the sound source data of the vocal sound and the guitar sound at different times is displayed, the user touches the display area change request button 105R or the display area change request button 105L as illustrated in Figure 14 . Touch information about the display area change request button 105L is input to the instruction analysis section 123 illustrated in Figure 7 .

[0255] When it is detected that the user has performed the touch processing of the display area change request button 105L, the instruction analysis section 123 notifies the display data generation section 122 that the display area change request has been made.

[0256] The display data generation section 122 changes the display area of the display data on the display unit 117 based on the notification input from the instruction analysis section 123.

[0257] (5-1-2. (Process example lb) Process example in which the timing offset region is clearly indicated in the sound source data and the user is allowed to select the sound source data to be corrected)

[0258] Next, the following process example will be described as (Process example lb): a process in which the timing offset region is clearly indicated in the sound source data and the user is allowed to select the sound source data to be corrected.

[0259] A specific example (processing example 1b) will be described with reference to Figure 15 and the subsequent drawings.

[0260] Figure 15 is a view that illustrates an example of display data (UI) displayed on the display unit of the information processing apparatus 100 in the present disclosure.

[0261] The display data (UI) is generated by the display data generation section 122 of the audio data editing processing section 115 illustrated in Figure 7

[0262] Figure 15 The display data (UI) illustrated in is similar to the UI described above with reference to Figures 8 to 14 and shows the change in pitch over time in the two sound source data on the pitch identification codes "C8, B7, A#7, A7, G#7, G7" indicating the pitch.

[0263] The two sound source data (vocal and guitar) are the two sound source data generated by the sound source separation processing performed by the sound source separation processing execution section 113 of the information processing apparatus 100 illustrated in Figure 5

[0264] In addition, the pitch change data of the two sound source data (vocal and guitar) over time is analyzed by the sound source data analysis section 121 described with reference to Figure 7 and the timing offset between the two sound source data (vocal and guitar) is analyzed by the comprehensive analysis section 132 in the sound source data analysis section 121.

[0265] The comprehensive analysis section 132 in the sound source data analysis section 121 detects that the audio output timing t1 of the vocal and the audio output timing t2 of the guitar illustrated in Figure 15 are offset from each other and outputs the detection result to the display data generation section 122.

[0266] The display data generation section 122 receives the detection result from the comprehensive analysis section 132, generates the display data (UI) illustrated in Figure 15 and outputs the display data to the display unit 117.

[0267] As illustrated in Figure 15 , the display data (UI) is display data including a timing offset region indication frame 201 and a timing offset correction sound source selection button 202.

[0268] The timing offset region indication frame 201 is an identification frame for indicating to the user in an easy-to-understand manner the region in which the audio output timing of the vocal and the audio output timing of the guitar are offset. ​​

[0269] The timing offset correction sound source selection button 202 is a button for allowing the user to select the sound source data (vocal or guitar sound) to be corrected.

[0270] The user can specify the sound source data to be corrected by identifying the timing offset of the two sound source data (vocal and guitar sound) in the area surrounded by the timing offset area indication frame 201, and then touching one of the timing offset correction sound source selection buttons 202 displayed in correspondence with the sound source data.

[0271] Figure 16 An example is illustrated in which the user has selected and touched the timing offset correction sound source selection button 202 on the vocal side.

[0272] This user input information is input to the instruction analysis section 123 in the audio data edit processing unit 115 of the information processing apparatus 100 illustrated in Figure 7 The instruction analysis section 123 determines that the user has selected the sound source data of the vocal as the sound source data whose timing offset is to be corrected, and outputs a correction execution request to the sound source data correction section 124 to match the output timing of the sound source data of the vocal with the output timing of the sound source data of the guitar.

[0273] For example, the sound source data correction section 124 performs the following vocal correction processing: correcting the audio output timing tl of the vocal so that the audio output timing tl of the vocal matches the audio output timing t2 of the guitar in the first sound source data correction section 141-1 illustrated in Figure 7 The audio output timing tl of the vocal matches the audio output timing t2 of the guitar in the first sound source data correction section 141-1 illustrated in

[0274] When the correction processing of the sound source data correction section 124 is completed, the sound source data correction section 124 notifies the display data generation section 122 that the correction has been completed.

[0275] Upon receiving the correction completion notification from the sound source data correction section 124, the display data generation section 122 generates the display data (UI) illustrated in Figure 17 and displays the display data on the display unit 117.

[0276] That is, the display data is display data indicating that the correction processing for matching the audio output timing of the vocal with the audio output timing t2 of the guitar has been completed, and is display data for notifying that the area surrounded by the timing offset correction area indication frame 203 is a correction portion. Furthermore, display data (UI) including a message asking whether or not to cancel the correction is generated and output to the display unit 117.

[0277] As illustrated in Figure 17The inquiry message about whether to cancel the correction is the following message illustrated in the pitch display part of the corrected audio output timing t2 of the vocal sound in the middle drawing.

[0278] "Corrected. Cancel the correction? (Yes / No)"

[0279] If the user views the message and determines to cancel the correction, the user touches Figure 17 "Yes" at the bottom of the display data (UI) illustrated in the middle drawing. On the other hand, if the user determines not to cancel the correction, the user touches Figure 17 "No" at the bottom of the display data (UI) illustrated in the middle drawing.

[0280] Figure 17 The example illustrated in the middle drawing is an example in which the user has determined not to cancel the correction and has touched "No".

[0281] This user input information is input to Figure 7 the instruction analysis section 123 in the audio data editing processing unit 115 of the information processing apparatus 100 illustrated in the middle drawing, and the instruction analysis section 123 determines that the user has requested not to cancel the correction, confirms the correction processing for matching the audio output timing of the vocal sound source data with the audio output timing t2 of the guitar sound, and the vocal sound source data is stored in the storage unit 112 as new sound source data that has undergone the correction processing.

[0282] Further, in a case where the sound source data of the vocal sound and the guitar sound at different times is displayed, the user touches the display area change request button 105R or the display area change request button 105L as illustrated in Figure 18 the middle drawing. The touch information about the display area change request button 105L is input to Figure 7 the instruction analysis section 123 illustrated in the middle drawing.

[0283] When it is detected that the user has performed the touch processing of the display area change request button 105L, the instruction analysis section 123 notifies the display data generation section 122 that the display area change request has been made.

[0284] The display data generation section 122 changes the display area of the display data on the display unit 117 on the basis of the notification input from the instruction analysis section 123.

[0285] Figure 19 The modification of the display data (UI) using the timing offset area indication frame 201 is illustrated. Figure 19 The display data (UI) illustrated in the middle drawing is an example in which the timing offset check line 204 is displayed in the timing offset area indication frame 201.

[0286] The timing offset check line 204 is a straight line extending from the audio output timing position of one sound source data to the same timing position of the pitch conversion data of another sound source, and is a line that makes it easy to understand the offset in the audio output timing of each sound source data (vocal and guitar).

[0287] As Figure 19 illustrated in (B), by further displaying the timing offset check line 204 in the timing offset region indication frame 201, the user can clearly confirm that the audio output timing of each sound source data is offset.

[0288] Figure 20 Another modification of the display data (UI) using the timing offset region indication frame 201 is illustrated. Figure 20 The display data (UI) illustrated in (B) is an example in which a timing offset check frame 205 is displayed in the timing offset region indication frame 201.

[0289] The timing offset check frame 205 is a frame that encloses the data region of the pitch conversion data of the sound source offset from the audio output timing position of one sound source data.

[0290] As Figure 20 illustrated in (B), by further displaying the timing offset check frame 205 in the timing offset region indication frame 201, the user can clearly confirm that the audio output timing of each sound source data is offset.

[0291] (5-1-3. (Process example 1c) Process example of displaying timing offset check lines of multiple sound source data)

[0292] Next, the following process example will be described as (Process example 1c): display of timing offset check lines of multiple sound source data.

[0293] Specific examples (Process example 1c) thereof will be described with reference to Figure 21 and the subsequent drawings.

[0294] Figure 21 The display data (UI) illustrated in (B) is also display data (UI) generated by the display data generation section 122 of the audio data editing processing unit 115 illustrated in (A). Figure 7 The display data (UI) illustrated in (B) is also display data (UI) generated by the display data generation section 122 of the audio data editing processing unit 115 illustrated in (A).

[0295] In Figure 21 The display data (UI) illustrated in (B) is also display data (UI) generated by the display data generation section 122 of the audio data editing processing unit 115 illustrated in (A).

[0296] The timing offset check line 204 is a straight line extending from a timing position of the audio output timing and the audio end timing of one of the sound source data (reference sound source data) selected from the plurality of sound source data to the same timing position of the pitch conversion data of another sound source, and is a line that makes it easy to understand the offset in the audio output timing of each of the sound source data (vocal and guitar).

[0297] In Figure 21 the example illustrated in FIG. 10, one of the sound source data selected from the plurality of sound source data (vocal and guitar) is the sound source data of the guitar, and a straight line extending from a timing position such as the audio output timing position or the audio end timing of the sound source data of the guitar (reference sound source data) to the same timing position of the pitch conversion data of another sound source (vocal) is shown as the timing offset check line 204.

[0298] By observing these timing offset check lines 204, the user can easily confirm the portion in which there is an audio output timing or audio end timing offset between the plurality of sound source data.

[0299] Note that whether or not to display the timing offset check line 204 can be selected by the user.

[0300] The timing offset check line 204 can be displayed by turning on the setting of the timing offset check line display / non-display selection box 206 at the bottom of the display data (UI) illustrated in Figure 21 FIG. 10.

[0301] On the other hand, the timing offset check line 204 can be removed by turning off the setting of the timing offset check line display / non-display selection box 206 at the bottom of the display data (UI) illustrated in Figure 22 FIG. 10.

[0302] Note that although the above has described the plurality of processing examples 1a to 1c of the UI in which the timing offset region between the plurality of sound source data can be identified and the timing offset between the plurality of sound source data is corrected in accordance with the user operation on the UI as (processing example 1), in any of these processing examples, the processing of playing back each of the sound source data or the processing of playing back all of the sound source data can be performed when each of the sound source data is corrected.

[0303] Figure 23 is an example of the display data (UI) that displays a message requesting determination of whether or not to perform playback and whether or not to perform the correction processing.

[0304] As illustrated in Figure 24 FIG. 10, if, for example, the user touches the all sound source playback request button 107 displayed at the top of the display data (UI) in accordance with this message, the processing of playing back all of the sound sources (vocal and guitar) is performed.

[0305] If the user touches the individual sound source playback request button 106, processing to play back each sound source (vocal and guitar) is performed.

[0306] The user can determine whether or not to perform the correction processing after hearing these playback sounds. In addition, by performing the playback processing after the correction, it can be determined whether or not the correction has been properly performed, whether or not to perform the undo processing, and the like.

[0307] Note that, although Figure 23 and 24 An example in which a message inquiring whether or not to perform playback is displayed is illustrated, but even in the case where such a message is not displayed, the user can perform the playback processing by touching the individual sound source playback request button 106 or the all sound source playback request button 107 displayed at the top of the display data (UI).

[0308] (Processing Example 2) Processing Example in Which a UI That Can Identify a Pitch Shift Region in Sound Source Data Is Displayed and a Pitch Shift in the Sound Source Data Is Corrected in Accordance with a User Operation on the UI

[0309] (5-2. (Processing Example 2) Processing Example in Which a UI That Can Identify a Pitch Shift Region in Sound Source Data Is Displayed and a Pitch Shift in the Sound Source Data Is Corrected in Accordance with a User Operation on the UI)

[0310] First, the following processing example will be described as (Processing Example 2): a UI that can identify a pitch shift region in sound source data is displayed, and a pitch shift in the sound source data is corrected in accordance with a user operation on the UI.

[0311] Figure 25 An example of display data (UI) in the present processing example 2 is illustrated.

[0312] The display data (UI) is generated by the display data generation section 122 of the audio data editing processing unit 115 illustrated in Figure 7

[0313] In the display data (UI) illustrated in Figure 25

[0314] Figure 25 The example illustrated in

[0315] The user can know from the display data that the pitch region of a part of the sound source data of the guitar deviates from the pitch of the subsequent data. ​​

[0316] Figure 25 The display data (UI) illustrated in FIG. 8 further displays the following inquiry message.

[0317] “Pitch shift. Correct shift? (Yes / No)

[0318] “Play it back and see?”

[0319] The user can respond to the inquiry message to request to perform or not to perform the correction, to perform the playback processing, and the like.

[0320] In a case where the correction is to be performed, as illustrated in FIG. 8, the user touches the “Yes” button at the bottom of the display data (UI). Figure 26 The user input information is input to the instruction analysis section 123 in the audio data editing processing unit 115 of the information processing apparatus 100 illustrated in FIG. 8. The instruction analysis section 123 determines that the user has requested the pitch shift correction processing of the guitar sound, and outputs the execution request of the pitch shift correction processing of the guitar sound to the sound source data correction section 124.

[0321] Figure 7 For example, in the second sound source data correction section 141-2 illustrated in FIG. 8, which performs the sound source data correction processing of the guitar sound, the sound source data correction section 124 performs the processing of correcting the pitch shift in the guitar sound.

[0322] For example, in the second sound source data correction section 141-2 illustrated in FIG. 8, which performs the sound source data correction processing of the guitar sound, the sound source data correction section 124 performs the processing of correcting the pitch shift in the guitar sound. Figure 7 When the correction processing of the sound source data correction section 124 is completed, the sound source data correction section 124 notifies the display data generation section 122 that the correction has been completed.

[0323] Upon receiving the correction completion notification from the sound source data correction section 124, the display data generation section 122 displays the sound source data of which the pitch of the guitar sound has been corrected, generates the display data (UI) that displays the pitch shift correction section frame 232 that encloses the pitch shift correction section, and displays the display data on the display unit 117, as illustrated in FIG. 8.

[0324] Figure 27 The display data generation section 122 further displays the following message, as illustrated in FIG. 8.

[0325] The display data generation section 122 further displays the following message, as illustrated in FIG. 8. Figure 27 “Corrected. Undo correction? (Yes / No)

[0326] “Play it back and see?”

[0327] If, for example, the user touches the full sound playback request button 107 displayed at the top of the display data (UI) in accordance with the message, the processing of playing back all the sound sources (the vocal sound and the guitar sound) is performed, as illustrated in FIG. 8.

[0328] Figure 28 If, for example, the user touches the full sound playback request button 107 displayed at the top of the display data (UI) in accordance with the message, the processing of playing back all the sound sources (the vocal sound and the guitar sound) is performed, as illustrated in FIG. 8. ​​​

[0329] Note that if a user touches the individual audio source playback request button 106, the process of playing back each audio source (vocals and guitar sounds) will be performed.

[0330] Users can decide whether to undo the correction after hearing these playback sounds. By performing playback after correction, it is possible to determine whether the correction has been performed correctly and whether undoing is necessary.

[0331] [6. Example of calculating sound source distance and performing sound source separation processing based on sound source distance]

[0332] Next, an embodiment of calculating the distance between sound sources and performing sound source separation processing based on the sound source distance will be described.

[0333] The above is for reference only. Figure 5 The sound source separation processing execution unit 113 of the information processing apparatus 100 described in this disclosure uses a trained model (that is, a trained model including neural networks such as DNNs) to perform sound source separation processing, as described in Reference Figure 6 Described.

[0334] Specifically, a source-specific trained model (DNN, etc.) is used that corresponds to the type of sound source to be separated (e.g., human voice, guitar, piano, violin, etc.).

[0335] In other words, sound source separation processing is performed using a trained model (such as a DNN) with parameters set specifically for the sound source to be separated.

[0336] In the embodiments described below, a trained model (DNN, etc.) is used to perform optimal sound source extraction processing based on the sound source distance (distance from the microphone to the sound source). That is, sound source separation processing is performed using a trained model (DNN, etc.) corresponding to the sound source distance.

[0337] The trained model (DNN, etc.) used to perform optimal sound source extraction processing based on the sound source distance (distance from the microphone to the sound source) is created in advance through training processes such as supervised training.

[0338] In addition, when recording audio data that will undergo source separation processing, the source distance (distance from the microphone to the source) included in the recorded data is calculated or input, and the source separation processing execution unit performs source separation processing using a trained model (DNN, etc.) corresponding to the source distance based on the calculated or input source distance.

[0339] Figure 29 The illustration shows a configuration example of the information processing apparatus 300 according to this embodiment, that is, an information processing apparatus 300 that performs sound source separation processing based on the distance of the sound source.

[0340] As Figure 29 illustrated in FIG. 12, the information processing apparatus 300 according to the present embodiment includes a main audio input unit (main microphone) 301, a sub audio input unit (sub microphone) 302, a motion sensor (gyro) 303, a sound source distance calculation unit 304, a sound source distance corresponding sound source separation processing execution unit 305, a trained model (DNN) 306, a storage unit 112, an audio data editing processing unit 115, a playback control unit 116, a display unit 117, and an audio output unit (speaker) 118.

[0341] Note that, the components (processing blocks) illustrated in the information processing apparatus 300 illustrated in FIG. 12 are main components necessary to perform the processing in the present embodiment. In addition to these components, the information processing apparatus 300 includes various other components. For example, there are various components such as a communication unit, an input unit, a control unit, and a camera. Figure 29

[0342] Further, among the components of the information processing apparatus 300 illustrated in FIG. 12, the storage unit 112, the audio data editing processing unit 115, the playback control unit 116, the display unit 117, and the audio output unit (speaker) 118 have similar configurations to those of the storage unit 112, the audio data editing processing unit 115, the playback control unit 116, the display unit 117, and the audio output unit (speaker) 118 in the information processing apparatus 100 described above with reference to FIG. 11, and perform similar processing. Figure 29 Figure 5

[0343] Hereinafter, details of components different from those of the information processing apparatus 100 described above with reference to FIG. 11 (that is, the main audio input unit (main microphone) 301, the sub audio input unit (sub microphone) 302, the motion sensor (gyro) 303, the sound source distance calculation unit 304, the sound source distance corresponding sound source separation processing execution unit 305, and the trained model (DNN) 306) will be described. Figure 5 The main audio input unit (main microphone) 301 includes a microphone, an amplifier, an A / D converter, and the like, receives a user's voice and a guitar instrument sound and the like input via the microphone, performs digitization processing, and stores digital audio data in the storage unit 112.

[0344] Further, the audio data input via the microphone (main microphone) 301 of the main audio input unit is also input to the sound source distance calculation unit 304, and is used for the processing of calculating the sound source distance (distance from the sound source to the main microphone) performed by the sound source distance calculation unit 304.

[0345] ​​​​

[0346] Audio data input via the microphone (sub microphone) 302 of the sub audio input unit is input to the sound source distance calculation unit 304, and is used for the processing of calculating the sound source distance (distance from the sound source to the main microphone) performed by the sound source distance calculation unit 304.

[0347] The motion sensor (gyroscope) 303 includes, for example, a gyroscope, detects the motion information (acceleration, each acceleration, and the like) of the information processing apparatus 300, inputs the detected value to the sound source distance calculation unit 304, and uses the detected value for the processing of calculating the sound source distance (distance from the sound source to the main microphone) performed by the sound source distance calculation unit 304.

[0348] The sound source distance calculation unit 304 receives the audio data input to the microphone (main microphone) of the main audio input unit 301, the audio data input to the microphone (sub microphone) of the sub audio input unit 302, and the sensor detection value of the motion sensor (gyroscope) 303, and calculates the sound source distance (distance from the sound source to the main microphone) based on these input data.

[0349] Details regarding the sound source distance calculation processing performed by the sound source distance calculation unit 304 will be described later.

[0350] The sound source distance corresponding sound source separation processing performing unit 305 receives the sound source distance calculated by the sound source distance calculation unit 304, and performs the sound source separation processing using the trained model (DNN) corresponding to the sound source distance.

[0351] Note that the sound source distance information can also be input by the user via, for example, a UI displayed on the display unit 117 of the information processing apparatus 300 or an input unit.

[0352] In this case, the sound source distance corresponding sound source separation processing performing unit 305 performs the sound source separation processing using the trained model (DNN) corresponding to the sound source distance input by the user.

[0353] The trained model (DNN) 306 stores the trained model (DNN) for performing the optimal sound source separation processing according to various sound source distances.

[0354] An example of the data stored in the trained model (DNN) 306 will be described with reference to Figure 30

[0355] Figure 30 The following corresponds to the data stored as the trained model (DNN) 306:

[0356] (a) a separation target sound source type identifier;

[0357] ​(b) separating the target sound source; and

[0358] (c) a trained model (DNN, etc.).

[0359] The separation target sound source type identifier = C01 is a trained model (DNN) for extracting a separation target sound source = human voice (vocal sound), and is a neural network such as a DNN in which parameters for extracting human voice (vocal sound) are set.

[0360] As Figure 30 As illustrated in FIG. 6, different trained models (DNN, etc.) including the following are generated and stored in advance for each type of sound source to be separated according to the sound source distance:

[0361] a trained model corresponding to a sound source distance = 0 m to 0.5 m;

[0362] a trained model corresponding to a sound source distance = 0.5 m to 1.5 m;

[0363] a trained model corresponding to a sound source distance = 1.5 m to 3.0 m; and

[0364] a trained model corresponding to a sound source distance = 3.0 m or more.

[0365] The trained model corresponding to the sound source type and the sound source distance is, for example, a trained model generated in advance through a supervised training process, and is a trained model in which parameters for enabling the most accurate sound source data extraction in a case where a sound source is set at various sound source distances are set for each type of sound source.

[0366] As described above, in the sound source separation processing in the present embodiment, a sound source specific neural network corresponding to the type of the sound source to be separated (that is, for example, the type of the sound source to be separated such as human voice, guitar, piano, or violin) and the sound source distance is used.

[0367] That is, a neural network in which parameters specific to the sound source to be separated and the sound source distance are set is used.

[0368] In order to generate a trained model specific to the sound source type and the sound source distance such as a specific neural network according to the sound source type and the sound source distance, a training process using a large amount of training data is required. That is, it is necessary to perform a training process by inputting various different audio data as training data to a DNN, etc.

[0369] A specific example of a process of generating a trained model (DNN) specific to the sound source type and the sound source distance will be described with reference to Figure 31

[0370] Figure 31 ​The example illustrated in the middle is a diagram illustrating a training process for generating a trained model (vocal sound separation DNN) for extracting vocal sound corresponding to a human voice from input data, and is a diagram illustrating a training process for generating a vocal sound separation section (vocal sound separation DNN) for extracting vocal sound at a sound source distance of 0 m to 0.5 m, which corresponds to the sound source distance.

[0371] Figure 31 The training process illustrated in the middle is a supervised training process.

[0372] That is, the training process is performed using a large amount of "input data" dataset including mixed data of various types of sound source data and "annotation data" which is extraction data of vocal sound data included in each input data.

[0373] In the supervised training process, as described above, annotation data for input data is prepared in advance, input data is input to a vocal sound separation section (vocal sound separation DNN) via a pre-processing section (STFT), and a separation processing result in the vocal sound separation section (vocal sound separation DNN) is output via a post-processing section (ISTFT) to obtain "output data".

[0374] The "output data" is compared with the "annotation data", and a process of setting and updating parameters that constitute a neural network of the vocal sound separation section (vocal sound separation DNN) is performed so that the difference between the two data becomes smaller.

[0375] A large amount of training data including pairs of "input data" and "annotation data" is prepared.

[0376] For example, as Figure 32 As illustrated in the middle, it is necessary to generate training data identifiers 00001 to nnnnn including a large amount of different pairs of "input data" and "annotation data", that is, a combination of "input data" including mixed data of various sound sources and "annotation data" including only vocal sound included in the input data, in advance.

[0377] Note that the vocal sound included in the input data is vocal sound whose distance (that is, its sound source distance) from the microphone is in the range of 0 m to 0.5 m.

[0378] Similarly, "annotation data" including only vocal sound included in the input data is also "annotation data" of vocal sound whose distance (that is, its sound source distance) from the microphone is in the range of 0 m to 0.5 m.

[0379] Figure 32The large amount of input data illustrated in the middle is sequentially input to the vocal separation section (vocal separation DNN), and the parameters of the neural network constituting the vocal separation section (vocal separation DNN) are sequentially updated so that the difference between the "output data" and the "annotation data" becomes smaller.

[0380] By repeating this processing, the separation performance of the vocal separation section (vocal separation DNN) can be gradually improved.

[0381] When the separation performance of the preset allowable level is obtained, the training processing can be ended.

[0382] By performing such training processing, the vocal separation section (vocal separation DNN) can be generated, that is, the neural network as a trained model for separating the vocal sound whose sound source type = vocal sound and whose sound source distance = 0 m to 0.5 m.

[0383] Figure 33 is a diagram illustrating an example of training processing for generating a vocal separation section (vocal separation DNN) corresponding to the sound source distance of 0.5 m to 1.5 m of the sound source whose sound source distance is 0.5 m to 1.5 m from the input data.

[0384] Similarly to Figure 31 , Figure 33 An example of supervised training processing is illustrated. That is, the training processing is performed using a data set of a large amount of "input data" including mixed data of various types of sound source data and "annotation data" that is extraction data of vocal sound data included in each input data.

[0385] Figure 34 An example of training data including pairs of "input data" and "annotation data" is illustrated.

[0386] For example, as Figure 34 illustrated in the middle, it is necessary to generate training data identifiers 00001 to nnnnn including a large amount of different pairs of "input data" and "annotation data", that is, a combination of "input data" including mixed data of various sound sources and "annotation data" including only vocal sound included in the input data.

[0387] Note that in this example, the vocal sound included in the input data is vocal sound whose distance from the microphone (that is, its sound source distance) is in the range of 0.5 m to 1.5 m.

[0388] Similarly, the "annotation data" including only the vocal sound included in the input data is also "annotation data" of the vocal sound whose distance from the microphone (that is, its sound source distance) is in the range of 0.5 m to 1.5 m.

[0389] Figure 34 The large amount of input data shown in the diagram is sequentially entered into... Figure 33 The diagram shows a voice separation unit (voice separation DNN), and the parameters of the neural network that constitutes the voice separation unit (voice separation DNN) are updated sequentially to reduce the difference between the "output data" and the "labeled data".

[0390] By repeating this process, the separation performance of the voice separation unit (voice separation DNN) can be gradually improved.

[0391] The training process can end when the preset allowable level of separation performance is achieved.

[0392] By performing such training processing, a voice separation unit (voice separation DNN) can be generated, that is, a neural network as a trained model for separating voices whose sound source type is human voice and whose sound source distance is 0.5 m to 1.5 m.

[0393] Although it has been referenced Figures 31 to 34 Examples of training processes for generating a trained model that performs the process of separating human voices whose source type is human and whose source distance is 0 m to 0.5 m are described; examples of training processes for generating a trained model that performs the process of separating human voices whose source type is human and whose source distance is 0.5 m to 1.5 m are also described, but by performing similar training processes on other source types and source distances, it is possible to generate voices similar to those in the above reference. Figure 30 The various types of sound sources and the corresponding trained models (DNN, etc.) describe the distances between them.

[0394] Next, we will describe Figure 29 The information processing device 300 shown in the figure contains a sound source separation processing execution unit 305 corresponding to the sound source distance.

[0395] The sound source separation processing execution unit 305, which corresponds to the sound source distance, uses a trained model (DNN) corresponding to the sound source distance calculated by the sound source distance calculation unit 304 or the sound source distance input by the user to perform sound source separation processing.

[0396] For example, when the distance from the microphone into which the human voice (singing) is input to the mouth of the user emitting the human voice (singing) is in the range of 0 m to 0.5 m, the sound source separation processing execution unit 305 corresponding to the sound source distance uses a trained model corresponding to the sound source distance = 0 m to 0.5 m to perform human voice extraction processing.

[0397] Further, in a case where the distance from the microphone in which the human voice (singing voice) is input to the mouth of the user from which the human voice (singing voice) is emitted is in the range of 1.5 m to 3.0 m, the sound source distance corresponding sound source separation processing execution unit 305 executes the human voice extraction processing using the trained model corresponding to the sound source distance = 1.5 m to 3.0 m.

[0398] As described above, each of the trained models different according to the sound source distance is a trained model in which a parameter enabling extraction of sound source data with the highest accuracy according to each sound source distance is set, and the parameter set for each of the trained models corresponding to various distances is a different parameter.

[0399] As described above with reference to Figure 30 The separation target sound source type identifier = C01 is a trained model (DNN) for extracting the separation target sound source = human voice (singing voice), and is a neural network such as a DNN in which a parameter for extracting the human voice (singing voice) is set.

[0400] As Figure 30 illustrated in FIG. 1, different trained models (DNNs, etc.) including the following are generated and stored in advance according to the sound source distance:

[0401] a trained model corresponding to the sound source distance = 0 m to 0.5 m;

[0402] a trained model corresponding to the sound source distance = 0.5 m to 1.5 m;

[0403] a trained model corresponding to the sound source distance = 1.5 m to 3.0 m; and

[0404] a trained model corresponding to the sound source distance = 3.0 m or more.

[0405] Further, as Figure 30 illustrated in FIG. 1, the separation target sound source type identifier = C02 is a trained model (DNN) for extracting the separation target sound source = guitar sound, and is a neural network such as a DNN in which a parameter for extracting the guitar sound is set.

[0406] As Figure 30 illustrated in FIG. 1, different trained models (DNNs, etc.) including the following are generated and stored in advance according to the sound source distance:

[0407] a trained model corresponding to the sound source distance = 0 m to 0.5 m;

[0408] a trained model corresponding to the sound source distance = 0.5 m to 1.5 m;

[0409] a trained model corresponding to the sound source distance = 1.5 m to 3.0 m; and

[0410] a trained model corresponding to a sound source distance = 3.0 m or more.

[0411] Similarly, a separation target sound source type identifier = C03 is a trained model (DNN) for extracting a separation target sound source = piano sound, and is a neural network such as a DNN in which parameters for extracting piano sound are set.

[0412] As illustrated in Figure 30 in advance according to a sound source distance, different trained models (DNNs, etc.) including the following are generated and stored:

[0413] a trained model corresponding to a sound source distance = 0 m to 0.5 m;

[0414] a trained model corresponding to a sound source distance = 0.5 m to 1.5 m;

[0415] a trained model corresponding to a sound source distance = 1.5 m to 3.0 m; and

[0416] a trained model corresponding to a sound source distance = 3.0 m or more.

[0417] In addition, the trained models corresponding to the extracted various sound source types and sound source distances are stored in the trained models (DNNs) 306.

[0418] The sound source distance corresponding sound source separation processing execution unit 305 executes sound source separation processing using the trained model (DNN) corresponding to the sound source distance calculated by the sound source distance calculation unit 304 or the sound source distance input by the user.

[0419] Next, a specific configuration example of the sound source distance corresponding sound source separation processing execution unit 305 and the processing executed by the sound source distance corresponding sound source separation processing execution unit 305 will be described with reference to Figure 35

[0420] The sound source distance corresponding sound source separation processing execution unit 305 receives the input data illustrated in the upper left part of Figure 35 from the storage unit 112.

[0421] For example, the input data is audio data in which a human voice uttered by a user 1 and a guitar sound are mixed, and is audio data that changes along a time axis.

[0422] The sound source distance corresponding sound source separation processing execution unit 305 receives the audio data in which the human voice and the guitar sound are mixed, and executes processing of extracting only the human voice and processing of extracting only the guitar sound.

[0423] ​The sound source distance corresponding sound source separation processing execution unit 305 receives the sound source distance calculated by the sound source distance calculation unit 304 or the sound source distance input by the user, and executes sound source separation processing using a trained model (for example, a neural network such as a deep neural network (DNN)) corresponding to the sound source distance.

[0424] As Figure 35 As illustrated in the upper part, the sound source distance corresponding sound source separation processing execution unit 305 includes:

[0425] a series of data processing sections including a pre-processing section (STFT) 311, a vocal separation section (vocal separation DNN) 312, and a post-processing section (ISTFT) 313; and

[0426] a series of data processing sections including a pre-processing section (STFT) 321, a guitar sound separation section (guitar sound separation DNN) 322, and a post-processing section (ISTFT) 323.

[0427] The vocal included in the input data is extracted by the components illustrated in the upper part, that is, the pre-processing section (STFT) 311, the vocal separation section (vocal separation DNN) 312, and the post-processing section (ISTFT) 313.

[0428] On the other hand, the guitar sound included in the input data is extracted by the components illustrated in the lower part, that is, the pre-processing section (STFT) 321, the guitar sound separation section (guitar sound separation DNN) 322, and the post-processing section (ISTFT) 323.

[0429] The pre-processing section (STFT) 311 illustrated in the upper part performs a short-time Fourier transform (STFT) on the input data.

[0430] With the STFT processing, the input data, which is data in the time domain in which an audio signal changes along a time axis, is converted into data in the frequency domain (spectrum).

[0431] This converted data is input to the vocal separation section (vocal separation DNN) 312.

[0432] The vocal separation section (vocal separation DNN) 312 is a sound source separation section configured using a trained model such as a neural network or more specifically, for example, a deep neural network (DNN).

[0433] However, the trained model used here is a trained model including a neural network corresponding to the sound source distance calculated by the sound source distance calculation unit 304 or the sound source distance input by the user, for example, a neural network such as a deep neural network (DNN).

[0434] That is, the sound source extraction processing is performed using the trained model in which the optimal sound source extraction parameter according to the sound source distance is set.

[0435] The vocal separation section (vocal separation DNN) 312 is a neural network (trained model) that extracts sound data having a feature (frequency or the like) of a human voice corresponding to a human voice from audio data of various sound sources included in the input data, and is a neural network (trained model) in which an optimal vocal extraction parameter corresponding to the sound source distance calculated by the sound source distance calculation unit 304 or the sound source distance input by the user is set.

[0436] The neural network (trained model) is generated by a training process performed in advance and is a neural network (trained model) in which a parameter for extracting sound data having a feature of a human voice corresponding to a human voice is set.

[0437] The vocal separation section (vocal separation DNN) 312 extracts audio data corresponding to a human voice corresponding to a human voice from audio data of various sound sources included in the input data.

[0438] The audio data extracted by the vocal separation section (vocal separation DNN) 312 is input to the post-processing section (ISTFT) 313.

[0439] The post-processing section (ISTFT) 313 performs inverse processing of the processing performed by the above-described pre-processing section (STFT) 311. That is, inverse short-time Fourier transform (ISTFT) is performed.

[0440] With the ISTFT processing, data in the frequency domain (spectrum) is converted into data in the time domain in which the audio signal changes along the original time axis.

[0441] The output of the post-processing section (ISTFT) 313 is the audio data extracted by the vocal separation section (vocal separation DNN) 312, that is, the audio data of the human voice.

[0442] On the other hand, the pre-processing section (STFT) 321 illustrated in the lower portion also performs short-time Fourier transform (STFT) on the input data to convert the input data which is data in the time domain into data in the frequency domain (spectrum).

[0443] The converted data is input to the guitar sound separation section (guitar sound separation DNN) 322.

[0444] The guitar sound separation section (guitar sound separation DNN) 322 is also a sound source separation section configured using a trained model such as a neural network or more specifically, for example, a deep neural network (DNN).

[0445] A trained model of a neural network (e.g., a neural network such as a deep neural network (DNN)) corresponding to the sound source distance calculated by the sound source distance calculation unit 304 or the sound source distance input by the user is also used for the guitar sound separation section (guitar sound separation DNN) 322.

[0446] That is, the sound source extraction processing is performed using a trained model in which optimal sound source extraction parameters according to the sound source distance are set.

[0447] The guitar sound separation section (guitar sound separation DNN) 322 is a neural network (trained model) that extracts audio data having characteristics (frequency, etc.) of guitar sound from audio data of various sound sources included in the input data, and is a neural network (trained model) in which optimal sound source extraction parameters corresponding to the sound source distance calculated by the sound source distance calculation unit 304 are set.

[0448] This neural network (trained model) is also generated by a training processing performed in advance and is a neural network (trained model) in which parameters for extracting sound data having characteristics of guitar sound are set.

[0449] The guitar sound separation section (guitar sound separation DNN) 322 extracts audio data corresponding to guitar sound from audio data of various sound sources included in the input data.

[0450] The audio data extracted by the guitar sound separation section (guitar sound separation DNN) 322 is input to the post-processing section (ISTFT) 323.

[0451] The post-processing section (ISTFT) 323 performs inverse processing of the pre-processing section (STFT) 321. That is, inverse short-time Fourier transform (ISTFT) is performed.

[0452] With the ISTFT processing, data in the frequency domain (spectrum) is converted into data in the time domain in which the audio signal changes along the original time axis.

[0453] The output of the post-processing section (ISTFT) 323 is the audio data extracted by the guitar sound separation section (guitar sound separation DNN) 322, that is, the audio data of the guitar sound.

[0454] As described above, in a configuration in which sound source separation processing is performed using a neural network such as a DNN, a sound source-specific neural network corresponding to the type of sound source to be separated (that is, for example, the type of sound source to be separated such as human voice, guitar, piano, or violin) is used.

[0455] Further, a neural network in which optimal parameters corresponding to the sound source distance according to the sound source distance are set is used.

[0456] As described above, the sound source distance corresponding sound source separation processing execution unit 305 of the information processing apparatus 300 in the present embodiment executes processing of extracting each sound source data using a neural network in which a specific parameter corresponding to the type of sound source to be separated and the sound source distance is set.

[0457] Next, specific processing examples of the sound source distance calculation processing executed by the sound source distance calculation unit 304 will be described with reference to Figure 36 and subsequent drawings.

[0458] Figure 36 A smartphone is illustrated as an example of the information processing apparatus 300.

[0459] As illustrated in Figure 36 , the information processing apparatus 300 includes a main microphone 331 and a sub microphone 332.

[0460] The main microphone 331 corresponds to the main audio input unit 301 that is a component of the information processing apparatus 300 illustrated in Figure 29 . The sub microphone 332 corresponds to the sub audio input unit 302 that is a component of the information processing apparatus 300 illustrated in Figure 29

[0461] The audio data input via the main microphone 331 and the sub microphone 332 is input to the sound source distance calculation unit 304 of the information processing apparatus 300 illustrated in Figure 29

[0462] Note that the main microphone 331 and the sub microphone 332 are disposed at positions separated by a predetermined distance from each other, such as the upper end and the lower end of the information processing apparatus 300.

[0463] Next, specific processing examples of the sound source distance calculation processing executed by the sound source distance calculation unit 304 will be described with reference to Figure 37 and subsequent drawings.

[0464] Figure 37 A user 1 and the information processing apparatus 300 are illustrated. The main microphone 331 and the sub microphone 332 are mounted on the information processing apparatus 300.

[0465] In this example, the sound source position of the user 1's voice is the position of the user 1's mouth, and the sound source distance d is the distance between the position of the user 1's mouth and the main microphone 331 of the information processing apparatus 300, and the object is to measure the sound source distance d.

[0466] The user 1 rotates the information processing apparatus 300 around the position of the main microphone 331 while outputting the voice. This rotation is performed so that a trajectory is drawn along the dashed arrow illustrated in Figure 37 .​​

[0467] In this state of rotation, the input sound of the main microphone 331 and the input sound of the sub microphone 332 are continuously input to Figure 29 the sound source distance calculating unit 304 illustrated in FIG. 6.

[0468] A specific example of the process of calculating the sound source distance d (distance between the sound source (mouth of the user) and the main microphone) by the sound source distance calculating unit 304 will be described with reference to Figure 38 and the subsequent drawings.

[0469] Figure 38 A user 1 outputting a human voice and the positions of the main microphone 331 and the sub microphone 332 of the information processing apparatus 300 such as a smartphone are illustrated.

[0470] The position of the sub microphone 332 is rotated around the position of the main microphone 331.

[0471] As illustrated in Figure 38 , it is assumed that

[0472] the distance between the sound source (mouth) and the main microphone = d (= sound source distance),

[0473] the distance between the sound source (mouth) and the sub microphone = I,

[0474] the angle between the straight line connecting the sound source (mouth) and the main microphone and the straight line connecting the main microphone and the sub microphone = θ,

[0475] the distance between the main microphone and the sub microphone = r, and

[0476] the difference between I and d = I - d.

[0477] The sound source distance calculating unit 304 uses these parameters to calculate the sound source distance d (distance between the sound source (mouth of the user) and the main microphone).

[0478] Note that the angle between the straight line connecting the sound source (mouth) and the main microphone and the straight line connecting the main microphone and the sub microphone = θ is angle information that can be obtained from the sensor detection value of the motion sensor (gyroscope) 303.

[0479] Hereinafter, two different calculation processing sequences will be described as specific examples of the process of calculating the sound source distance d performed by the sound source distance calculating unit 304.

[0480] (Sound Source Distance Calculation Example 1)

[0481] An outline of "Sound Source Distance Calculation Example 1" will be described.

[0482] In this "Sound Source Distance Calculation Example 1",

[0483] First, the angle θ at which the offset between the sounds obtained by the primary microphone 331 and the secondary microphone 332 becomes zero is calculated, that is, the angle θ at which the distance between the sound source (mouth) and the primary microphone becomes equal to the distance between the sound source (mouth) and the secondary microphone is calculated.

[0484] The distance between the sound source (mouth) and the primary microphone = d (= sound source distance),

[0485] The distance between the sound source (mouth) and the secondary microphone = I, and

[0486] The position of the secondary microphone at which the difference ΔI between I and d = I - d = 0.

[0487] Next, the sound source distance d (the distance between the sound source (mouth of the user) and the primary microphone) is calculated at this position using the following values:

[0488] The angle between the straight line connecting the sound source (mouth) and the primary microphone and the straight line connecting the primary microphone and the secondary microphone = θ, and

[0489] The distance between the primary microphone and the secondary microphone = r.

[0490] This is the sequence for calculating the sound source distance d by this processing.

[0491] The detailed sequence of "Sound Source Distance Calculation Example 1" performed by the sound source distance calculation unit 304 will be described with reference to the flowchart illustrated in Figure 39 and Figure 40

[0492] Hereinafter, the processing details in each step in the flowchart illustrated in Figure 39 and Figure 40

[0493] (Step S11)

[0494] First, the sound source distance calculation unit 304 performs the following processing in step S11.

[0495] ∙ The distance between the sound source (mouth) and the primary microphone = d

[0496] ∙ The distance between the sound source (mouth) and the secondary microphone = I,

[0497] ∙ The angle between the straight line connecting the sound source (mouth) and the primary microphone and the straight line connecting the primary microphone and the secondary microphone = θ

[0498] ∙ The distance between the primary microphone and the secondary microphone = r

[0499] Using these parameters, the calculation formula (Expression 1a) of the distance between the sound source (mouth) and the secondary microphone = I is generated. ​​

[0500] I = √(r 2 sin 2 θ + (d - rcosθ) 2 ... (Expression 1a)

[0501] (Step S12)

[0502] Next, the sound source distance calculation unit 304 performs the following processing in step S12.

[0503] By substituting the formula for calculating "the distance between the sound source (mouth) and the secondary microphone = I" shown above (Expression 1a) into the formula for calculating the difference ΔI between I and d (ΔI = I - d),

[0504] The formula for calculating the difference ΔI between I and d (ΔI=Id) is transformed into the following (expression 1b).

[0505] ΔI = I - d

[0506] = √(r 2 sin 2 θ + (d - rcosθ) 2 ) - d

[0507] = √(r 2 + d 2 - 2drcosθ) - d ... (expression 1b)

[0508] (Step S13)

[0509] Next, the sound source distance calculation unit 304 performs the following processing in step S13.

[0510] Analyze the sound offset between the main microphone and the secondary microphone to obtain the angle θ at the position where the sound offset = zero (phase difference = 0). Note that when the difference between I and d becomes ΔI = Id = 0, the sound offset between the main microphone and the secondary microphone is zero.

[0511] (Step S14)

[0512] Next, the sound source distance calculation unit 304 performs the following processing in step S14.

[0513] By substituting ΔI=0 into (Expression 1b), the following relational expression (Expression 1c) is derived.

[0514] 0 = √(r 2 + d 2 - 2drcosθ) - d

[0515] d 2 = r 2 + d 2 - 2drcosθ

[0516] 0 = r 2 - 2drcosθ

[0517] cosθ = r / 2d... (Expression 1c)

[0518] (Step S15)

[0519] Next, the sound source distance calculation unit 304 executes the following processing in Step S15.

[0520] From (Expression 1c), the following sound source distance calculation formula (Expression 1d) is derived.

[0521] Sound source distance d = (r / 2cosθ)... (Expression 1d)

[0522] (Step S16)

[0523] Next, the sound source distance calculation unit 304 executes the following processing in Step S16.

[0524] The sound source distance d (distance between the sound source (mouth) and the main microphone) is calculated by substituting the following items into (Expression 1d):

[0525] • Distance between the main microphone and the sub microphone = r, and

[0526] • Angle θ at the position where the sound offset between the main microphone and the sub microphone = zero.

[0527] The sound source distance calculation unit 304 can calculate the sound source distance d (distance between the sound source (mouth) and the main microphone) according to such a sequence.

[0528] Note that, although an example in which the sound source is a human voice and the sound source position is the mouth of the user is described here, by setting the sound source position to the position of a guitar or the position of a piano instead of the mouth of the user, the sound source distance can be calculated for each of other musical instruments such as a guitar and a piano according to a similar sequence.

[0529] (Sound source distance calculation example 2)

[0530] Next, a sequence different from the sequence described in “Sound source distance calculation example 1” described in the flowchart illustrated in Figure 39 and Figure 40 , that is, “Sound source distance calculation example 2” performed by the sound source distance calculation unit 304 will be described.

[0531] An outline of "Example 2 of sound source distance calculation" will be described.

[0532] In this "Example 2 of sound source distance calculation",

[0533] First, a formula "Va•Δt = f(d, θ, r)" for calculating an offset amount "Va•Δt" between sounds obtained by the primary microphone 331 and the secondary microphone 332 is generated, and the offset amount is calculated as a product of a sound speed Va and a time difference Δt between sounds obtained by the primary microphone 331 and the secondary microphone 332 using the following items:

[0534] A distance between the sound source (mouth) and the primary microphone = d (= sound source distance);

[0535] An angle between a straight line connecting the sound source (mouth) and the primary microphone and a straight line connecting the primary microphone and the secondary microphone = θ; and

[0536] A distance between the primary microphone and the secondary microphone = r.

[0537] Next, the generated formula "Va•Δt = f(d, θ, r)" is transformed into a formula for calculating the sound source distance d, and parameters constituting the generated calculation formula of the sound source distance d are substituted, that is,

[0538] An angle between a straight line connecting the sound source (mouth) and the primary microphone and a straight line connecting the primary microphone and the secondary microphone = θ;

[0539] A distance between the primary microphone and the secondary microphone = r;

[0540] A sound speed Va; and

[0541] A time difference Δt between sounds obtained by the primary microphone and the secondary microphone,

[0542] to calculate the sound source distance d (a distance between the sound source (mouth of the user) and the primary microphone).

[0543] This is a sequence for calculating the sound source distance d by the processing.

[0544] The detailed sequence of "Example 2 of sound source distance calculation" performed by the sound source distance calculation unit 304 will be described with reference to the flowchart illustrated in Figures 41 to 43 Hereinafter, the processing details in each step in the flowchart illustrated in

[0545] Figures 41 to 43 will be sequentially described.

[0546] (Step S21)

[0547] ​​First, the sound source distance calculation unit 304 performs the following processing in step S21.

[0548] * Distance between sound source (mouth) and main microphone = d

[0549] * Distance between sound source (mouth) and sub microphone = I,

[0550] * Angle between straight line connecting sound source (mouth) and main microphone and straight line connecting main microphone and sub microphone = θ

[0551] * Distance between main microphone and sub microphone = r

[0552] Using these parameters, a calculation formula (Expression 2a) of the distance between the sound source (mouth) and the sub microphone = I is generated.

[0553] I = √(r 2 sin 2 θ + (d - rcosθ) 2 )... (Expression 2a)

[0554] (Step S22)

[0555] Next, the sound source distance calculation unit 304 performs the following processing in step S22.

[0556] * Time difference between sounds obtained by main microphone and sub microphone = Δt

[0557] * Time at which sound emitted by user reaches main microphone = t1,

[0558] * Time at which sound emitted by user reaches sub microphone = t2,

[0559] * Speed of sound = Va

[0560] Using these parameters, a calculation formula (Expression 2b) of the "time difference Δt between sounds obtained by the main microphone and the sub microphone" is generated.

[0561] Δt = t1 - t2 = d / Va - I / Va... (Expression 2b)

[0562] (Step S23)

[0563] Next, the sound source distance calculation unit 304 performs the following processing in step S23.

[0564] By substituting "distance between sound source (mouth) and sub microphone = I" shown in (Expression 2a) obtained in step 21 into the calculation formula (Expression 2b) of "time difference At between sounds obtained by the main microphone and the sub microphone" calculated in step S22 (that is, At = tl - t2 = d / Va - I / Va... (Expression 2b)), the following calculation formula (Expression 2c) of "time difference At between sounds obtained by the main microphone and the sub microphone" is generated.

[0565] At = d / Va - (r 2 + d 2 - 2drcosθ) - d) / Va... (Expression 2c)

[0566] (Step S24)

[0567] Next, the sound source distance calculation unit 304 executes the following processing in step S24.

[0568] A calculation formula (Expression 2d) of the offset (VaAt) of sounds obtained by the main microphone and the sub microphone is generated.

[0569] VaAt = 2d - (r 2 + d 2 - 2drcosθ)... (Expression 2d)

[0570] (Step S25)

[0571] Next, the sound source distance calculation unit 304 executes the following processing in step S25.

[0572] Using the relational expression (Expression 2d) obtained in step S24, a calculation formula (Expression 2e) of the sound source distance d (distance between sound source (mouth) and main microphone = d) is generated.

[0573] VaAt = 2d - (r 2 + d 2 - 2drcosθ)... (Expression 2d)

[0574] r 2 + d 2 - 2drcosθ = (2d - VaAt) 2

[0575] r 2 + d 2 - 2drcosθ = 4d 2 - 4dVaAt + (VaAt) 2

[0576] 3D 2 + (2rcosθ - 4VaΔt)d + (VaΔt) 2 -r 2 = 0

[0577] d = (4VaΔt - 2rcosθ) ± √((2rcosθ - 4VaΔt) 2 - 12((VaΔt) 2 - r2))) / 6... (expression 2e)

[0578] (Step S26)

[0579] Next, the sound source distance calculation unit 304 performs the following processing in step S26.

[0580] The distance d from the sound source is calculated by substituting the following terms into the above (expression 2e):

[0581] • The speed of sound = Va;

[0582] • The distance between the main microphone and the secondary microphone = r;

[0583] • The time difference between the sounds obtained by the main microphone and the secondary microphone = Δt (measured value);

[0584] • The angle between the line connecting the sound source (mouth) and the main microphone and the line connecting the main microphone and the secondary microphone is θ (measured value).

[0585] The sound source distance calculation unit 304 can calculate the sound source distance d (the distance between the sound source (mouth) and the main microphone) according to such a sequence.

[0586] Note that although this example describes a sound source as a human voice and a sound source location as the user's mouth, the sound source distance can be calculated for each of other instruments such as guitars and pianos based on a similar sequence by setting the sound source location to the position of a guitar or piano instead of the user's mouth.

[0587] The sound source distance (distance from the sound source to the main microphone) information calculated by the sound source distance calculation unit 304 is input to the sound source separation processing execution unit 305 corresponding to the sound source distance.

[0588] The sound source separation processing execution unit 305, which corresponds to the sound source distance, receives the sound source distance calculated by the sound source distance calculation unit 304 and uses the trained model (DNN) corresponding to the sound source distance to perform sound source separation processing.

[0589] Note that the referenceFigures 38 to 43 An example of the sound source distance calculation processing of the sound source distance calculation unit 304 described above is one example, and alternatively, another method can be used to perform the sound source distance calculation processing.

[0590] Further, as described above, for example, the user can directly input the sound source distance information to the information processing apparatus 300 via the UI or the input unit of the information processing apparatus 300.

[0591] In this case, the sound source distance corresponding sound source separation processing execution unit 305 uses the sound source distance information input by the user, and uses a trained model (DNN) corresponding to the input sound source distance to perform the sound source separation processing.

[0592] [7. Sequence of sound source separation processing performed by information processing apparatus in the present disclosure]

[0593] Next, the sequence of the sound source separation processing performed by the information processing apparatus in the present disclosure will be described.

[0594] The sequence of the sound source separation processing performed by the information processing apparatus in the present disclosure will be described with reference to the flowchart illustrated in Figure 44

[0595] The flowchart illustrated in Figure 44 is a flowchart illustrating the sequence of the processing performed by the information processing apparatus 300 in the present disclosure described above with reference to Figure 29 According to the program stored in the storage unit of the information processing apparatus 300, the flow is executed under the control of the control unit. The control unit includes, for example, a processor such as a CPU having a program execution function, and by using the program execution processing of the processor, the processing can be performed according to the flow.

[0596] The processing in each step in the flow illustrated in

[0597] will be described sequentially. Figure 44 Note that, before the flow illustrated in

[0598] begins, the audio data to be subjected to sound source separation has already been stored in the memory (storage unit) of the information processing apparatus. Figure 44 The user starts the sound source separation processing of the audio data stored (recorded) in the memory (storage unit) inside the information processing apparatus.

[0599] (Step S101)

[0600] First, in step S101, the information processing apparatus 300 calculates or inputs the distance of the sound source to be subjected to the sound source separation processing.

[0601] ​

[0602] The sound source distance calculation processing is executed by Figure 29 the sound source distance calculation unit 304 of the information processing apparatus 300 illustrated in

[0603] Specifically, the sound source distance calculation processing is executed in accordance with the processing described above with reference to Figure 39 and Figure 40 the flowchart illustrated in Figures 41 to 43 or the processing described above with reference to

[0604] Note that, as described above, the user can input the sound source distance directly to the information processing apparatus 300 via the UI or the input unit.

[0605] In a case where a plurality of sound sources exist, the sound source distance is calculated or input for each sound source. However, in a case where the user sings with a guitar, since the sound source position of the voice and the sound source position of the guitar sound are substantially the same, the same sound source distance can be calculated or input.

[0606] (Step S102)

[0607] Next, in step S102, the information processing apparatus 100 obtains a trained model of a neural network such as a DNN including a sound source distance corresponding to the sound source distance calculated or input in step S101.

[0608] This is the processing of: Figure 29 the sound source distance corresponding sound source separation processing execution unit 305 of the information processing apparatus 300 illustrated in Figure 29 obtains a trained model of a neural network such as a DNN including a sound source distance corresponding to the calculated or input sound source distance from the trained model 306 illustrated in

[0609] As described above with reference to Figure 30 , Figure 29 the trained model 306 illustrated in stores trained models including different neural networks such as DNNs in accordance with the sound source type and the sound source distance.

[0610] The sound source distance corresponding sound source separation processing execution unit 305 selects and obtains a trained model (DNN, etc.) corresponding to the sound source type and the sound source distance from among a large number of trained models illustrated in Figure 30

[0611] Note that, in the sound source type information, the sound source type information selected by the user using the UI described above with reference to Figure 2 is recorded as metadata corresponding to the recording data stored in the information processing apparatus, and the sound source distance corresponding sound source separation processing execution unit 305 determines the type of the sound source with reference to the metadata of the recording data to be subjected to sound source separation.​

[0612] (Step S103)

[0613] Next, in step S103, the information processing apparatus 100 performs a sound source separation process using a trained model (DNN or the like) corresponding to the type of the sound source and the sound source distance of the sound source selected and obtained in step S102.

[0614] This process is a process performed by the sound source distance corresponding sound source separation process performing unit 305 of the information processing apparatus 300 illustrated in Figure 29

[0615] As described above with reference to Figures 8 to 28 the sound source distance corresponding sound source separation process performing unit 305 receives the recorded data stored in the storage unit, and performs a sound source separation process using a trained model (DNN or the like) corresponding to the sound source type and the sound source distance of each sound source included in the recorded data.

[0616] (Step S104)

[0617] Next, in step S104, the information processing apparatus 300 displays display data (UI) for playback processing or editing processing of the audio data of each sound source generated by the sound source separation process performed in step S103 on the display unit.

[0618] The display data (UI) displayed on the display unit is, for example, the display data (UI) as described above with reference to Figures 8 to 28

[0619] That is, it is display data (UI) indicating the audio data of each sound source and including display data (UI) for notifying the user of a problem (timing shift, pitch shift, or the like), an editing target region, and the like.

[0620] (Step S105)

[0621] In step S105, the user can perform playback processing and editing processing of the audio data of each sound source using the display data for editing displayed in step S104.

[0622] In a case where the editing processing of the audio data of each sound source is performed, the user performs an operation on the UI according to, for example, a message or the like of the display data (UI), as described above with reference to Figure 7

[0623] This user operation information is input to the instruction analysis section 123 in the audio data editing processing unit 115 of the information processing apparatus 100 illustrated in Figure 7

[0624] ​​​​The instruction analysis section 123 analyzes the correction content requested by the user based on the user operation, and outputs a request to perform the correction processing obtained as a result of the analysis to the sound source data correction section 124.

[0625] In Figure 45 The sound source data correction section 141 illustrated in FIG. 12, the sound source data correction section 124 performs correction of each sound source (vocal sound, guitar sound, etc.) (for example, main characteristics of the audio output timing, sound correction processing, etc.).

[0626] (Step S106)

[0627] When the data editing processing of each sound source in step S105 is completed, in step S106, the information processing apparatus 100 stores the edited data file of each sound source in the storage unit 106, and ends the processing.

[0628] Note that, when each sound source data corrected by the sound source data correction section 124 and the synthesized audio data of the plurality of corrected sound source data generated by the sound source data synthesis unit 125 are stored in the storage unit 112, the data can be recorded as an overwritten data file with respect to the data file before the correction, or the data can be recorded as a new data file different from the data file before the correction.

[0629] [8. Configuration example of performing sound source separation processing in external device]

[0630] Next, a configuration example of performing sound source separation processing in an external device will be described.

[0631] In the above-described embodiment, for example, the following configuration example has been described: the information processing apparatus 300 such as a smartphone performs sound source separation processing using the sound source distance corresponding sound source separation processing execution unit 305 configured inside the information processing apparatus 300.

[0632] That is, the audio data input by the user 1 via the microphone of the information processing apparatus 300 such as a smartphone is recorded in the storage unit 112 of the information processing apparatus 300, and the sound source distance corresponding sound source separation processing execution unit 305 inside the information processing apparatus 300 performs sound source separation processing on the recorded audio data using a trained model (DNN, etc.) according to the sound source type and the sound source distance.

[0633] The sound source separation processing is not limited to being performed inside the information processing apparatus 300, and instead, can be performed by an external device (for example, a data processing server) that can communicate with the information processing apparatus 300.

[0634] An example of a system that performs sound source separation processing using an external device will be described with reference to Figure 45 FIG. 13.

[0635] Figure 45 An information processing apparatus 300 used by a user such as a smartphone and a data processing server 400 that is an external device communicable with the information processing apparatus 300 are illustrated.

[0636] In Figure 35 the system illustrated in, instead of the information processing apparatus 300, the data processing server 400 performs a sound source separation process.

[0637] The data processing server 400 includes a sound source separation process execution unit corresponding to the sound source distance corresponding to the sound source separation process execution unit 305 described above with reference to Figure 4

[0638] The information processing apparatus 300 transmits audio data stored in a storage unit inside the information processing apparatus 300 to the data processing server 400.

[0639] The audio data to be transmitted is, for example, data recorded in advance by the user of the information processing apparatus 300.

[0640] Note that, when transmitting the audio data to the data processing server 400, the information processing apparatus 300 adds and transmits the audio data as metadata information indicating the type of the sound source included in the audio data to be transmitted and the sound source distance information of each sound source.

[0641] Note that, as the information indicating the type of the sound source included in the audio data, for example, information specified by the user using the UI described above with reference to Figure 39

[0642] The sound source distance information of each sound source is calculated according to the process described above with reference to the flowcharts illustrated in Figure 40 and Figures 41 to 43 or the process described above with reference to Figure 35 Alternatively, the user can directly input the distance information to the information processing apparatus 300 via the UI or the input unit.

[0643] The data processing server 400 receives the audio data and the metadata including the information indicating the type of the sound source included in the audio data and the sound source distance information from the information processing apparatus 300.

[0644] The data processing server 400 performs a sound source separation process using a trained model (DNN or the like) according to the sound source type and the sound source distance based on the metadata received from the information processing apparatus 300.

[0645] The sound source separation process is similar to the process described above with reference to Figures 8 to 28 .​​

[0646] The audio data of each sound source generated as a result of the sound source separation processing performed by the data processing server 400 is transmitted from the data processing server 400 to the information processing apparatus 300, and stored in the storage unit of the information processing apparatus 300.

[0647] Thereafter, the information processing apparatus 300 can perform playback processing and editing processing of the audio data of each sound source. That is, the UI described above with reference to Figure 46 is displayed, and playback processing and editing processing of the audio data of each sound source can be performed.

[0648] The sound source separation processing is thus not limited to being performed inside the information processing apparatus 300, and instead can be performed by an external device (e.g., a data processing server) that can communicate with the information processing apparatus 300.

[0649] Next, a configuration example of the information processing apparatus 300 and a configuration example of the data processing server 400 in a case where the sound source separation processing is performed by the data processing server 400 will be described with reference to Figure 46

[0650] As illustrated in Figure 29 , the information processing apparatus 300 implemented as a smartphone or the like used by a user includes a main audio input unit (main microphone) 301, a sub audio input unit (sub microphone) 302, a motion sensor (gyroscope) 303, a sound source distance calculation unit 304, a storage unit 112, an audio data editing processing unit 115, a playback control unit 116, a display unit 117, an audio output unit (speaker) 118, and a communication unit 119.

[0651] This configuration corresponds to a configuration in which the sound source distance corresponding sound source separation processing execution unit 305 and the trained model (DNN) 306 are removed from the configuration of the information processing apparatus 300 described above with reference to Figure 30

[0652] On the other hand, the data processing server 400 includes a communication unit 401, a sound source distance corresponding sound source separation processing execution unit 402, and a trained model (DNN) 403.

[0653] The communication unit 401 performs communication with the information processing apparatus 300 such as, for example, a smartphone.

[0654] The sound source distance corresponding sound source separation processing execution unit 402 performs sound source separation processing that inputs audio data received from the information processing apparatus 300 and separates the audio data into data corresponding to each sound source included in the audio data. ​​

[0655] Note that the sound source distance corresponding sound source separation processing performed by the sound source distance corresponding sound source separation processing execution unit 402 is processing performed using a trained model (DNN or the like) selected according to the sound source type and the sound source distance of the metadata included in the audio data received from the information processing apparatus 300.

[0656] The data processing server 400 stores a plurality of different trained models for sound source separation in the trained model (DNN) 403.

[0657] That is, as described above with reference to Figure 47 the trained model including different neural networks such as DNNs is stored according to the sound source type and the sound source distance.

[0658] The sound source distance corresponding sound source separation processing execution unit 402 of the data processing server 400 performs separation processing of each sound source using a trained model (DNN or the like) selected from the trained model (DNN) 403 according to the sound source type and the sound source distance of the metadata included in the audio data received from the information processing apparatus 300.

[0659] The processing result of the sound source distance corresponding sound source separation processing performed by the sound source distance corresponding sound source separation processing execution unit 402 of the data processing server 400 (that is, each sound source data) is transmitted to the information processing apparatus 300 via the communication unit 401.

[0660] [9. Hardware configuration example of information processing apparatus in the present disclosure]

[0661] Next, a hardware configuration example of an information processing apparatus in the present disclosure will be described with reference to Figure 47

[0662] Figure 5 The hardware configuration example illustrated in FIG. 12 is a hardware configuration example of the information processing apparatus 100 described above with reference to Figure 29 and the information processing apparatus 300 described above with reference to Figure 47

[0663] Each element of the hardware configuration illustrated in FIG. 12 will be described. ​

[0664] The central processing unit (CPU) 501 functions as a data processing unit that performs various processing according to programs stored in a read only memory (ROM) 502 or a storage unit 508. For example, processing according to the sequence described in the above-described embodiments is performed. A random access memory (RAM) 503 stores programs, data, and the like executed by the CPU 501. The CPU 501, the ROM 502, and the RAM 503 are connected to each other through a bus 504. ​​​

[0665] The CPU 501 is connected to an input / output interface 505 via a bus 504, and an input unit 506 including various sensors, a camera, a switch, a keyboard, a mouse, a microphone, and the like, and an output unit 507 including a display, a speaker, and the like are connected to the input / output interface 505.

[0666] A storage unit 508 connected to the input / output interface 505 includes, for example, a USB memory, an SD card, a hard disk, and the like, and stores programs executed by the CPU 501 and various data. A communication unit 509 functions as a data transfer transmission / reception unit via a network such as the Internet or a local area network, and communicates with an external device.

[0667] A drive 510 connected to the input / output interface 505 drives a removable medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory such as a memory card, to record or read data.

[0668] [10. Summary of the configuration in the present disclosure]

[0669] Embodiments of the present disclosure have been described in detail with reference to specific embodiments. However, it is obvious to those skilled in the art that the embodiments can be modified or replaced without departing from the spirit of the present disclosure. That is, the present invention has been disclosed by way of example, and should not be construed in a limiting manner. In order to determine the spirit of the present disclosure, the claims should be considered.

[0670] Note that the technology disclosed in this specification can have the following configurations.

[0671] (1) An information processing apparatus comprising:

[0672] a sound source separation processing execution unit that executes sound source separation processing; and

[0673] an audio data editing processing unit that generates display data corresponding to sound sources separated by the sound source separation processing, and outputs the display data to a display unit.

[0674] (2) The information processing apparatus according to (1), wherein the audio data editing processing unit generates pitch change data of the sound source data over time as the display data corresponding to the sound sources, and outputs the pitch change data to the display unit.

[0675] (3) The information processing apparatus according to (1) or (2), wherein the audio data edit processing unit generates display data corresponding to the plurality of different sound sources separated by the sound source separation processing, and displays the generated display data corresponding to the plurality of different sound sources adjacent to each other on the display unit.

[0676] (4) The information processing apparatus according to any one of (1) to (3), wherein the audio data edit processing unit performs sound source data analysis processing that detects a shift between an audio output timing or an audio end timing of sound source data of the plurality of different sound sources separated by the sound source separation processing.

[0677] (5) The information processing apparatus according to any one of (1) to (4), wherein the audio data edit processing unit generates display data indicating a timing shift occurrence position at which at least an audio output timing or an audio end timing of sound source data of the plurality of different sound sources separated by the sound source separation processing is shifted from each other, and outputs the display data to the display unit.

[0678] (6) The information processing apparatus according to (5), wherein the audio data edit processing unit generates display data including a correction example for correcting the timing shift at the timing shift occurrence position, and outputs the display data to the display unit.

[0679] (7) The information processing apparatus according to (5) or (6), wherein the audio data edit processing unit generates display data including a display frame that surrounds the timing shift occurrence position, and outputs the display data to the display unit.

[0680] (8) The information processing apparatus according to any one of (5) to (7), wherein the audio data edit processing unit generates display data including a straight line connecting the same time positions of the display data corresponding to the plurality of different sound sources as the display data indicating the timing shift occurrence position, and outputs the display data to the display unit.

[0681] (9) The information processing apparatus according to any one of (1) to (8), wherein the audio data edit processing unit generates display data corresponding to the plurality of different sound sources separated by the sound source separation processing, and displays the generated display data corresponding to the plurality of different sound sources adjacent to each other on the display unit, and generates display data including a selection button for selecting display data corresponding to a sound source to be corrected, and outputs the display data to the display unit.

[0682] (10) The information processing apparatus according to any one of (1) to (9), wherein the audio data edit processing unit generates display data including a message asking whether or not to correct the sound source data together with the display data corresponding to the sound source, and outputs the display data to the display unit.

[0683] (11) The information processing apparatus according to any one of (1) to (10), wherein the audio data edit processing unit performs processing of correcting the sound source data in accordance with a user input to the display data corresponding to the sound source displayed on the display unit.

[0684] (12) The information processing apparatus according to any one of (1) to (11), wherein the audio data edit processing unit performs sound source data correction processing of correcting an offset between an audio output timing or an audio end timing of sound source data of a plurality of different sound sources separated by the sound source separation processing in accordance with a user input to the display data corresponding to the sound source displayed on the display unit.

[0685] (13) The information processing apparatus according to any one of (1) to (12), wherein the audio data edit processing unit performs sound source data analysis processing of detecting a pitch offset in the sound source data of the sound source separated by the sound source separation processing.

[0686] (14) The information processing apparatus according to any one of (1) to (13), wherein the audio data edit processing unit generates display data including pitch transform data corresponding to the sound source and position identification data indicating a position where a pitch offset occurs in the pitch transform data, and outputs the display data to the display unit.

[0687] (15) The information processing apparatus according to any one of (1) to (14), wherein the audio data edit processing unit performs sound source data correction processing of correcting a pitch offset in the sound source data separated by the sound source separation processing in accordance with a user input to the display data corresponding to the sound source displayed on the display unit.

[0688] (16) The information processing apparatus according to any one of (1) to (15), wherein the audio data edit processing unit generates display data including an individual sound source playback request button operable by a user to perform processing of playing back each sound source in accordance with a user input, and outputs the display data to the display unit.

[0689] (17) The information processing apparatus according to any one of (1) to (16), wherein the sound source separation processing execution unit performs the sound source separation processing using a trained model in accordance with a sound source type and a sound source distance.

[0690] (18) The information processing apparatus according to (17), in which the trained model is a trained model including a deep neural network (DNN).

[0691] (19) An information processing method executed by an information processing apparatus, the information processing method comprising:

[0692] a sound source separation processing execution step of executing sound source separation processing using a sound source separation processing execution unit; and

[0693] a step of generating display data corresponding to sound sources separated by the sound source separation processing using an audio data editing processing unit and outputting the display data to a display unit.

[0694] (20) A program causing an information processing apparatus to execute an information processing including:

[0695] a sound source separation processing execution step of executing sound source separation processing using a sound source separation processing execution unit; and

[0696] a step of generating display data corresponding to sound sources separated by the sound source separation processing using an audio data editing processing unit and outputting the display data to a display unit.

[0697] In addition, a series of processes described herein can be executed by hardware, software, or a combination of hardware and software. In the case of executing a software-based process, a program in which a process sequence is recorded can be installed in a memory incorporated in a dedicated hardware and executed, or the program can be installed in a general-purpose computer capable of executing various types of processes and executed. For example, the program can be recorded in a recording medium in advance. In addition to being installed on a computer from a recording medium, the program can be received via a network such as a local area network (LAN) or the Internet and installed on a recording medium such as a built-in hard disk.

[0698] Note that the various kinds of processes described herein can not only be executed in time series according to the description, but can be executed in parallel or separately depending on the processing capacity of the apparatus that executes the processes or in dependence on the need. Furthermore, the system in the present specification is a logical collection configuration of a plurality of devices, and is not limited to a system in which the respective configured devices are in the same housing.

[0699] Industrial applicability

[0700] As described above, with the configuration according to the embodiment of the present disclosure, a device and a method for generating and outputting display data corresponding to sound sources separated by sound source separation processing and performing correction on each sound source data according to user input are realized.

[0701] Specifically, for example, a sound source separation processing execution unit that executes sound source separation processing, and an audio data editing processing unit that generates display data corresponding to sound sources separated by the sound source separation processing and outputs the display data to a display unit. The audio data editing processing unit generates display data that can identify timing shifts, pitch shifts, and the like between different sound source data and outputs the display data together with pitch change data of the sound source data over time, and performs correction processing on each sound source data according to a user input.

[0702] With these configurations, a device and a method for generating and outputting display data corresponding to sound sources separated by sound source separation processing and performing correction on each sound source data according to a user input are realized.

[0703] (List of Reference Signs)

[0704] 1 User

[0705] 100 Information processing device

[0706] 101 Microphone

[0707] 102 Memory (storage unit)

[0708] 104 Pitch identification code

[0709] 105 Display region change request button

[0710] 106 Individual sound source playback request button

[0711] 107 All sound source playback request button

[0712] 111 Audio input unit (microphone)

[0713] 112 Storage unit

[0714] 113 Sound source separation processing execution unit

[0715] 114 Trained model (DNN)

[0716] 115 Audio data editing processing unit

[0717] 116 Playback control unit

[0718] 117 Display unit

[0719] 118 Audio output unit (speaker)

[0720] 119 Communication unit

[0721] 121 Sound source data analysis section

[0722] 122 display data generation section

[0723] 123 instruction analysis section (correction / playback, etc.)

[0724] 124 sound source data correction section

[0725] 125 sound source synthesis section

[0726] 131-1 to 131-n 1st to nth sound source data analysis sections

[0727] 141-1 to 141-n 1st to nth sound source data correction sections

[0728] 201 timing offset region indication box

[0729] 202 timing offset correction sound source selection button

[0730] 203 timing offset correction region indication box

[0731] 204 timing offset check line

[0732] 205 timing offset check box

[0733] 206 timing offset check line display / non-display selection box

[0734] 231 pitch offset check box

[0735] 232 pitch offset correction section check box

[0736] 300 information processing apparatus

[0737] 301 main audio input unit (main microphone)

[0738] 302 sub audio input unit (sub microphone)

[0739] 303 motion sensor (gyroscope)

[0740] 304 sound source distance calculation unit

[0741] 305 sound source distance corresponding sound source separation processing execution unit

[0742] 306 trained model (DNN)

[0743] 331 main microphone

[0744] 332 sub microphone

[0745] 400 data processing server

[0746] 401 communication unit

[0747] 402 sound source distance corresponding sound source separation processing execution unit

[0748] 403 trained model (DNN)

[0749] 501 CPU

[0750] 502 ROM

[0751] 503 RAM

[0752] 504 bus

[0753] 505 input / output interface

[0754] 506 input unit

[0755] 507 output unit

[0756] 508 storage unit

[0757] 509 communication unit

[0758] 510 driver

[0759] 511 removable medium

Claims

1. An information processing apparatus comprising: a sound source separation processing execution unit that executes sound source separation processing; and an audio data editing processing unit that generates display data corresponding to sound sources separated by the sound source separation processing, and outputs the display data to a display unit.

2. The information processing apparatus according to claim 1, wherein The audio data editing processing unit generates pitch change data of sound source data over time as the display data corresponding to the sound sources, and outputs the pitch change data to the display unit.

3. The information processing apparatus according to claim 1, wherein The audio data editing processing unit generates display data corresponding to a plurality of different sound sources separated by the sound source separation processing, and displays the generated display data corresponding to the plurality of different sound sources adjacent to each other on the display unit.

4. The information processing apparatus according to claim 1, wherein The audio data editing processing unit executes sound source data analysis processing that detects a shift between an audio output timing or an audio end timing of sound source data of a plurality of different sound sources separated by the sound source separation processing.

5. The information processing apparatus according to claim 1, wherein The audio data editing processing unit generates display data indicating a timing shift occurrence position at which at least an audio output timing or an audio end timing of sound source data of a plurality of different sound sources separated by the sound source separation processing is shifted from each other, and outputs the display data to the display unit.

6. The information processing apparatus according to claim 5, wherein The audio data editing processing unit generates display data including a correction example for correcting the timing shift at the timing shift occurrence position, and outputs the display data to the display unit.

7. The information processing apparatus according to claim 5, wherein The audio data editing processing unit generates display data including a display frame that surrounds the timing shift occurrence position, and outputs the display data to the display unit.

8. The information processing apparatus according to claim 5, wherein The audio data editing processing unit generates display data including a straight line connecting the same time positions of the display data corresponding to the plurality of different sound sources as the display data indicating the timing shift occurrence position, and outputs the display data to the display unit.

9. The information processing apparatus according to claim 1, wherein The audio data editing processing unit generates display data corresponding to a plurality of different sound sources separated by the sound source separation processing, and displays the generated display data corresponding to the plurality of different sound sources adjacent to each other on the display unit, and generates display data including a selection button for selecting display data corresponding to a sound source to be corrected, and outputs the display data to the display unit.

10. The information processing apparatus according to claim 1, wherein The audio data editing processing unit generates display data including a message asking whether to correct sound source data together with the display data corresponding to the sound source, and outputs the display data to the display unit.

11. The information processing apparatus according to claim 1, wherein The audio data editing processing unit executes processing of correcting sound source data in accordance with a user input to the display data corresponding to the sound source displayed on the display unit.

12. The information processing apparatus according to claim 1, wherein The audio data edit processing unit performs a sound source data correction process that corrects a shift between an audio output timing or an audio end timing of sound source data of a plurality of different sound sources separated by the sound source separation process, according to a user input to the display data corresponding to the sound source displayed on the display unit. 13.The information processing apparatus according to claim 1, wherein The audio data edit processing unit performs a sound source data analysis process that detects a pitch shift in the sound source data of the sound source separated by the sound source separation process.

14. The information processing apparatus according to claim 1, wherein The audio data edit processing unit generates display data including pitch conversion data corresponding to the sound source and position identification data indicating a position where a pitch shift occurs in the pitch conversion data, and outputs the display data to the display unit.

15. The information processing apparatus according to claim 1, wherein The audio data edit processing unit performs a sound source data correction process that corrects a pitch shift in the sound source data separated by the sound source separation process, according to a user input to the display data corresponding to the sound source displayed on the display unit.

16. The information processing apparatus according to claim 1, wherein The audio data edit processing unit generates display data including an individual sound source playback request button that can be operated by a user to perform a process of playing back each sound source according to a user input, and outputs the display data to the display unit.

17. The information processing apparatus according to claim 1, wherein The sound source separation process execution unit executes the sound source separation process using a trained model according to a sound source type and a sound source distance.

18. The information processing apparatus according to claim 17, wherein The trained model is a trained model including a deep neural network (DNN).

19. An information processing method executed by an information processing apparatus, the information processing method comprising: a sound source separation process execution step of executing a sound source separation process using a sound source separation process execution unit; and a step of generating display data corresponding to sound sources separated by the sound source separation process and outputting the display data to a display unit using an audio data edit processing unit.

20. A program causing an information processing apparatus to execute an information processing including: a sound source separation process execution step of executing a sound source separation process using a sound source separation process execution unit; and a step of generating display data corresponding to sound sources separated by the sound source separation process and outputting the display data to a display unit using an audio data edit processing unit.

21. An information processing apparatus comprising: a sound source separation process execution unit configured to execute a sound source separation process; and an audio data edit processing unit configured to generate display data corresponding to sound sources separated by the sound source separation process and output the display data to a display unit.

Images