Acoustic device, program, and control method

Abstract

Provided is a sound device (100) that performs control pertaining to a separation process of separating audio data of a musical piece into multiple parts that constitute the musical piece, wherein the sound device comprises a display control unit (131) that causes a display to display labels representing each of the multiple parts, an operation acquisition unit (132) that acquires information indicating a user operation for selecting at least one of the labels, and a selection unit (134) that selects at least one part included in the multiple parts in accordance with the information indicating the user operation.

Description

[Technical Field]

[0001] The present invention relates to an audio device, a program, and a control method. [Background technology]

[0002] In recent years, various technologies have been developed for audio devices and applications related to DAWs (Digital Audio Workstations) and DTMs (Desktop Music). For example, Non-Patent Document 1 discloses an application that can perform audio separation after the fact on a computer. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Djay Neural Mix (algoriddim GmbH) application introduction homepage (https: / / www.algoriddim.com / neural-mix) Summary of the Invention [Problem to be solved by the invention]

[0004] However, it cannot be said that the above-mentioned technologies have yet been fully proposed for practical use. Therefore, an object of the present invention is to provide an audio device, a program, and a control method that can achieve high operability and flexibility in relation to an audio device that controls separation processing. [Means for solving the problem]

[0005] [1] An audio device that controls a separation process that separates the audio data of a song into multiple parts that make up the song, and that includes a display control unit that displays labels representing each of the multiple parts on a display, an operation acquisition unit that acquires information indicating a user operation that selects at least some of the labels, and a selection unit that selects at least one part included in the multiple parts in accordance with the information indicating the user operation. [2] The audio device described in [1], wherein the multiple parts include candidate parts from which the audio data of the music can be separated in an unexecuted separation process, and further comprising an output unit that outputs information indicating at least one part selected by the selection unit to a device that executes the separation process. [3] The multiple parts include parts from which the audio data of the music piece was separated in the separation process that was executed, and the operation acquisition unit further acquires information indicating a user operation of inputting a label representing at least a part of the multiple parts, and further includes an output unit that outputs the input label together with information identifying the part associated with the label to a device that stores the results of the separation process. [4] The multiple parts include a first part from which the audio data of the song was separated in the separation process executed, and a second part from which the audio data of the song was not separated in the separation process, and the display control unit displays a label representing the first part and a label representing the second part on the display in different display forms. [5] A program configured to cause a computer to function as the acoustic device described in any one of [1] to [4]. [6] A control method implemented by an audio device that controls a separation process that separates audio data of a song into multiple parts that make up the song, the control method including the steps of: displaying labels representing each of the multiple parts on a display; acquiring information indicating a user operation that selects at least a portion of the labels; and selecting at least one part included in the multiple parts in accordance with the information indicating the user operation. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a block diagram showing a schematic functional configuration of an audio device according to an embodiment of the present invention; [Figure 2] FIG. 10 is a diagram illustrating an example of a separation process according to an embodiment of the present invention. [Figure 3] FIG. 2 is a diagram showing an example of a display screen according to an embodiment of the present invention. [Figure 4] FIG. 10 is a diagram showing another example of a display screen according to an embodiment of the present invention. [Figure 5] 3 is a flowchart illustrating a control method according to an embodiment of the present invention. [Figure 6A] FIG. 10 is a diagram showing another example of a display screen according to an embodiment of the present invention. [Figure 6B] FIG. 10 is a diagram showing another example of a display screen according to an embodiment of the present invention. [Figure 7A] FIG. 10 is a diagram showing another example of a display screen according to an embodiment of the present invention. [Figure 7B] FIG. 10 is a diagram showing another example of a display screen according to an embodiment of the present invention. [Figure 8] 10 is another flowchart illustrating a control method according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0007] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0008] 1 is a block diagram showing a schematic functional configuration of an audio device according to one embodiment of the present invention, which is an audio device configured by a computer. 1, the audio device 100 includes a display 110, an operation unit 120, a control unit 130, and a storage unit 140. The functions of each of the above units are realized by a processor operating according to a program in an audio device having, for example, a computer hardware configuration. The functions of each unit are further described below.

[0009] The display 110 is a display device that displays a screen including various types of information and controls for accepting user operations in the audio device 100. As the display 110, various types of display devices can be used, such as a liquid crystal display device or an organic EL display device. The operation unit 120 includes input devices such as a keyboard and a mouse.

[0010] The control unit 130 is implemented in the acoustic device 100 by, for example, a communication interface, a processor such as a CPU (Central Processing Unit), and a memory that serves as a working area, and controls the operation of the acoustic device 100. The control unit 130 includes a display control unit 131, an operation acquisition unit 132, a separation processing unit 133, a selection unit 134, and an output unit 135, which are realized by the processor operating in accordance with a program stored in the memory or received via the communication interface.

[0011] The display control unit 131 causes the display 110 to display a screen including information on the audio data of the music pieces related to the acoustic device 100 and a GUI (Graphical User Interface) for accepting user operations related to setting and controlling the audio data. The operation acquisition unit 132 acquires information indicating the user operation based on the user operation performed via the operation unit 120. Then, the operation acquisition unit 132 supplies the information indicating the user operation to each unit in the control unit .

[0012] The separation processing unit 133 executes a separation process to separate the audio data of a song into multiple parts that make up the song. The multiple parts separated from the audio data correspond to the sounds of each part of the song played back using the audio data. Here, each part corresponds to, for example, vocals or the sound of each instrument. Fig. 2 is a diagram showing an example of separation processing. The example in Fig. 2 shows an example in which separation processing is performed on audio data and the data is separated into five parts: vocals, bass, drums, piano, and others. In the example in Fig. 2, the piano part is a part that has not been separated, and no data exists for it.

[0013] In addition to the above examples, parts may be further subdivided and separated by instrument type, such as bass drum, snare drum, and hi-hat. In this case, direct separation processing into the bass drum, snare drum, and hi-hat parts may be performed from the audio data of the music to be separated, or stepwise separation processing into each part may be performed by separating the drum part into music. In either case, the parts of the music include parts obtained by separating each part that makes up the music, for example, parts that make up some of the instruments that are obtained by further subdividing each part that makes up the music. Since various known technologies can be used for such separation processing into multiple parts, detailed description will be omitted.

[0014] The acoustic device 100 may be provided with a communication interface, and the separation process may be performed by an external acoustic device, computer, etc., or audio data on which separation process has been performed may be acquired from an external acoustic device, computer, etc. via a communication interface (not shown) and stored in the storage unit 140. In this case, the acoustic device 100 does not include a device that performs separation process (separation processing unit 133), and the external acoustic device and computer function as a device that performs separation process (separation processing unit 133).

[0015] The selection unit 134 selects at least one part from the plurality of parts in accordance with the information indicating the user operation, and then supplies information indicating the selection result to each unit in the control unit 130. The output unit 135 outputs information to the storage unit 140 and to the outside of the acoustic device 100. Specific processing by the output unit 135 will be described later.

[0016] The storage unit 140 is configured to be able to store audio data of songs using an HDD (Hard Disk Drive), flash memory, or the like. The storage unit 140 stores audio data of multiple songs in a predetermined format, such as MP3 format. In addition to audio information, the audio data includes tag information such as the song's BPM, artwork, title, artist name, album name, key, number of DJ plays, and genre. A timestamp, which is information about the playback position, is associated with the audio data stored in the storage unit 140. The storage unit 140 also functions as a device for storing the results of the separation process. The storage of the results of the separation process will be described later.

[0017] The acoustic device 100 may be configured to include a communication interface, and acquire audio data stored in an external storage device, a computer, or the like via the communication interface (not shown), and store the data in the storage unit 140. In this case, the acoustic device 100 does not include the storage unit 140, and the external storage device functions as the storage unit 140.

[0018] In the acoustic device 100 described above, a display screen that is displayed on the display 110 when the user controls the separation process will be described.

[0019] (1) Control before separation Regarding control before the separation process, the display control unit 131 of the acoustic device 100 displays labels on the display 110 representing candidate parts into which the audio data of the music can be separated in the separation process that has not yet been executed, and the operation acquisition unit 132 acquires information indicating a user operation to select at least a portion of the labels displayed on the display 110. 3 and 4 are diagrams showing examples of display screens in this embodiment.

[0020] As shown in Fig. 3, the display control unit 131 displays a list of candidate parts from which audio data of a song can be separated in the separation process in the display area A1 of the display 110. Fig. 3 shows an example of a list of text indicating the type of part and check boxes. In the example of Fig. 3, the drums, guitar, and other parts are checked. The text representing the type of part is a label that identifies candidate parts into which the audio data of the song can be separated, and the operation acquisition unit 132 acquires whether or not the check box for each part is checked as information indicating the user operation. The user can select the parts to be separated by the separation process prior to the separation process by operating the operation unit 120 based on the information displayed on the display 110.

[0021] Fig. 4 is a modified example of Fig. 3. As shown in Fig. 4, the display control unit 131 may separately display lists of candidate parts into which audio data of a song can be separated in the separation process in display areas A2 and A3 of the display 110. In the example of Fig. 4, parts to be separated in the separation process are displayed in display area A2, parts not to be separated in the separation process are displayed in display area A3, and controls for moving parts between display area A2 and display area A3 are displayed in display area A4. The user can separate the parts that he / she wants to separate by the separation process from other parts by operating the operation unit 120 based on the information displayed on the display 110, prior to the separation process.

[0022] As described above, the display control unit 131 displays labels on the display 110 representing candidate parts into which the audio data of a song can be separated in an unexecuted separation process, and the operation acquisition unit 132 acquires information indicating a user operation to select at least a portion of the labels displayed on the display 110. The selection unit 134 selects at least one part from candidate parts from which audio data of the music can be separated in the separation process, based on information indicating the user operation acquired by the operation acquisition unit 132. Then, the separation processing unit 133 executes the separation process on the part selected by the selection unit 134.

[0023] As described above, when the acoustic device 100 does not include the separation processing unit 133 and the separation processing is performed by an external acoustic device, computer, etc., the output unit 135 outputs information indicating the part selected by the selection unit 134 to the external acoustic device, computer, etc. that performs the separation processing.

[0024] Next, the operation of each part during control before separation processing will be described. FIG. 5 is a flowchart showing the operation of each unit during control before separation processing. 5, first, the display control unit 131 displays candidates for parts into which audio data of a song can be separated in the separation process on the display 110 (step S101). Then, the operation acquisition unit 132 determines whether a user operation for selecting a part has been performed via the operation unit 120 (step S102). When the operation acquisition unit 132 determines that a user operation to select a part has been performed (YES in step S102), the display control unit 131 updates the display content of the display 110 (step S103). The audio device 100 repeats the processes of steps S102 and S103 until the part selection is completed (YES in step S104).

[0025] As described above, the display control unit 131 of the acoustic device 100 displays labels representing candidate parts that can be separated in the separation process on the display 110, and the operation acquisition unit 132 acquires information indicating a user operation to select at least some of the labels. Based on the acquired information indicating the user operation, the selection unit 134 selects at least one part, and the separation processing unit 133 executes the separation process for the selected part. Therefore, in the past, all parts were subject to the separation process, for example, even for audio data of a piece of music that does not have a piano part, so the piano part was subject to the separation process. However, the acoustic device 100 of this embodiment allows for greater flexibility in setting the separation process. Therefore, it is possible to select parts according to the characteristics of the audio data to be separated, or to select parts according to the user's wishes.

[0026] (2) Control after separation Regarding control after the separation process, the display control unit 131 of the audio device 100 causes the display 110 to display labels indicating the parts into which the audio data of the music piece has been separated in the separation process executed by the separation processing unit 133 . 6A, 6B, 7A, and 7B are diagrams showing examples of display screens in this embodiment.

[0027] As shown in Fig. 6A, the display control unit 131 displays a waveform indicating the separation result in a display area A5 of the display 110, and displays a list indicating the parts into which the audio data of the music piece has been separated in the separation process executed by the separation processing unit 133 in a display area A6 superimposed on the display area A5. An example of a list of text indicating the type of part is shown in the display area A6 of Fig. 6A. The text indicating the type of part is a label that identifies the part into which the audio data of the music piece has been separated in the separation process executed.

[0028] In addition, if the separation process results in parts where the audio data of the music piece has been separated and parts where it has not been separated, the display control unit 131 causes the display 110 to display labels representing parts where the audio data of the music piece has been separated and labels representing parts where it has not been separated in different display formats. For example, if a piano part has not been separated, the label representing the piano part where the audio data of the music piece has not been separated is displayed in reverse color from the other labels, as shown in display area A7 in FIG. 6B . Furthermore, for example, the label representing the part where the audio data of the music piece has not been separated may be displayed in grayout, hidden, or with a message such as "No separation result." By displaying in this manner, the user can easily distinguish between parts where the audio data of the music piece has been separated and parts where it has not been separated. The parts from which the audio data of the music piece has been separated and the parts from which it has not been separated can be determined, for example, depending on whether the separation result by the separation processing unit 133 includes audio data for each part.

[0029] As described above, the results of the separation process can be confirmed by displaying on the display 110 labels representing the parts into which the audio data of the music piece has been separated, but the acoustic device 100 can also edit the labels described above. The display control unit 131 of the audio device 100 displays on the display 110 a screen for accepting user operations to input a label for any of the labels displayed as a result of the separation process, and the operation acquisition unit 132 acquires information indicating the user operations to select and input a label.

[0030] As shown in Figure 7A, the display control unit 131 displays a waveform showing the separation results in display area A5 of the display 110, as in Figure 6A, displays a list showing the parts from which the audio data of the song has been separated in display area A6, and displays a text box for entering a label in display area A8 superimposed on display area A6. The user selects any part from the list by operating the operation unit 120, and inputs a label for the selected part into the text box by operating the operation unit 120. For example, if a piano sound part is separated by the separation process, but the user's sensory evaluation of that part reveals that the guitar sound part is correct, that is, if the guitar sound part was erroneously separated into a piano sound part, the user can correct the label of the piano sound part to the guitar sound part.

[0031] Fig. 7B is a modification of Fig. 7A. As shown in Fig. 7B, display control unit 131 may display a select box for changing the label in display area A9 that overlaps display area A6 of display 110. The user can select the correct label from the selection box displayed on the display 110 by operating the operation unit 120 .

[0032] It is preferable that the user can set whether or not to display each of the displays described with reference to Figures 6A, 6B, 7A, and 7B. For example, in the examples of Figures 7A and 7B, when a part is selected from the list displayed in display area A6, a display related to inputting the label of that part (display area A8 or display area A9) should be displayed. 6A, 6B, 7A, and 7B show examples in which text indicating the type of part (vocals, bass, etc.) is displayed in display area A6 or display area A7 as a label for identifying the part into which the audio data of the song has been separated. However, a configuration may be adopted in which the user can input new labels for some or all of the parts. Such a configuration is useful, for example, when the type of any separated part is unclear.

[0033] Next, the operation of each part during control after separation processing will be described. FIG. 8 is a flowchart showing the operation of each unit during control after separation processing. 8, first, the display control unit 131 causes the display 110 to display the parts into which the audio data of the music piece has been separated in the separation process executed by the separation processing unit 133 (step S201). Then, the operation acquisition unit 132 determines whether or not a user operation to select a part has been performed via the operation unit 120 (step S202). If the operation acquisition unit 132 determines that a user operation to select a part has been performed (YES in step S202), the operation acquisition unit 132 determines whether a user operation to input a label has been performed via the operation unit 120 (step S203). When the operation acquisition unit 132 determines that a user operation to input a label has been performed (YES in step S203), the display control unit 131 updates the display content of the display 110 (step S204). The acoustic device 100 repeats the processes from step S202 to step S204 until the label input is completed (step S205 YES), and when the label input is completed (step S205 YES), the output unit 135 outputs the input label to the memory unit 140 together with information identifying the part associated with that label (step S206).

[0034] As described above, the display control unit 131 of the acoustic device 100 displays on the display 110 labels representing parts into which the audio data of the music piece has been separated in the executed separation process, and the operation acquisition unit 132 acquires information indicating a user operation for selecting and inputting a label. Then, based on the acquired information indicating the user operation, the selection unit 134 selects at least one part and outputs the input label to the storage unit 140 together with information identifying the part associated with that label. This makes it possible to edit the results of the separation process based on user operations. This makes it possible to make the results of the separation process more realistic, thereby improving the accuracy of the separation process.

[0035] (3) Application of controls to other processes The input result of the label described in (2) may be applied to other controls in the audio device 100. For example, when setting filter or effect parameters for each part separated by the separation process, the label for each part when setting the parameters may be dynamically changed based on the label input results described in (2). Furthermore, when performing various edits on a piece of music based on parts separated by the separation process, the labels of each part may be dynamically changed during editing based on the label input results described in (2). For example, this type of control is useful when editing to mute any part or replace it with a different instrument part. Also, during other processing, labels representing parts of music whose audio data has been separated and labels representing parts that have not been separated may be displayed in different display formats.

[0036] According to one embodiment of the present invention as described above, labels representing each of a plurality of parts are displayed on a display, information indicating a user operation to select at least some of the labels is acquired, and at least one part from the plurality of parts is selected in accordance with the information indicating the user operation. Therefore, by providing a user-friendly display for an audio device that controls the separation process, high operability and flexibility can be achieved.

[0037] According to one embodiment of the present invention, the plurality of parts include candidate parts from which audio data of a song can be separated in an unexecuted separation process, and information indicating at least one selected part is output to a device that executes the separation process, thereby improving the flexibility of setting up the separation process.

[0038] According to one embodiment of the present invention, the multiple parts include parts from which the audio data of the music piece was separated in the separation process. The system further acquires information indicating a user operation to input labels representing at least some of the multiple parts, and outputs the input labels together with information identifying the parts associated with the labels to a device that stores the results of the separation process. Therefore, the results of the separation process can be edited based on the user operation. Furthermore, according to one embodiment of the present invention, the labels representing the parts (first parts) from which the audio data of the music piece was separated in the separation process and the parts (second parts) from which the audio data of the music piece was not separated in the separation process are displayed in different display formats, thereby enabling a display that is easy for the user to understand.

[0039] The separation process described in the above embodiment may be performed on a song-by-song basis, or may be performed on a portion of a song. For example, a known music analysis process may be performed, and the separation process may be performed for each section based on the development of the music. In this case, the control before the separation process described in (1) may be configured to set the separation process for each section. Furthermore, the control after the separation process described in (2) may be configured to edit the results of the separation process for each section. This configuration allows for more detailed control of the separation process.

[0040] Furthermore, the display screens described in the above embodiment are merely examples, and the elements, designs, layouts, etc. are not limited to those of the present embodiment. For example, labels identifying parts may be illustrations instead of text, or may be a combination of illustrations and text. Furthermore, the elements, designs, layouts, etc. of each display screen may be user-configurable.

[0041] In the above embodiment, the control unit 130 of the acoustic device 100 performs the above-described processing by reading and executing a program from a storage unit or recording medium (not shown). However, this is not limiting, and for example, the control unit 130 may obtain and execute a program from a device on a network. Examples of recording media include disk-type recording media, HDDs, and semiconductor memories.

[0042] Furthermore, the audio device with the above-described functions is not limited to the examples described in the above embodiments, but may be, for example, a mixer or a DJ controller with a mixer function. Furthermore, the present invention is not limited to DJ equipment and DJ applications, but may also be applied to music applications and streaming services using the Internet. It can also be applied to electronic musical instruments, as well as audio devices such as DAWs (Digital Audio Workstations) and DTMs (Desktop Music).

[0043] Although the preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings, the present invention is not limited to these examples. It is clear that a person skilled in the art to which the present invention pertains can conceive of various modifications or alterations within the scope of the technical idea described in the claims, and it is understood that these also naturally fall within the technical scope of the present invention. [Explanation of symbols]

[0044] 100...acoustic device, 110...display, 120...operation unit, 130...control unit, 131...display control unit, 132...operation acquisition unit, 133...separation processing unit, 134...selection unit, 135...output unit, 140...storage unit.

Claims

1. An audio device that controls a separation process that separates multiple parts from audio data of a piece of music that includes multiple parts each corresponding to an instrument or vocal, and outputs waveforms of each of the multiple parts, a display control unit that displays labels representing each of the plurality of parts on a display; an operation acquisition unit that acquires information indicating a user operation for selecting at least a part of the label; a selection unit that selects at least one part included in the plurality of parts as a target part in accordance with the information indicating the user operation, When multiple target parts are selected by the selection unit, the display control unit associates labels representing each of the selected target parts with waveforms showing the separation results for each of the multiple target parts and displays them on the display.

2. the plurality of parts include candidates for parts that can be separated from the audio data of the music piece in the separation process that has not yet been executed, The audio device according to claim 1 , further comprising an output unit that outputs information indicating the at least one part selected by the selection unit to a device that executes the separation process.

3. An acoustic device that controls a separation process that separates multiple parts from audio data of a piece of music that includes multiple parts each corresponding to an instrument or vocal, and outputs waveforms of each of the multiple parts, a display control unit that displays labels representing each of the plurality of parts on a display; an operation acquisition unit that acquires information indicating a user operation of selecting at least a part of the label and inputting the label; a selection unit that selects at least one part from the plurality of parts in accordance with the information indicating the user operation; an output unit that outputs the input label together with information identifying the part associated with the label to a device that stores a result of the separation process; The plurality of parts include parts separated from the audio data of the music piece in the separation process that was executed.

4. An acoustic device that controls a separation process that separates multiple parts from audio data of a piece of music that includes multiple parts each corresponding to an instrument or vocal, and outputs waveforms of each of the multiple parts, a display control unit that displays labels representing each of the plurality of parts on a display; an operation acquisition unit that acquires information indicating a user operation of selecting at least a part of the label and inputting the label; a selection unit that selects at least one part from the plurality of parts in accordance with the information indicating the user operation; an output unit that outputs the input label together with information that identifies the part associated with the label.

5. the plurality of parts include a first part into which the audio data of the music piece has been separated in the separation process that has been executed, and a second part into which the audio data of the music piece has not been separated in the separation process, The acoustic device according to claim 3 , wherein the display control unit causes the display to display the label representing the first part and the label representing the second part in different display forms.

6. A program configured to cause a computer to function as the acoustic device according to any one of claims 1 to 5.

7. A control method implemented by an acoustic device that controls a separation process for separating a plurality of parts from audio data of a piece of music, the parts corresponding to instruments or vocals, and outputting waveforms of each of the plurality of parts, comprising: displaying a label representing each of the plurality of parts on a display; obtaining information indicating a user action of selecting at least a portion of the label; selecting at least one part included in the plurality of parts as a target part in accordance with the information indicating the user operation; When a plurality of target parts are selected, a label representing each of the selected target parts is associated with a waveform showing the separation results of each of the plurality of target parts and displayed on the display.

Images