PROGRAM, INFORMATION PROCESSING METHOD AND INFORMATION PROCESSING APPARATUS

Abstract

To provide a program capable of generating reference data which is further appropriate as reference when scoring singing from sound source data.SOLUTION: A program causes a computer of an information processing device to perform a sound source data acquisition step for acquiring sound source data including a melody part of music; a singing voice separation extraction step for separating and extracting first singing voice data of a first singer and second singing voice data of a second singer different from the first singer from the sound source data, and a reference data generation step for generating first reference data of a portion which the first singer sings in the melody part on the basis of the first singing voice data and generating second reference data of a portion which the second singer sings in the melody part on the basis of the second singing voice data.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] The present disclosure relates to a program, an information processing method, and an information processing device for generating reference data from audio source data of a song. [Background technology]

[0002] A karaoke device is known which has a program main vocal data extraction means which extracts main vocal data, which is the singing data of the main melody, from song data by canceling the band of the vocal signal that is centered in the singing data of the song data and subtracting the remaining data from the original data (singing data) to extract it as main vocal data, and a reference data generation means which generates reference data from the extracted main vocal data as a standard for scoring singing (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2015-225302 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the technology shown in Patent Document 1, the extracted main vocal data may not necessarily match the part sung as karaoke. For example, a production part in which a human voice is used other than the singing part, or a sound of an instrument that is erroneously extracted due to a problem of extraction accuracy, etc., may be mixed into the extracted main vocal data. If reference data is generated using such main vocal data including a non-singing part that does not match the singing part of karaoke, it may not be appropriate as a standard for scoring singing.

[0005] The present disclosure has been made to solve such problems. It is an object of the present invention to provide a program, an information processing method, and an information processing device that can generate more appropriate reference data as a standard for scoring singing from sound source data including the melody part of a song. [Means for solving the problem]

[0006] The program of the present disclosure is a program to be executed on a computer of an information processing device, and the following steps are executed on the computer of the information processing device: a sound source data acquisition step for acquiring sound source data including a melody part of a song; a singing voice separation and extraction step for separating and extracting first singing voice data of a first singer and second singing voice data of a second singer different from the first singer from the sound source data; and a reference data generation step for generating first reference data for a portion of the melody part sung by the first singer based on the first singing voice data, and generating second reference data for a portion of the melody part sung by the second singer based on the second singing voice data.

[0007] The information processing method of the present disclosure is an information processing method executed on a computer of an information processing device, and includes: a sound source data acquisition step of acquiring sound source data including a melody part of a song; a singing voice separation and extraction step of separating and extracting first singing voice data of a first singer and second singing voice data of a second singer different from the first singer from the sound source data; and a reference data generation step of generating first reference data for a portion of the melody part sung by the first singer based on the first singing voice data, and generating second reference data for a portion of the melody part sung by the second singer based on the second singing voice data.

[0008] The information processing device of the present disclosure includes a sound source data acquisition unit that acquires sound source data including a melody part of a song, a singing voice separation and extraction unit that separates and extracts first singing voice data of a first singer and second singing voice data of a second singer different from the first singer from the sound source data, and a reference data generation unit that generates first reference data for a portion of the melody part sung by the first singer based on the first singing voice data, and generates second reference data for a portion of the melody part sung by the second singer based on the second singing voice data. Effect of the Invention

[0009] The program, information processing method, and information processing device disclosed herein have the effect of making it possible to generate reference data that is more appropriate as a standard for scoring singing, from audio source data including the melody part of a song. [Brief description of the drawings]

[0010] [Figure 1] 1 is a diagram showing an overall configuration of a communication system according to a first embodiment; [Diagram 2] 2 is a block diagram showing a configuration of a terminal included in the communication system according to the first embodiment. FIG. [Diagram 3] 2 is a block diagram showing a configuration of a server included in the communication system according to the first embodiment. FIG. [Figure 4] FIG. 2 is a diagram showing an example of synchronized lyrics data stored in a server included in the communication system according to the first embodiment. [Diagram 5] 4 is a flowchart showing an example of processing in the communication system according to the first embodiment. [Figure 6] 13 is a block diagram showing a configuration of a main part of a modified example of a server included in the communication system according to the first embodiment. FIG. [Figure 7] 13 is a flowchart showing an example of processing in a modification of the communication system according to the first embodiment. [Figure 8]FIG. 11 is a block diagram showing a configuration of a server included in a communication system according to a second embodiment. [Figure 9] 11 is a flowchart showing an example of a process in the communication system according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] The embodiments for implementing the program, information processing method, and information processing device according to the present disclosure will be described with reference to the accompanying drawings. In each drawing, the same or corresponding parts are given the same reference numerals, and duplicated descriptions are appropriately simplified or omitted. In the following description, for convenience, the positional relationship of each structure may be expressed based on the illustrated state. Note that the present disclosure is not limited to the following embodiments, and it is possible to freely combine each embodiment, modify any component of each embodiment, or omit any component of each embodiment within the scope of the present disclosure.

[0012] Embodiment 1 A first embodiment of the present disclosure will be described with reference to Figs. 1 to 7. Fig. 1 is a diagram showing an overall configuration of a communication system. Fig. 2 is a block diagram showing a configuration of a terminal included in the communication system. Fig. 3 is a block diagram showing a configuration of a server included in the communication system. Fig. 4 is a diagram showing an example of synchronized lyrics data stored in a server included in the communication system. Fig. 5 is a flowchart showing an example of processing in the communication system. Fig. 6 is a block diagram showing a configuration of a main part in a modified example of a server included in the communication system. Fig. 7 is a flowchart showing an example of processing in the modified example of the communication system.

[0013] As shown in Fig. 1, a communication system 400 according to this embodiment includes a server 100 and a terminal 200. In the communication system 400, the server 100 and the terminal 200 are communicably connected via a network 300. In the configuration example described here, the server 100 provides a music sound source data distribution service, a karaoke scoring service, and the like to the terminal 200 owned by a user via the network 300. Note that the number of terminals 200 connected to the network 300 is not limited to two, and may be one or three or more.

[0014] The network 300 serves to connect one or more terminals 200 and one or more servers 100. In other words, the network 300 refers to a communication network that provides a connection path so that the terminals 200 can transmit and receive data after connecting to the server 100. One or more parts of the network 300 may or may not be a wired network or a wireless network.

[0015] The network 300 may include, for example, an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), a portion of the Internet, a portion of a public switched telephone network (PSTN), a mobile phone network, an integrated service digital network (ISDN), a wireless LAN, a long term evolution (LTE), a code division multiple access (CDMA), Bluetooth (registered trademark), or a satellite communication, or a combination of two or more of these. The network 300 may include one or more networks 300.

[0016] The terminal 200 may be any information processing terminal capable of implementing the functions of the embodiments of the present disclosure. The terminal 200 includes, for example, a smartphone, a mobile phone (feature phone), a computer (e.g., a desktop PC, a laptop PC, a tablet PC, etc.), a media computer platform (e.g., a cable or satellite set-top box, a digital video recorder), a handheld computer device (e.g., a PDA (Personal Digital Assistant), an email client, etc.), a wearable terminal (a glasses-type device, a watch-type device, etc.), or other types of computers or communication platforms. The terminal 200 may also be expressed as an information processing terminal.

[0017] The server 100 has a function of providing a predetermined service to the terminal 200. The server 100 may be any device as long as it is an information processing device capable of realizing the functions of the embodiment according to the present disclosure. The server 100 includes, for example, a server device, a computer (e.g., a desktop PC, a laptop PC, a tablet PC, etc.), a media computer platform (e.g., a cable or satellite set-top box, a digital video recorder), a handheld computer device (e.g., a PDA, an email client, etc.), or other types of computers or communication platforms. The server 100 may also be expressed as an information processing device. When it is not necessary to distinguish between the server 100 and the terminal 200, the server 100 and the terminal 200 may or may not be expressed as information processing devices.

[0018] The configuration of each terminal 200 is basically the same. Next, the configuration of the terminal 200 will be described with reference to FIG. 2. The terminal 200 includes a terminal control unit 230, a terminal storage unit 220, a terminal communication unit 210, an input / output unit 240, a display unit 250, a microphone 260, a speaker 270, and a camera 280. The hardware components of the terminal 200 are connected to each other, for example, via a bus. It is not essential that the hardware configuration of the terminal 200 includes all the components described here. For example, the terminal 200 may or may not be configured such that individual components such as the camera 280 or multiple components are detachable.

[0019] The terminal communication unit 210 transmits and receives various data via the network 300. The communication may be performed either wired or wirelessly, and any communication protocol may be used as long as the communication between the terminals can be performed. The terminal communication unit 210 has a function of performing communication with the server 100 via the network 300. The terminal communication unit 210 includes a terminal transmission unit 211 and a terminal reception unit 212. The terminal transmission unit 211 transmits various data to the server 100 according to an instruction from the terminal control unit 230. The terminal reception unit 212 receives various data transmitted from the server 100 and transmits the data to the terminal control unit 230. The terminal communication unit 210 may also be expressed as a terminal communication I / F (interface). When the terminal communication unit 210 is configured with a physically structured circuit, it may also be expressed as a terminal communication circuit.

[0020] The input / output unit 240 includes an input unit and an output unit. The input unit is a device for inputting various operations to the terminal 200. The output unit is a device for outputting the processing results processed by the terminal 200. The input / output unit 240 may be integrated with the input unit and the output unit, may be separated into the input unit and the output unit, or may not be.

[0021] The input unit is realized by any one or a combination of all kinds of devices that can receive input from a user and transmit information related to the input to the terminal control unit 230. The input unit includes, for example, a touch panel, a touch display, hardware keys such as a keyboard, a pointing device such as a mouse, a camera (operation input via moving images), and a microphone (operation input by voice).

[0022] The output unit is realized by any one or a combination of all kinds of devices capable of outputting the processing results processed by the terminal control unit 230. The output unit includes, for example, a touch panel, a touch display, a speaker (audio output), a lens (for example, 3D (Three Dimensions) output or hologram output), a printer, etc.

[0023] The display unit 250 is realized by any one of all kinds of devices capable of displaying according to the display data written in the frame buffer, or a combination thereof. The display unit 250 includes, for example, a touch panel, a touch display, a monitor (for example, a liquid crystal display or an OELD (Organic Electroluminescence Display)), a head mounted display (HDM), a projection mapping, a hologram, a device capable of displaying images, text information, etc. in the air (which may or may not be a vacuum). Note that these display units 250 may or may not be capable of displaying display data in 3D.

[0024] When the input / output unit 240 has a touch panel, the input / output unit 240 and the display unit 250 may be disposed facing each other and have approximately the same size and shape.

[0025] The terminal control unit 230 has a circuit physically structured to execute the functions realized by the code or instructions included in the program, and is realized by, for example, a data processing device built into the hardware. Therefore, the terminal control unit 230 may or may not be expressed as a control circuit.

[0026] The terminal control unit 230 includes, for example, a central processing unit (CPU), a microprocessor, a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc.

[0027] The terminal storage unit 220 has a function of storing various programs and various data required for the operation of the terminal 200. The terminal storage unit 220 includes various storage media such as a hard disk drive (HDD), a solid state drive (SSD), a flash memory, a random access memory (RAM), and a read only memory (ROM). The terminal storage unit 220 may or may not be expressed as a memory.

[0028] Terminal 200 stores a program in terminal storage unit 220, and executes this program, causing terminal control unit 230 to execute processing as each unit included in terminal control unit 230. That is, the program stored in terminal storage unit 220 causes terminal 200 to realize each function executed by terminal control unit 230. In other words, a processor executes a program stored in memory in terminal 200, and the hardware and software of terminal 200 work together to realize the function of each unit included in terminal 200. Note that this program may or may not be expressed as a program module.

[0029] The microphone 260 is used to input audio data, the speaker 270 is used to output audio data, and the camera 280 is used to obtain moving image data and / or still image data.

[0030] Next, the configuration of the server 100 will be described with reference to Fig. 3. The server 100 includes a server control unit 130, a server storage unit 120, and a server communication unit 110. The hardware components of the server 100 are connected to each other via a bus, for example.

[0031] The server control unit 130 has a physically structured circuit for executing functions realized by codes or instructions included in a program, and is realized by, for example, a data processing device built into hardware. The server control unit 130 is typically a central processing unit (CPU), and may or may not be a microprocessor, a processor core, a multiprocessor, an ASIC, an FPGA, or the like. In the present disclosure, the server control unit 130 is not limited to these.

[0032] The server storage unit 120 has a function of storing various programs and various data required for the operation of the server 100. The server storage unit 120 is realized by various storage media such as a HDD, an SSD, and a flash memory. However, in this disclosure, the server storage unit 120 is not limited to these. Furthermore, the server storage unit 120 may or may not be expressed as a memory.

[0033] The server communication unit 110 transmits and receives various data via the network 300. The communication may be performed either wired or wirelessly, and any communication protocol may be used as long as the communication between the two units can be performed. The server communication unit 110 has a function of performing communication with the terminal 200 via the network 300. The server communication unit 110 includes a server transmission unit 111 and a server reception unit 112. The server transmission unit 111 transmits various data to the terminal 200 according to an instruction from the server control unit 130. The server communication unit 110 also receives various data transmitted from the terminal 200 and transmits it to the server control unit 130. The server communication unit 110 may also be expressed as a server communication I / F (interface). When the server communication unit 110 is configured with a physically structured circuit, it may also be expressed as a server communication circuit.

[0034] The server 100 may include an input / output unit and a display as hardware configurations. The input / output unit is realized by a device that inputs various operations to the server 100. The input / output unit is realized by any of all types of devices that can receive input from a user and transmit information related to the input to the server control unit 130, or a combination thereof. The display is typically realized by a monitor (e.g., a liquid crystal display, an OELD, etc.). In this case, for example, the hardware of the server 100 may or may not be configured to have a detachable display.

[0035] Server 100 stores a program in server storage unit 120, and executes this program, causing server control unit 130 to execute processing as each unit included in server control unit 130. That is, the program stored in server storage unit 120 causes server 100 to realize each function executed by server control unit 130. In other words, a processor executes the program stored in memory in server 100, and the hardware and software of server 100 work together to realize the function of each unit included in server 100. Note that this program may or may not be expressed as a program module.

[0036] The terminal control unit 230 of the terminal 200 and / or the server control unit 130 of the server 100 may or may not realize each process by a CPU having a control circuit, but may also realize each process by a logic circuit (hardware) formed in an integrated circuit (IC (Integrated Circuit) chip, LSI (Large Scale Integration)) or a dedicated circuit. These circuits may be realized by one or more integrated circuits, and the multiple processes shown in the embodiments of the present disclosure may or may not be realized by one integrated circuit. LSIs may also be called VLSIs, super LSIs, ultra LSIs, etc., depending on the degree of integration.

[0037] In addition, the programs (e.g., software programs, computer programs, or program modules) of the embodiments of the present disclosure may or may not be provided in a state stored in a computer-readable storage medium. The storage medium can store the programs in a "non-transitory tangible medium." In addition, the programs may or may not be for realizing part of the functions of the embodiments of the present disclosure. Furthermore, the programs may or may not be what can realize the functions of the embodiments of the present disclosure in combination with a program already recorded in the storage medium, that is, a so-called difference file (difference program).

[0038] The storage medium may include one or more semiconductor-based or other integrated circuits (ICs) (e.g., field programmable gate arrays (FPGAs) or application specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical drives, floppy diskettes, floppy disk drives (FDDs), magnetic tapes, solid state drives (SSDs), RAM drives, secure digital cards, or drives, any other suitable storage media, or any suitable combination of two or more of these. The storage medium may be volatile, non-volatile, or a combination of volatile and non-volatile, where appropriate. It should be noted that the storage medium is not limited to these examples and may be any device or medium capable of storing a program. The storage medium may or may not be referred to as a memory.

[0039] Furthermore, the program of the present disclosure may or may not be provided to the server 100 and / or the terminal 200 via any transmission medium capable of transmitting the program (such as a communication network or broadcast waves). When the program is provided via a transmission medium, the server 100 and / or the terminal 200 can realize the functions of the multiple functional units shown in each embodiment by executing a program downloaded via the Internet or the like.

[0040] Moreover, the embodiment according to the present disclosure may be realized in the form of a data signal embedded in a carrier wave in which a program is embodied by electronic transmission. Moreover, at least a part of the processing in the server 100 and / or the terminal 200 may or may not be realized by cloud computing configured by one or more computers. Moreover, at least a part of the processing in the terminal 200 may or may not be configured to be performed by the server 100. In this case, at least a part of the processing of each functional unit of the terminal control unit 230 of the terminal 200 may or may not be configured to be performed by the server 100. Moreover, at least a part of the processing in the server 100 may or may not be configured to be performed by the terminal 200. In this case, at least a part of the processing of each functional unit of the server control unit 130 of the server 100 may or may not be configured to be performed by the terminal 200.

[0041] The program of the present disclosure may be implemented using, for example, a scripting language such as ActionScript or JavaScript (registered trademark), an object-oriented programming language such as Objective-C or Java (registered trademark), or a markup language such as HTML5.

[0042] In the communication system 400 according to this embodiment, as shown in Fig. 3, the server storage unit 120 of the server 100 includes a sound source data storage unit 121 and a lyrics data storage unit 122. The sound source data storage unit 121 stores sound source data of a song. The sound source data is data consisting of waveform data in which the accompaniment part and the melody part of the song are mixed. The melody part is the main melody part, and is usually a vocal part sung by a person. The sound source data can be in various audio file formats such as WAVE, AIFF, MP3, AAC, and FLAC.

[0043] The lyrics data storage unit 122 stores lyrics data for songs. The lyrics data is data for lyrics of the melody part of a song. In the configuration example described here, the lyrics data is synchronized lyrics data. The synchronized lyrics data includes a plurality of lyrics phrases. In the synchronized lyrics data, each lyrics phrase is associated with a display start time so that the phrases are displayed in synchronization with the playback of the sound source data of the song.

[0044] An LRC file, for example, can be used as this type of synchronized lyrics data. An example of an LRC file that is synchronized lyrics data is shown in Fig. 4. An LRC file is a text file that follows a specific format. In the LRC file format, each line is a lyric phrase to be displayed at one time. The number enclosed in "[" and "]" at the beginning of each line indicates the display start time of the lyric phrase on that line. The display start time is written in the format mm:ss.xx, where mm indicates minutes, ss indicates seconds, and xx indicates 1 / 100ths of a second.

[0045] In the communication system 400 according to this embodiment, the server 100 includes a sound source data acquisition unit 131, a lyrics data acquisition unit 132, a singing part identification unit 133, and a reference data generation unit 134 as functions realized by a server control unit 130. The sound source data acquisition unit 131 performs a process of acquiring sound source data stored in the sound source data storage unit 121. The lyrics data acquisition unit 132 performs a process of acquiring lyrics data stored in the lyrics data storage unit 122.

[0046] The singing part identification unit 133 identifies the singing part in the sound source data acquired by the sound source data acquisition unit 131 based on the lyrics data acquired by the lyrics data acquisition unit 132. In the configuration example described here, the lyrics data acquired by the lyrics data acquisition unit 132 is synchronized lyrics data. Then, the singing part identification unit 133 identifies the singing part in the sound source data based on the display start time of each lyrics phrase included in the synchronized lyrics data.

[0047] The identification of the singing portion by the singing portion identification unit 133 will be described with a concrete example. For example, in the case of the synchronized lyrics data shown in Fig. 4, the period from the start of the song until 11.14 seconds is an introduction without lyrics. Then, from 11.14 seconds to 1 minute 10.68 seconds is the singing portion of the first verse. Also, from 1 minute 10.68 seconds to 1 minute 34.16 seconds is an interlude without lyrics. Then, from 1 minute 34.16 seconds to 2 minutes 16.71 seconds is the singing portion of the second verse. And from 2 minutes 16.71 seconds onwards is the postlude. The introduction, interlude and postlude are non-sung portions.

[0048] The reference data generating unit 134 generates reference data for the melody part of the sound source data. The reference data is data related to the sound elements of the melody part. The sound elements that are the subject of the reference data include pitch, pronunciation timing, and sound duration.

[0049] The reference data generating unit 134 first extracts human singing voice data from the sound source data. The extraction of the singing voice data from the sound source data can be performed using a known method, such as, for example, taking advantage of the fact that if the sound source data is a stereo sound source, human singing voice (vocal part) is often located in the center, extracting frequency components of human singing voice, or removing waveform components of timbres of musical instruments that are not human voices. Then, the reference data generating unit 134 detects the pitch of the extracted singing voice data using, for example, a known pitch detection algorithm, and outputs it as reference data.

[0050] Particularly, in the server 100 according to this embodiment, the reference data generating unit 134 generates reference data for a melody part by using the sound source data of the singing part identified by the singing part identifying unit 133. In other words, the reference data generating unit 134 generates reference data for a melody part by excluding non-singing parts from the sound source data of the song.

[0051] In the example of the synchronized lyrics data shown in Fig. 4, the reference data generating unit 134 generates reference data for the melody part using the singing part of the first verse from 11.14 seconds to 1 minute 10.68 seconds and the singing part of the second verse from 1 minute 34.16 seconds to 2 minutes 16.71 seconds in the sound source data. Therefore, the reference data generating unit 134 generates reference data for the melody part using the parts from 11.14 seconds to 1 minute 10.68 seconds and from 1 minute 34.16 seconds to 2 minutes 16.71 seconds in the sound source data of the song. This can be said to generate reference data for the melody part after excluding the non-singing parts from the start of the song up to 11.14 seconds, from 1 minute 10.68 seconds to 1 minute 34.16 seconds, and from 2 minutes 16.71 seconds onwards.

[0052] 3, the server storage unit 120 further includes a reference data storage unit 123. The reference data storage unit 123 stores reference data for each piece of music generated by a reference data generation unit .

[0053] 3, the server 100 further includes an accompaniment part acquisition unit 135 as a function realized by the server control unit 130. The accompaniment part acquisition unit 135 performs a so-called vocal cancellation process on the sound source data acquired by the sound source data acquisition unit 131, and extracts and acquires the accompaniment part of the music piece. A known method can be used for the vocal cancellation process. The data of the accompaniment part acquired by the accompaniment part acquisition unit 135 can be used as so-called karaoke sound source data.

[0054] The server transmitting unit 111 of the server 100 transmits the sound source data of the music stored in the sound source data storage unit 121 to the terminal 200. This realizes a music sound source data distribution service. The server transmitting unit 111 may also transmit the lyric data of the music stored in the lyric data storage unit 122 to the terminal 200. Furthermore, the server transmitting unit 111 may transmit the data of the accompaniment part acquired by the accompaniment part acquisition unit 135 to the terminal 200.

[0055] The terminal receiving unit 212 of the terminal 200 receives the sound source data, lyrics data, and accompaniment part data of the music piece transmitted from the server 100. In the configuration example shown in Fig. 2, the terminal 200 includes a playback processing unit 231, a display processing unit 232, and a singer voice acquisition unit 233 as functions realized by the terminal control unit 230. The playback processing unit 231 plays the sound source data of the music piece received by the terminal receiving unit 212, and outputs it from the speaker 270 or the like.

[0056] Furthermore, the playback processing unit 231 plays back the data of the accompaniment part of the music piece received by the terminal receiving unit 212, and outputs it from the speaker 270 or the like. At this time, the display processing unit 232 displays the lyric data of the music piece received by the terminal receiving unit 212 on the display unit 250. As described above, in the configuration example described here, the lyric data is synchronous lyric data. Therefore, the display processing unit 232 can display the lyric data in synchronization with the playback of the data of the accompaniment part of the music piece by the playback processing unit 231. The singer voice acquisition unit 233 acquires the voice input to the microphone 260 as the singer's singing voice data while the playback processing unit 231 is playing back the data of the accompaniment part of the music piece.

[0057] The terminal transmitting unit 211 transmits the singing voice data acquired by the singer voice acquiring unit 233 to the server 100. The server receiving unit 112 of the server 100 receives the singing voice data of the singer transmitted from the terminal 200.

[0058] In the configuration example shown in FIG. 3, the server 100 further includes a singing voice acquisition unit 136 and an evaluation unit 137 as functions realized by the server control unit 130. The singing voice acquisition unit acquires singing voice data received by the server receiving unit 112. The evaluation unit 137 of the server 100 compares the singing voice data of the singer acquired by the singing voice acquisition unit 136 with the reference data stored in the reference data storage unit 123 to evaluate it. For example, the evaluation unit 137 detects the pitch of the singing voice data of the singer using the known pitch detection algorithm described above. The evaluation unit 137 then compares the pitch with the reference data, and the higher the degree of agreement between these pitches, the better the evaluation. The evaluation result by the evaluation unit 137 is transmitted to the terminal 200 via the server transmitting unit 111 and the terminal receiving unit 212, and is displayed on the display unit 250, for example.

[0059] Also, the reference data may be transmitted from the server 100 to the terminal 200. In this case, for example, while the terminal 200 is reproducing data of an accompaniment part which is a karaoke sound source, a so-called guide melody can be displayed on the display unit 250 using the reference data. Also, the evaluation unit 137 may be provided in the terminal 200 instead of the server 100.

[0060] Next, an example of the operation of the communication system 400 according to this embodiment will be described with reference to the flow diagram of Fig. 5. First, in step S10, the sound source data acquisition unit 131 of the server 100 acquires the sound source data stored in the sound source data storage unit 121. In the following step S11, the lyrics data acquisition unit 132 acquires the lyrics data stored in the lyrics data storage unit 122. After step S11, the server control unit 130 then performs the process of step S12.

[0061] In step S12, the singing part identification unit 133 identifies the singing part in the sound source data acquired in step S10 based on the lyric data acquired in step S11. In the following step S13, the reference data generation unit 134 generates reference data for the melody part using the sound source data of the singing part identified in step S12. After step S13, the server 100 next performs the process of step S14.

[0062] In step S14, the reference data storage unit 123 of the server 100 stores the reference data generated in step S13. Also, the accompaniment part acquisition unit 135 extracts and acquires the accompaniment part from the sound source data acquired in step S10. Then, the server transmission unit 111 transmits the sound source data, the lyrics data, and the data of the accompaniment part to the terminal 200. After step S14, the terminal 200 next performs the process of step S15.

[0063] In step S15, the terminal receiving unit 212 of the terminal 200 receives the sound source data, lyrics data, and accompaniment part data of the music transmitted from the server 100 in step S14. Then, the playback processing unit 231 of the terminal 200 plays the accompaniment part data of the music received by the terminal receiving unit 212, and outputs it from the speaker 270 or the like. In addition, the display processing unit 232 of the terminal 200 displays the lyrics data of the music received by the terminal receiving unit 212 on the display unit 250. After step S15, the terminal 200 next performs the process of step S16.

[0064] In step S16, the singing voice acquisition unit 136 of the terminal 200 acquires the singing voice data of the singer input to the microphone 260. In the following step S17, the terminal transmission unit 211 transmits the singing voice data acquired in step S16 to the server 100. The server reception unit 112 of the server 100 receives the singing voice data of the singer transmitted from the terminal 200. After step S17, the server 100 next performs the process of step S18.

[0065] In step S18, the singing voice acquisition unit 136 of the server 100 acquires the singing voice data received by the server receiving unit 112 in step S16. In the following step S19, the evaluation unit 137 of the server 100 evaluates the singing voice data of the singer acquired in step S18 by comparing it with the reference data stored in the reference data storage unit 123 in step S14. In further following step S20, the evaluation result by the evaluation unit 137 is transmitted to the terminal 200 via the server transmitting unit 111 and the terminal receiving unit 212, and is displayed on the display unit 250, for example.

[0066] In the above flow, step S10 corresponds to a voice source data acquisition step, step S11 corresponds to a lyric data acquisition step, step S12 corresponds to a vocal part identification step, step S13 corresponds to a reference data generation step, step S18 corresponds to a vocal voice acquisition step, and step S19 corresponds to an evaluation step.

[0067] According to the server 100 of the communication system 400 configured as above, the lyric data is used to identify the vocal part in the sound source data, and the audio data of the identified vocal part is used to generate reference data for the melody part. This makes it possible to prevent the reference data from reflecting the sound of a human voice other than the vocal part, or the sound of an instrument that is erroneously extracted due to problems with extraction accuracy, etc., and to generate reference data that is more appropriate as a standard for scoring singing.

[0068] Next, a modified example of the server 100 included in the communication system 400 according to this embodiment will be described. In this modified example, the singing part identification unit 133 of the server 100 includes a singing voice extraction unit 141, a text conversion unit 142, and a matching unit 143, as shown in Fig. 6. The singing voice extraction unit 141 extracts human singing voice data from the sound source data acquired by the sound source data acquisition unit 131. The human singing voice data can be extracted from the sound source data by using the known method described above.

[0069] The text conversion unit 142 converts the singing voice data extracted by the singing voice extraction unit 141 into text by a known voice recognition process to generate singing text data. The collation unit 143 collates the singing text data generated by the text conversion unit 142 with the lyrics data acquired by the lyrics data acquisition unit 132 to identify the singing portion in the sound source data. That is, the collation unit 143 identifies the portion where the singing text data and the lyrics data match as the singing portion. Conversely, the collation unit 143 identifies the portion where the singing text data and the lyrics data do not match as the non-singing portion.

[0070] Next, an example of the operation of the server 100 according to this modification will be described with reference to the flow diagram of Fig. 7. The flow diagram of Fig. 7 shows a sub-process of step S12 (singing part identification step) in the flow diagram of Fig. 5. First, in step S30, the singing voice extraction unit 141 extracts human singing voice data from the sound source data acquired in step S10 of Fig. 5, and acquires the extracted singing voice data. In the following step S31, the text conversion unit 142 converts the singing voice data acquired in step S30 into text by voice recognition processing to generate singing text data, and acquires the generated singing text data. In the further following step S32, the collation unit 143 collates the singing text data acquired in step S31 with the lyric data acquired in step S11 of Fig. 5 to identify the singing part in the sound source data.

[0071] In the above flow, step S30 corresponds to a singing voice extraction step, step S31 corresponds to a text conversion step, and step S32 corresponds to a comparison step.

[0072] In this modification, the lyric data does not need to include the display start time for each lyric phrase. Therefore, according to this modification, the lyric data that is not synchronized lyric data is used to identify the sung / non-sung parts of the sound source data, and the sound of the non-sung parts can be prevented from being reflected in the reference data, and more appropriate reference data can be generated as a standard for scoring singing.

[0073] Embodiment 2 A second embodiment of the present disclosure will be described with reference to Fig. 8 and Fig. 9. Fig. 8 is a block diagram showing a configuration of a server included in the communication system. Fig. 9 is a flowchart showing an example of processing in the communication system.

[0074] The program, information processing method, and information processing device according to this embodiment 2 will be described below, focusing on the differences from embodiment 1. Configurations whose description is omitted are basically the same as those in embodiment 1. In the following description, configurations similar to or corresponding to those in embodiment 1 will be described, in principle, with the same reference numerals as those used in the description of embodiment 1.

[0075] In the communication system 400 according to this embodiment, as shown in FIG. 8, the server 100 further includes a singing voice separation extraction unit 150 as a function realized by the server control unit 130. The singing voice separation extraction unit 150 separates and extracts each of the singing voice data of a plurality of people from the sound source data acquired by the sound source data acquisition unit 131. For example, the singing voice separation extraction unit 150 first extracts the singing voice data of a person from the sound source data. The extraction of the singing voice data of a person from the sound source data can be performed by using the above-mentioned known method. Then, when the extracted singing voice data includes singing voice data of a plurality of people, the singing voice separation extraction unit 150 separates the singing voice data for each person. The separation at this time can be performed by a known method such as waveform analysis of the singing voice data. The singing voice separation extraction unit 150 may directly separate each of the singing voice data of a plurality of people from the sound source data.

[0076] In this embodiment, the singing part identification unit 133 identifies the singing part in the sound source data based on the singing voice data of a specific person among the singing voice data of multiple people extracted by the singing voice separation extraction unit 150. The singing voice data of the specific person here is, for example, the singing voice data of the main vocal. The singing part identification unit 133 determines the singing voice data of the specific person using, for example, the localization, volume, and proportion of the singing voice data of the multiple people to the playback time of the song. In addition, when the gender of the main vocalist is known from the artist name of the song, the singing voice data of the specific person may be determined based on whether each of the singing voice data of the multiple people is a male voice or a female voice.

[0077] The singing part identification unit 133 identifies the singing part of the singing voice data of the specific person thus determined, and sets this as the singing part of the sound source data of the song. Then, the reference data generation unit 134 generates reference data of the melody part using the sound source data of the singing part identified by the singing part identification unit 133.

[0078] The singing voice data of the specific person is not limited to one person, and may be two or more singing voice data. In this case, the singing part identification unit 133 identifies each of the singing parts of the multiple people in the sound source data based on each of the singing voice data of the specific multiple people. Then, the reference data generation unit 134 generates reference data for the melody part using the sound source data of each of the singing parts of the multiple people identified by the singing part identification unit 133. In this way, in the case of a song in which multiple singers take turns to sing the melody part, reference data can be generated separately for the parts of the multiple singers who are in charge of the melody part. Therefore, for example, a karaoke accompaniment in which only an arbitrary one of the singers who are in charge of the melody part is canceled can be played, and only the part sung by that person can be scored using the reference data.

[0079] Next, an example of the operation of the communication system 400 according to this embodiment will be described with reference to the flow diagram of Fig. 9. Steps S10 and S11 in the flow diagram of this figure are similar to steps S10 and S11 in the flow diagram of Fig. 5, so their description will be omitted here. After step S11, the server control unit 130 next performs the process of step S40.

[0080] In step S40, the singing voice separation extraction unit 150 separates and extracts each of the singing voice data of the multiple people from the sound source data acquired in step S10. This step S40 corresponds to a singing voice separation extraction step. After step S40, the server control unit 130 next performs the process of step S12. In step S12, the singing part identification unit 133 identifies the singing part in the sound source data based on the singing voice data of a specific person among the singing voice data of the multiple people extracted in step S40. Steps S13 to S20 after step S12 are similar to steps S13 to S20 in the flow diagram of FIG. 5, so their description will be omitted here.

[0081] In this embodiment, the server 100 may not include the singing part identification unit 133. In this case, the reference data generation unit 134 generates reference data for the melody part based on the singing voice data of a specific person among the singing voice data of the multiple people extracted by the singing voice separation extraction unit 150. In this case, the process of step S12 in the flow diagram of FIG. 9 is not performed, and after step S40, the server control unit 130 next performs the process of step S13. Then, in step S13, the reference data generation unit 134 generates reference data for the melody part based on the singing voice data of a specific person among the singing voice data of the multiple people extracted in step S40. Even with this configuration, by generating reference data based on the singing voice data of a specific person among the singing voice data of the multiple people, it is possible to suppress the reflection of the production part in which a human voice is used other than the singing part in the reference data, and it is possible to generate reference data that is more appropriate as a standard for scoring singing.

[0082] In the configuration examples of each embodiment described above, the sound source data acquisition unit 131, the lyrics data acquisition unit 132, the singing part identification unit 133 (including the singing voice extraction unit 141, the text conversion unit 142, and the matching unit 143), the reference data generation unit 134, the accompaniment part acquisition unit 135, the singing voice acquisition unit 136, the evaluation unit 137, and the singing voice separation and extraction unit 150 are provided in a single server 100. However, each of these units may not be provided in a single server 100, but may be distributed among multiple server devices. Also, the functions of each unit of the server 100 may be realized by a collaboration between multiple devices.

[0083] Although the invention according to the present disclosure has been described based on various drawings and examples, it should be noted that a person skilled in the art can easily make various modifications and corrections based on the present disclosure. Therefore, it should be noted that these modifications and corrections are included in the scope of the invention according to the present disclosure. For example, the functions included in each part, each means, each step, etc. can be rearranged so as not to cause logical contradictions, and multiple means, steps, etc. can be combined into one or divided. In addition, the configurations shown in the above-described embodiments may be appropriately combined.

[0084] The program, information processing method, and information processing device according to the present disclosure can be utilized as a program, information processing method, and information processing device for generating reference data from audio source data of a song. [Explanation of symbols]

[0085] 100 Servers 110 Server Communication Department 111 Server transmission unit 112 Server Reception Unit 120 Server memory unit 121 Sound source data storage unit 122 Lyrics data storage section 123 Reference Data Storage Unit 130 Server control unit 131 Sound source data acquisition unit 132 Lyrics data acquisition section 133 Singing part specific part 134 Reference Data Generation Unit 135 Accompaniment part acquisition section 136 Singing Voice Acquisition Unit 137 Evaluation Department 141 Singing Voice Extraction Unit 142 Text conversion section 143 Matching Unit 150 Singing voice separation extraction unit 200 terminals 210 Terminal communication unit 211 Terminal transmitter 212 Terminal receiving unit 220 Terminal memory unit 230 Terminal Control Unit 231 Reproduction Processing Unit 232 Display processing unit 233 Singer Voice Acquisition Unit 240 Input / output section 250 Display section 260 Mike 270 Speaker 280 Camera 300 Network 400 Communication Systems

Claims

1. A program to be executed by a computer of an information processing device, A sound source data acquisition step of acquiring sound source data including a melody part of a piece of music; a singing voice separation and extraction step of separating and extracting a first singing voice data of a first singer and a second singing voice data of a second singer different from the first singer from the sound source data; a singing portion identification step of identifying the singing portions of the first singer and the second singer in the sound source data, respectively, using the ratio of the first singing voice data and the second singing voice data to the playback time of the song, and at least one of whether the voice is a male voice or a female voice; A reference data generating step is executed by the computer of the information processing device to generate first reference data for the part of the melody part sung by the first singer based on the sound source data for the singing part of the first singer identified in the singing part identifying step, and to generate second reference data for the part of the melody part sung by the second singer based on the sound source data for the singing part of the second singer identified in the singing part identifying step.

2. The program according to claim 1, A singing voice acquiring step of acquiring singing voice data of a singer; The computer of the information processing device further executes an evaluation step of comparing the singing voice data acquired in the singing voice acquisition step with one of the first reference data and the second reference data and evaluating the same.

3. The program according to claim 1, The computer of the information processing device further executes a step of generating first accompaniment data by canceling from the sound source data the portion of the melody part sung by the first singer, and generating second accompaniment data by canceling from the sound source data the portion of the melody part sung by the second singer.

4. The program according to claim 3, A singing voice acquiring step of acquiring singing voice data of a singer; The computer of the information processing device further executes an evaluation step of comparing and evaluating the singing voice data acquired in the singing voice acquisition step by comparing it with the first reference data when the first accompaniment data is played back, and of comparing and evaluating the singing voice data acquired in the singing voice acquisition step by comparing it with the second reference data when the second accompaniment data is played back.

5. An information processing method executed by a computer of an information processing device, A sound source data acquisition step of acquiring sound source data including a melody part of a piece of music; a singing voice separation and extraction step of separating and extracting a first singing voice data of a first singer and a second singing voice data of a second singer different from the first singer from the sound source data; a singing portion identification step of identifying the singing portions of the first singer and the second singer in the sound source data, respectively, using the ratio of the first singing voice data and the second singing voice data to the playback time of the song, and at least one of whether the voice is a male voice or a female voice; and a reference data generating step of generating first reference data for a portion of the melody part sung by the first singer based on the sound source data for the singing part of the first singer identified in the singing part identifying step, and generating second reference data for the portion of the melody part sung by the second singer based on the sound source data for the singing part of the second singer identified in the singing part identifying step.

6. An information processing device, a sound source data acquisition unit for acquiring sound source data including a melody part of a piece of music; a singing voice separation and extraction unit for separating and extracting a first singing voice data of a first singer and a second singing voice data of a second singer different from the first singer from the sound source data; a singing part identification unit that identifies the singing parts of the first singer and the singing parts of the second singer in the sound source data by using at least one of a ratio of the first singing voice data and the second singing voice data to a playback time of the song and whether the voice is a male voice or a female voice; and a reference data generating unit that generates first reference data for a portion of the melody part sung by the first singer based on the sound source data for the singing part of the first singer identified by the singing part identifying unit, and generates second reference data for a portion of the melody part sung by the second singer based on the sound source data for the singing part of the second singer identified by the singing part identifying unit.

Images