Server terminal, playback terminal, program, data processing method, and instrument practice system for practicing musical instruments at home

Abstract

To provide a musical instrument practice system which detects mistakes made during playing and reproduces points pointed out during lessons when practicing a musical instrument. [Solution] The musical instrument practice system comprises a server terminal having a first receiving unit that receives video transmitted from a first transmitting unit of the video acquisition terminal, an editing unit that automatically edits the video received by the first receiving unit according to the sound contained in the video, and a second transmitting unit that transmits the video edited by the editing unit to a playback terminal, and a playback terminal having a second receiving unit that receives the edited video, a second memory unit that stores the video received from the second receiving unit, a recording unit that records the sound being practiced by the practitioner, and a playback unit that plays back the sound of the instructor that corresponds to the sound being practiced by the practitioner based on the sound recorded by the recording unit and the edited video.

Description

[Technical field]

[0001] The present invention relates to a server terminal, a playback terminal, a program, a data processing method, and a musical instrument practice system for practicing at home, which detects incorrect playing parts when a practitioner is practicing at home, and provides re-instruction to the practitioner by utilizing edited video containing an instructor's comments. [Background technology]

[0002] In recent years, the number of people who practice musical instruments has been increasing, and they are learning how to play musical instruments by attending music schools or music classes. Generally, instructors at music schools or music classes set practice assignments before each lesson, and the students practice by playing songs at home in response to the practice assignments. After that, the students show the results of their practice to the instructor in the next lesson. At this time, the instructor discovers problems with the students' performance and provides appropriate guidance accordingly. However, when students try to review their performance at home after attending a lesson, they are influenced by various factors (e.g., they may not be able to properly understand the instructor's comments during the lesson, they may have low motivation to practice at home, or they may forget what they were taught), and they are unable to practice efficiently.

[0003] Therefore, in order to solve the above problems, it is essential to develop a home practice and support system for learners. Patent Document 1 proposes a pronunciation practice system that uses a human characteristic of spontaneous behavior base and automatically arranges audio data at a timing that matches the tempo of background music. The pronunciation practice system disclosed in Patent Document 1 provides a pronunciation practice function based on background music, signal sounds, model pronunciations, and images corresponding to the model pronunciations. Learners practice pronunciation in an environment where background music is always playing. In one embodiment, the pronunciation practice system stores multiple model pronunciations and videos and images corresponding to the model pronunciations as learning contents. The learner inputs instructions for pronunciation practice into an input device, and the control unit reads out and arranges each of the stored audio data (background music, signal sounds, and model pronunciations). The control unit also plays a signal sound (referred to as an audio signal for notifying the learner) by overlapping each of the audio data to be heard by the learner with the beat of the background music. The control unit of the pronunciation practice system repeatedly plays the model pronunciation and the signal, while also displaying a delivery image corresponding to the model pronunciation on the display. At this time, the learner may effectively predict the timing of overlapping by the cue sound while listening to the model pronunciation, while experiencing the rhythm of the background music and being synchronized with its tempo.

[0004] Patent Document 2 discloses a system that supports a learner when practicing a piece of music. In this practice support system, music score data is stored in the system and the music score is displayed via a display terminal. In addition, this practice support system can determine the performance position corresponding to the music score based on the performance data of the learner, and play music from any specified position. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2021-67844 [Patent Document 2] JP 2013-200455 A Summary of the Invention [Problem to be solved by the invention]

[0006] According to the above Patent Documents 1 and 2, while a student is playing an instrument, the student's own performance data or standard performance data (i.e., model performance data used as a model for practice, or performance data used by an instructor for instruction) is played back and presented, but there is no description of the causes of errors that may occur while the student is playing, or how to correct errors after they are discovered. In addition, the support systems described in the prior art documents have the student actively practice by himself in response to the model, and there is a problem in that the student cannot receive any guidance from the instructor.

[0007] The present disclosure has been made in consideration of these problems, and aims to provide an instrument practice system that allows a practitioner to repeatedly practice at home using video footage of lessons. [Means for solving the problem]

[0008] One embodiment of the present invention is a musical instrument practice system including a video acquisition terminal that acquires practice video, a server terminal that edits the practice video, and a playback terminal that plays back the edited practice video received from the server terminal, wherein the video acquisition terminal includes a first image capture unit that captures an image of a practitioner playing and its audio data, and an image of an instructor instructing and its audio data, a first storage unit that stores the images captured by the first image capture unit and the acquired audio data, and a first transmission unit that transmits the video including the images and audio data stored in the first storage unit to the server terminal. the server terminal comprises a first receiving unit that receives the video transmitted from the first transmitting unit, an editing unit that automatically edits the video received by the first receiving unit according to the sound contained in the video, and a second transmitting unit that transmits the video edited by the editing unit to a playback terminal, and the playback terminal comprises a second receiving unit that receives the edited video, a second storage unit that stores the video received from the second receiving unit, a recording unit that records the sound being practiced by the practitioner, and a playback unit that plays back the sound of the instructor that corresponds to the sound being practiced by the practitioner, based on the sound recorded by the recording unit and the edited video.

[0009] In the above-mentioned musical instrument practice system, the editing unit may be characterized in that it extracts the sounds of the practitioner and the sounds of the instructor's performance and speech from the video received by the first receiving unit, each for multiple pieces of performance data and speech data, and automatically clusters the performance data based on the similarity of the sounds made by the practitioner during performance. In the above-mentioned musical instrument practice system, the editing unit may further be characterized by having a second memory unit that assigns corresponding utterances to clusters obtained by clustering the performance data based on the time period of the video, and stores them as a performance practice set. In the above musical instrument practice system, the playback terminal may include a third memory unit that stores sample sounds uploaded by the practitioner, and may be characterized in that the sounds of the instructor's speech are replaced according to the performance practice set based on the sample sounds stored in the third memory unit. In the above-mentioned musical instrument practice system, the playback terminal may include a setting unit that sets a total practice time based on a performance practice set, and the setting unit may further be characterized in that the practice time is set individually according to the performance proficiency of the practitioner. In the above-mentioned musical instrument practice system, the playback terminal may be characterized by having a second imaging unit that captures an image and sound of the practitioner playing the instrument, and a detection unit that detects whether the practitioner is playing correctly based on the actions of the practitioner playing the instrument and the sound of the action based on the image captured by the second imaging unit. In the above-mentioned musical instrument practice system, the playback terminal may be characterized in that when the detection unit detects an incorrectly played portion, it estimates the similarity between the image captured by the second imaging unit and the performance practice set, and plays back the performance practice set if the similarity is equal to or greater than a predetermined threshold. In the above musical instrument practice system, a feature may be that before a learner practices a performance, the learner refers to a performance practice set, and specifies and plays back the performance practice set.

[0010] The server terminal includes a first receiving unit that receives video of a lesson, an editing unit that automatically edits the video received by the first receiving unit according to the sound contained in the video, and a second transmitting unit that transmits the video edited by the editing unit to a playback terminal.

[0011] This data processing method includes executing a first receiving step of receiving lesson video, an editing step of automatically editing the video received from the first receiving step in accordance with the sound contained in the video, and a second transmitting step of transmitting the video edited by the editing step to a playback terminal.

[0012] This is a program for realizing a first receiving function of receiving video of a lesson, an editing function of automatically editing the video received by the first receiving function according to the sound contained in the video, and a second transmitting function of transmitting the video edited by the editing function to a playback terminal.

[0013] This is a playback terminal that includes a second receiving unit that receives edited lesson video, a second memory unit that stores the video received from the second receiving unit, a recording unit that records the sound that the learner is practicing, and a playback unit that plays back the instructor's sound that corresponds to the sound that the learner is practicing based on the sound recorded by the recording unit and the edited video.

[0014] This is a data processing method that includes executing a second receiving step of receiving edited lesson video, a second storage step of storing the video received from the second receiving step, a recording step of recording the sound being practiced by the practitioner, and a playback step of playing back the instructor's sound that corresponds to the sound being practiced by the practitioner based on the sound recorded by the recording step and the edited video.

[0015] This is a program for realizing a second receiving function that receives edited lesson video, a second storage function that stores the video received from the second receiving function, a recording function that records the sound that the learner is practicing, and a playback function that plays back the instructor's sound that corresponds to the sound that the learner is practicing based on the sound recorded by the recording function and the edited video. Effect of the Invention

[0016] According to the musical instrument practice system of the present invention, the situation in which a learner is being taught by an instructor is filmed and recorded. When the learner practices an instrument by himself, the musical instrument practice system records the sound, and when he makes a mistake similar to the mistake he made in the lesson, it can play back the contents of the instruction given by the instructor during the lesson, so that the learner can remember the contents of the instruction (including words of praise and scolding) given by the instructor even when he is practicing alone. [Brief description of the drawings]

[0017] [Figure 1] 1 is a schematic diagram showing a configuration of a practice assistance system according to an embodiment of the present disclosure. [Diagram 2] 2 is a block diagram showing the configuration and functions of the video acquisition terminal of FIG. 1. [Diagram 3] 2 is a block diagram showing the configuration and functions of the server terminal of FIG. 1. [Figure 4] A figure showing a portion of possible video data that can be used in the practice support system in an embodiment of the present disclosure. [Diagram 5] 4 is a flowchart showing an example of automatic editing of data in the server terminal of FIG. 3; [Figure 6] 2 is a flowchart showing an example of re-editing the video related to the schematic diagram of FIG. 1 and obtaining a performance practice set. [Figure 7] 2 is a block diagram showing the configuration and functions of the playback terminal of FIG. 1. [Figure 8] 8 is an image showing an example of image replacement in the setting section of FIG. 7. [Figure 9] 8 is a flowchart showing an example of setting a practice time in the playback terminal of FIG. 7. [Figure 10] 8 is a flowchart showing an example of a flow for extracting similar performance content and linking the instructor's comments during a lesson to a practice video and playing them back on the playback terminal of FIG. 7. [Figure 11]8 is a flowchart showing an example of presentation of the atmosphere of a practice piece on the playback terminal of FIG. 7. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, the same or equivalent components, processes, and signals shown in each drawing will be denoted by the same reference numerals, and duplicated explanations will be omitted as appropriate. Also, in each drawing, some components that are not important for explanation will be omitted.

[0019] In the following description, "photographing" may mean simultaneously acquiring image data showing the appearance and movements of a target and audio data of the target using a camera and microphone. "Video" photographed in this embodiment may include both image data and audio data. "Capture" means acquiring only image data showing the appearance and movements of a target via a camera. "Recording" means recording the audio of a target. "Sound source" may include speech data of a trainee, speech data of an instructor, performance data of a trainee (instrument sound), and performance data of an instructor (instrument sound). Meanwhile, a "musical instrument" in this embodiment is an instrument for playing music, and may include not only keyboard instruments but also wind instruments, string instruments, percussion instruments, etc.

[0020] In addition, in order to record the image data and audio data according to this embodiment, the camera and microphone may be installed separately. Specifically, the camera according to this embodiment is configured to acquire image data capturing an image of the practice, and the microphone is configured to acquire audio data capturing the audio of the practice and the audio during instruction. However, the functions of the camera and microphone are not limited to the above embodiment, and although the camera and microphone are described separately in this specification, the audio data may be collected by a microphone attached to the camera.

[0021] <Summary> As shown in FIG. 1, the musical instrument practice system according to this embodiment is composed of a video acquisition terminal 100, a server terminal 200, a playback terminal 300, and a network NW, which are connected to each other so that they can communicate with each other. In the musical instrument practice system, the video acquisition terminal 100 acquires a practice video (hereinafter referred to as video data) including audio and image data, and passes it to the server terminal 200, which then automatically edits the audio and image data. The server terminal 200 then passes the edited video data to the playback terminal 300. Based on the edited video data received, the playback terminal 300 can detect whether the practicer's playing matches the musical score when practicing the musical instrument at home, or whether the practicer follows the instructor's instruction in class. When the practicer performs an incorrect performance, the playback terminal 300 plays the corresponding edited video. An "incorrect performance" includes a performance that does not follow the notes or beat of the music, or a performance that does not follow the instructor's intention of instruction, such as the way of adding dynamics, the shape of the hand, the way of fingering, or the way of playing the keys with the fingers. In addition, the video data of the instructor can be replaced with a virtual character, and the image and voice of the virtual character can be used. In this case, it is expected that the trainee will become more interested in the practice and will be more motivated to practice. The musical instrument practice system may have a function of judging the movement of the practicer or the way of playing the musical instrument. After acquiring image data of the practicer, the image acquisition terminal 100 stores the image data in the second storage unit 220 of the server terminal 200. When the practicer re-practices the musical instrument at home, it becomes possible to judge based on the image data whether or not the practicer is performing the movement of playing the musical instrument properly in order to correct the incorrect playing method according to the instructor's instruction.

[0022] 1, the musical instrument practice system in this embodiment is composed of a video acquisition terminal 100, a server terminal 200, and a playback terminal 300, and each function may be realized through data sharing on a network NW. The configuration of the musical instrument practice system is not limited to the above, and the video acquired from the video acquisition terminal 100 may be uploaded directly to the server terminal 200 without going through the network NW (for example, the video data acquired by the video acquisition terminal 100 may be stored in a flash memory, and the flash memory may be connected to the server terminal 200 to be uploaded).

[0023] Furthermore, the video data acquired by the video acquisition terminal 100 may be divided into (1) performance data of the practicer in the class (sound of the practicer's instrument), (2) image data of the practicer in the class, (3) performance data of the instructor in the class (sound of the instructor's instrument), (4) image data of the instructor in the class, (5) speech data of the practicer in the class, and (6) speech data of the instructor in the class. In addition to acquiring the above six types of data, the video acquisition terminal 100 may also acquire voice data (including the instrument sounds of the attendees and the questions of the attendees) and image data of the accompanying practicers, and speech data of the parents who are present at the lesson, if there are multiple practicers.

[0024] In this embodiment, the server terminal 200 analyzes the audio and image data based on the video data acquired from the video acquisition terminal 100, dividing the data into three groups: (i) the instrumental sounds of the learner and the instructor, (ii) the questioning voice of the learner and the instruction voice of the instructor, and (iii) the movements of the learner and the movements of the instructor. The practitioner's instrument sound is the sound of the instrument when the practitioner is playing. The instructor's instrument sound is the sound of the instrument that the instructor is playing for the purpose of instruction during a lesson. The practitioner's questioning voice is the voice of the practitioner asking the instructor about a problem that he or she is having while playing. The instructor's instruction voice is the voice made by the instructor during lessons (including voices pointing out problems and voices giving praise). The practitioner's actions are the actions of the practitioner playing the instrument. The instructor's actions are the actions of playing the instrument to instruct the practitioner.

[0025] The learner's musical instrument sound and the learner's action, and the instructor's musical instrument sound and the instructor's action may be extracted as a set according to time. For example, after acquiring the video data, the server terminal 200 receives the video data including the learner's musical instrument sound and the learner's action (or the instructor's musical instrument sound and the instructor's action at the time of the instructor) as a set. In addition, the server terminal 200 transmits to the playback terminal 300 separately the audio signals (practitioner's question voice, instructor's instruction voice, practicer's instrument sound and instructor's instrument sound) recorded via the microphone of the video acquisition terminal 100 and the image information (practitioner's actions and instructor's actions) captured via the camera of the video acquisition terminal 100, and it is not necessary to combine them into one set in the video acquisition terminal 100.

[0026] The playback terminal 300 mainly functions to allow the learner to review the contents of the lesson at home. Each data compiled in the server terminal 200 is detection material for judging whether the learner is playing correctly when practicing at home, and also serves as training data for the learner who practices at home. In other words, if the learner is playing the instrument correctly at home (i.e., no guidance from the instructor is required), there is no need to play the video including the instructor's guidance. If the learner plays incorrectly and repeats the mistake made during the lesson, the playback terminal 300 can detect the part where the performance is incorrect, and plays the video showing the guidance given when the learner played incorrectly during practice with the instructor.

[0027] The playback terminal 300 according to the present embodiment may have the above-mentioned automatic editing function, or may further edit the image data and audio data played by the playback terminal 300. Specifically, the playback terminal 300 replaces each image data and audio data acquired from the server terminal 200 with a new image and tone of voice acquired from an external terminal or the Internet without changing the audio content. As an example, the original instructor image and instructor voice are replaced with an animated character image and an animated character voice without changing the audio content. In this case, the trainee can relieve stress during training, and when a trainee uses a voice and character that he or she finds interesting, the trainee's training efficiency can be expected to improve.

[0028] The instrument practice system provides a function for the learner to re-learn and review points that he or she could not understand during lessons. In addition, when practicing an instrument at home, the learner can efficiently find problems that frequently arise during practice based on the data obtained from lessons and receive re-guidance. The following describes in detail the structure of the instrument practice system according to this embodiment and the mechanism for implementing the home practice function.

[0029] <Configuration> <Video Acquisition Terminal 100> FIG. 2 is a block diagram showing an example of the configuration of the video acquisition terminal 100 that acquires the voice data of the trainee, the image data of the trainee, the voice data of the instructor, and the image data of the instructor. As shown in FIG. 2, the video acquisition terminal 100 has a RAM (Random Access Memory, not shown), a ROM (Read Only Memory, not shown), a first image capture unit 110, a first storage unit 120, and a first transmission unit 130. The video acquisition terminal 100 is a computer system that includes a processor and a memory, and the processor executes a program to realize a function of capturing an image of a performance and transmitting the image data to the server terminal 200. In order to acquire the voice data and image data of the trainee and the instructor, the first image capture unit 110 is further provided with a right-facing camera 111, a left-facing camera 112, a foot-facing camera 113, a downward-facing camera 114, an overall image capture camera 115, and a microphone 116. Each component is connected to each other so as to be able to communicate with each other via the Internet.

[0030] The first image capturing unit 110 has a function of capturing an image of a learner playing and its audio data, and an image of an instructor teaching and its audio data. The first image capturing unit 110 is only required to capture an image of a lesson, and is composed of at least one camera and microphone. In the present embodiment, the instrument is described as a piano, but the instrument is not limited to a piano and may be various other instruments such as a guitar, a ukulele, a violin, a saxophone, etc. When the instrument is a piano, the first image capturing unit 110 may include a right-facing camera 111, a left-facing camera 112, a foot-facing camera 113, and a downward-facing camera 114. The right-facing camera 111 of the piano keyboard is installed on the left side of the practicer (the left side of the piano keyboard) so as to capture the right side (the keyboard side) when the practicer faces the keyboard so that he can play the piano. The left-facing camera 112 of the piano keyboard is installed on the right side of the practicer (the right side of the piano keyboard) so as to capture the left side (the keyboard side) when the practicer faces the keyboard so that he can play the piano. The foot-facing camera 113 of the piano keyboard is installed below the practicer (the piano pedals) so as to capture the upward direction when the practicer faces the keyboard so that he can play the piano. The downward-facing camera 114 of the piano keyboard is installed above the practicer (the piano top) so as to capture the downward direction when the practicer faces the keyboard so that he can play the piano. By filming the way the learner plays, it is possible to grasp the mistakes that are made when playing the piano. In addition, it is equipped with a foot-facing camera 113 that films how the pedals are pressed, and a downward-facing camera 114 that films from above the keyboard, making it possible to comprehensively obtain image data of the learner using the piano.

[0031] Meanwhile, right-facing camera 111, left-facing camera 112, foot-facing camera 113, and downward-facing camera 114 can not only obtain image data of the learner, but also obtain image data of the instructor playing the piano to provide guidance when the learner plays the piano without following the musical score. In other words, image data of the person playing the musical instrument can be obtained.

[0032] The first photographing section 110 may also include an overall imaging camera 115 that captures an image of the entirety of the practicer and the instructor. That is, while the practicer is taking a lesson, image data of the practicer himself and image data of the instructor himself can all be captured from other angles. In this case, the overall imaging camera 115 may be able to supplement image data of angles of view that cannot be captured using the right-facing camera 111, the left-facing camera 112, the foot-facing camera 113, and the downward-facing camera 114.

[0033] The number and positions of the cameras provided in the first photographing unit 110 may be changed depending on the type of instrument, the practitioner, etc. Also, depending on the type and brand of the camera, the camera may have not only a photographing function but also a sound recording function. In this case, the microphone 116 does not need to be installed. Also, a camera for photographing the instructor may be provided in the first photographing unit 110. This camera may have a function of automatically tracking and photographing the instructor so that the instructor is captured within a predetermined range in the captured image.

[0034] Image data acquired from each camera mounted on first photographing unit 110 and audio data recorded by microphone 116 are stored in an information storage database of first storage unit 120. The audio data and image data may be transmitted to first storage unit 120 as a set, or may be transmitted to first storage unit 120 separately.

[0035] The first storage unit 120 stores programs and data required for the operation of the video acquisition terminal 100. The first storage unit 120 may be configured mechanically with a ROM (Read Only Memory) or a RAM (Random Access Memory). The first storage unit 120 may store each piece of data by using a cloud storage service.

[0036] The voice data and image data are stored in an information storage database of the first storage unit 120, and then transmitted to the server terminal 200 via the first transmission unit 130. The transmission method for each data is not limited to one, and the voice data and image data may be transmitted to the server terminal 200 together (i.e., the voice data acquired from the microphone and the image data acquired from the camera are transmitted as two types of data without being integrated), or the voice data and the image data may be transmitted to the server terminal 200 together while corresponding to each other in chronological order (i.e., the voice data and the image data are transmitted as a set, as one type of data).

[0037] <Server terminal 200> 3, the server terminal 200, as a cloud server, is composed of a first receiving unit 210, a second storage unit 220, an editing unit 230, and a second transmitting unit 240. Not limited to the above configurations, the editing unit 230 may be subdivided into a voice data editing unit, an image data editing unit, and an integrating unit.

[0038] The first receiving unit 210 mounted on the server terminal 200 has a function of receiving video data including audio data and image data from the video acquisition terminal 100 as a communication path for performing processing related to data exchange.

[0039] Furthermore, the first receiving unit 210 temporarily stores each data received from the first transmitting unit 130 in the second storage unit 220 of the server terminal 200, and outputs it to the editing unit 230. The configuration of the first receiving unit 210 is not limited to this, and the received video data may be output directly to the editing unit 230 without being temporarily stored in the second storage unit 220.

[0040] The editing unit 230 inputs the data from the first receiving unit 210, and then divides the received data according to characteristics and uses, and performs individual or comprehensive analysis.

[0041] Specifically, in this embodiment, the data used for editing is video data. As shown in Fig. 4, the video data is divided into audio data and image data according to the voice, action, and the time when each voice or action was recorded or photographed, and each audio data and image data is accompanied by a time stamp. Next, each data may be further classified according to the purpose. For example, the image data is divided into performer image data showing the actions of the performer, instructor image data showing the actions of the instructor, and lesson image data showing lesson images (i.e., images including both the performer and the instructor).

[0042] Furthermore, since the voice data includes both human voices (i.e., including performers and instructors) and instrument sounds, the voice data is divided into performance data and speech data, and the server terminal 200 can link the data to each piece of editing image data using a time stamp. In this case, the performance data and speech data may be classified in more detail. As an example, the performance data may be divided into performance data of the practitioner and performance data of the instructor, and the speech data may be divided into speech data of the practitioner and speech data of the instructor.

[0043] When receiving video data from the video acquisition terminal 100, the editing unit 230 can classify the video data into audio data and image data. As an example, the editing unit 230 selects and extracts the part of the video data in which audio is recorded, that is, the so-called sound track. After the sound track is extracted, the part remaining in the video data becomes image data. In this case, the image data may be linked to the audio data using a time stamp.

[0044] <Sound source separation> The editing unit 230 can distinguish who is speaking by separating the sound sources according to the frequency of the speaker. Specifically, the sound of an instrument or the voice of a learner or instructor has its own characteristics, and each piece of audio data also represents its own signal (hereinafter referred to as "audio signal"), and the editing unit 230 may separate a plurality of sound sources according to frequency, etc. based on these audio signals. Furthermore, the audio signals in this embodiment may include a learner speech signal, an instructor speech signal, a learner performance signal, and an instructor performance signal.

[0045] The frequencies of the trainee speech signal and the instructor speech signal may be determined by factors such as the function of the human body (e.g., gender, age, height, weight, etc.) and the space (e.g., a recording room, a soundproof room, etc.). As an example, if the trainee is a child, the trainee's tone is usually higher than that of an adult instructor, so the frequency is relatively higher and therefore the frequency is higher than that of the instructor. Conversely, the tone of an instructor who is an adult is usually lower than that of a trainee who is a child, so the frequency is relatively lower and therefore is shown as a low frequency.

[0046] Since the performance by the trainee and the instructor use the same instrument, the frequency of the sound produced by both is basically the same. Therefore, since the performance by the trainee and the instructor cannot be distinguished from each other based on the frequency of the sound alone, in this case, it is possible to identify who is performing by referring to the image data as well.

[0047] Specifically, the sound source separation method relates to a technique that uses data such as changes in wavelength and amplitude due to frequency, and time from image data. That is, in this sound source separation method, the editing unit 230 uses wavelength and amplitude due to frequency to separate a mixed audio signal (that is, an audio signal acquired through a microphone, which usually includes an audio signal of the learner's speech, an audio signal of the instructor's speech, an audio signal of the learner's performance, and an audio signal of the instructor's performance) into speech data of the learner, speech data of the instructor, and performance data. Then, the editing unit 230 uses a time stamp to identify the speech data of the learner, speech data of the instructor, and performance data as image data, and determines who is performing.

[0048] 5 is a flowchart showing classification of video data and a process of combining the classified data. Before the editing unit 230 extracts specific audio data from the mixed audio signal to be separated, the method of separating a plurality of sound waves according to the present embodiment includes: A1, the editing unit 230 divides the mixed audio signal according to the frequency of the mixed audio signal, and while specifying a time period (which is the real time corresponding to the time of shooting) for the divided audio section (the audio section is a section in which sound exists in the recorded audio signal, specifically, a section in which there is a fluctuation in the amplitude of the frequency indicating audio), assigns a time stamp corresponding to this time period (step S101); A2, the editing unit 230 divides the image data into a plurality of video sections according to time periods corresponding to audio sections, and assigns time stamps to the video sections (step S102); A3, the editing unit 230 matches the audio section and the image section using the time stamp (step S103). If the time when each sound or each action was recorded or photographed is the same, the audio section belongs to the image section, and the set is treated as one set (hereinafter, referred to as the edited video). A4, the editing unit 230 classifies the learner speech data, the instructor speech data, and the performance data included in the edited video according to the physical characteristics of the audio signal (step S104); A5 further includes the editing unit 230 determining who the player is by detecting the playing actions of the learner and the instructor based on the image section (step S105).

[0049] In the above-mentioned step A1, the editing unit 230 can extract the target voice data from the mixed voice signal containing a plurality of sound sources by the feature quantity of the voice data. Specifically, the voice signal is expressed by the fluctuation of the frequency according to the passage of time, and basically, the sound source can be separated by the frequency band of the sound. In addition, the intonation of the voice and the tempo (speed) of the conversation of the voice data can also be separated by the timing of the frequency vibration on the time axis. As an example, since the waveform of the voice signal of playing a musical instrument is different from the voice signal of a human speaking, if the change in the voice signal is large, it is considered that the sound source has changed.

[0050] Moreover, the editing unit 230 divides the mixed audio signal into a plurality of audio segments according to frequency, wavelength, and amplitude, and specifies a time period corresponding to each audio segment in the real time (hereinafter referred to as time series) recorded during shooting, and sets a time stamp. Specifically, the editing unit 230 divides the mixed audio signal into a plurality of audio segments according to frequency, thereby generating an audio segment including any of the speech of the learner, the speech of the instructor, and the performance. The editing unit 230 sets the time of the audio segment before division as the time of each divided audio segment. For example, if the audio segment before division is located in the time period from 10 minutes to 11 minutes on the original time axis, after division, the editing unit 230 sets the time stamp of the corresponding time period (from 10 minutes to 11 minutes) as the time of the audio segment after division.

[0051] In step A2 described above, the editing unit 230 may divide the image data into a plurality of image segments according to the time period corresponding to the voice segment. As an example, the editing unit 230 extracts images corresponding to the time period (for example, from 10 minutes to 11 minutes) corresponding to the voice segment from the image data to set the extracted images as image segments. After that, the editing unit 230 simultaneously assigns a time stamp corresponding to the time period to the image segment.

[0052] In step A3 described above, the editing unit 230 matches the voice section and the image section using the timestamps. If the timestamps are the same, the voice section belongs to the image section, and so they may be treated as one set (hereinafter referred to as edited video). If the timestamps do not match, the voice section and the image section are separate sections and are not treated as one set. Furthermore, since the image section is divided according to the time zone of the voice section, the final data set (i.e., a set of data including voice data and image data matched by timestamps) always includes one voice section and one image section.

[0053] In step A4 described above, the editing unit 230 determines whether the audio data is human speech data or performance data based on the physical feature of the audio signal. The edited video that has been matched using the timestamp includes the learner speech data, the instructor speech data, the learner performance data, and the instructor performance data. Among them, the learner speech data and the instructor speech data can be distinguished based on the characteristics of the subjects (e.g., age and gender), but the learner performance data and the instructor performance data use the same instrument, so the frequencies of the learner performance data and the instructor performance data are the same, making it difficult to separate them directly.

[0054] In general, it is possible to determine whether someone is speaking in edited video by motion information such as human mouth movement. As an example, the editing unit 230 has a function of detecting human mouth movement from edited video, and when mouth movement is detected, the person making the sound in the edited video is deemed to be the person whose mouth is moving, and it is possible to determine who is making the sound. When sound is included as audio data and no person whose mouth is moving is detected in the corresponding image data, the editing unit 230 determines that the sound is the sound of a musical instrument.

[0055] However, although it is possible to determine who is making the sound based on the mouth movements, there are cases where the trainees and instructors in the video data (or image data) are wearing masks or the like covering their faces, or the video loaded on the video acquisition terminal 100 has blind spots, and it may not be possible to detect the mouth movements.

[0056] In this case, the editing unit 230 may distinguish the sound source based on the physical feature amount of each audio signal. As an example, the editing unit 230 classifies the instructor's speech data, the trainee's speech data, and the performance data by frequency. Since the frequency of a human audio signal is different from the frequency of an instrument's audio signal, there is a band in which the audio signals do not overlap. Therefore, the editing unit 230 may set the frequency in the band in which they do not overlap as the reference value of the threshold. Specifically, if the frequency of the audio signal in the edited video is higher than a certain threshold, the audio signal in the edited video may be performance data, and if the frequency of the audio signal in the edited video is lower than a certain threshold, the audio signal in the edited video may be instructor's speech data or trainee's speech data.

[0057] However, due to individual differences between people, the frequency band of a child learner may overlap with that of an adult instructor, and instrument performance data may overlap with human voice data (for example, the frequency band of an adult's normal speaking voice is 150Hz to 500Hz, the frequency band of a child's speaking voice is about 1000Hz to 2000Hz, and the frequency band of a piano is about 27Hz to 4186Hz, and the frequency band of instruments has a band that overlaps with the human frequency band). In this case, there is a risk that the sound source cannot be separated by frequency alone.

[0058] Therefore, the editing unit 230 performs the first voice separation using frequency, and then introduces a second voice separation technique that identifies and quantifies the waveform (wavelength, amplitude, etc.) of the sound source. The higher the tone of the sound data, the higher the amplitude of the sound source waveform contained in the mixed sound signal, but the shorter the wavelength. Conversely, the lower the tone of the sound data, the lower the amplitude and the longer the wavelength, making it possible to determine whose voice it is. However, the physical feature amount used to separate the sound in this embodiment is not limited to this.

[0059] As another method for distinguishing the voice of the trainee, the voice of the instructor, and the sound of the instrument, the editing unit 230 may use a voice recognition model to distinguish between the voices of the trainee, the instructor, and the instrument. The voice recognition model is a model that has learned the voice of the trainee, the voice of the instructor, and the sound of the instrument. That is, the voice recognition model is a learning model that has learned teacher data in which the voice data of the trainee's voice is annotated with information indicating that the trainee is a trainee, teacher data in which the voice data of the instructor's voice is annotated with information indicating that the trainee is an instructor, and teacher data in which the voice data of the instrument's voice is annotated with information indicating that the instrument is an instrument, and is a model that accepts input of voice data and estimates whether the voice data is the voice of the trainee, the voice of the instructor, or the voice of the instrument. By inputting the speech data into the voice recognition model, it is possible to identify who is making the sound, and the editing unit 230 can classify the voice data more efficiently.

[0060] In step A5 described above, after distinguishing between the speech data and the performance data between the learner and the instructor, the editing unit 230 must further separate the performance data. Unlike speech data, performance data is data acquired while playing the same musical instrument, and there is a possibility that it is not possible to distinguish who played the performance data using the frequency.

[0061] As one method of identifying the performance data according to the present embodiment, the editing unit 230 may use the motion information of playing the instrument in the acquired edited video to identify the subject of the performance data. Specifically, the editing unit 230 detects a person playing the instrument in each edited video, and the person playing the instrument is the performer in the edited video. Also, if the performer is a practicer in the edited video and there is no speech data, the edited video does not include the instructor's instructions and may be deleted. The detection target is not limited to the above, and the performer may be identified by detecting the movement of the hand playing the instrument or the magnitude of the force applied to the instrument. That is, the more physical features that can indicate the instructor's performance data, the easier it is to identify who the performer is, and the higher the accuracy.

[0062] <Processing each data after separation> Processing of each data after audio separation according to this embodiment will now be described with reference to Fig. 6. Based on the edited video obtained according to the physical feature amount of each data, the editing unit 230 extracts edited video having only instructor utterance data, edited video having only instructor performance data, and edited video having both instructor utterance data and instructor performance (hereinafter, non-silence sections) (step S201). Edited video that is not extracted may be deleted as a silent space.

[0063] In order to determine whether the extracted non-silence sections are similar to each other, in this embodiment, each non-silence section is analyzed as a "language of performance" and the similarity of each non-silence section is measured using a similarity calculation method of natural language processing. The "language of performance" refers to information in which sound is made expressible in language, and more specifically, it refers to information in which the pitch, length, and importance of each sound (i.e., the number of times each note appears) are expressed as vectors after the sound is converted into a mel spectrogram. A mel spectrogram is information that expresses the amplitude of sound on a time axis and a frequency axis in the mel scale.

[0064] Specifically, in this embodiment, the audio data of each non-silence section may be treated like a natural language. The editing unit 230 can generate a mel spectrogram based on each non-silence section. The mel spectrogram is a spectrogram whose frequency axis is in the mel scale, and is usually used in voice recognition and the like. The editing unit 230 creates a feature vector expressing the "language of performance" in the non-silence section from the generated mel spectrogram. The feature vector may include the pitch of the voice (hereinafter referred to as "pitch"), the continuity of the voice (i.e., the longest vocalization duration), and the like.

[0065] As an example of generating a feature vector, when there are five pieces of audio data in one non-silence section, the editing unit 230 treats each of the five pieces of audio data as independent data and creates a feature vector based on the pitch of each piece of audio data, the continuity of the audio, and other audio features. The method of creating a feature vector is not limited, and the editing unit 230 may calculate the pitch difference between adjacent pieces of audio data based on the order of performance, and use this as a feature vector. In the non-silence section, the order of performance is a1, a2, and a3, and when creating a feature vector based on the pitch difference, the editing unit 230 may calculate the pitch difference using the calculation methods of (a2-a1) and (a3-a2). After that, the editing unit 230 may use the calculation result as the feature vector ((a2-a1), (a3-a2)) of the non-silence section.

[0066] Then, the editing unit 230 can calculate the relative similarity between the non-silence sections using the created feature vector. Also, the method of calculating the similarity in this embodiment is not limited. Each method will be described below.

[0067] <Similarity calculation method 1> Specifically, the editing unit 230 calculates the cosine similarity between each of the non-quiet periods based on the feature vector of each of the quiet periods to determine the similarity between the non-quiet periods. As an example, the editing unit 230 generates a mel spectrogram from a non-silence section A having five pieces of audio data, and then creates a feature vector A[a1, a2, a3, a4, a5] according to the pitch of the audio. The editing unit 230 generates a mel spectrogram from a non-silence section B having five pieces of audio data, and then creates a feature vector B[b1, b2, b3, b4, b5] according to the pitch of the audio. The editing unit 230 then calculates the cosine similarity between the non-silence section A and the non-silence section B based on the feature vector A and the feature vector B.

[0068]

number

[0069] Feature vector A = (a1, a2, , a n ), feature vector A∈R n Feature vector B = (b1, b2, , b n ), feature vector B∈R n Feature vector A and feature vector B form a certain angle, and editing unit 230 can calculate the relative similarity between non-quiet section A and non-quiet section B based on the cosine value of the angle between feature vector A and feature vector B. In general, when the value indicating cos(a,b) is close to 1, the relative similarity between non-quiet section A and non-quiet section B is considered to be high, and when it is close to -1, the relative similarity between non-quiet section A and non-quiet section B is considered to be low.

[0070] <Similarity calculation method 2> Specifically, the editing unit 230 replaces the feature vector of each silent section with a TF-IDF value (abbreviation of term frequency-inverse document frequency), and calculates the cosine similarity between each non-silent section based on the TF-IDF value to determine the similarity of the non-silent sections. Among these, the TF value (abbreviation of Term Frequency) is an index representing the frequency of occurrence of a word, and the IDF value (abbreviation of Inverse Document Frequency) is an index representing the inverse document frequency.

[0071]

number

[0072] In the formula 2 of this embodiment, tf(t, d) is the TF value of a certain voice data t in a non-silence section d. t,d is the number of times that the voice data t appears in the non-quiet section d. Also, Σ (S∈d) n (S,d) is the sum of the occurrence counts of all sounds recognized as different sounds in the non-quiet section d.

[0073]

number

[0074] In the formula 3 of this embodiment, idf(t) is an index representing the importance of each piece of audio data relative to the non-quiet section, and is called the inverse document frequency. N is the number of pieces of audio data in the non-quiet section, and df(t) is the number of times that audio data t appears. As an example of the similarity calculation method 2, the editing unit 230 generates a mel spectrogram from a non-silence section A having five pieces of audio data, and then creates a feature vector A [a1, a2, a3, a4, a5] according to the pitch of the audio. The editing unit 230 also generates a mel spectrogram from a non-silence section B having five pieces of audio data, and then creates a feature vector B [b1, b2, b3, b4, b5] according to the pitch of the audio. Next, the editing unit 230 calculates a TF-IDF value for each pitch of the feature vector A, and also calculates a TF-IDF value for each pitch of the feature vector B. Then, the cosine similarity is calculated between the vectors with the TF-IDF value for each pitch. The accuracy of the similarity calculation method 2, which calculates the cosine similarity using the TF-IDF value, is higher than that of the similarity calculation method 1. The reason is that there are multiple parts in the same musical score where the rhythm is similar due to modulated performance (for example, transposition or modulation). In this case, similarity calculation method 1 may conversely determine that parts with similar rhythms are dissimilar. As a result, when the cosine similarity based on the TF-IDF value is close to 1, the relative similarity between non-quiet section A and non-quiet section B is considered to be high, and when it is close to -1, the relative similarity between non-quiet section A and non-quiet section B is considered to be low.

[0075] Furthermore, a threshold value for the similarity of each non-quiet section may be freely set according to the feature vector. The set threshold value is stored in the second storage unit 220. If the similarity between the multiple non-quiet sections is higher than the set threshold value, the multiple non-quiet sections may be clustered into one group (cluster). Conversely, if the similarity is lower than the set threshold value, the multiple non-quiet sections may be considered to be different, and the multiple non-quiet sections may be clustered into different groups.

[0076] After calculating the similarity of each non-silence section, the editing unit 230 clusters the non-silence sections according to a threshold value (step S202). Each non-silence section cannot be completely identical due to external factors, and for example, when a performer (including a learner and an instructor) plays the same musical score fast and when a performer plays it slowly (or modulated performance), the similarity calculated by the feature vector is different. Therefore, by comparing the similarity of multiple input performance sound signals, if the deviation between the similarities is within a certain range, the multiple non-silence sections can be determined to be similar. However, the threshold value for determining the similarity may not be a fixed value, but may be a relative value according to the similarity of each non-silence section.

[0077] The editing unit 230 may calculate the deviation between the similarities of each non-silence section according to the physical feature amount of the sound, or may calculate the deviation between the similarities using an externally connected sound similarity determination program.

[0078] The editing unit 230 ranks the clustered non-silence sections along a time axis (step S203). Specifically, the editing unit 230 ranks the non-silence sections according to the time when each sound or action was recorded or photographed. Even in the case of overlapping performances (for example, when there are multiple identical sound sections in a musical score, the repeated performance of those parts is called overlapping performance), the editing unit 230 can identify the ranking of the non-silence sections using the time stamps.

[0079] Furthermore, the editing unit 230 may integrate the edited video having only the instructor's speech data and the edited video having only the instructor's performance data based on the time when each sound or each action was recorded or photographed. Specifically, the editing unit 230 uses the time stamp to compare the edited video having only the instructor's speech data with the edited video having only the instructor's performance data, and if there is a time correlation, the edited video having only the instructor data is combined into one set with the edited video having only the instructor's performance data. For example, when the editing unit 230 has three edited videos (edited video A (10 minutes 10 seconds to 10 minutes 20 seconds) having only instructor speech data, edited video B (10 minutes 20 seconds to 10 minutes 40 seconds) having only instructor performance data, and edited video C (15 minutes 24 seconds to 15 minutes 30 seconds) having instructor speech data and instructor performance data), the time information of the time period of edited video A and the second edited video B is compared using time stamps, and since the edited videos A and B after comparison have a relationship on the time axis (i.e., time continuity), they may be integrated as one set (hereinafter referred to as matched video). In this case, since the edited video C that was not compared already has instructor speech data and instructor performance data, it may be stored as an independent edited video as review material for the learner without comparison. All the edited images and the collated images finally obtained are recorded as a performance practice set. The performance practice set is information that associates the learner's performance data for a predetermined period of time where the learner plays incorrectly in the practice video with the content of the instruction (the instructor's speech data and image data) of the instructor's instruction when the learner plays in that manner (step S204).

[0080] The instructor's scolding and praise included in the musical performance practice set may be summarized. As an example of summarizing, when there is a plurality of pieces of instructor utterance data for the same part, the editing unit 230 may organize the plurality of pieces of instructor utterance data in the musical performance practice set including that part and combine them into one musical performance practice set. Also, when there are a plurality of musical performance practice sets, the editing unit 230 may organize the instructor utterance data and combine them into one musical performance practice set. As another example of summarization, when the editing unit 230 detects speech data having the same meaning from multiple performance practice sets at the same location, for example, when there is speech data such as "Play more slowly!" and "Slow down here!" from the instructor, the editing unit 230 may detect adjectives and verbs in the two pieces of speech data by using a speech recognition means, for example, a morphological analysis method for speech, or by connecting to an external speech recognition application. Specifically, the editing unit 230 extracts the verb "play" and the adjective "slowly" from the two pieces of speech data by using a morphological analysis method. After that, when the editing unit 230 detects the adjective "slowly" multiple times and can determine that the two pieces of speech data have the same meaning, the editing unit 230 may combine the two pieces of speech data into one piece of "play slowly!". Conversely, when no speech data having the same meaning is detected, i.e., when the same verb or adjective is not detected, the editing unit 230 may extract the multiple pieces of speech data and list them together in one performance practice set.

[0081] In addition, the editing unit 230 may convert the instructor's speech data in the performance practice set into text in order to organize it. As an example, the editing unit 230 extracts the instructor's speech data from the performance practice set, inserts punctuation marks, and converts it into text (i.e., speech-to-text conversion). The editing unit 230 may then use a morphological analysis method to identify frequently appearing keywords in the text, and organize phrases contained therein into a single group. When a student is practicing a performance, he or she can review the points pointed out during the lesson while referring to the organized results.

[0082] <Playback terminal 300> As shown in FIG. 7, the playback terminal 300 includes a reception processing unit 310 including a second receiving unit 311, a third storage unit 312, and a setting unit 313, and a playback processing unit 320 including a second shooting unit 321, a fourth storage unit 324, a detection unit 325, and a playback unit 326. The playback terminal 300 has a re-editing function, a detection function, and a playback function for each musical performance practice set received from the server terminal 200. Specifically, the playback terminal 300 includes a processor and a memory. The setting unit 313 may correspond to the processor, and the third storage unit 312 and the fourth storage unit 324 may correspond to the memory. The setting unit 313, which is a processor, may replace the image data and audio data in the musical performance practice set with other image data and audio data and reset it based on the musical performance practice set stored in the third storage unit 312. In this case, the obtained video data may be called post-setting video data.

[0083] On the other hand, when the practicer is practicing the instrument at home, the second image capturing unit 321 mounted on the playback terminal 300 may record video of the practice (hereinafter referred to as home practice video data). The home practice video data acquired by the second image capturing unit 321 is stored in the fourth storage unit 324 as detection data. If the practicer makes a mistake while practicing at home, the detection unit 325 may detect the mistake from the fourth storage unit 324 and cause the playback unit 326 to play the video data after setting.

[0084] The receiving and processing unit 310 according to this embodiment can re-process each piece of data received from the second transmitting unit 240. Specifically, the setting unit 313 may replace the instructor's portrait and voice appearing in the musical performance practice set with different images and voices, based on the musical performance practice set received by the second receiving unit 311. The replacement images or GIF (Graphics Interchange Format) images and voices may be stored in advance in an information storage database of the third storage unit 312.

[0085] As shown in FIG. 8, each figure appearing in the image (including a figure 801 of a practitioner and a figure 802 of an instructor; i.e., a part of the image) may be changed to a replacement image 804 (as shown in the upper right figure (1)), and the video played as a performance practice set may be video including the instructor in class, the practitioners themselves, instruments, etc., but this video may be changed so that only the image of the character shown in replacement image 804 (as shown in the lower right figure (2)) is an image of the instructor speaking the content of his / her instruction. The data regarding the replacement image 804 may be data stored in the third storage unit 312, or the practitioner may upload a preferred replacement image 804 to the playback terminal 300. When the practitioner uploads an image, the receiving processing unit 310 may be provided with an image control unit for changing the size, color, etc. of the image. A replacement image is an image displayed in place of the instructor, and changing the instructor to a replacement image 804 means that when the playback terminal 300 plays back the instruction content, image processing is performed to change the image of the instructor to the replacement image 804 specified by the practitioner. By changing the replacement image 804 to an anime or game character, entertainer, virtual idol, or the like that the practitioner likes, the musical instrument practice system (playback terminal 300) can improve the motivation of the practitioner to practice.

[0086] On the other hand, the playback terminal 300 can also reset the instructor's speech. Specifically, the reception processing unit 310 may connect to an external sound processing device, a sound processing application, a sound processing program, etc., and change the speech data in the performance practice set to a replacement sound (hereinafter, sample sound) according to the frequency. Specifically, the reception processing unit 310 inputs the sample sound uploaded by the learner based on the speech data in the performance practice set to a sound quality change model such as ChatGPT (registered trademark) or S0-VITS-SVC-V4 (registered trademark) connected to the setting unit 313. The setting unit 313 causes the sound quality change model to learn the sound quality, and outputs the learned speech data. Also, the more the playback terminal 300 collects the instructor's speech data, the higher the accuracy of reproducing the learned speech data.

[0087] Alternatively, the speech data in the performance practice set received from the second receiving unit 311 may be read by extracting words and phrases contained in the speech data and using the voice stored in the third storage unit 312. As one example, the words and phrases extracted from the speech data are introduced into a voice changer application and played back in a different voice (for example, replacing the original instructor's voice with a child's voice). The above image and sound replacement method can reduce the tension felt by practitioners when practicing at home and increase their passion during practice.

[0088] As shown in Figure 9, when a practitioner is practicing playing an instrument at home, the practitioner may set various indicators for practice according to the practitioner's own needs (e.g., setting practice time, selecting a song to practice, playing a lesson video, etc.). Specifically, before practicing at home, a student can play back the video of the lesson and can set in advance the music to be practiced going forward (step 301). Then, the total practice time that the learner is going to practice can be set via the terminal (step 302). For example, the learner freely inputs his / her practice time using a smartphone app. Alternatively, the playback terminal 300 sets the practice time of the learner through multiple voice conversations with the learner. This setting may be performed by the learner inputting a numerical value indicating the practice time, or may be realized via an automatic response function realized by the playback terminal 300. The automatic response function may be fuzzy designed, and may be programmed to negotiate the practice time with the learner when there is a difference between the time that the learner is going to set and the time that is desired for practice. In this case, the playback terminal 300 may be equipped with a microphone 323 and connected to an external voice recognition system (e.g., ChatGPT (registered trademark)) to recognize the voice of the learner. After that, the learner can freely specify the piece of music that he / she is going to practice and play it.

[0089] After the learner has decided on the piece of music he or she has selected, the playback terminal 300 presents a performance practice set that corresponds to the piece of music, and the learner can review the points that were pointed out in the lesson before practicing at home (step 303).

[0090] On the other hand, the learner may set the difficulty of each practice piece according to his / her own subjective judgment (step 304). Specifically, the playback terminal 300 may set each piece according to the learner's own level of proficiency in the order of "high to low difficulty: S>A>B>C>D." For a piece with a high level of difficulty, the speed of the piece may be slower, the practice time may be relatively longer, and the learner may play the instrument with one hand. Conversely, for a piece with a low level of difficulty, the speed of the piece may be the same as the standard performance data, the practice time may be relatively shorter, and the learner may play with both hands together. However, the method of setting the difficulty level is not limited to this, and the receiving and processing unit 310 may determine the difficulty level of a piece according to the number of pointed out points in the performance practice set.

[0091] Since the time required for each piece of music varies depending on the level of difficulty, the learner sets a final practice plan according to his / her needs (e.g., which pieces he / she is not good at practicing, etc.) (step 305). After the practice plan is determined, the learner may start practicing at home.

[0092] The playback processing unit 320 according to this embodiment can detect errors from the video of the practicer practicing at home, and can simultaneously play back the performance practice set. Taking the piano as an example, the second image capturing unit 321 captures the way the learner presses the black and white keys and the pedals, while the microphone 323 captures the performance voice of the learner. The obtained video data including the performance data and voice data is stored in the information storage database of the fourth storage unit 324.

[0093] Furthermore, the playback terminal 300 uses the musical performance practice set received from the server terminal 200 as material for home practice. Before passing the musical performance practice set to the playback processing unit 320, the playback terminal 300 determines whether or not to replace the images and sounds in the musical performance practice set via the setting unit 313. If the practitioner is attempting to re-set the images and sounds in each musical performance practice set (i.e., replacing the instructor image in the musical performance practice set with another character, or replacing the instructor's voice information with another person's voice information without changing the contents of the instructions), the playback terminal 300 may pass the set musical performance practice set to the detection unit 325 and use it as practice material. If the practitioner does not set a musical performance practice set, the playback terminal 300 may pass the musical performance practice set received from the server terminal 200 directly to the playback processing unit 320 and use it as practice material.

[0094] When the correctness of the movement of the learner is to be judged, the detection unit 325 uses the performance practice set as the basis data and compares it with the image data of the learner captured by the second image capture unit 321. As an example, the detection unit 325 may judge the image data based on the movement of the learner's fingers or the fluctuation of the keys or strings of the musical instrument. On the other hand, when playing a musical instrument, the detection unit 325 detects whether the position of the movement of the learner is correct in the detection unit 325 according to this embodiment, even though the sound that the instrument can produce differs depending on the force applied to the keys or strings, and does not need to measure the force applied (for example, if the position of the fingers is correct, the movement is considered to be correct. However, without being limited thereto, the shape of the hand, the way of fingering, the way of playing the keys with the fingers, and other movements may be detected to judge whether the performance is correct or not). In addition, the detection unit 325 may measure the sound pressure due to the force applied to the keys or strings, and detect the correctness of the movement based on the volume of the collected sound.

[0095] When attempting to determine the correctness of the practicer's performance, the detection unit 325 can use the performance practice set as basis data and compare it with the performance data obtained from the microphone 323 mounted on the playback terminal 300. The detection unit 325 can also calculate the similarity between the performance data during practice at home and the performance data included in the performance practice set, and compare it with a predetermined threshold value.

[0096] If the difference is greater than the predetermined threshold, it is determined that the home practice video data matches the performance data in the performance practice set, and it is determined that the student has made the same mistake during the lesson again. In this case, the playback terminal 300 plays back the speech data and the image data corresponding to it that are included in the performance practice set.

[0097] On the other hand, if the difference is lower than the predetermined threshold, it is determined that the home practice video data does not match the performance data included in the performance practice set. In this case, a re-evaluation may be performed to see if the practicer will play correctly or will make another mistake. As an implementation measure for the re-evaluation, standard performance data is introduced into the playback terminal 300, and the home practice video data is compared with the standard performance data, and the similarity is used to determine whether the practicer will play correctly. When the home practice video data and the standard performance data are similar and the so-called correct performance is performed, the playback unit 326 does not need to play back the performance practice set. In cases where the home practice video data and the standard performance data are not similar, i.e., the practitioner makes a different mistake, since there is no speech data from the instructor, the playback unit 326 cannot provide any pointers and may play back only the parts that do not match.

[0098] In short, when the detection unit 325 determines the performance data that the practitioner is practicing at home, it first compares it with the performance data included in the performance practice set, and if the home practice video data and the performance practice set are similar to each other to a certain extent, it may play the speech data and image data included in the performance practice set. Next, if the home practice video data and the musical performance practice set do not match, the detection unit 325 introduces the standard musical performance data and further compares it with the home practice video data. If the standard musical performance data matches, the musical performance data may be skipped. Finally, if the home practice video data and the standard performance data do not match, the audio data of the portion that does not match may be reproduced.

[0099] FIG. 10 is a flowchart showing the process of the playback terminal 300 when a practitioner practices a musical instrument at home in this embodiment. The playback terminal 300 detects the position where the practicer is performing (the position where the practicer is seen performing in the video captured by the second image capturing unit 321) (step 401). At the same time that the practicer is performing at home, the second image capturing unit 321 mounted on the playback terminal 300 captures at-home practice video data of the practicer practicing at home (step 402). At this time, the second image capturing unit 321 may capture both audio and video, or only audio or only video. The playback terminal 300 calculates the similarity for each piece of home practice video data by comparing it with the musical performance practice set and the standard musical performance data, and presents to the practicer a list of information indicating the musical performance practice sets in order of high to low similarity (step 403). In this case, the playback terminal 300 may present a list of multiple musical performance practice sets ranked in order of high to low similarity, or may present only the musical performance practice set with the highest similarity. Next, the playback terminal 300 judges whether the learner will have a conversation with the playback terminal 300 (step 404). The learner may input voice into the microphone 323 mounted on the playback terminal 300. The voice may be a question or an inquiry about the practice. At that time, if the playback terminal 300 detects a frequency fluctuation, it may identify that the learner is speaking. If the playback terminal 300 can detect a voice signal, it may consider that the learner is speaking and proceed to the next stage (YES in step S404). If the playback terminal 300 does not detect a frequency change and does not identify the learner's speech (NO in step 404), the learner has understood that part of the practice piece and may proceed to the next stage, and the playback terminal 300 determines whether the learner wants to continue practicing (step 407). If the learner is still planning to practice (YES in step 407), the process returns to the step where the learner is playing (step 401). If the learner does not want to continue practicing (NO in step 407), the instrument practice ends (step 408). Conversely, when the playback terminal 300 detects frequency fluctuations and identifies so-called learner speech (YES in step 404), it judges whether the learner is having trouble with the list of information showing the performance practice set in order of high and low similarity (step 405). If the learner is not having trouble with the video being played and understands it well (NO in step 405), it may proceed to the next stage. It is judged whether the learner will continue practicing (step 407). If the learner is still trying to practice (YES in step 407), the process returns to the step where the learner is playing (step 401). Conversely, if the learner is not yet trying to practice (NO in step 407), the current instrument practice is terminated (step 408). On the other hand, if the learner is unsure about the order according to the similarity (YES in step 405), the playback terminal 300 may allow the learner to play the performance practice set that corresponds to that time according to the similarity. As an example, the learner may specify and play the performance practice set himself, or the playback terminal 300 may automatically play the performance practice set with the highest similarity according to the similarity.

[0100] The embodiment for determining whether or not the learner is having difficulty may be realized by morphological analysis, face recognition, and adding functions to the terminal.

[0101] <Example 1: Morphological analysis method> As an example, the playback terminal 300 connects to an external terminal (usually an application for voice analysis) and compares the performance data of the practicer at home with the standard performance data, and then displays the similarity. The practice performance set with the highest similarity may be displayed immediately before the practicer asks a question. If the practicer has a question about the practice performance set, he or she may ask a question or negation such as "why?", "how?", or "how?" and input it as speech data to the playback terminal 300. At this time, the playback terminal 300 divides the speech data input by the learner into subjects, predicates, and interrogatives, and compares them with keywords stored in the third storage unit 312, the fourth storage unit 324, or a newly-installed storage unit. For example, when the learner says, "How do you play like this?", the playback terminal 300 divides the data into "how to play" as the subject, "to do" as the predicate, and "how...?" as the interrogative by morphological analysis. If the unit indicating the divided natural language, i.e., a part of the subject and interrogative, matches or highly matches a preset criterion, the playback unit 326 will play the performance practice set with the highest similarity. The playback terminal 300 may detect only interrogative words or negative words without detecting subjects and predicates. For example, when the learner directly utters "Why?" or "I don't understand!", the playback terminal 300 cannot detect keywords because the utterance does not contain a subject or predicate. At this time, the playback terminal 300 judges whether or not the interrogative words and negative words stored in the storage unit described above match. When there is a perfect match or a high degree of match, the playback unit 326 may directly play the performance practice set with the highest similarity from the list showing the order of similarity.

[0102] <Example 2: Face Recognition> For example, if a learner has questions about the performance practice set presented in order of similarity, he or she may show a gloomy or questioning expression, such as frowning, closing the eyes, or pursing the lips, which show dissatisfaction or doubt. The playback terminal 300 compares the facial expression captured by the overall imaging camera 322 of the second imaging unit 321. If an awkwardness or unnaturalness is detected, the playback terminal 300 may play the musical performance practice set with the highest similarity.

[0103] <Example 3: Adding a button or other function to freely select review points on the device> As an example, a selection button may be added to the playback terminal 300. The learner may select the musical performance practice set that he or she most wants to review from among the musical performance practice sets presented in order of the degree of similarity.

[0104] In addition, the system according to the present embodiment in which the practitioner practices a musical instrument at home may be connected to an external system such as ChatGPT (registered trademark) to realize a voice conversation function. Also, the practitioner may play a specific performance practice set through the external system while talking to the external system.

[0105] As shown in FIG. 11, the musical instrument practice system of this embodiment is not limited to judging whether or not a performance is being performed correctly, but may also be able to introduce the atmosphere and background of a practice piece as one example to the practice person. Before practicing an instrument, the learner may check the practice piece in advance (step S501). When the learner practices an instrument at a designated time, the playback unit 326 of the instrument practice system according to this embodiment may automatically distribute information on the piece to the learner at the designated time. When the learner specifies the practice time taking into consideration his / her own performance proficiency, the learner can select the piece stored in the playback terminal 300 by himself / herself.

[0106] After deciding the etude, it is confirmed whether there is an image showing the atmosphere of the etude (step 502). As an example, the playback terminal 300 according to the present embodiment may connect to ChatGPT (registered trademark) (YES in step 502), and the learner may decide the etude and search for atmosphere data related to the etude on the Internet via ChatGPT (registered trademark) (step 503). If the atmosphere related to the etude is not detected using ChatGPT (registered trademark) (NO in step 502), a commentary showing the atmosphere of the piece may be compiled from the speech data of the instructor of the performance practice set belonging to the etude stored in the third storage unit 312 or the fourth storage unit 324 (step 504).

[0107] <Supplementary Information> Although the musical instrument practice system according to the present invention has been described in the above embodiment, it goes without saying that the present invention is not limited to the above embodiment. Various modified examples will be described below.

[0108] (1) In the above embodiment, the editing unit 230 of the server terminal 200 may be mounted on the playback terminal 300. As an example, after the first shooting unit 110 and microphone 116 mounted on the video acquisition terminal 100 acquire the video and audio of the trainee, they may not store them in the first storage unit 120 and the second storage unit 220, but may directly pass them over the Internet to the editing unit 230. Furthermore, the video capturing terminal 100 may be configured as one unit with the server terminal 200. In this case, the integrated terminal may automatically edit the captured audio data and image data as the lesson progresses.

[0109] (2) Furthermore, the embodiment of acquiring video including images and audio of the learner during the lesson is not limited to this, and the video may be acquired via a smartphone, tablet, or the like.

[0110] (3) In the embodiment, the frequency fluctuation and the number of vibrations per unit time nT are used for the explanation, but the present invention is not limited to this. For example, the playback terminal 300 may convert the performance data into text while receiving it, or may recognize notes including the music title and convert the performance data into text using machine learning. In this case, the playback terminal 300 may compare the performance data when the learner plays in class and the performance data when the learner plays at home when converting them into text. If the results of the conversion into text are different, this part may be played incorrectly, and may be detected by the playback terminal 300 and played back.

[0111] (4) In the embodiment according to the present invention, the playback terminal 300 may search for and play back videos of famous musicians performing, commentary from music experts, and the like.

[0112] (5) In addition, in the functions and control procedures of each component described in this specification, particularly the processing procedures described using flow charts, the processing methods and procedures may omit some parts, or new parts may be added, or procedures may be substituted or changed in order. Processing procedures in which such omissions, additions, or changes in order have been made are also included in the scope of this disclosure as long as they do not deviate from the spirit of this disclosure. [Explanation of symbols]

[0113] 100 Video acquisition terminal 110 First Filming Section 111 Right facing camera 112 Left facing camera 113 Foot Camera 114 Downward Camera 115 Overall Imaging Camera 116 Mike 120 1st memory section 130 First Transmission Unit 200 Server terminal 210 First Receiving Unit 220 2nd memory section 230 Editorial Department 240 Second Transmission Unit 300 playback devices 310 Receiving processing unit 311 Second Receiving Unit 312 Third memory section 313 Settings 320 Reproduction Processing Unit 321 2nd Filming Division 322 Whole-view camera 323 Mike 324 4th memory section 325 Detector 326 Playback Department 800 images 801 Practice Images 802 Leader Images 803 Audio Data 804 Replacement Image

Claims

1. A musical instrument practice system including: a video acquisition terminal that acquires a practice video; a server terminal that edits the practice video; and a playback terminal that plays back the edited practice video received from the server terminal, The image acquisition terminal includes: a first photographing unit for photographing an image of a learner playing and audio data thereof, and an image of an instructor teaching and audio data thereof; a first storage unit that stores the image captured by the first image capture unit and the acquired audio data; a first transmission unit that transmits the video including the image and audio data stored in the first storage unit to a server terminal; The server terminal includes: a first receiving unit that receives the video transmitted from the first transmitting unit; an editing unit that automatically edits the video received by the first receiving unit according to a sound included in the video; a second transmission unit that transmits the video edited by the editing unit to the playback terminal; The playback terminal includes: A second receiving unit for receiving the edited video; a second storage unit that stores the video received from the second receiving unit; a recording unit for recording the sound being practiced by the person practicing; A musical instrument practice system comprising: a playback unit that plays back the instructor's sound corresponding to the sound being practiced by the learner based on the sound recorded by the recording unit and the edited video.

2. The editorial department: From the video received by the first receiving unit, extracting the sounds of the learner and the sounds of the instructor's performance and speech for each of a plurality of pieces of performance data and speech data; The performance data is automatically clustered according to the similarity of the sounds being played by the practice person.

2. The musical instrument practice system according to claim 1.

3. The editorial department further The speech data based on a time period of the video, a second storage unit for storing the performance data as a performance practice set by dividing the corresponding utterances into clusters obtained by clustering the performance data.

3. The musical instrument practice system according to claim 2.

4. The playback terminal includes: a third storage unit for storing sample sounds uploaded by the learner; The sound of the instructor's speech is replaced in accordance with the performance practice set based on the sample sound stored in the third storage unit.

4. The musical instrument practice system according to claim 3.

5. The playback terminal includes: a setting unit for setting an overall practice time based on the musical performance practice set, The setting unit further sets the practice time according to the performance proficiency of the learner.

4. The musical instrument practice system according to claim 3.

6. The playback terminal includes: A second photographing unit that photographs an image of the person practicing and the sound thereof; and a detection unit that detects whether the person is playing the instrument correctly based on the motion and sound of the person playing the instrument based on the video captured by the second image capture unit.

4. The musical instrument practice system according to claim 3.

7. The playback terminal includes: When detecting incorrectly played parts, A similarity between the image captured by the second image capture unit and the musical performance practice set is estimated, and when the similarity is equal to or greater than a predetermined threshold, the musical performance practice set is played back.

7. The musical instrument practice system according to claim 6.

8. Before the learner practices playing a piece, the learner refers to the playing practice set; and Select the relevant practice set and play it 4. The musical instrument practice system according to claim 3.

Images