Music performance device using a mobile device

Abstract

To provide a method and device for, when performing concert using a plurality of music playing devices that read electronically expressed musical score data to play music according to an operation made by a player, enabling musical performance while easily matching timing and speed of musical performance in a device set as a member of concert with musical performance in a device set as a leader.SOLUTION: The present invention includes: first display means for transmitting performance position data from one device set as a leader of concert by communication means of the device, and continuously displaying the performance position data received by communication means in at least one device set as a non-leader, utilizing the performance position data as a display position of musical score data of a reference musical performance; and second display means for continuously displaying the musical score data displayed in conformity with musical performance of a player in the device set as the non-leader in such an overlapping manner that the reference musical score image is visible and independently from the first display means.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a music performance device using a mobile terminal device such as a smartphone.

Background Art

[0002] Conventionally, for example, as shown in Patent Document 1, a line representing musical notes is printed and recorded on a roll of paper or the like to form a musical score. From digital video data created by photographing, recording, or other methods of moving the roll of paper or the like, an image input means for continuously cutting out still images and inputting them, and an image processing means for detecting musical notes recorded by image processing from the input still images are provided. Thus, a music performance toy that simulates an automatic-playing piano and plays the musical score by producing the sound of the corresponding musical note based on the detected musical note is known.

[0003] Furthermore, as a developed form of this toy, in an electronic device such as a mobile terminal device as shown in Patent Document 2, the rotation speed of a gesture is detected from a rotation operation on a touch panel recognized as a gesture event by a single contact point, the performance position of a musical score for performance is determined from the detected rotation speed, and a method and apparatus for playing the musical score based on the determined performance position information are known. [[ID=as18]]

[0004] If it is a method and apparatus mounted on a mobile terminal device as shown in Patent Document 2, it can be easily brought along, and it is conceivable to perform a performance, that is, an ensemble, with multiple people using multiple devices. Therefore, when actually performing an ensemble, as disclosed in FIGS. 1 and FIGS. 7 of Patent Document 2, all that is displayed on the display of each performer's mobile terminal is a thin line that moves from the top to the bottom of the screen according to the rotation speed of the performer's own gesture, with different positions and lengths. It is difficult to understand which part of the musical score the performer himself / herself is playing, and even if a leader of the ensemble is determined as in a general ensemble and other performers (referred to as members of the ensemble) try to match the timing and speed of their own performance by listening to the leader's performance, it is difficult to accurately match them.

[0005] On the other hand, with the spread of network technology, techniques are now known that allow multiple musicians to perform together using network communication functions even when they are in different locations.

[0006] For example, Patent Document 3 discloses a technology relating to ensemble playing and automatic performance by multiple people via a network, which is a technology that uses a performance control unit to select at least one performance part and the necessary automatic performance parts to realize ensemble playing via a network.

[0007] Furthermore, according to Patent Document 4, a technology is disclosed that transmits control information of a specified electronic musical instrument via a network and displays the performance status of electronic musical instruments at other locations on a display.

[0008] Furthermore, Patent Documents 5, 6, and 7 also disclose technologies related to real-time ensemble systems using network functions.

[0009] Furthermore, Patent Documents 8 and 9 disclose a method for correcting the delay in performance caused by network communication when performing a real-time ensemble performance using network functionality.

[0010] Patent documents 10 and 11 disclose a method for coordinating timing when performing real-time ensemble playing using network functions. This method involves predicting the motion state, generating performance information according to the predicted motion state, and providing notification actions using information other than visual / auditory information, such as operation information input from other devices.

[0011] Furthermore, Patent Document 12 discloses a technique for displaying a variety of musical scores by displaying a standard musical staff on multiple layers, setting a display mode for each layer and various elements such as musical notes, and then overlaying the layers.

[0012] On the other hand, Apple provides a framework called Apple Multipeer Connectivity (hereinafter abbreviated as Multipeer Connectivity), which is a mechanism that enables the discovery of services provided by nearby terminal devices via Wi-Fi networks and Bluetooth personal area networks, and enables message-based data communication and file transfers through those services. Using this, it is possible to transmit the performance position information of a musical score on a specific terminal device (hereinafter referred to as the leader terminal device) that is playing as the leader of an ensemble, using the technology disclosed in Patent Document 2, to at least one other terminal device (hereinafter referred to as the member terminal device) via data communication.

[0013] On the other hand, member terminal devices receive performance position information of the sheet music from the reader terminal device via a communication means utilizing the Multipey Connectivity framework, and can play the sheet music stored in their own terminal device based on the received performance position information of the sheet music. Therefore, this method makes it possible to play together with a performance part and an automated performance part without using the performance control unit shown in Patent Document 3. The performance position data used for data communication over this network is a simple data that can be represented by a single value, as will be described later, and does not require complex data such as phrases or audio information as shown in Patent Documents 5, 6, and 7.

[0014] Furthermore, since the data used for communication via this network is only one-way information from the reader terminal device to the member terminal device, the delay caused by network communication is only in one direction. Since the member terminal device stores the sheet music it will play, by advancing the performance position of the sheet music played by the member terminal device relative to the performance position of the sheet music received by the reader terminal device by the amount of the delay caused by communication, it becomes possible to correct the performance delay caused by network communication without using the techniques shown in Patent Documents 8 and 9.

[0015] Furthermore, in cases where the reader terminal device outputs sound using an external speaker via Bluetooth connection, for example, the delay of data transfer from the reader terminal device to the external speaker may be longer than the delay of network communication due to the performance. However, even in such cases, the present invention can address this by setting a delay in the reading position of the musical score played by the member terminal device.

[0016] In this way, it became possible to automatically synchronize the timing and speed of each member terminal's performance with the leader terminal's performance. However, the objective of allowing the member terminal players to easily manually play in sync with the leader terminal's performance has not yet been achieved.

[0017] Furthermore, the technologies described in Patent Documents 10 and 11 require means to predict movement states and means to notify users of operation information input from other devices using information other than visual / auditory information, in order to synchronize timing when performing ensemble playing in real time using network functions. This has the drawback of complicating the device, and there is a need for a simpler method to achieve this.

[0018] Furthermore, Patent Document 12 discloses a technique for displaying a standard musical staff using multiple layers and overlaying each layer. However, even if this technique is used to overlay the performance of the leader terminal device onto the member terminal device, as already mentioned, what is displayed on the display of the mobile terminal targeted by this invention is only thin lines of different positions and lengths moving from top to bottom of the screen. Since the musical scores played by the leader and the musical scores played by the members are usually different, this was not sufficient for the members to visually recognize whether the timing and speed of their performance matched or were out of sync with the leader's performance. [Prior art documents] [Patent Documents]

[0019] [Patent Document 1] Japanese Patent Application No. 2020-210850 [Patent Document 2] Japanese Patent Application No. 2021-096161 [Patent Document 3] Japanese Patent Publication No. 2016-521381 [Patent Document 4] Japanese Unexamined Patent Application Publication No. 6-35456 [Patent Document 5] Japanese Unexamined Patent Application Publication No. 2005-258267 [Patent Document 6] Japanese Patent No. 3277875 [Patent Document 7] Japanese Unexamined Patent Application Publication No. 2001-356764 [Patent Document 8] Japanese Unexamined Patent Application Publication No. 2005-195982 [Patent Document 9] Japanese Unexamined Patent Application Publication No. 2007-178860 [Patent Document 10] Japanese Unexamined Patent Application Publication No. 2006-162904 [Patent Document 11] Japanese Unexamined Patent Application Publication No. 2005-250148 [Patent Document 12] Japanese Unexamined Patent Application Publication No. 2009-230007 [Summary of the Invention] [Problems to be Solved by the Invention]

[0020] The problem to be solved by the present invention is that when performing an ensemble using a music performance device such as a plurality of portable terminal devices equipped with music score input means for reading electronically represented music score data, event detection means for detecting performance events generated along with the performance of a performer, and performance position determination means for determining the performance position of the music score data from the events detected by the event detection means, it is to provide a method and a device that enable a device set as a member of the ensemble to easily synchronize with the performance of a device set as the leader of the ensemble. [Means for Solving the Problems]

[0021] The present invention is characterized by comprising: a first display means that, in one device set as the leader of the ensemble by the leader setting means, transmits performance position data determined by the performance position determination means via a communication means provided in the same device; in at least one other device set as a non-leader by the leader setting means, uses the performance position data received by the communication means as the display position of reference performance score data by the display position determination means, and continuously displays the reference performance score data as a score image; and a second display means that displays the score image displayed in accordance with the performance of the performer in the device set as a non-leader, superimposed on the reference performance score image so that it is visible, continuously and independently of the first display means. [Effects of the Invention]

[0022] According to the present invention, the first display means continuously displays sheet music data images that the performer should play based on performance position data transmitted from the reader terminal device by the reader, so that the displayed images can be used as a reference for the performance. The second display means continuously overlays sheet music images of the performer's own performance using the same sheet music data onto the reference sheet music images, so that the performer can visually recognize how far ahead or behind their own performance is compared to the reference performance.

[0023] In this way, if a performer can recognize how far ahead or behind their own performance is compared to the reference performance, they can continuously adjust their own performance speed based on the results of that recognition, making it possible to synchronize the timing and speed of the member terminals' performances with the leader terminal's performance in an ensemble setting. [Brief explanation of the drawing]

[0024] [Figure 1] Figure 1 is an explanatory diagram showing the state when acting as the leader of the ensemble in the first implementation method. (Example 1) [Figure 2]Figure 2 is an explanatory diagram showing the state when acting as a member of an ensemble in the first implementation method. (Example 1) [Figure 3] Figure 3 is an explanatory diagram showing the hardware and software configuration in Example 1. [Figure 4] Figure 4 is a flowchart showing the operation of the first software in Example 1. [Figure 5] Figure 5 is a flowchart showing the operation of the second software in Example 1. [Figure 6] Figure 6 is a flowchart showing the operation of the third software in Example 1. [Figure 7] Figure 7 is an explanatory diagram illustrating a typical musical score. [Figure 8] Figure 8 is an explanatory diagram illustrating the digital musical score image in Example 1. [Figure 9] Figure 9 is the first explanatory diagram illustrating the overlay of digital musical score images in Example 1. [Figure 10] Figure 10 is a second explanatory diagram illustrating the overlay of digital musical score images in Example 1. [Figure 11] Figure 11 is a third explanatory diagram illustrating the overlay of digital musical score images in Example 1. [Figure 12] Figure 12 is a flowchart showing the operation of the second software in Example 2. [Modes for carrying out the invention]

[0025] The objective of providing a method and apparatus for performing an ensemble using multiple musical performance devices, each equipped with a musical score input means for reading electronically represented musical score data, an event detection means for detecting performance events that occur during performance, and a performance position determination means for determining the performance position of the musical score data from the events detected by the event detection means, is to enable the timing and speed of performances by devices set as members of an ensemble to easily match the performance of a device set as the leader. This is achieved by providing a first display means that continuously displays the reference musical score data as a musical score image, using a display position determination means, and a second display means that continuously displays the musical score data displayed in accordance with the performance of the performer in the device set as the non-leader, superimposed on the reference musical score image displayed by the first display means, so that it is visible. [Examples]

[0026] Figure 1 is an explanatory diagram showing the state when the device of the present invention is operating as the leader of an ensemble in the first method of implementation, where 100 is the smartphone body and 101 is the image output unit using a liquid crystal panel. The liquid crystal panel is a touch panel and also serves as an information input unit 102 for user operation. The area indicated by the dashed frame 1011 is a musical score image display unit that is read appropriately from the first musical score input means in the smartphone body and displays the movement of the musical score in real time according to the progress of the performance. The dashed line 1020 is a virtual timing line indicating the performance position, and it indicates that a sound is produced when the note is on that line. The black lines shown at 2005, 2006, and 2007 (there are other similar black lines, but they are omitted for explanation) are musical notes in the musical score image. 2010 and 2011 are marker lines recorded at both ends of the musical score image for detecting the position of the notes displayed on the panel 101. In this embodiment, these marker lines are used in the image processing means described later, but they are not essential for producing sound from notes using digitally recorded sheet music. The area indicated by the dashed frame 1012 is the keyboard display section. When there is a note at the playing position, the white and black keys are displayed in gray to indicate that the corresponding key has been pressed. When the note that was at the playing position moves away over time, the white and black keys are displayed in their original colors to indicate that the corresponding key has been released. 300 is a single finger used for gesture operation on the liquid crystal panel 101, and 301 is an arrow that simulates the rotational gesture made by that finger 300. When finger 300 is rotated in the direction of arrow 301, the notes indicated by the black lines 2005, 2006, and 2007 move in the direction of arrow 302 according to the rotation speed. When a note comes onto the virtual timing line 1020, which is the performance position, it produces a sound, and when a note moves off the timing line 1020, it stops producing a sound, thus achieving performance. 3000 schematically represents that this device is communicating with other devices that are members of the ensemble via data communication means, and in this embodiment, it transmits performance position information that accompanies the progress of the performance through gesture control.

[0027] Figure 2 is an explanatory diagram showing the state of the device of the present invention when it is operating as a member of an ensemble in the first method of implementation. The device itself is the same as in Figure 1, so the physical configuration is the same, but the operation is different, so the image displayed on the score image display unit 1011, which is indicated by the dashed frame on the liquid crystal panel 101, is different from that in Figure 1. Also, 3000 schematically represents that the device is communicating with the device that is acting as the leader of the ensemble via a data communication means, similar to Figure 1, and in this case, it is receiving performance position information transmitted by the other device that is acting as the leader of the ensemble. The black lines shown in 2001, 2002, and 2003 are notes on a reference score image that are appropriately read and displayed from the second score input means based on performance position information received from another device acting as the leader of the ensemble. The white lines shown in 2001-2, 2002-2, and 2003-2 are notes on a score image played by a performer that are appropriately read and displayed from the first score input means based on rotational gestures made by finger 300. These are displayed by inverting the original score image to black and white and setting its transparency to 75%, overlaying it on top of the reference score image. Both the black and white lines move in the direction of arrow 302 as the performance progresses. However, the events that trigger their movement are independent of each other: the former is performance position information received via data communication from a device acting as the leader of the ensemble, while the latter is an event based on the performer's own gestures. When a note represented by the white line comes onto the virtual timing line 1020, which is the performance position, the sound is produced, and when the note moves off the timing line 1020, the sound is silenced, thus realizing the performer's participation as a member of the ensemble.

[0028] Figure 3 is a configuration diagram showing the hardware and software of an embodiment of the present invention. 1 is a CPU (Central Processing Unit), which realizes the present invention by performing various processes in various means according to a program stored in non-volatile memory (not shown). 2 is a leader setting means for setting whether the present invention's device operates as a leader in an ensemble or as a member (non-leader) of an ensemble. This setting is configured from the device's various operation setting screens (not shown), and the setting is stored in non-volatile memory (not shown). 3 is a gesture event detection means (abbreviated as gesture detection means) which is a performance event detection means in this embodiment. It recognizes a rotational gesture made by a finger 300 on the touch panel 102 and detects the position coordinates (x,y) of the finger 300 on the touch panel 102 at regular time intervals. From the detected position coordinates (x,y) of the finger 300, the CPU 1 detects the rotational speed using a rotational speed detection means 4 configured by a software program. The method for detecting the rotational speed is disclosed in Patent Document 2 and is therefore omitted here.

[0029] Similarly, 5 is a performance position determination means configured by a software program, and similarly, it uses the method disclosed in Patent Document 2 to determine the performance position of the musical score from the rotation speed detected by the CPU 1. 6 is a first musical score input means for reading musical score image data from the musical score data electronically stored in a musical score storage means (not shown) within the main body of the device, which is composed of non-volatile memory, by trimming it to the required size (details will be described later) based on the performance position determined by the performance position determination means 5, and then displaying it to the required size (details will be described later) for the image processing means 7, the first display means 12, or the second display means 13.

[0030] 7 is an image processing means which identifies the position of a note at the performance timing (corresponding to the timing line 1020) from the musical score image data read by the first musical score input means 6 using the method shown in Patent Documents 1 and 2, and detects whether or not there is a note at the identified position. 11 is a sound output means which, based on the information of whether or not there is a note at the specific position of the performance timing detected by the image processing means 7 and its history, has the function of playing or stopping the sound of the corresponding note with a preset timbre and volume, according to the CPU 1's decision to play or stop the sound of the corresponding note.

[0031] 8 is a data communication means that transmits and receives data via a network between multiple devices having the same configuration as the present invention. For example, when the present device is set as the leader of the ensemble by the leader setting means 2, the data communication means 8 transmits the performance position determined by the performance position determination means 5 as performance position data to the devices set as members of the ensemble sequentially. On the other hand, when the present device is set as a member of the ensemble by the leader setting means 2, it receives the performance position data transmitted from the device set as the leader and uses that data as input to the display position determination means 9, which will be described later.

[0032] 9 is a display position determination means, which, when the device is set as a member of an ensemble, is used to display the sheet music data electronically stored in the sheet music storage means within the device on the liquid crystal panel 101 within the device, based on the performance position data of the ensemble leader received by the data communication means 8. The received performance position data of the ensemble leader becomes the display position of the sheet music data. 10 is a second sheet music input means, similar to the first sheet music input means 6, for reading image data from the sheet music data electronically stored in the sheet music storage means within the device, after trimming it to the size required for the first display means 12 (described later) based on the display position of the sheet music determined by the display position determination means 9.

[0033] 12 is the first display means, and based on the settings in the leader setting means 2, if this device is set as the leader of the ensemble, it displays the score data image read by the first score input means 6, and if this device is set as a member of the ensemble, it displays the score data image read by the second score input means 10 directly on the score image display unit 1011 on the liquid crystal panel 101.

[0034] 13 is a second display means, which operates independently of the operation 12 of the first display means. Only when this device is set as a member of the ensemble based on the settings in the leader setting means 2, it overlays the score data image read by the first score input means 10 onto the score image display unit 1011 on the liquid crystal panel 101 so that the image already displayed by the first display means 12 is visible.

[0035] Figure 4 is a flowchart showing the operation of the first software of the present invention, illustrating the operation when the device is set as the leader in an ensemble performance by the leader setting means 2 in this embodiment.

[0036] First, in step S1, CPU1 waits for a gesture event to be detected by gesture detection means 3. In this embodiment, when a gesture operation by finger 300 begins, an event occurs at intervals of 1 / 60th of a second, and the position where finger 300 first touches the panel is set as the origin O:(x,y)=(0,0), and thereafter, the relative coordinates from the origin are detected each time until finger 300 leaves the panel. If a gesture is detected, the process proceeds to step S2, where rotation speed detection means 4 determines the speed of rotation caused by the gesture.

[0037] The method for determining the rotation speed by the velocity detection means 4 from the position coordinates of finger 300 on the panel due to a gesture event is disclosed in Patent Document 2, so it will be omitted here.

[0038] If the rotation speed is determined by the rotation speed detection means 4 through the gesture operation of the finger 300, the CPU 1 determines the performance position in step S3 using the performance position determination means 5.

[0039] This section describes the digital representation of musical scores as musical score images. Figures 5 and 6 illustrate how a typical musical score is represented and played as a digital musical score image according to this invention. Figure 5 shows a typical musical score, where 2004, 2005, 2006, 2007, and 2008 represent quarter notes of different pitches. The symbol 400 is a tempo marking, indicating that the score should be played at a speed of 80 quarter notes per minute. Figure 6 shows the beginning of the digital musical score image obtained by converting the musical score shown in Figure 5 into a digital image. Its coordinates are with the positive x-axis on the right and the positive y-axis on the top, and the notes contained within the ellipse shown at 401 in Figure 5 are shown as 2004, 2005, 2006, 2007, and 2008. 2010 and 2011 are marker lines recorded at both ends of the digital musical score image for note position detection. In this embodiment, the width of the musical score image is 384 pixels, the width of each note is 2 pixels, and the gap between adjacent notes is 1 pixel. Since 85 notes (= 7 octaves [1 octave includes 12 notes including semitones] + 1 note) can be played, the space between marker lines 2010 and 2011 is 256 pixels = (2 + 1) × 85 + 1 (the + 1 is the gap between the marker lines). In this embodiment, marker lines 2010 and 2011 are placed on the musical score image to detect the position of the notes, but these marker lines 2010 and 2011 are not essential if the position of a given note on the x-axis is predetermined. Although Figure 6 only shows the beginning of the musical score shown in Figure 5, the actual musical score data is an image of the entire musical score which is electronically stored in the musical score storage means within the device.

[0040] In Figure 6, quarter note 2004 is located three octaves above the lowest note in this embodiment, so it corresponds to the 36th note (= 12 notes × 3 octaves) from the lowest note. Therefore, there is a gap of 109 pixels = (2 + 1) × 36 + 1 between it and marker line 2010 in Figure 6. Similarly, there are gaps of 115, 121, and 124 pixels between quarter notes 2005, 2006, and 2007 and marker line 2010, respectively.

[0041] Furthermore, the lengths in the y-axis direction of quarter notes 2004, 2005, 2006, 2007, and 2008 in Figure 6 correspond to the length of a quarter note. In this embodiment, the distance from the beginning of quarter note 2004 to the beginning of quarter note 2005 is 180 pixels. Similarly, the distances between 2005 and 2006, 2006 and 2007, and 2007 and 2008 are also 180 pixels. Although not shown here, the length of an eighth note would be half that, 90 pixels; a sixteenth note would be 45 pixels; and a half note would be 360 ​​pixels.

[0042] In Figure 5, the performance of the musical score progresses from left to right, as indicated by arrow 500, following the score. On the other hand, in the case of the musical score image, it progresses from bottom to top, as indicated by arrow 500 in Figure 6. The dashed line 501 in Figures 5 and 6 indicates the performance position, and by advancing this dashed line in the direction of arrow 500 at regular time intervals, the performance on the musical score and musical score image will progress. Therefore, the distance that the dashed line 501 advances is determined according to the rotation speed at the timing of the gesture event (1 / 60 second interval in this embodiment). For example, in this embodiment, the tempo marking 400 in Figure 5 indicates playing at a speed of counting quarter notes 80 times per minute, so on the digital musical score image in Figure 6, it should advance 14,400 pixels (= 180 pixels × 80 times) per minute. Therefore, since gesture events occur at intervals of 1 / 60th of a second, following the instruction for speed symbol 400, we can see that we need to advance 4 pixels (= 14400 pixels / 60 seconds / 60 times) for each gesture event. On the other hand, the rotation speed V obtained in step S2 has been measured and found to have an approximate average value of 15.0, so if we find the constant C such that the number of pixels is 4 when the speed V is 15.0, we get 4 pixels = 15.0 × C, so C = 4 / 15.0 = 0.2667. Therefore, if N is the number of pixels that move when the rotation speed is V,

number

[0043] Once the performance position is determined in step S3, the process proceeds to step S4, where the image data is trimmed and read from the first score input means 6 based on the performance position determined in step S3. Trimming and reading means partially reading the score image based on the performance position so that it can be displayed on the score display unit 1011 on the liquid crystal panel 101. For example, if the performance position is indicated by the dashed line 501 in Figure 6, then only the portion enclosed by the dashed line 502, which is 384 pixels vertically and 384 pixels horizontally, with the dashed line placed in the vertical center, is read.

[0044] In step S4, the sheet music data read out as trimmed image data is processed by the image processing means 7 in the next step S5 to determine the position of each of the 85 notes at the performance position (on the dashed line 501 in Figure 6) determined in step S3, and to detect the presence or absence of notes at each position. The details of the image processing performed by the image processing means 7 in this embodiment to determine the position of each note from the sheet music data and to detect the presence or absence of notes at each determined position are as described in Patent Document 1.

[0045] Based on the information about the presence or absence of musical notes at each position detected in step S5, in step S6 the CPU 1 compares this information with the information about the presence or absence of musical notes at each position detected during the previous gesture event, which is stored in a memory means (not shown), to determine whether a new musical note has appeared at each position or whether a previously appearing musical note has disappeared. Based on this determination, the CPU 1 instructs the sound output means 11 to play or stop the sound of the musical note corresponding to each position at a predetermined timbre and volume.

[0046] Once the instructions to the sound output means 11 are finished, the process proceeds to step S7, where the CPU 1 passes the musical score data read in step S4 to the first display means 12, and the first display means 12 displays the passed musical score data image as is on the musical score image display unit 1011 on the liquid crystal panel 101.

[0047] Finally, in step S8, the CPU1 transmits the performance position determined in step S3 as performance position data to multiple other devices having the same configuration as the present invention, which are connected to the network via the data communication means 8 and are set as members of the ensemble.

[0048] With the above steps complete, the series of processes resulting from the occurrence of a gesture event are finished, so we return to S1 and wait for the gesture detection means 7 to detect another gesture event.

[0049] In step S8, performance position data is transmitted via the data communication means 8, but volume and other information may be added and transmitted simultaneously. Also, in step S1, the following steps proceed triggered by a gesture event from the gesture detection means 3, but for example, an event triggered by a timer that occurs at 1 / 60 second intervals may be used, and the following steps may proceed with the rotation speed determined in step S2 remaining constant. In this way, the leader's performance in an ensemble can be automated.

[0050] Figure 7 is a flowchart showing the operation of the second software of the present invention in the same embodiment, and shows the operation when the device is set as a member in an ensemble by the leader setting means 2 in the same embodiment, and data is received by the data communication means 8.

[0051] First, in step S9, CPU1 waits for data to be received from data communication means 8. If data is received by data communication means 8, the process proceeds to step S10, where display position determination means 9 determines the display position of the musical score data from the received data. At this time, the data received from communication means 8 is performance position data of another device that has been set as the leader in the ensemble. Therefore, this value is used as the reference display position of its own musical score data, and a correction value is added or subtracted to account for delays in inter-device communication in the network. The resulting value is determined as the display position of the musical score, i.e., the performance position that serves as the reference for its own performance.

[0052] Once the display position of the musical score data is determined in step S10, the process proceeds to step S11, where image data is trimmed and read from the second musical score input means 10 based on the display position determined in step S10. This reading method is similar to the method of trimming and reading image data from the first musical score input means 6 described in step S4. In step S4, the image data was trimmed and read based on the performance position determined in step S3, but in step S11, the musical score data is trimmed and read based on the display position of the musical score data determined in step S10.

[0053] The process then proceeds to step S12, where the CPU1 passes the musical score data read in step S11 to the first display means 12, and the first display means 12 displays the passed musical score data image as is on the musical score image display unit 1011 on the liquid crystal panel 101.

[0054] As described above, the series of processes that occur when this device is set as a member in an ensemble and receives data via the data communication means 8 are completed, so the process returns to S9 and waits for the data communication means 8 to receive data again.

[0055] Figure 8 is a flowchart showing the operation of the third software in this embodiment of the present invention. This software operates within the same device in parallel with the operation of the second software shown in Figure 7 when the device is set as a member in an ensemble by the leader setting means 2 in this embodiment, and it shows the operation when a gesture event is detected by the gesture detection means 3.

[0056] First, in step S13, CPU1 waits for a gesture event to be detected by the gesture detection means 3. If a gesture is detected, the process proceeds to step S14, where the rotation speed detection means 4 determines the rotation speed caused by the gesture.

[0057] If the rotation speed is determined by the rotation speed detection means 4 through the gesture operation by the finger 300, the CPU 1 determines the performance position using the performance position determination means 5 in step S15.

[0058] Once the performance position is determined in step S15, the process proceeds to step S16, where image data is trimmed and read from the first score input means 6 based on the performance position determined in step S15.

[0059] In step S16, the sheet music data read out as trimmed image data is processed by the image processing means 7 in the next step S17 to determine the position of each of the 85 notes at the performance position determined in step S15, and to detect the presence or absence of notes at each position.

[0060] Based on the information detected in step S17 regarding the presence or absence of musical notes at each position, in step S18 the CPU 1 compares this information with the information stored in the memory means regarding the presence or absence of musical notes at each position when the previous gesture event occurred to determine whether a new musical note has appeared at each position or whether a previously appearing musical note has disappeared. Based on this determination, the CPU 1 instructs the sound output means 11 to play or stop the sound of the musical note corresponding to each position with a predetermined timbre and volume. These operations from steps S13 to 18 are identical to the operations from steps S1 to S6 of the operation of the first software shown in Figure 4.

[0061] If the instruction to the sound output means 11 is completed in step S18, the process proceeds to step S19, where the musical score data read out as trimmed image data in step S16 is passed to the second display means 13. Here, the second display means 13, independently of the first display means 12, overlays and displays the passed musical score data image so that the musical score data image already displayed on the musical score image display unit 1011 on the liquid crystal panel 101 by the first display means is visible.

[0062] Here, we will explain how to display the superimposed sheet music data images. Figure 9 shows how to superimpose the sheet music image data read in step S11 and the sheet music image data read in step S16. In the figure, 2100 is the sheet music image data read in step S11, 2101 is the sheet music image data read in step S16 with the colors inverted and the transparency set to 75%, and 2102 is image data with a virtual timing line 1020 indicating the performance position drawn on a transparent background. Figures 10 and 11 show these superimposed in the order of 2100 → 2101 → 2102 from bottom to top.

[0063] In Figure 10, the black lines labeled 2001, 2002, 2003, and 2005 represent notes in the musical score image data read in step S11. These are reference notes for performance, displayed based on the performance position of the reader terminal device received by the data communication means. On the other hand, the white lines labeled 2001-2, 2002-2, 2003-2, and 2005-2 represent notes in the musical score image data read in step S16 in accordance with the performer's gesture operations. Lines 2001, 2002, 2003, and 2005 advance independently in the direction of the arrows indicated by 302 in the figure, following the progress of the performance on the reader terminal device, while lines 2001-2, 2002-2, 2003-2, and 2005-2 advance independently in the direction of the arrows indicated by 302 in the figure, following the progress of the performer's own performance on the member terminal device. Incidentally, since notes 2001, 2002, 2003, and 2005, as well as notes 2001-2, 2002-2, 2003-2, and 2005-2, were originally read from the same musical score data, if the timing and speed of the performance on the reader terminal device and the performer's own performance on the member terminal device match, they should overlap and progress in the same place. However, in Figure 10, the white notes are behind the black notes in the direction of progression, indicating that the notes 2001-2, 2002-2, 2003-2, and 2005-2, which display the performer's performance, are behind the reference notes 2001, 2002, 2003, and 2005. Similarly, in Figure 11, the white notes are ahead of the black notes in the direction of movement, indicating that the notes 2001-2, 2002-2, 2003-2, and 2005-2, which represent the performer's playing, are ahead of the reference notes 2001, 2002, 2003, and 2005. By overlaying the score image data representing the performer's playing with the score image data representing the reference playing and displaying them sequentially, it becomes possible to visually recognize whether the performer's playing is ahead or behind the reference playing.

[0064] As a result of the above, the series of processes resulting from the occurrence of a gesture event by the third software is completed, so we return to step S13 and wait for a gesture event to be detected by the gesture detection means 7.

[0065] Furthermore, when the performance position is determined by the performance position determination means 5 in step S15, the gesture detection means 3 uses the position where the finger 300 first touches the panel as the origin, and detects events at intervals of 1 / 60 seconds until the finger 300 leaves the panel. For the first and last events, it is also possible to detect information indicating the start and end. Therefore, in determining the performance position in step S15 in a processing flow where a gesture start event occurs, instead of simply setting the performance position to 0 pixels, the display position of the reference image determined by the display position determination means 9 immediately before that point is copied and used as the initial value of the performance position. Subsequently, the performance start position in step S15 for each gesture event detection is determined based on this value, making it easier for the performer to synchronize the timing of the start of their performance.

[0066] As described above, this embodiment provides a first display means for displaying a reference performance score image and a second display means for overlaying a performer's performance score image onto a reference performance score video image. By independently displaying both images in accordance with the progress of the performance, it becomes possible to visually recognize whether the performer's own performance is ahead or behind the reference performance. Furthermore, by adjusting the rotation speed of the performer's own performance image using gesture control to match the reference performance image, it becomes possible to easily realize ensemble playing with multiple devices, which was previously difficult.

[0067] In this embodiment, the gesture was described as a single-point rotation gesture using a finger 300 on a touch panel, but any single-point gesture using a stylus or the like is acceptable as long as the gesture can be recognized.

[0068] Furthermore, in this embodiment, the image processing means 7 is used to detect musical notes from a digital musical score image through image processing, but the musical score data can be digital data such as MIDI data instead of a digital image. For example, in the case of MIDI data, the data is stored in groups called tracks in the form of delta time and event repetitions, where delta time is the time obtained by specifying the length of a quarter note and the unit of time separately, and represents the elapsed time from the previously specified delta time, and events specify the pitch, volume, and on or off of the note at that time, so if the performance position is represented by time instead of the number of pixels, and the MIDI data is converted into a digital musical score image as shown in this invention and displayed, the exact same function can be realized. Moreover, in this embodiment, performance position data transmitted from the device that is the leader of the ensemble using the data communication means 8 is used as a reference, but if periodic events are generated by a timer within the device that is a member of the ensemble, thereby replacing step S9 in the flowchart showing the operation of the second software in Figure 7, and processing equivalent to S17 and S18 in the flowchart showing the operation of the third software in Figure 8 is added after step S12, then an ensemble by a single performer within the same device can also be achieved.

[0069] Figure 12 shows a second embodiment of the present invention, in which steps S20, S21, and S22 are added between steps S11 and S12 of the flowchart showing the operation of the second software in Figure 7 of Embodiment 1. Since the gesture detection means 3 can identify the start and end events of a gesture, in step S20 it is determined whether the device is performing a gesture operation. If it is performing a gesture, it proceeds to step S17 as in Figure 7, and if it is not performing a gesture, it proceeds to step S21 and step S21. Steps S21 and S21 are the same processes as steps S17 and S18 of the flowchart showing the operation of the third software in Figure 8. By adding this step, the device set as a member of the ensemble by the leader setting means 3 will operate as an automatic playing device that automatically plays according to the playing of the ensemble leader when the performer is not performing a gesture operation. In this case, if the data received via the data communication means 8 includes data other than the playing position data, it is possible to use that data as well. For example, if volume information is included, it is possible to reflect that in step S22. [Industrial applicability]

[0070] By detecting rotational speed using electrical signals instead of gestures to determine the performance position, and then playing the musical score based on the determined position, this system can be applied not only to mobile devices but also to personal computer-based devices and standalone music performance devices. Furthermore, the operation of visually matching the video image of the musical score can also be used as a game. [Explanation of Symbols]

[0071] 1 CPU (Central Processing Unit) 2. Leader setting means 3. Gesture detection means 4. Means for determining the speed of rotation 5 Performance position determining means 6. First method of inputting musical scores 7 Image processing means 8. Data communication methods 9 Display position determining means 10. Second method of inputting musical scores 11. Sound output means 12. First means of display 13. Second means of display

Claims

1. A music performance device comprising a music score input means for reading electronically represented music score data, an event detection means for detecting performance events, and a performance position determination means for determining the performance position of the music score data from the events detected by the event detection means, wherein the device further comprises a first display means for continuously displaying the music score data of a reference performance as a music score image, and a second display means for continuously displaying the same music score data, which is displayed in accordance with the performer's performance, superimposed on the music score image of the reference performance so that it is visible, and independently of the first display means, wherein, as the reference performance, one device set as the leader of the ensemble by a leader setting means transmits the performance position data determined by the performance position determination means via a communication means provided in that device, and at least one other device set as a non-leader by the leader setting means uses the performance position data received by the communication means as the display position of the music score data of the reference performance by a display position determination means.

2. The music performance device according to claim 1, characterized in that the event detection means is a gesture detection means that recognizes a rotation operation on a touch panel as a gesture.

Images