Performance part tracking device and program of the same

Abstract

To provide a performance part tracking device which can correctly track a performance part and reduces a burden on a player.SOLUTION: A performance part tracking device 2 includes: a musical score image analysis unit 11 for performing musical score analysis processing on a musical score image and generating a marker for performance part tracking; a performance operation analysis unit 12 for detecting timing when a player plays a next note by performing performance operation analysis processing on a player image; and a performance part determination unit 13 for determining a performance part from a timing at which the performance operation analysis unit 12 detects and synthesizing the markers.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a performance portion tracking device and a program therefor. [Background technology]

[0002] Camerawork and switching in music programs are done by following the progression of the music. In music with lyrics, such as pop and enka, the lyrics can be tracked, so camerawork and switching are relatively easy. On the other hand, in music without lyrics, such as jazz and classical music, the progression of the music must be understood mainly by counting bars and beats and the number of seconds, making camerawork and switching more difficult. Orchestral music in particular involves many different instruments and often lasts for several tens of minutes, so even experienced musicians can lose track of the progression of the music.

[0003] Therefore, methods have been proposed for tracking the playing part of the score. For example, Non-Patent Document 1 proposes a method that combines keystroke information of a piano or the like with gaze information. In addition, Non-Patent Document 2 proposes a method that predicts the playing position based on the local tempo of the performance using a probabilistic model that represents the fluctuation of the performance. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] “Music score tracking system using eye gaze information”, [online], [searched on September 6, 2023], Internet <URL:https: / / hiratakelab.jp / %E7%A0%94%E7%A9%B6%E3%83%86%E3%83%BC%E3%83%9E / %E8%AA%8D%E7%9F%A5%E5%BF%83%E7%90%86 / %E8%A6%96%E7%B7%9A%E6%83%85%E5%A0%B1%E3%82%92%E6%B4%BB%E7%94%A8%E3%81%97%E3%81%9F%E6%A5%BD%E8%AD%9C%E8%BF%BD%E8%B7%A1%E3%82%B7%E3%82%B9%E3%83%86%E3%83%A0 / > [Non-Patent Document 2] “Performance Tracking and Automatic Accompaniment System”, [online], [searched on September 6, 2023], Internet<URL:https: / / sakoweb.net / joomla3 / research / topics / score-following.html> Summary of the Invention [Problem to be solved by the invention]

[0005] The techniques described in the above-mentioned non-patent documents 1 and 2 record actual performances, so they are susceptible to the effects of differences in reverberation between halls and fluctuations in performance, and may not be able to accurately track the performance part. Furthermore, in the technique described in non-patent document 1, in order to obtain the player's line of sight information, a camera must be installed in a position where the player's eyeball can be photographed, or the player must wear measurement goggles, which places a heavy burden on the player.

[0006] An object of the present invention is to provide a performance part tracking device and a program therefor that can accurately track a performance part and reduce the burden on a player. [Means for solving the problem]

[0007] In order to solve the above problems, the performance point tracking device of the present invention is a performance point tracking device that tracks the performance points on a musical score by using a musical score image obtained by photographing the musical score and a player image obtained by photographing a player playing an instrument, and is configured to include a musical score image analysis unit, a playing action analysis unit, and a performance point determination unit.

[0008] According to this configuration, the score image analysis unit performs score analysis processing on the score image to determine the coordinates of the staff contained in the score and the coordinates and order of the notes contained in the score, and generates markers for tracking the performance point based on the coordinates of the staff and the coordinates and order of the notes. The performance motion analysis unit detects the timing at which the player plays the next note by performing a performance motion analysis process on the player image. The played portion determination unit determines the notes being played by the player as played portions from the timing and note order detected by the performance motion analysis unit, and combines a marker with the determined played portion.

[0009] In this way, the performance part tracking device does not need to record the actual performance, but only uses images such as the score image and the player image. Therefore, the performance part tracking device can accurately track the performance part without being affected by the difference in reverberation between halls or the fluctuation of the performance. Furthermore, since the performance location tracking device uses the player's line of sight information, there is no need to install a camera in a position where the player's eyes can be photographed, nor is there a need for the player to wear measurement goggles, reducing the burden on the player.

[0010] The present invention can also be realized by a program for causing a computer to function as the above-mentioned performance portion tracking device. Effect of the Invention

[0011] According to the present invention, the performance part can be accurately tracked, and the burden on the player can be reduced. [Brief description of the drawings]

[0012] [Figure 1]FIG. 1 is a schematic diagram of a performance portion tracking system according to an embodiment. [Diagram 2] 1 is a block diagram showing a configuration of a performance portion tracking device according to an embodiment; [Diagram 3] 4 is a flowchart showing the operation of the performance portion tracking device according to the embodiment. [Figure 4] 11 is a flowchart of advance preparation in an embodiment. [Diagram 5] 1 is a flowchart of a musical score analysis process according to an embodiment. [Figure 6] FIG. 1A is an explanatory diagram for explaining the elimination of vertical lines, and FIG. 1B is an explanatory diagram for explaining the elimination of horizontal lines in an embodiment. [Figure 7] 11 is a flowchart of a staff coordinate estimation process in an embodiment. [Figure 8] 11 is a flowchart of a process for estimating coordinates of a musical note and a treble clef in an embodiment. [Figure 9] FIG. 11 is an explanatory diagram for explaining erasure of vertical and horizontal lines in the embodiment. [Figure 10] 6(a) to 6(c) are explanatory diagrams illustrating the estimation of coordinates of notes in an embodiment. [Figure 11] FIG. 11 is an explanatory diagram for explaining treble clef coordinate estimation in the embodiment. [Figure 12] 11(a) to 11(d) are explanatory diagrams illustrating note order estimation in an embodiment. [Figure 13] 11 is a flowchart of a note order estimation / treble clef determination process in an embodiment. [Figure 14] FIG. 11 is an explanatory diagram of marker generation in an embodiment. [Figure 15] 11 is a flowchart of a marker generation process according to an embodiment. [Figure 16] 13 is a flowchart of performance tracking and camera switching in an embodiment. [Figure 17] FIG. 11 is an explanatory diagram for explaining a motion analysis of a player in an embodiment. [Figure 18] 11 is a flowchart of a performance action analysis process according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the embodiment described below is for embodying the technical idea of ​​the present invention, and unless otherwise specified, the present invention is not limited to the following. In addition, the same symbols are used for the same means, and the description may be omitted.

[0014] (Outline of the performance tracking system) With reference to FIG. 1, an overview of a performance portion tracking system 1 according to an embodiment will be described. As shown in FIG. 1, the performance part tracking system 1 tracks the parts played by an orchestra and automatically performs camera work according to the parts played. In order to present the parts played to a user of the performance part tracking system 1, the performance part tracking system 1 colors the parts that have already been played on the musical score, like lyrics in karaoke that color the parts that have already been sung. Specifically, the performance part tracking system 1 prepares markers (e.g., colored rectangles) according to the number of notes included in the musical score, and synthesizes the markers on the musical score at appropriate coordinates, order, and timing. At this time, the performance part tracking system 1 obtains the coordinates and order for synthesizing the markers by analyzing the musical score. In addition, since the player must perform some kind of action to play the instrument, the performance part tracking system 1 obtains the timing for synthesizing the markers by analyzing the player's movement.

[0015] Many instruments are used in an orchestra, but it is preferable that the instruments to be analyzed satisfy the following conditions 1 to 6. 1) To prevent losing track of the performance, instruments are used frequently. 2) Image processing is used, so the player can be seen 3) To make it easier to analyze the movements of the players, the players' movements are large. 4) To make it easier to analyze the movements of the players, the players’ movements have patterns. 5) There are few rests to allow for constant tracking of the performance 6) To enable detailed tracking of performance parts, many notes are used.

[0016] The violin is always played in the front row of the orchestra, the player's movements during the performance are large, the movements are somewhat fixed, and there are few rests and many notes. In other words, the violin satisfies the above conditions 1 to 6. Therefore, the performance part tracking system 1 takes the violin as an example of an analysis target.

[0017] As shown in FIG. 1, a performance part tracking system 1 includes a performance part tracking device 2 and a camera C (C1 to C3).

[0018] [camera] Camera C is a general camera for shooting, for example, music programs such as orchestras. Camera C also shoots score images and player images used by performance part tracking device 2. In this embodiment, the score image is an image of a violin score, and the player image is an image of a violinist.

[0019] In this embodiment, the performance portion tracking system 1 is equipped with three cameras C (C1 to C3). In this case, in the advance preparation described below, at least one of the cameras C may capture a score image. Also, in the performance tracking and camera switching described below, at least one of the cameras C may capture a player image.

[0020] Camera C outputs the score image to the score image analysis unit 11 of the performance portion tracking device 2, and outputs the player image to the performance motion analysis unit 12 (FIG. 2). Camera C also outputs the camera footage it has taken to the camera switching unit 30. In FIG. 1, the footage taken by camera C1 is referred to as camera 1 footage, the footage taken by camera C2 as camera 2 footage, and the footage taken by camera C3 as camera 3 footage.

[0021] The number of cameras C is not particularly limited. For example, the performance part tracking system 1 may include more than three cameras C. Furthermore, the performance part tracking system 1 may include a dedicated camera (not shown) for capturing sheet music images and player images in addition to the three cameras C. Furthermore, the performance part tracking system 1 may include only one camera C when switching between cameras C is not required.

[0022] [Performance Tracking Device] The performance part tracking device 2 tracks the performance part on the score by using a score image obtained by photographing the score and a player image obtained by photographing a player playing an instrument. The performance part tracking device 2 automatically switches the camera C according to the performance part.

[0023] (Configuration of the performance tracking device) The configuration of the performance portion tracking device 2 will be described in detail with reference to FIG. 2, the performance part tracking device 2 includes a performance part tracking unit 10, a camera switching destination / timing information storage unit (camera switching destination information storage unit) 20, and a camera switching unit 30. The performance part tracking unit 10 performs processing related to tracking of the performance part, and includes a musical score image analysis unit 11, a performance action analysis unit 12, and a performance part determination unit 13.

[0024] A score image is input to the score image analysis unit 11 from camera C (FIG. 1), and a player image is input to the performance motion analysis unit 12 from camera C. Furthermore, switching destination information for camera C, which will be described later, is input by the user to the camera switching destination / timing information storage unit 20. Camera images (camera 1 image to camera 3 image) are input to the camera switching unit 30 from camera C.

[0025] The score image analysis unit 11 performs score analysis processing on the score image to determine the coordinates of the staff and the coordinates and order of the notes contained in the score, and generates markers for tracking the performance points based on the coordinates of the staff and the coordinates and order of the notes.

[0026] Specifically, the score image analysis unit 11 generates a binary score image by binarizing the score image as a score analysis process (step S100 in FIG. 7). Next, the score image analysis unit 11 detects the staff by performing a logical sum between the horizontally moved score image obtained by translating the binary score image in the horizontal direction and the binary score image, thereby obtaining the coordinates of the staff (steps S110 to S140 in FIG. 7). Next, the score image analysis unit 11 obtains a logical sum between the vertically moved score image obtained by translating the binary score image in the vertical direction, the horizontally moved score image, and the binary score image, and obtains the coordinates of the notes by detecting the balls of the notes determined by the line intervals of the staff from the logical sum image representing the result of the logical sum (step S20 in FIG. 5). Next, the score image analysis unit 11 classifies the detected balls of the notes by lines of the score according to the staff and arranges them horizontally, thereby obtaining the order of the notes (step S30 in FIG. 5).

[0027] The playing motion analysis unit 12 detects the timing when the player plays the next note by performing a playing motion analysis process on the player image. Specifically, as the playing motion analysis process, the playing motion analysis unit 12 performs a skeleton estimation process on the player image to calculate a motion vector of the playing motion, and detects the timing when the direction of the calculated motion vector is reversed as the timing when the player plays the next note (step S500 in FIG. 16).

[0028] The played part determination unit 13 determines the notes being played by the performer as played parts from the timing and note order detected by the playing motion analysis unit, and combines markers with the determined played parts (steps S510 and S530 in FIG. 16). The played part determination unit 13 may also present the user with a musical score image with a marker combined with the played part.

[0029] The played portion determination unit 13 may not combine markers with notes that have finished being played, but may combine markers with notes that are still being played. The played portion determination unit 13 may also combine markers with notes that have finished being played and notes that are still being played. The method of combining markers can be manually set by the user of the played portion tracking device 2.

[0030] The camera switching destination / timing information storage unit 20 stores in advance switching destination information of camera C in association with a specific note included in the musical score. The switching destination information is information that indicates the switching destination and switching timing of camera C, and is manually input by a user to the performance part tracking device 2 using operation means such as a mouse or keyboard (not shown). The camera switching destination / timing information storage unit 20 then stores the switching destination information in a memory (not shown) and outputs it to the performance part determination unit 13 and the camera switching unit 30.

[0031] The camera switching unit 30 switches the camera C based on the performance part and switching destination information determined by the performance part determination unit 13. In this embodiment, when the orchestra performance reaches the performance part determined by the performance part determination unit 13, the camera switching unit 30 switches and outputs the camera 1 video to camera 3 video according to the switching destination information input from the camera switching destination / timing information storage unit 20 (step S520 in FIG. 16).

[0032] (The performance tracking device in operation) The operation of the performance portion tracking device 2 will be described in detail with reference to FIG. 3 and FIG. As shown in FIG. 3, the performance portion tracking device 2 executes advance preparation S1 and performance tracking / camera switching S2.

[0033] The advance preparation S1 may be performed before tracing the played portion. Also, if the musical piece to be traced is the same, the advance preparation S1 may be performed only once at the beginning. Moreover, performance tracking / camera switching S2 may be executed when tracking a performance part.

[0034] [Advance preparation] As shown in FIG. 4, in advance preparation S1, the performance portion tracking device 2 sequentially executes a musical score analysis process S3 and an input of switching destination information S4. In the score analysis process S3, the played portion tracking device 2 obtains the coordinates of the staff, the coordinates of the notes, and the order of the notes contained in the score, and generates markers for tracking the played portion. In input of switching destination information S4, the performance portion tracking device 2 receives switching destination information for camera C from the user.

[0035] <Music score analysis processing> The score analysis process S3 in FIG. 4 will be described in detail with reference to FIG. 5, the score image analysis unit 11 performs a staff coordinate estimation process S10. In this staff coordinate estimation process S10, the image is translated and a logical operation is performed to estimate the staff coordinates.

[0036] As shown in FIG. 6(a), the original image P0 and the image P1 obtained by translating the image P0 in the X direction (horizontal direction) are X When a logical OR operation is performed with this, the vertical line L included in the image P0 is Y Here, we obtain an image P1 from which the pixels P0 and P1 have been erased. X Then, the pixel value of the white pixels, which is the background color of the music score, is set to 1, and the pixel value of the black pixels, which is the color of the notes and treble clefs, is set to 0, and logical sum is performed. As a result, images P0 and P X Horizontal line L that is black on both sides X The vertical line L remains. Y will be erased.

[0037] As shown in FIG. 6(b), the vertical line L Y Similarly, horizontal line L X Specifically, the original image P0 and the image P1 obtained by translating the image P0 in the Y direction (vertical direction) can be erased. Y When the logical OR is performed with this, the horizontal line L included in the image P0 X The image P2 is obtained by erasing the image.

[0038] All symbols on music notation, except for the staff, are indicated by a vertical line L. Y Therefore, by applying the process shown in FIG. 6(a) to the image of the musical score, everything other than the staff can be removed from the image of the musical score, and as a result, the coordinates of the staff can be estimated.

[0039] Strictly speaking, this method creates a vertical line L Y If you erase the horizontal line L X The end of the horizontal line L is also erased. XThe horizontal line L X If you erase the vertical line L Y is slightly shorter, but this can also be neglected.

[0040] <<Staff coordinate estimation processing>> The staff coordinate estimation process S10 of FIG. 5 will be described in detail with reference to FIG. 7, in step S100, the score image analysis unit 11 generates a binary score image by binarizing the score image. This binary score image is a binary image (black and white image) in which white areas that are the background of the score image have a pixel value of 1, and black areas that are notes or treble clefs in the score image have a pixel value of 0.

[0041] In step S110, the score image analysis unit 11 copies the binarized score image, resulting in a total of two binarized score images. In step S120, the score image analysis unit 11 generates a horizontally moved score image by moving one binarized score image in the X direction.

[0042] In step S130, the score image analysis unit 11 calculates a logical sum for each pixel between the horizontally-moving score image and the binarized score image. That is, the score image analysis unit 11 calculates a logical sum between corresponding pixels at the same coordinates between the horizontally-moving score image and the binarized score image. In step S140, the score image analysis unit 11 detects black pixels as staff notation from the result of the logical sum operation.

[0043] <<Note and treble clef coordinate estimation process>> Returning to FIG. 5, the explanation of the musical score analysis process S3 will be continued. The musical score image analysis unit 11 performs a process S20 of estimating the coordinates of the notes and the treble clef. In this process S20, the image is translated, a logical operation is performed, and filtering is performed to estimate the coordinates of the notes and the treble clef.

[0044] The note and treble clef coordinate estimation process S20 in FIG. 5 will be described in detail with reference to FIG. As shown in FIG. 8, in step S200, the score image analysis unit 11 binarizes the score image to generate a binarized score image. In step S210, the score image analysis unit 11 copies the binarized score image, resulting in a total of three binarized score images.

[0045] In step S220, the score image analysis unit 11 translates one binary score image in the X direction to generate a horizontally moved score image. In step S230, the score image analysis unit 11 translates one binary score image in the Y direction to generate a vertically moved score image.

[0046] In step S240, the score image analysis unit 11 calculates a logical sum for each pixel at the same coordinates for the vertically moved score image, the horizontally moved score image, and the binarized score image. The process in step S240 is as shown in FIG. 9. First, the binarized score image G0 and the horizontally moved score image G X A binary music score image G1 is obtained by ORing the vertical lines and the music score image G2. Since the vertical lines are removed from this binary music score image G1, the staff 90 and the notes 91 remain. Next, the binary music score image G0 and the vertically moved music score image G Y A binary music score image G2 is obtained by performing a logical OR with the above. In this binary music score image G2, the horizontal lines of the staff 90 and the like have been erased, so that only the notes 91 remain. Then, a logical OR is performed on the binary music score images G1 and G2.

[0047] In step S250, the score image analysis unit 11 detects black pixels from the logical sum image R that represents the result of the logical sum. In this state, as shown in Fig. 10(a) in the logical sum image R, not only the note balls 93 but also clusters such as the note flags 94 remain, so filtering is performed as described below.

[0048] In step S260, the score image analysis unit 11 estimates the coordinates of the note 91 and the treble clef by detecting the note head 93 and the treble clef determined by the line spacing of the staff 90 from the logical sum image R. As shown in FIG. 10(b), the score image analysis unit 11 estimates the coordinates of the note 91 and the treble clef by detecting the width D between the lines of the staff 90. Y Filtering is performed by taking advantage of the fact that is equal to the vertical width of the note ball 93.

[0049] Specifically, the score image analysis unit 11 calculates the line spacing D Y The area of ​​a circle having a diameter of is calculated as the area of ​​the ball portion 93 of the note. Then, the score image analysis unit 11 erases from the logical sum image R an area equal to or larger than the area of ​​the ball 93 of the note plus a predetermined value. Note that this predetermined value is a threshold value that takes into consideration that the ball portion of the note or the treble clef is not strictly a circle, and can be set to any value. Furthermore, the score image analysis unit 11 erases from the logical sum image R an area equal to or smaller than the area of ​​the ball 93 of the note minus a predetermined value. As a result, the ball 93 of the note remains in the logical sum image R as shown in FIG. 10(c). Similarly, the ball 96 of the treble clef 95 remains as shown in FIG. 11. In this way, since everything other than the note 91 and the ball portions 93 and 96 of the treble clef 95 can be erased from the score image, the coordinates of the note 91 and the treble clef 95 located at the beginning of each line can be estimated.

[0050] <<Note order estimation and treble clef determination processing>> Returning to FIG. 5, the explanation of the musical score analysis process S3 will be continued. The score image analysis unit 11 performs a note order estimation / treble clef determination process S30. In this note order estimation / treble clef determination process S30, the order of notes to be played is estimated, and treble clefs that do not need to be played are determined. Here, the order of notes to be played is not simply determined by their position up and down or left and right, but from left to right for each row. Therefore, a musical staff 90 is used to determine the order of notes for each row.

[0051] 12(a), the score image analysis unit 11 classifies the notes 91 by row based on the coordinates of the musical staff 90, and then sorts the notes 91 by the X coordinate (horizontal coordinate) to assign an order to the notes 91. Note that for ease of explanation, the notes 91 and treble clefs 95 are shown in their entirety, but in reality, only the bead parts of the notes 91 and treble clefs 95 remain.

[0052] 12(b), in the first line of the upper row and the second line of the lower row, the first treble clef 95 is assigned the number 1, and the notes 91 are assigned the numbers 2 to 7. Note that, for ease of explanation, the notes 91 are assigned the numbers 1 to 12, but the numbers are not actually synthesized into the musical score.

[0053] At the beginning of each line is a treble clef 95. Therefore, when determining the order of the notes 91, the score image analysis unit 11 excludes the notes 91 at the beginning of each line as the notes of the treble clef 95. As a result, as shown in FIG. 12(c), in the first line of the upper row and the second line of the lower row, the notes 91 are assigned the orders 1 to 6.

[0054] After that, the score image analysis unit 11 determines the order of notes in the entire score by merging all the lines. As shown in Fig. 12(d), when the first line in the upper row and the second line in the lower row are merged, the notes 91 in the first row are assigned the numbers 1 to 6, and the notes 91 in the second row are assigned the numbers 7 to 12.

[0055] The note order estimation / G clef determination process S30 in FIG. 5 will be described in detail with reference to FIG. 13, in step S300, the score image analysis unit 11 classifies the detected elements into lines of the score based on the coordinates of the staff (see FIG. 12(a)). Note that the elements refer to the note balls and treble clef balls remaining after filtering. In step S310, the score image analysis unit 11 sorts the detected elements by X coordinate to determine the order of each element (see FIG. 12(b)).

[0056] In step S320, the score image analysis unit 11 determines the first element in each line. In step S330, the score image analysis unit 11 excludes the element determined to be the head of each line as the treble clef head (see FIG. 12(c)). In step S340, since a note order is assigned to each line, the score image analysis unit 11 determines the note order for the entire score by merging all the lines (see FIG. 12(d)).

[0057] <<Marker generation process>> Returning to FIG. 5, the explanation of the musical score analysis process S3 will be continued. The score image analysis unit 11 performs a marker generation process S40. In the marker generation process S40, a marker for tracking a played portion is generated based on the coordinates of the staff and the coordinates and order of the notes.

[0058] 14, the score image analysis unit 11 superimposes a rectangle M corresponding to each note 91 as a marker for tracking the played portion onto the score image when the note is played. The inside of this rectangle M is colored a predetermined color (e.g., light red) so that it can be distinguished from the score. This allows the score to be colored like lyrics in karaoke, where the parts to be sung are colored, and the played portion can be tracked.

[0059] In order to determine the position of rectangle M, it is necessary to determine the coordinates of each side of rectangle M. As shown in FIG. 14, the coordinates of the left side of rectangle M are the coordinates of the corresponding note 91. N The note before 91 B The X coordinate of the rectangle M is the right side coordinate of the note 91. N The top coordinate of rectangle M is the X coordinate of note 91. N The top staff 90 in the row where U The Y coordinate (vertical coordinate) of the rectangle M is the bottom coordinate of the note 91. N The lowest staff 90 in the row where D For the first note in each row, the left side coordinate of the rectangle M should be the X coordinate of the treble clef.

[0060] As shown in FIG. 15, in step S400, the score image analysis unit 11 sets the order n of the notes and rectangles to an initial value of 1, and sets the order m of the lines to an initial value of 1. In step S410, the score image analysis unit 11 sets the left side coordinate of the rectangle to the X coordinate of the treble clef in the mth line, and the right side coordinate of the rectangle to the X coordinate of the nth note. The score image analysis unit 11 also sets the top side coordinate of the rectangle to the Y coordinate of the topmost staff in the mth line, and sets the bottom side coordinate of the rectangle to the Y coordinate of the bottommost staff in the mth line.

[0061] In step S420, the score image analysis unit 11 generates the n-th rectangle based on the coordinate values ​​set in step S410 or step S450.

[0062] In step S430, the score image analysis unit 11 determines whether the (n+1)th note is in the mth line, the (m+1)th line, or does not exist. If the (n+1)th note is in the mth line, the score image analysis unit 11 proceeds to the process of step S440. If the (n+1)th note does not exist, the score image analysis unit 11 ends the marker generation process S40. If the (n+1)th note is in the (m+1)th line, the score image analysis unit 11 proceeds to the process of step S460.

[0063] In step S440, the score image analysis unit 11 increments the sequence number n. In step S450, the score image analysis unit 11 sets the left side coordinate of the rectangle to the X coordinate of the n-1th note, and sets the right side coordinate of the rectangle to the X coordinate of the nth note. The score image analysis unit 11 also sets the top side coordinate of the rectangle to the Y coordinate of the top staff in the mth line, and sets the bottom side coordinate of the rectangle to the Y coordinate of the bottom staff in the mth line. After that, the score image analysis unit 11 returns to the process of step S420.

[0064] In step S460, the score image analysis unit 11 increments the order n and the order m. After that, the score image analysis unit 11 returns to the process of step S410.

[0065] [Performance tracking and camera switching processing] Returning to FIG. 3, the performance tracking / camera switching S2 will now be described. In performance tracking / camera switching S2, the performance part tracking device 2 performs performance action analysis processing and automatically switches camera C. In performance tracking / camera switching S2, one note constitutes one cycle. In other words, the performance part tracking device 2 determines that the note being played has advanced by one, and combines the musical score with a marker. The performance part tracking device 2 then determines whether that note is the timing to switch camera C, and if so, switches camera C. The performance part tracking device 2 repeats this processing from the first note to the last note of the musical score.

[0066] The performance tracking / camera switching S2 in FIG. 3 will be described in detail with reference to FIG. 16, the playing motion analysis unit 12 performs playing motion analysis processing S500 on the player image. Details of the playing motion analysis processing S500 will be described later.

[0067] In step S510, the played portion determination unit 13 determines whether or not the note at the played portion analyzed by the playing motion analysis process S500 is at the switching timing designated by the switching destination information. If it is time to switch camera C (Yes in step S510), the performance portion tracking device 2 proceeds to the process of step S520. If it is not the timing to switch the camera C (No in step S510), the performance portion tracking device 2 proceeds to the process of step S530.

[0068] In step S520, the camera switching unit 30 switches to camera C designated by the switching destination information. In step S530, the played portion determination unit 13 combines a marker with the played portion.

[0069] In step S540, the played portion determination unit 13 determines whether or not it is the last note. If it is the last note (Yes in step S540), the performance portion tracking device 2 ends the performance tracking / camera switching S2. If it is not the last note (No in step S540), the performance part tracking apparatus 2 returns to the performance action analysis process S500.

[0070] <<Performance movement analysis processing>> The playing motion analysis process S500 in FIG. 16 will be described in detail with reference to FIG. 17 and FIG. In this embodiment, the musical instrument is a violin. The violin is a bowed string instrument that produces sound by rubbing the strings with a bow. For this reason, the violin requires the player's right arm to move up and down in order to produce sound.

[0071] As shown in FIG. 17(a), when the player 96 raises his right arm and rubs the string, the first note 91 is played. Also, as shown in FIG. 17(b), when the player lowers his right arm 96 and rubs the string, the second note 91 is played. Furthermore, as shown in FIG. 17(c), when the player 96 raises his right arm again and rubs the string, the third note 91 is played. In this way, the up and down movement of the player 96's right arm is linked to the progression of the music, so it can be used as a timing for synthesizing a marker at the performance point. Note that in FIG. 17, for ease of explanation, the movement vector of the player 96's right arm is illustrated with an arrow.

[0072] For example, the playing motion analysis unit 12 can capture a player image using the image processing library described in Reference 1, and analyze the movement of the player's right arm using the skeleton estimation process described in Reference 2. Then, the playing motion analysis unit 12 can calculate the movement vector of the right arm by acquiring the coordinates of the player's right arm.

[0073] Reference 1: “OpenCV”, [online], [searched on September 28, 2023], Internet<URL:https: / / opencv.org / > Reference 2: “MediaPipe”, [online], [searched on September 28, 2023], Internet<URL:https: / / developers.google.com / mediapipe>

[0074] 18, in step S600, the playing motion analysis unit 12 acquires the Y coordinate y' and the motion vector v' of the player's right arm from the memory. Note that the Y coordinate y' and the motion vector v' are the Y coordinate and the motion vector of the right arm detected from the previous frame of the player image.

[0075] In step S610, the playing motion analysis unit 12 detects the Y-coordinate y of the right arm in the current frame from the player image. In step S620, the playing motion analysis unit 12 calculates a right arm motion vector v. This motion vector v is the difference between the Y coordinate y of the right arm in the current frame and the Y coordinate y' of the right arm in the previous frame (v=y-y').

[0076] In step S630, the playing motion analysis unit 12 judges whether the directions of the motion vectors v and v' match or not, that is, whether the signs of the motion vectors v and v' match or not. If the directions of the motion vectors v and v' match (Yes in step S630), the playing motion analysis section 12 proceeds to the process of step S640. If the directions of the motion vectors v and v' do not match (No in step S630), the playing motion analysis section 12 proceeds to the process of step S650.

[0077] In step S640, the playing motion analysis unit 12 stores the Y coordinate y and the movement vector v of the right arm in memory as the Y coordinate y' and the movement vector v'.

[0078] In step S650, the playing motion analysis unit 12 stores the Y coordinate y and the movement vector v of the right arm in memory as the Y coordinate y' and the movement vector v'. In step S660, the playing motion analysis unit 12 judges that the performance has moved to the next note, and ends the playing motion analysis process S500. In other words, the timing when the direction of the right arm's motion vector is reversed is the timing when the performance moves to the next note.

[0079] [Actions and Effects] As described above, the performance part tracking device 1 according to the embodiment only uses images such as a score image and a player image, and does not need to record the actual performance. Therefore, the performance part tracking device 1 can accurately track the performance part without being affected by the difference in reverberation between halls or fluctuations in the performance. Furthermore, since the performance part tracking device 1 does not use the gaze information of the player, there is no need to install a camera in a position where the player's eyes can be photographed, and the player does not need to wear measurement goggles, which reduces the burden on the player.

[0080] Furthermore, since the performance portion tracking device 1 performs the process of FIG. 4 as the score analysis process, it is possible to accurately estimate the coordinates of the staff and the coordinates and order of the notes, thereby improving the accuracy of the performance portion. Furthermore, when determining the order of notes, the performance portion tracking device 1 excludes the treble clef ball located at the beginning of each line, so that the performance portion can be accurately tracked for instruments that use treble clefs in musical scores, such as the violin. Furthermore, since the performance portion tracking device 1 performs the processing of FIG. 18 as the performance movement analysis processing, it is possible to accurately analyze the movement of the player and improve the accuracy of the performance portion. In this way, the performance portion tracking device 1 can accurately track the performance portion of a stringed instrument, particularly a violin.

[0081] (Modification) Although the embodiment has been described in detail above, the present invention is not limited to the above-described embodiment, and includes design modifications and the like within the scope of the present invention.

[0082] In the above embodiment, the musical instrument is described as a violin, but it is not particularly limited as long as it is capable of analyzing the movement of the player playing the musical instrument. For example, the musical instrument may be a stringed instrument such as a viola, cello, or double bass. For example, in the case of a cello or double bass, unlike a violin, the player's arm moves left and right. For this reason, in the player movement analysis process, the X coordinate may be detected instead of the Y coordinate of the right arm. Also, in the case of an instrument that does not use a treble clef in the musical score, such as a cello or double bass, it is not necessary to determine the treble clef (steps S320 and S330 in FIG. 13).

[0083] In the above embodiment, an orchestral piece is described as an example, but the type of the piece is not particularly limited. The performance part tracking device can track the performance part not only in an orchestral piece without lyrics, but also in a piece with lyrics.

[0084] In the above embodiment, the marker is described as being rectangular, but there is no particular limitation as long as it can identify the playing part. For example, the marker may be a predetermined figure synthesized above and below the note of the playing part, or the color of the note of the playing part may be changed.

[0085] In the above embodiment, the performance point tracking device is described as switching the camera, but the use is not particularly limited. For example, the performance point tracking device may have a performance point tracking unit as an independent device and only track the performance point. Also, for example, the performance point tracking device can output a voltage contact according to the performance point, thereby instructing a desired external device. Usually, in news subtitles, a script is prepared in advance with settings such as camera placement, audio ON / OFF, and video playback, and the program can be progressed by simply pressing one button. Therefore, the performance point tracking device can automate the progress of the program by replacing the button pressing action with a voltage contact.

[0086] In the above embodiment, the playing portion tracking device is described as an independent hardware, but the present invention is not limited to this. For example, the present invention can be realized by a program for making hardware resources such as a CPU, memory, and hard disk of a computer function as the playing portion tracking device. This program may be distributed via a communication line, or may be written on a recording medium such as a CD-ROM or a flash memory and distributed. [Explanation of symbols]

[0087] 1. Performance Tracking System 2. Performance Tracking Device 10 Tracking section 11 Music score image analysis section 12 Performance motion analysis section 13. Performance part determination section 20 Camera switching destination / timing information storage unit (camera switching destination information storage unit) 30 Camera switching section C Camera

Claims

1. A performance portion tracking device that tracks a performance portion on a musical score by using a musical score image obtained by photographing a musical score and a player image obtained by photographing a player playing an instrument, comprising: a score image analysis unit that performs score analysis processing on the score image to obtain coordinates of a staff included in the score and coordinates and order of notes included in the score, and generates markers for tracking a portion of the score based on the coordinates of the staff and the coordinates and order of notes; a performance motion analysis unit that performs a performance motion analysis process on the player image to detect a timing at which the player plays the next note; a played portion determination unit that determines the note being played by the player as the played portion based on the timing and the order of the notes detected by the playing motion analysis unit, and combines the marker with the determined played portion; A performance part tracking device comprising:

2. The score image analysis unit performs the score analysis process by generating a binary score image by binarizing the score image; The coordinates of the staff are obtained by detecting the staff by performing a logical sum of a horizontally moved score image obtained by translating the binary score image in the horizontal direction and the binary score image; a logical sum is calculated between a vertically moved score image obtained by translating the binary score image in the vertical direction, the horizontally moved score image, and the binary score image, and the coordinates of the notes are calculated by detecting the balls of the notes that are determined by the line spacing of the staff from a logical sum image that represents the result of the logical sum; 2. The performance portion tracking device according to claim 1, wherein the detected note balls are classified by lines of the musical score and arranged horizontally based on the coordinates of the musical staff, thereby determining the order of the notes.

3. 3. The performance portion tracking device according to claim 2, wherein the musical score image analysis unit excludes the head of the note located at the beginning of each line as a treble clef head when determining the order of the notes.

4. The playing motion analysis unit performs the playing motion analysis process by The performance part tracking device according to claim 1, characterized in that a skeleton estimation process is performed on the player image to calculate a motion vector of the performance movement, and the timing at which the direction of the calculated motion vector is reversed is detected as the timing at which the player plays the next note.

5. 2. The performance portion tracking device according to claim 1, wherein the musical instrument is a stringed instrument.

6. 2. The performance part tracing device according to claim 1, wherein the musical instrument is a violin.

7. a camera switching destination information storage unit that stores camera switching destination information in advance in association with a predetermined note included in the musical score; a camera switching unit that switches the camera based on the performance part determined by the performance part determination unit and the switching destination information; The performance portion tracking device according to claim 1, further comprising:

8. A program for causing a computer to function as the performance portion tracking device according to any one of claims 1 to 7.

Images