IMAGE PROCESSING METHOD, IMAGE PROCESSING SYSTEM, AND PROGRAM

Abstract

To provide an image processing system, an image processing method, and a program that improve convenience of a performance image.SOLUTION: In a performance analysis system connected with a keyboard instrument by wire or wirelessly and having a control device, a storage device, a scanning device, a display device, and an imaging device, image extraction processing includes region estimation processing Sb1 and region extraction processing Sb2. The region estimation processing Sb1 estimates a specific region for a performance image represented by image data. The specific region includes a keyboard image and a finger image in the performance image. When the control device executes the region estimation processing Sb1, an element (a region estimation unit) that estimates the specific region of the performance image is achieved. The region extraction processing Sb2 extracts a specific region from the performance image represented by image data. When the control device executes the region extraction processing Sb2, an element (a region extraction unit) that extracts the specific region of the performance image is achieved.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] The present disclosure relates to a technique for analyzing a performance by a user. [Background technology]

[0002] For example, a technique for estimating an area in an image captured by an image capture device where a specific object exists has been proposed. For example, Patent Literature 1 discloses a technique for detecting an object using a deep neural network. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2020-528176 Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, if it were possible to extract a specific area, such as a keyboard area, from a performance image captured of a performance of a musical instrument such as a keyboard instrument, it would be useful for, for example, analyzing a user's fingering. In consideration of the above circumstances, one aspect of the present disclosure has an object to improve the convenience of the performance image. [Means for solving the problem]

[0005] In order to solve the above problems, an image processing method according to one aspect of the present disclosure estimates a specific area that includes an image of an instrument from a performance image that includes an image of an instrument and images of the fingers of a user playing the instrument, and extracts the specific area from the performance image.

[0006] An image processing system according to one aspect of the present disclosure includes an area estimation unit that estimates a specific area including an image of an instrument from a performance image including an image of an instrument and an image of a plurality of fingers of a user playing the instrument, and an area extraction unit that extracts the specific area from the performance image.

[0007] A program according to one aspect of the present disclosure causes a computer system to function as an area estimation unit that estimates a specific area including an image of an instrument from a performance image including an image of an instrument and an image of the fingers of a user playing the instrument, and an area extraction unit that extracts the specific area from the performance image. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram illustrating the configuration of a performance analysis system according to a first embodiment. [Diagram 2] FIG. [Diagram 3] 1 is a block diagram illustrating an example of a functional configuration of a performance analysis system. [Figure 4] FIG. 13 is a schematic diagram of an analysis screen. [Diagram 5] 13 is a flowchart of a finger position estimation process. [Figure 6] 13 is a flowchart of left / right determination processing. [Figure 7] FIG. 11 is an explanatory diagram of an image extraction process. [Figure 8] 13 is a flowchart of an image extraction process. [Figure 9] FIG. 1 is an explanatory diagram of machine learning for establishing an estimation model. [Figure 10] FIG. 2 is a schematic diagram of a reference image. [Figure 11] 13 is a flowchart of a matrix generation process. [Figure 12] 13 is a flowchart of an initial setting process. [Figure 13] FIG. 4 is a schematic diagram of a setting screen. [Figure 14] 13 is a flowchart of a performance analysis process. [Figure 15] FIG. 1 is an explanatory diagram regarding a problem of fingering estimation. [Figure 16] FIG. 11 is a block diagram illustrating the configuration of a performance analysis system according to a second embodiment. [Figure 17]FIG. 11 is a schematic diagram of control data in the second embodiment. [Figure 18] 13 is a flowchart of a performance analysis process in the second embodiment. [Figure 19] 13 is a flowchart of a performance analysis process in the third embodiment. [Figure 20] 13 is a flowchart of an initial setting process in the fourth embodiment. [Figure 21] FIG. 13 is a block diagram illustrating the configuration of a performance analysis system according to a fifth embodiment. [Figure 22] FIG. 13 is a block diagram illustrating an example of the functional configuration of an image processing system according to a sixth embodiment. [Figure 23] 23 is a flowchart of a first image processing according to the sixth embodiment. [Figure 24] FIG. 13 is a block diagram illustrating an example of the functional configuration of an image processing system according to a seventh embodiment. [Diagram 25] 23 is a flowchart of second image processing in the seventh embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] 1: First embodiment FIG. 1 is a block diagram illustrating the configuration of a musical performance analysis system 100 according to the first embodiment. A keyboard instrument 200 is connected to the musical performance analysis system 100 by wire or wirelessly. The keyboard instrument 200 is an electronic musical instrument having a keyboard 22 on which a plurality of (N) keys 21 are arranged. Each of the plurality of keys 21 of the keyboard 22 corresponds to a different pitch n (n=1 to N). A user (i.e., a performer) operates desired keys 21 of the keyboard instrument 200 sequentially with his / her left and right hands. The keyboard instrument 200 supplies the musical performance analysis system 100 with performance data P representing a performance by the user. The performance data P is time-series data that specifies the pitch n of each of a plurality of notes that the user sequentially plays. For example, the performance data P is data in a format that complies with the MIDI (Musical Instrument Digital Interface) standard.

[0010] The performance analysis system 100 is a computer system that analyzes a user's performance of the keyboard instrument 200. Specifically, the performance analysis system 100 analyzes the user's fingering. Fingering is the way in which the user uses the fingers of his or her left and right hands when playing the keyboard instrument 200 (i.e., fingering). In other words, information regarding which fingers the user uses to operate each key 21 of the keyboard instrument 200 is analyzed as the user's fingering.

[0011] The performance analysis system 100 comprises a control device 11, a storage device 12, an operation device 13, a display device 14, and a photographing device 15. The performance analysis system 100 is realized by a portable information device such as a smartphone or a tablet terminal, or a portable or stationary information device such as a personal computer. The performance analysis system 100 may be realized as a single device, or may be realized as multiple devices configured separately from each other. The performance analysis system 100 may also be mounted on a keyboard instrument 200.

[0012] The control device 11 is composed of one or more processors that control each element of the performance analysis system 100. For example, the control device 11 is composed of one or more types of processors such as a CPU (Central Processing Unit), an SPU (Sound Processing Unit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or an ASIC (Application Specific Integrated Circuit).

[0013] The storage device 12 is one or more memories that store the programs executed by the control device 11 and various data used by the control device 11. The storage device 12 is configured with a known recording medium such as a magnetic recording medium or a semiconductor recording medium, or a combination of multiple types of recording media. Note that the storage device 12 may be a portable recording medium that is detachably attached to the performance analysis system 100, or a recording medium (e.g., cloud storage) to which the control device 11 can write or read via a communication network such as the Internet.

[0014] The operation device 13 is an input device that accepts instructions from a user. The operation device 13 is, for example, an operator operated by the user, or a touch panel that detects contact by the user. Note that the operation device 13 (for example, a mouse or a keyboard) separate from the performance analysis system 100 may be connected to the performance analysis system 100 by wire or wirelessly.

[0015] The display device 14 displays images under the control of the control device 11. For example, various display panels such as a liquid crystal display panel or an organic EL (Electroluminescence) panel are used as the display device 14. Note that the display device 14, which is separate from the performance analysis system 100, may be connected to the performance analysis system 100 by wire or wirelessly.

[0016] The photographing device 15 is an image input device that generates a time series of image data D1 by photographing a subject. The time series of image data D1 is video data representing a video. For example, the photographing device 15 includes an optical system such as a photographing lens, an image sensor that receives incident light from the optical system, and a processing circuit that generates image data D1 according to the amount of light received by the image sensor. Note that the photographing device 15, which is separate from the performance analysis system 100, may be connected to the performance analysis system 100 by wire or wirelessly.

[0017] The user adjusts the position or angle of the image capturing device 15 relative to the keyboard instrument 200 so that the image capturing conditions recommended by the provider of the performance analysis system 100 are realized. Specifically, the image capturing device 15 is installed above the keyboard instrument 200 and captures the keyboard 22 of the keyboard instrument 200 and the left and right hands of the user. Therefore, as illustrated in FIG. 2, a time series of image data D1 representing a performance image G1 including an image of the keyboard 22 of the keyboard instrument 200 (hereinafter referred to as a "keyboard image") g1 and an image of the left and right hands of the user (hereinafter referred to as a "fingering image") g2 is generated by the image capturing device 15. That is, video data representing a video of the user playing the keyboard instrument 200 is generated in parallel with the performance. The image capturing conditions by the image capturing device 15 are, for example, the image capturing range or the image capturing direction. The image capturing range is the range (angle of view) captured by the image capturing device 15. The image capturing direction is the direction of the image capturing device 15 relative to the keyboard instrument 200.

[0018] FIG. 3 is a block diagram illustrating a functional configuration of the performance analysis system 100. The control device 11 executes a program stored in the storage device 12 to function as a performance analysis unit 30 and a display control unit 40. The performance analysis unit 30 generates fingering data Q representing the fingering of the user by analyzing the performance data P and the image data D1. The fingering data Q specifies which of the user's fingers has operated each of the multiple keys 21 of the keyboard instrument 200. Specifically, the fingering data Q specifies a pitch n corresponding to the key 21 operated by the user and a finger number (hereinafter referred to as a "finger number") k used by the user to operate the key 21. The pitch n is, for example, a note number in the MIDI standard. The finger number k is a number assigned to each finger of the user's left and right hands.

[0019] The display control unit 40 causes various images to be displayed on the display device 14. For example, the display control unit 40 causes the display device 14 to display an image (hereinafter referred to as an "analysis screen") 61 showing the results of the analysis by the performance analysis unit 30. FIG. 4 is a schematic diagram of the analysis screen 61. The analysis screen 61 is an image in which a plurality of note images 611 are arranged on a coordinate plane in which a horizontal time axis and a vertical pitch axis are set. The note images 611 are displayed for each note played by the user. The position of the note image 611 in the direction of the pitch axis is set according to the pitch n of the note represented by the note image 611. The position and total length of the note image 611 in the direction of the time axis is set according to the sounding period of the note represented by the note image 611.

[0020] A symbol (hereinafter referred to as a "fingering symbol") 612 corresponding to the finger number k designated for the note by the fingering data Q is arranged on the note image 611 of each note. The letter "L" of the fingering symbol 612 means the left hand, and the letter "R" of the fingering symbol 612 means the right hand. The numbers of the fingering symbol 612 mean each finger. Specifically, the number "1" of the fingering symbol 612 means the thumb, the number "2" means the index finger, the number "3" means the middle finger, the number "4" means the ring finger, and the number "5" means the little finger. Therefore, for example, the fingering symbol 612 "R2" means the index finger of the right hand, and the fingering symbol 612 "L4" means the ring finger of the left hand. The note image 611 and the fingering symbol 612 are displayed in different manners (for example, hues or gradations) for the right hand and the left hand. The display control unit 40 uses the fingering data Q to cause the display device 14 to display the analysis screen 61 of FIG.

[0021] It should be noted that, among the multiple note images 611 in the analysis screen 61, for notes for which the estimation result of the finger number k is unreliable, the note image 611 is displayed in a manner different from that of a normal note image 611 (for example, with a dashed border), and a specific symbol "??" is displayed, which means that the estimation result of the finger number k is invalid.

[0022] As illustrated in FIG. 3, the performance analysis unit 30 includes a finger position data generation unit 31 and a fingering data generation unit 32. The finger position data generation unit 31 generates finger position data F by analyzing the performance image G1. The finger position data F is data representing the positions of each finger of the user's left hand and each finger of the right hand. As described above, in the first embodiment, the positions of each finger of the user are distinguished between the left hand and the right hand, so that the fingering for the left hand and the right hand of the user can be estimated. On the other hand, the fingering data generation unit 32 generates fingering data Q by using the performance data P and the finger position data F. The finger position data F and the fingering data Q are generated for each unit period on the time axis. Each unit period is a period (frame) of a predetermined length.

[0023] A: Finger position data generation unit 31 The finger position data generating unit 31 includes an image extracting unit 311 , a matrix generating unit 312 , a finger position estimating unit 313 , and a projective transformation unit 314 .

[0024] [Finger position estimation unit 313] The finger position estimation unit 313 estimates the position c[h,f] of each finger of the left hand and the right hand of the user by analyzing the performance image G1 represented by the image data D1. The position c[h,f] of each finger is the position of each fingertip in the xy coordinate system set in the performance image G1. The position c[h,f] is expressed by a combination (x[h,f], y[h,f]) of the coordinate x[h,f] on the x-axis and the coordinate y[h,f] on the y-axis in the xy coordinate system of the performance image G1. The positive direction of the x-axis corresponds to the right direction of the keyboard 22 (the direction from low to high notes), and the negative direction of the x-axis corresponds to the left direction of the keyboard 22 (the direction from high to low notes). The symbol h is a variable indicating either the left hand or the right hand (h=1,2). Specifically, the value "1" of the variable h means the left hand, and the value "2" of the variable h means the right hand. The variable f is the number (f=1-5) of each finger on the left and right hands. The value "1" of the variable f means the thumb, the value "2" means the index finger, the value "3" means the middle finger, the value "4" means the ring finger, and the value "5" means the little finger. Therefore, for example, the position c[1,2] illustrated in FIG. 2 is the position of the fingertip of the index finger (f=2) of the left hand (h=1), and the position c[2,4] is the position of the fingertip of the ring finger (f=4) of the right hand (h=2).

[0025] 5 is a flowchart illustrating a specific procedure of a process (hereinafter, referred to as a "finger position estimation process") in which the finger position estimation unit 313 estimates the positions of the fingers of the user. The finger position estimation process includes an image analysis process Sa1, a left / right determination process Sa2, and an interpolation process Sa3.

[0026] The image analysis process Sa1 is a process for estimating the position c[h,f] of each finger on one of the user's left hand and right hand (hereinafter referred to as the "first hand") and the position c[h,f] of each finger on the other of the user's left hand and right hand (hereinafter referred to as the "second hand") by analyzing the performance image G1. Specifically, the finger position estimation unit 313 estimates the positions c[h,1]-c[h,5] of each finger on the first hand and the positions c[h,1]-c[h,5] of each finger on the second hand by image recognition processing for estimating the skeleton or joints of the user by analyzing the image. The image analysis process Sa1 uses a known image recognition process such as MediaPipe or OpenPose. If the fingertip is not detected from the performance image G1, the coordinate x[h,f] of the fingertip on the x-axis is set to an invalid value such as "0".

[0027] In the image analysis process Sa1, the positions c[h,1]-c[h,5] of the fingers of the first hand and the positions c[h,1]-c[h,5] of the fingers of the second hand of the user are estimated, but it is not possible to determine whether the first hand and the second hand correspond to the left hand or the right hand of the user. In addition, since the right arm and the left arm of the user may cross when playing the keyboard instrument 200, it is not appropriate to determine the left hand / right hand only from the coordinates x[h,f] of each position c[h,f] estimated by the image analysis process Sa1. If the user's arms and a part including the torso are photographed by the photographing device 15, the user's left hand / right hand can be estimated from the performance image G1 from the coordinates of the user's shoulders and arms. However, there are problems in that a wide range needs to be photographed by the photographing device 15, and that the processing load of the image analysis process Sa1 increases.

[0028] Considering the above circumstances, the finger position estimation unit 313 of the first embodiment executes left / right determination process Sa2 in Fig. 5 to determine whether each of the first and second hands corresponds to the left or right hand of the user. That is, the finger position estimation unit 313 determines the variable h in the finger position c[h,f] of each of the first and second hands to be either the value "1" indicating the left hand or the value "2" indicating the right hand.

[0029] When playing the keyboard instrument 200, the backs of both the left and right hands are positioned vertically upward, so the performance image G1 captured by the image capture device 15 includes images of the backs of both the left and right hands of the user. Therefore, in the left hand in the performance image G1, the thumb position c[h,1] is located to the right of the little finger position c[h,5], and in the right hand in the performance image G1, the thumb position c[h,1] is located to the left of the little finger position c[h,5]. Taking the above into consideration, in the left-right determination process Sa2, the finger position estimation unit 313 determines that the hand whose thumb position c[h,1] is located to the right (positive direction of the x-axis) of the little finger position c[h,5] is the left hand (h=1) out of the first and second hands. On the other hand, finger position estimation unit 313 determines that the hand whose thumb position c[h,1] is located to the left (in the negative direction of the x-axis) of the little finger position c[h,5] of the first or second hand is the right hand.

[0030] 6 is a flowchart illustrating a specific procedure of the left-right determination process Sa2. The finger position estimation unit 313 calculates a determination index γ[h] for each of the first hand and the second hand (Sa21). The determination index γ[h] is calculated, for example, by the following formula (1).

number

[0031] The image analysis process Sa1 and the left-right determination process Sa2 estimate the position c[h,f] of each finger of the user for each unit period. However, due to various circumstances such as noise present in the performance image G1, the position c[h,f] may not be estimated properly. Therefore, when the position c[h,f] is missing in a specific unit period (hereinafter referred to as a "missing period"), the finger position estimation unit 313 calculates the position c[h,f] in the missing period by an interpolation process Sa3 using the position c[h,f] in the unit periods before and after the missing period. For example, when the position c[h,f] is missing in the central unit period (missing period) of three consecutive unit periods on the time axis, the average of the position c[h,f] in the unit period immediately before the missing period and the position c[h,f] in the unit period immediately after the missing period is calculated as the position c[h,f] in the missing period.

[0032] [Image Extraction Unit 311] As described above, the performance image G1 includes the keyboard image g1 and the finger image g2. The image extraction unit 311 in Fig. 3 extracts a specific area (hereinafter referred to as "specific area") B from the performance image G1, as illustrated in Fig. 7. The specific area B is an area of ​​the performance image G1 that includes the keyboard image g1 and the finger image g2. The finger image g2 corresponds to an image of at least a part of the user's body.

[0033] 8 is a flowchart illustrating a specific procedure of the process (hereinafter, referred to as "image extraction process") in which the image extraction unit 311 extracts the specific region B from the performance image G1. The image extraction process includes a region estimation process Sb1 and a region extraction process Sb2.

[0034] The area estimation process Sb1 is a process for estimating a specific area B for the performance image G1 represented by the image data D1. Specifically, the image extraction unit 311 generates an image processing mask M representing the specific area B from the image data D1 by the area estimation process Sb1. As illustrated in FIG. 7, the image processing mask M is a mask of the same size as the performance image G1, and is composed of a plurality of elements corresponding to different pixels of the performance image G1. Specifically, the image processing mask M is a binary mask in which each element in an area corresponding to the specific area B of the performance image G1 is set to a numerical value of "1", and each element in an area other than the specific area B is set to a numerical value of "0". The control device 11 executes the area estimation process Sb1 to realize an element (area estimation unit) for estimating the specific area B of the performance image G1.

[0035] As illustrated in FIG. 3, the image extraction unit 311 uses an estimation model 51 to generate an image processing mask M. That is, the image extraction unit 311 generates an image processing mask M by inputting image data D1 representing a performance image G1 to the estimation model 51. The estimation model 51 is a statistical model that learns the relationship between the image data D1 and the image processing mask M by machine learning. The estimation model 51 is configured, for example, by a deep neural network (DNN). For example, a deep neural network of any type, such as a convolutional neural network (CNN) or a recurrent neural network (RNN), is used as the estimation model 51. The estimation model 51 may be configured by combining a plurality of types of deep neural networks. In addition, an additional element, such as a long short-term memory (LSTM), may be installed in the estimation model 51.

[0036] 9 is an explanatory diagram of machine learning that establishes the estimation model 51. For example, the estimation model 51 is established by machine learning by a machine learning system 900 that is separate from the performance analysis system 100, and the estimation model 51 is provided to the performance analysis system 100. The machine learning system 900 is a server system that can communicate with the performance analysis system 100 via a communication network such as the Internet. The estimation model 51 is transmitted from the machine learning system 900 to the performance analysis system 100 via the communication network.

[0037] A plurality of pieces of learning data T are used for the machine learning of the estimation model 51. Each of the plurality of pieces of learning data T is composed of a combination of learning image data Dt and learning image processing mask Mt. The image data Dt represents a known image including a keyboard image g1 of a keyboard instrument and an image of the surroundings of the keyboard instrument. The type of keyboard instrument and the shooting conditions (e.g., shooting range and shooting direction) are different for each piece of image data Dt. That is, the image data Dt is prepared in advance by shooting each of a plurality of types of keyboard instruments under different shooting conditions. Note that the image data Dt may be prepared by a known image synthesis technique. The image processing mask Mt of each piece of learning data T is a mask representing a specific region B of the known image represented by the image data Dt of the learning data T. Specifically, the elements in the area of ​​the image processing mask Mt corresponding to the specific region B are set to the numerical value "1", and the elements in the area other than the specific region B are set to the numerical value "0". That is, the image processing mask Mt means a correct answer that the estimation model 51 should output in response to the input of the image data Dt.

[0038] The machine learning system 900 calculates an error function that represents an error between an image processing mask M output by an initial or provisional model (hereinafter referred to as a "provisional model") 51a when image data Dt of each learning data T is input, and the image processing mask M of the learning data T. Then, the machine learning system 900 updates multiple variables of the provisional model 51a so as to reduce the error function. The provisional model 51a at the time when the above process is repeated for each of the multiple learning data T is determined as the estimated model 51. Therefore, the estimated model 51 outputs an image processing mask M that is statistically appropriate for the unknown image data D1 under the potential relationship between the image data Dt and the image processing mask Mt in the multiple learning data T. In other words, the estimated model 51 is a trained model that has learned the relationship between the image data Dt and the image processing mask Mt.

[0039] As described above, in the first embodiment, the image processing mask M representing the specific region B is generated by inputting the image data D1 of the performance image G1 to the machine-learned estimation model 51. Therefore, the specific region B can be identified with high accuracy for various unknown performance images G1.

[0040] The region extraction process Sb2 in FIG. 8 is a process for extracting a specific region B from the performance image G1 represented by the image data D1. Specifically, the region extraction process Sb2 is an image process for selectively removing a region other than the specific region from the performance image G1 to relatively emphasize the specific region B. The image extraction unit 311 in the first embodiment generates image data D2 by applying an image processing mask M to the image data D1 (performance image G1). Specifically, the image extraction unit 311 multiplies the pixel value of each pixel in the performance image G1 by an element of the image processing mask M corresponding to the pixel. The region extraction process Sb2 generates image data D2 representing an image (hereinafter referred to as a "performance image G2") from which a region other than the specific region B from the performance image G1 has been removed, as illustrated in FIG. 7. That is, the performance image G2 represented by the image data D2 is an image from which the keyboard image g1 and the fingering image g2 have been extracted from the performance image G1. The control device 11 executes the area extraction process Sb2, thereby realizing an element (area extraction unit) that extracts the specific area B of the performance image G1.

[0041] [Projective transformation part 314] The position c[h,f] of each finger estimated by the finger position estimation process is a coordinate in the xy coordinate system set in the performance image G1. The shooting conditions of the keyboard instrument 200 by the image capture device 15 may differ depending on various circumstances such as the environment in which the keyboard instrument 200 is used. For example, it is assumed that the shooting range is too wide (or too narrow) compared with the ideal shooting conditions exemplified in FIG. 2, or the shooting direction is inclined with respect to the vertical direction. The numerical values ​​of the coordinates x[h,f] and coordinates y[h,f] at each position c[h,f] depend on the shooting conditions of the performance image G1 by the image capture device 15. Therefore, the projection transformation unit 314 of the first embodiment transforms (image registration) the position c[h,f] of each finger in the performance image G1 into a position C[h,f] in the XY coordinate system that is substantially independent of the shooting conditions by the image capture device 15. The finger position data F generated by the finger position data generation unit 31 is data representing the position C[h,f] after transformation by the projection transformation unit 314. That is, the finger position data F specifies the positions C[1,1] to C[1,5] of the fingers on the user's left hand and the positions C[2,1] to C[2,5] of the fingers on the user's right hand.

[0042] The XY coordinate system is set in a predetermined image (hereinafter referred to as the "reference image") Gref, as exemplified in FIG. 10. The reference image Gref is an image of a standard keyboard instrument (hereinafter referred to as the "reference instrument") photographed under standard photographing conditions. Note that the reference image Gref is not limited to an image of an actual keyboard. For example, an image synthesized by a known image synthesis technique may be used as the reference image Gref. Image data (hereinafter referred to as the "reference data") Dref representing the reference image Gref and auxiliary data A related to the reference image Gref are stored in the storage device 12.

[0043] The auxiliary data A is data that specifies a combination of an area (hereinafter referred to as a "unit area") Rn in which each key 21 of the reference instrument exists in the reference image Gref, and a pitch n corresponding to the key 21. In other words, the auxiliary data A is data that defines the unit area Rn corresponding to each pitch n in the reference image Gref.

[0044] To convert a position c[h,f] in the xy coordinate system to a position C[h,f] in the XY coordinate system, a projective transformation using a transformation matrix W is used, as expressed in the following formula (2). The symbol X in formula (2) means the coordinate on the X axis in the XY coordinate system, and the symbol Y means the coordinate on the Y axis. The symbol s is an adjustment value for matching the scale between the xy coordinate system and the XY coordinate system.

number

[0045] [Matrix generation unit 312] The matrix generation unit 312 in FIG. 3 generates a transformation matrix W of Equation (2) that the projection transformation unit 314 applies to the projection transformation. FIG. 11 is a flowchart illustrating a specific procedure of the process (hereinafter referred to as the "matrix generation process") in which the matrix generation unit 312 generates the transformation matrix W. The matrix generation process of the first embodiment is executed with the performance image G2 (image data D2) after the image extraction process as the processing target. With the above configuration, it is possible to generate an appropriate transformation matrix W that accurately approximates the keyboard image g1 to the reference image Gref, compared to a configuration in which the matrix generation process is executed with the entire performance image G1, including areas other than the specific area B, as the processing target.

[0046] The matrix generation process includes an initial setting process Sc1 and a matrix update process Sc2. The initial setting process Sc1 is a process for setting an initial matrix W0 that is an initial value of the transformation matrix W. The initial setting process Sc1 will be described later in detail.

[0047] The matrix update process Sc2 is a process for generating a transformation matrix W by iteratively updating the initial matrix W0. That is, the projective transformation unit 314 generates the transformation matrix W by iteratively updating the initial matrix W0 so that the keyboard image g1 of the performance image G2 approaches the reference image Gref by projective transformation using the transformation matrix W. For example, the transformation matrix W is generated so that the coordinate X / s on the X axis of a specific point in the reference image Gref approximates or matches the coordinate x on the x axis of a point corresponding to the point in the keyboard image g1, and the coordinate Y / s on the Y axis of a specific point in the reference image Gref approximates or matches the coordinate y on the y axis of a point corresponding to the point in the keyboard image g1. That is, the transformation matrix W is generated so that the coordinates of the key 21 corresponding to a specific pitch in the keyboard image g1 are transformed into the coordinates of the key 21 corresponding to the pitch in the reference image Gref by projective transformation to which the transformation matrix W is applied. The control device 11 executes the matrix update process Sc2 exemplified above, whereby an element that generates the transformation matrix W (matrix generation unit 312) is realized.

[0048] Incidentally, the matrix update process Sc2 is assumed to be a process of updating the transformation matrix W so that image features such as SIFT (Scale-Invariant Feature Transform) etc. are closer between the reference image Gref and the keyboard image g1. However, since the keyboard image g1 has a repeated pattern in which multiple keys 21 are arranged in the same way, it may not be possible to appropriately estimate the transformation matrix W using the image features.

[0049] Considering the above circumstances, the matrix generation unit 312 of the first embodiment iteratively updates the initial matrix W0 in the matrix update process Sc2 so that the enhanced correlation coefficient (ECC) between the reference image Gref and the keyboard image g1 increases (ideally maximizes). According to the above embodiment, it is possible to generate an appropriate transformation matrix W that can accurately approximate the keyboard image g1 to the reference image Gref, compared to the above embodiment that uses image features. The generation of the transformation matrix W using the enhanced correlation coefficient is also disclosed in Georgios D. Evangelidis and Emmanouil Z. Psarakis, "Parametric Image Alignment Using Enhanced Correlation Coefficient Maximization", IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 30, NO. 10, October 2008. As mentioned above, the extended correlation coefficient is suitable for generating the transformation matrix W used to transform the keyboard image g1, but the transformation matrix W may also be generated so that the image features such as the SIFT described above are closer between the reference image Gref and the keyboard image g1.

[0050] The projection transformation unit 314 in FIG. 3 executes a projection transformation process. The projection transformation process is a projection transformation of the performance image G1 using the transformation matrix W generated by the matrix generation process. The projection transformation process transforms the performance image G1 into an image (hereinafter referred to as a "transformed image") captured under the same shooting conditions as the reference image Gref. For example, the area of ​​the converted image corresponding to the key 21 of the pitch n substantially coincides with the unit area Rn of the pitch n in the reference image Gref. Also, the xy coordinate system of the converted image substantially coincides with the XY coordinate system of the reference image Gref. In the projection transformation process described above, the projection transformation unit 314 transforms the position c[h,f] of each finger into the position C[h,f] of the XY coordinate system, as expressed by the above-mentioned formula (2). The control device 11 executes the projection transformation process exemplified above, thereby realizing an element (projection transformation unit 314) that executes the projection transformation of the performance image G1.

[0051] The display control unit 40 causes the transformed image generated by the projective transformation process to be displayed on the display device 14. For example, the display control unit 40 causes the transformed image and the reference image Gref to be displayed in a mutually overlapping state on the display device 14. As described above, the area of ​​the transformed image corresponding to the key 21 of each pitch n and the unit area Rn of the reference image Gref corresponding to the pitch n overlap each other.

[0052] As described above, in the first embodiment, the transformation matrix W is generated so that the keyboard image g1 of the performance image G1 approaches the reference image Gref, and the projective transformation process using the transformation matrix W is executed on the performance image G1. Therefore, the performance image G1 of the keyboard instrument 200 played by the user can be transformed into a transformed image that corresponds to the shooting conditions of the reference instrument in the reference image Gref.

[0053] FIG. 12 is a flow chart illustrating a specific procedure of the initial setting process Sc1. When the initial setting process Sc1 is started, the projective transformation unit 314 displays the setting screen 62 illustrated in FIG. 13 on the display device 14 (Sc11). The setting screen 62 includes a performance image G1 captured by the image capture device 15 and instructions 622 for the user. The instructions 622 are a message to select an area (hereinafter referred to as a "target area") 621 corresponding to one or more specific pitches (hereinafter referred to as a "target pitch") n in the keyboard image g1 in the performance image G1. The user operates the operation device 13 while viewing the setting screen 62 to select the target area 621 corresponding to the target pitch n in the performance image G1. The projective transformation unit 314 accepts the selection of the target area 621 by the user (Sc12).

[0054] The projective transformation unit 314 identifies one or more unit areas Rn that the auxiliary data A specifies for the target pitch n in the reference image Gref represented by the reference data Dref (Sc13). The projective transformation unit 314 then calculates an initial matrix W0 as a matrix for projectively transforming the target area 621 of the performance image G1 onto one or more unit areas Rn specified from the reference image Gref (Sc14). As can be understood from the above explanation, the initial setting process Sc1 of the first embodiment is a process for setting the initial matrix W0 so that the target area 621 specified by the user in the keyboard image g1 approaches the unit area Rn corresponding to the target pitch n in the reference image Gref by projective transformation using the initial matrix W0.

[0055] In order to generate an appropriate transformation matrix W by the matrix update process Sc2, it is important to set the initial matrix W0. In particular, in a form in which an extended correlation coefficient is used in the matrix update process Sc2, the suitability of the initial matrix W0 tends to easily affect the suitability of the final transformation matrix W. In the first embodiment, the initial matrix W0 is set so that the target area 621 in the performance image G1 according to an instruction from the user approaches the unit area Rn corresponding to the target pitch n in the reference image Gref. Therefore, an appropriate transformation matrix W that can accurately approximate the keyboard image g1 to the reference image Gref can be generated. Also, in the first embodiment, the area in the performance image G1 designated by the user through an operation on the operation device 13 is used as the target area 621 to set the initial matrix W0. Therefore, compared to a form in which, for example, the area corresponding to the target pitch n in the performance image G1 is estimated by calculation processing, an appropriate initial matrix W0 can be generated while reducing the processing load. In the above description, the initial setting process Sc1 is performed on the performance image G1, but the initial setting process Sc1 may be performed on the performance image G2.

[0056] B: Fingering data generator 32 As described above, the fingering data generating unit 32 in FIG. 3 generates fingering data Q by using the performance data P generated by the keyboard instrument 200 and the finger position data F generated by the finger position data generating unit 31. The fingering data Q is generated for each unit period. The fingering data generating unit 32 in the first embodiment includes a probability calculating unit 321 and a fingering estimating unit 322. Note that in the above description, one finger of the user is expressed by a combination of variables h and f, but in the following description, one finger of the user is expressed by finger number k (k=1 to 10). Therefore, the position C[h,f] specified for each finger by the finger position data F will be expressed as position C[k] in the following description.

[0057] [Probability Calculation Section 321] The probability calculation unit 321 calculates the probability p that the pitch n specified by the performance data P is played by the finger with finger number k for each finger number k. The probability p is an index (likelihood) of the certainty that the finger with finger number k operated the key 21 of pitch n. The probability calculation unit 321 calculates the probability p depending on whether the position C[k] of the finger with finger number k is within the unit area Rn of pitch n. The probability calculation unit 321 calculates the probability p for each unit period on the time axis. Specifically, when the performance data P specifies pitch n, the probability calculation unit 321 calculates the probability p(C[k]|ηk=n) by the calculation of Equation (3) shown below.

number

[0058] The condition "ηk=n" in the probability p(C[k]|ηk=n) means that the finger with finger number k is playing pitch n. In other words, the probability p(C[k]|ηk=n) means the probability that the position C[k] is observed for the finger with finger number k under the circumstances where the finger is playing pitch n.

[0059] The symbol I(C[k]∈Rn) in formula (3) is an indicator function that is set to the value "1" when the position C[k] is inside the unit region Rn, and is set to the value "0" when the position C[k] is outside the unit region Rn. The symbol |Rn| means the area of ​​the unit region Rn. In addition, the symbol ν(0,σ 2 E) denotes the observation noise, with mean 0 and variance σ 2 The symbol E is a 2-by-2 unit matrix. The symbol * denotes the observation noise ν(0, σ 2 E) convolution.

[0060] As can be understood from the above explanation, the probability p(C[k]|ηk=n) calculated by the probability calculation unit 321 is the likelihood that, under the condition that the pitch n specified by the performance data P is played by the finger with finger number k, the position of the finger is the position C[k] specified for the finger by the finger position data F. Therefore, the probability p(C[k]|ηk=n) is maximum when the position C[k] of the finger with finger number k is within the unit area Rn of the performance state, and decreases as the position C[k] is farther away from the unit area Rn.

[0061] On the other hand, when the performance data P does not specify any pitch n, that is, when the user does not operate any of the N keys 21, the probability calculation unit 321 calculates the probability p(C[k]|ηk=0) of each fingering using the following formula (4).

number

[0062] As described above, during the period in which the performance data P specifies the pitch n, multiple probabilities p(C[k]|ηk=n) corresponding to different fingers are calculated for each unit period on the time axis. On the other hand, during each unit period in which the performance data P does not specify the pitch n, multiple probabilities p(C[k]|ηk=0) corresponding to different fingers are set to a sufficiently small fixed value (1 / |R|).

[0063] [Fingering Estimation Section 322] The fingering estimation unit 322 estimates the fingering of the user. Specifically, the fingering estimation unit 322 estimates the finger (finger number k) that played the pitch n specified by the performance data P from the probability p(C[k]|ηk=n) of each finger. The fingering estimation unit 322 estimates the finger number k (generates the fingering data Q) every time the probability p(C[k]|ηk=n) of each finger is calculated (i.e., every unit period). Specifically, the fingering estimation unit 322 identifies the finger number k that corresponds to the maximum value among multiple probabilities p(C[k]|ηk=n) corresponding to different fingers. Then, the fingering estimation unit 322 generates the fingering data Q that specifies the pitch n specified by the performance data P and the finger number k identified from the probability p(C[k]|ηk=n).

[0064] In addition, if the maximum value of the multiple probabilities p(C[k]|ηk=n) falls below a predetermined threshold during the period in which the performance data P specifies the pitch n, this means that the reliability of the fingering estimation result is low. Therefore, the fingering estimation unit 322 sets the finger number k to an invalid value, which means that the estimation result is invalid, during the unit period in which the maximum value of the multiple probabilities p(C[k]|ηk=n) falls below the threshold. For notes in which the finger number k is set to an invalid value, the display control unit 40 displays the note image 611 in a manner different from the normal note image 611, as shown in the example of FIG. 4, and displays the symbol "??", which means that the estimation result of the finger number k is invalid. The configuration and operation of the fingering data generation unit 32 are as described above.

[0065] 14 is a flowchart illustrating a specific procedure of the process (hereinafter referred to as the “performance analysis process”) executed by the performance analysis unit 30. For example, the performance analysis process is started in response to an instruction from the user via the operation device 13.

[0066] When the performance analysis process is started, the control device 11 (image extraction unit 311) executes the image extraction process of Fig. 8 (S11). That is, the control device 11 generates a performance image G2 by extracting a specific area B including a keyboard image g1 and a fingering image g2 from the performance image G1. As described above, the image extraction process includes an area estimation process Sb1 and an area extraction process Sb2.

[0067] After executing the image extraction process, the control device 11 (matrix generation unit 312) executes the matrix generation process of Fig. 11 (S12). That is, the control device 11 generates the transformation matrix W by repeatedly updating the initial matrix W0 so that the extended correlation coefficient between the reference image Gref and the keyboard image g1 increases. As described above, the matrix generation process includes the initial setting process Sc1 and the matrix update process Sc2.

[0068] Once the transformation matrix W is generated, the control device 11 repeats the following process (S13 to S18) for each unit period. First, the control device 11 (finger position estimation unit 313) executes the finger position estimation process of FIG. 5 (S13). That is, the control device 11 estimates the positions c[h, f] of the fingers of the user's left and right hands by analyzing the performance image G1. As described above, the finger position estimation process includes the image analysis process Sa1, the left / right determination process Sa2, and the interpolation process Sa3.

[0069] The control device 11 (projective transformation unit 314) executes a projective transformation process (S14). That is, the control device 11 generates a transformed image by projective transformation of the performance image G1 using the transformation matrix W. In the projective transformation process, the control device 11 transforms the positions c[h,f] of each finger of the user to positions C[h,f] in the XY coordinate system, and generates finger position data F representing the positions C[h,f] of each finger.

[0070] After generating the finger position data F by the above process, the control device 11 (probability calculation unit 321) executes a probability calculation process (S15). That is, the control device 11 calculates the probability p(C[k]|ηk=n) that the pitch n specified by the performance data P was played by each finger with finger number k. Then, the control device 11 (fingering estimation unit 322) executes a fingering estimation process (S16). That is, the control device 11 estimates the finger number k of the finger that played the pitch n from the probability p(C[k]|ηk=n) of each finger, and generates fingering data Q that specifies the pitch n and the finger number k.

[0071] When the fingering data Q is generated by the above process, the control device 11 (display control unit 40) updates the analysis screen 61 in accordance with the fingering data Q (S17). The control device 11 also determines whether a predetermined end condition is met (S18). For example, when the user issues an instruction to end the performance analysis process by operating the operation device 13, the control device 11 determines that the end condition is met. If the end condition is not met (S18: NO), the control device 11 repeats the processes (S13 to S18) following the finger position estimation process for the immediately following unit period. On the other hand, if the end condition is met (S18: YES), the control device 11 ends the performance analysis process.

[0072] As described above, in the first embodiment, the fingering data Q is generated using the finger position data F generated by analyzing the performance image G1 and the performance data P representing the performance by the user. Therefore, the fingering can be estimated with a higher accuracy than in a configuration in which the fingering is estimated only from the performance data P.

[0073] In the first embodiment, the position c[h,f] of each finger estimated by the finger position estimation process is transformed using a transformation matrix W for projective transformation that brings the keyboard image g1 closer to the reference image Gref. That is, the position C[h,f] of each finger is estimated based on the reference image Gref. Therefore, fingering can be estimated with high accuracy compared to a configuration in which the position c[h,f] of each finger is not transformed to a position based on the reference image Gref.

[0074] In the first embodiment, a specific region B including the keyboard image g1 is extracted from the performance image G1. Therefore, as described above, an appropriate transformation matrix W can be generated that can accurately approximate the keyboard image g1 to the reference image Gref. Furthermore, by extracting the specific region B, it is possible to improve the usability of the performance image G1. In particular, in the first embodiment, a specific region B including the keyboard image g1 and the fingering image g2 is extracted from the performance image G1. Therefore, a performance image G2 can be generated that allows efficient visual recognition of the state of the keyboard 22 of the keyboard instrument 200 and the state of the user's fingers.

[0075] 2: Second embodiment A second embodiment will be described. In each of the following exemplary embodiments, elements having the same functions as those in the first embodiment will be denoted by the same reference numerals as those used in the description of the first embodiment, and detailed descriptions thereof will be omitted as appropriate.

[0076] In the first embodiment, the probability p(C[k]|ηk=n) is calculated depending on whether the position C[k] of the finger with finger number k exists within the unit area Rn of the pitch n. Assuming that only one finger exists within the unit area Rn, the fingering can be estimated with high accuracy even in the first embodiment. However, in an actual performance of the keyboard instrument 200, there may be cases where multiple finger positions C[k] exist within one unit area Rn.

[0077] For example, as illustrated in FIG. 15, when a user operates one key 21 with the middle finger of the left hand and moves the index finger of the left hand vertically upward, the middle finger and index finger of the left hand overlap each other in the performance image G1. That is, the position C[k] of the middle finger and the position C[k] of the index finger of the left hand exist within one unit area Rn. In addition, in a performance method (finger passing) in which a user operates a key 21 with one finger and passes other fingers above or below the finger, multiple fingers may overlap each other. When multiple fingers overlap each other within one unit area Rn as described above, the method of the first embodiment may not be able to estimate fingering with high accuracy. The second embodiment is a form for solving the above problems. Specifically, in the second embodiment, the positional relationship of multiple fingers and the temporal fluctuation (variation) of the position of each finger are taken into account in the fingering estimation.

[0078] 16 is a block diagram illustrating the functional configuration of a performance analysis system 100 in the second embodiment. The performance analysis system 100 in the second embodiment has the same elements as the first embodiment, and further includes a control data generator 323.

[0079] The control data generating unit 323 generates N pieces of control data Z[1] to Z[N] corresponding to different pitches n. Fig. 17 is a schematic diagram of the control data Z[n] corresponding to any one pitch n. The control data Z[n] is vector data representing the characteristics of the relative position (hereinafter referred to as "relative position") C'[k] of each finger with respect to the unit area Rn of pitch n. The relative position C'[k] is information obtained by converting the position C[k] represented by the finger position data F into a relative position with respect to the unit area Rn.

[0080] The control data Z[n] corresponding to one pitch n includes the pitch n, and also includes a position average Za[n,k], a position variance Zb[n,k], a velocity average Zc[n,k], and a velocity variance Zd[n,k] for each of a plurality of fingers. The position average Za[n,k] is the average of the relative positions C'[k] in a period of a predetermined length including the current unit period (hereinafter referred to as the "observation period"). The observation period is, for example, a period corresponding to a plurality of unit periods arranged forward on the time axis with the current unit period at the end. The position variance Zb[n,k] is the variance of the relative positions C'[k] in the observation period. The velocity average Zc[n,k] is the average of the velocity (i.e., the rate of change) at which the relative position C'[k] changes in the observation period. The velocity variance Zd[n,k] is the variance of the velocity at which the relative position C'[k] changes in the observation period.

[0081] As described above, the control data Z[n] includes information (Za[n,k], Zb[n,k], Zc[n,k], Zd[n,k]) regarding the relative position C'[k] for each of the multiple fingers. Therefore, the control data Z[n] is data that reflects the positional relationship of the user's multiple fingers. The control data Z[n] also includes information (Zb[n,k], Zd[n,k]) regarding the fluctuation of the relative position C'[k] for each of the multiple fingers. Therefore, the control data Z[n] is data that reflects the temporal fluctuation of the position of each finger.

[0082] The probability calculation process by the probability calculation unit 321 of the second embodiment uses multiple estimation models 52[k] (52[1] to 52

[10] ) prepared in advance for different fingers. The estimation model 52[k] for each finger is a trained model that has learned the relationship between the control data Z[n] and the probability p[k] for that finger. The probability p[k] is an index (probability) of the likelihood that the finger with finger number k has played the pitch n specified by the performance data P. The probability calculation unit 321 calculates the probability p[k] for each of the multiple fingers by inputting N pieces of control data Z[1] to Z[N] into the estimation model 52[k] for that finger.

[0083] The estimation model 52[k] corresponding to any one finger number k is a logistic regression model expressed by the following formula (5).

number

[0084] The variables βk and ωk,n in formula (5) are set by machine learning by the machine learning system 900. That is, each estimation model 52[k] is established by machine learning by the machine learning system 900, and each estimation model 52[k] is provided to the performance analysis system 100. For example, the variables βk and ωk,n of each estimation model 52[k] are transmitted from the machine learning system 900 to the performance analysis system 100.

[0085] A finger located above a finger in a key-pressing state, or a finger moving above or below a finger in a key-pressing state, tends to move more easily than a finger in a key-pressing state. Considering the above tendency, the estimation model 52[k] learns the relationship between the control data Z[n] and the probability p[k] so that the probability p[k] is a small value for a finger having a high rate of change in the relative position C'[k]. The probability calculation unit 321 inputs the control data Z[n] to each of the multiple estimation models 52[k] to calculate multiple probabilities p[k] for different fingers for each unit period.

[0086] The fingering estimation unit 322 estimates the fingering of the user by a fingering estimation process that applies multiple probabilities p[k]. Specifically, the fingering estimation unit 322 estimates the finger (finger number k) that played the pitch n specified by the performance data P from the probability p[k] of each finger. The fingering estimation unit 322 estimates the finger number k (generates the fingering data Q) every time the probability p[k] of each finger is calculated (i.e., every unit period). Specifically, the fingering estimation unit 322 identifies the finger number k that corresponds to the maximum value among the multiple probabilities p[k] corresponding to different fingers. Then, the fingering estimation unit 322 generates the fingering data Q that specifies the pitch n specified by the performance data P and the finger number k identified from the probability p[k].

[0087] 18 is a flow chart illustrating the specific steps of the performance analysis process in the second embodiment. In the performance analysis process in the second embodiment, the generation of control data Z[n] (S19) is added to the processes similar to those in the first embodiment. Specifically, the control device 11 (control data generating unit 323) generates N pieces of control data Z[1] to Z[N] corresponding to different pitches n from the finger position data F (i.e., the positions C[h,f] of each finger) generated by the finger position data generating unit 31.

[0088] The control device 11 (probability calculation unit 321) calculates a probability p[k] corresponding to finger number k by a probability calculation process in which N pieces of control data Z[1] to Z[N] are input to each estimation model 52[k] (S15). In addition, the control device 11 (fingering estimation unit 322) estimates the user's fingering by a fingering estimation process that applies the multiple probabilities p[k] (S16). The operations (S11 to S14, S17 to S18) of the elements other than the fingering data generation unit 32 are the same as those in the first embodiment.

[0089] The second embodiment also achieves the same effects as the first embodiment. In the second embodiment, the control data Z[k] input to the estimation model 52[k] includes the average Za[n,k] and variance Zb[n,k] of the relative positions C'[k] of each finger, and the average Zc[n,k] and variance Zd[n,k] of the change rates of the relative positions C'[k]. Therefore, even when multiple fingers overlap each other due to, for example, finger underpassing, the fingering of the user can be estimated with high accuracy.

[0090] In the above description, a logistic regression model is exemplified as the estimation model 52[k], but the type of the estimation model 52[k] is not limited to the above example. For example, a statistical model such as a multi-layer perceptron may be used as the estimation model 52[k]. Also, a deep neural network such as a convolutional neural network or a recurrent neural network may be used as the estimation model 52[k]. A combination of multiple types of statistical models may be used as the estimation model 52[k]. The various estimation models 52[k] exemplified above are collectively expressed as learned models that have learned the relationship between the control data Z[n] and the probability p[k].

[0091] 3: Third embodiment 19 is a flow chart illustrating a specific procedure of the performance analysis process in the third embodiment. After executing the image extraction process and the matrix generation process, the control device 11 judges whether or not the user is playing the keyboard instrument 200 by referring to the performance data P (S21). Specifically, the control device 11 judges whether or not any of the multiple keys 21 of the keyboard instrument 200 is being operated.

[0092] When the keyboard instrument 200 is being played (S21: YES), the control device 11 generates finger position data F (S13-S14), generates fingering data Q (S15-S16), and updates the analysis screen 61 (S17) in the same manner as in the first embodiment. On the other hand, when the keyboard instrument 200 is not being played (S21: NO), the control device 11 shifts the process to step S18. That is, the generation of finger position data F (S13-S14), the generation of fingering data Q (S15-S16), and the update of the analysis screen 61 (S17) are not executed.

[0093] The third embodiment also achieves the same effects as the first embodiment. Moreover, in the third embodiment, when the keyboard instrument 200 is not being played, the generation of the finger position data F and the fingering data Q is stopped. Therefore, the processing load required for generating the fingering data Q can be reduced compared to a configuration in which the generation of the finger position data F continues regardless of whether the keyboard instrument 200 is being played or not. The third embodiment is also applicable to the second embodiment.

[0094] 4: Fourth embodiment The fourth embodiment is a modification of the initial setting process Sc1 in each of the above-described embodiments. Fig. 20 is a flowchart illustrating a specific procedure of the initial setting process Sc1 executed by the control device 11 (matrix generating unit 312) of the fourth embodiment.

[0095] When the initial setting process Sc1 is started, the user operates a key 21 corresponding to a desired pitch (hereinafter referred to as a "specific pitch") n among the multiple keys 21 of the keyboard instrument 200 with a specific finger (hereinafter referred to as a "specific finger"). The specific finger is, for example, a finger (e.g., the index finger of the right hand) notified to the user by a display on the display device 14 or an instruction manual for the keyboard instrument 200. As a result of the performance by the user, performance data P specifying the specific pitch n is supplied from the keyboard instrument 200 to the performance analysis system 100. The control device 11 recognizes the performance of the specific pitch n by the user by acquiring the performance data P from the keyboard instrument 200 (Sc15). The control device 11 specifies a unit area Rn corresponding to the specific pitch n among the N unit areas R1 to RN of the reference image Gref (Sc16).

[0096] On the other hand, the finger position data generating unit 31 generates finger position data F by finger position estimation processing. The finger position data F includes the positions C[h,f] of the specific fingers used by the user to play the specific pitch n. The control device 11 acquires the finger position data F to identify the positions C[h,f] of the specific fingers (Sc17).

[0097] The control device 11 sets an initial matrix W0 using the unit area Rn corresponding to the specific pitch n and the position C[h,f] of the specific finger represented by the finger position data F (Sc18). That is, the control device 11 sets the initial matrix W0 so that the position C[h,f] of the specific finger represented by the finger position data F approaches the unit area Rn of the specific pitch n in the reference image Gref. Specifically, a matrix for projectively transforming the position C[h,f] of the specific finger to the center of the unit area Rn is set as the initial matrix W0.

[0098] The fourth embodiment also achieves the same effect as the first embodiment. In the fourth embodiment, when a user plays a desired specific pitch n with a specific finger, the initial matrix W0 is set so that the position c[h,f] of the specific finger in the performance image G1 approaches a portion (unit area Rn) corresponding to the specific pitch n in the reference image Gref. Since the user only needs to play the desired pitch n, the burden of the user's work required for setting the initial matrix W0 is reduced, compared to the first embodiment in which the user needs to select the target area 621 by operating the operation device 13. On the other hand, according to the first embodiment in which the user specifies the target area 621, it is not necessary to estimate the position C[h,f] of the user's finger, so that an appropriate initial matrix W0 can be set while reducing the influence of estimation errors, compared to the second embodiment. The fourth embodiment is also applicable to the second or third embodiment.

[0099] In the fourth embodiment, it is assumed that the user plays one specific pitch n, but the user may play multiple specific pitches n with specific fingers. The control device 11 sets an initial matrix W0 for each of the multiple specific pitches n so that the position C[h,f] of the specific fingers when playing the specific pitch n approaches the unit area Rn of the specific pitch n.

[0100] 5: Fifth embodiment FIG. 21 is a block diagram illustrating a functional configuration of a performance analysis system 100 according to the fifth embodiment. The performance analysis system 100 according to the fifth embodiment includes a sound collection device 16. The sound collection device 16 collects sounds reproduced from a keyboard instrument 200 during a performance by a user to generate an audio signal V. The audio signal V is a time-domain audio signal representing the waveform of the sound reproduced by the keyboard instrument 200. The sound collection device 16, which is separate from the performance analysis system 100, may be connected to the performance analysis system 100 by wire or wirelessly. The time series of samples constituting the audio signal V may be interpreted as "performance data P."

[0101] The control device 11 of the performance analysis system 100 functions as a performance analysis unit 30 by executing a program stored in the storage device 12. The performance analysis unit 30 generates fingering data Q using the audio signal V provided by the sound collection device 16 and the image data D1 provided by the image capture device 15. As in the first embodiment, the fingering data Q specifies the pitch n corresponding to the key 21 operated by the user and the finger number k of the finger used by the user to operate the key 21. In the first embodiment, the pitch n is specified by the performance data P, but the audio signal V in the fifth embodiment is not a signal that directly specifies the pitch n. Therefore, the performance analysis unit 30 simultaneously estimates the pitch n and the finger number k using the audio signal V and the image data D1.

[0102] To estimate the pitch n and finger number k, we use the latent variable w t,n,kAssume that the symbol t is a variable indicating time. One unit period on the time axis may be indicated by the variable t. Furthermore, the finger number k in the fifth embodiment is set to one of eleven values ​​including ten values ​​(k=1 to 10) corresponding to different fingers and a predetermined invalid value (k=0).

[0103] For each combination of pitch n and finger number k, a latent variable w t,n,k The latent variable w t,n,k is a variable for one-hot representation that is set to either "0" or "1". The latent variable w t,n,k The value "1" of means that the pitch n is played by finger number k, and the latent variable w t,n,k A value of "0" means that none of the fingers are used in playing.

[0104] In addition, the posterior probability U t,n and probability π t,n,k Assume that the posterior probability U t,n is the posterior probability that pitch n is produced at time t under the conditions in which audio signal V is observed. Therefore, the probability (1-U t,n ) is the latent variable w t,n,0 is the number "1" (the probability that no pitch n is played). t,n is the acoustic signal V and the posterior probability U t,n The estimation model is a trained model for automatic music transcription. For example, a deep neural network such as a convolutional neural network or a recurrent neural network estimates the posterior probability U t,n It is used as an estimation model to estimate the probability π t,n,k is the probability that pitch n is being played by finger number k when that pitch is being played.

[0105] Acoustic signal V and probability π t,n,k The latent variable w when t,n,k The probability p(w|V,π) is expressed by the following equation (6).

number

[0106] Also, the latent variable w t,n,k The probability p(C[k]|w) that position C[k] is observed from the performance image G1 when is observed is expressed by the following formula (7).

number

[0107] Also, probability π t,n,k As the prior distribution of , we assume a symmetric Dirichlet distribution (Dir) expressed by the following equation (8).

number

[0108] Under the above assumptions, the latent variable w t,n,k By performing maximum a posteriori (MAP) estimation to maximize the a posteriori probability p(z|V,π,C[k]), it is possible to simultaneously estimate the presence or absence of pitch n and finger number k. However, since it is difficult to estimate the probability distribution of the a posteriori probability p(z|V,π,C[k]), in the fifth embodiment, a mean field approximation (variational Bayesian estimation) is considered.

[0109] Specifically, among the distributions factorized as shown in the following formula (9), a distribution that is most similar to the probability distribution of the posterior probability p(z|V,π,C[k]) is identified. For example, a distribution that has the smallest KL (Kullback-Leibler) distance from the posterior probability p(z|V,π,C[k]) is identified.

number

[0110] Specifically, the performance analysis unit 30 repeats the calculations of the following formulas (10) and (11).

number

number

[0111] Specifically, the performance analysis unit 30 repeats the calculation of formula (10) and formula (11) for all combinations of pitch n and finger number k for one time t on the time axis. The performance analysis unit 30 calculates the calculation result of formula (10) at the time point when the calculation of formula (10) and formula (11) is repeated a predetermined number of times as a latent variable w t,n,k The probability distribution of ρ t,n,k For each time t on the time axis, the probability distribution ρ t,n,k is calculated.

[0112] By the way, the probability distribution ρ t,n,k Therefore, in the case where the pitch n and the finger number k are calculated for each time t, the finger number k may change between the times t before and after the user plays one note, or the period during which the pitch n continues may be too short. t,n,k We use a Hidden Markov Model (HMM) to generate a time series of combinations of pitch n and finger number k (i.e., fingering data Q).

[0113] Specifically, the HMM for fingering estimation is composed of latent states corresponding to each of the sounding (key pressing) and muting of pitch n, and multiple latent states corresponding to different finger numbers k. Only three types of state transitions are allowed: (1) self-transition, (2) silence → any finger number k, and (3) any finger number k → silence, and the transition probability for other state transitions is set to "0." The above conditions are constraints for not changing the finger number k during the period in which one note is sounded. In addition, the probability distribution ρ t,n,k The expected value of is set as the observation probability for each latent state of the HMM. The performance analysis unit 30 uses the above-described HMM to estimate a state sequence by dynamic programming such as the Viterbi algorithm. The performance analysis unit 30 generates a time series of fingering data Q according to the result of estimating the state sequence.

[0114] According to the fifth embodiment, fingering data Q is generated using the audio signal V and image data D1. That is, fingering data Q can be generated even in a situation where performance data P cannot be acquired. In addition, in the fifth embodiment, pitch n and finger number k are simultaneously estimated using the audio signal V and image data D1, so fingering can be estimated with high accuracy while reducing the processing load compared to a form in which pitch n and finger number k are estimated separately. The fifth embodiment is also applied to the second to fourth embodiments.

[0115] 6: Sixth embodiment As exemplified in each of the above embodiments, the projective transformation unit 314 generates a transformed image from the performance image G1. That is, the projective transformation unit 314 changes the shooting conditions of the performance image G1. The sixth embodiment is an image processing system 700 that utilizes the above function of changing the shooting conditions of the performance image G1. Note that the performance analysis systems 100 of the first to fifth embodiments can also be expressed as image processing systems 700 if attention is focused on the processing of the performance image G1 by the projective transformation unit 314. Note that in the sixth embodiment, it is not essential to estimate the user's fingering.

[0116] 22 is a block diagram illustrating the functional configuration of an image processing system 700 in the sixth embodiment. As with the performance analysis system 100 in the first embodiment, the image processing system 700 includes a control device 11, a storage device 12, an operation device 13, a display device 14, and a photographing device 15. As with the first embodiment, the photographing device 15 photographs a keyboard instrument 200 under specific photographing conditions to generate a time series of image data D1 representing a performance image G1.

[0117] The storage device 12 stores a plurality of reference data Dref. Each of the plurality of reference data Dref represents a reference image Gref obtained by photographing a reference instrument, which is a standard keyboard instrument. The photographing conditions of the reference instrument differ for each reference image Gref (each reference data Dref). Specifically, for example, one or more conditions of the photographing range or the photographing direction differ for each reference image Gref. The storage device 12 also stores auxiliary data A for each reference data Dref.

[0118] The control device 11 executes a program stored in the storage device 12 to realize a matrix generation unit 312, a projective transformation unit 314, and a display control unit 40. The matrix generation unit 312 selectively uses one of a plurality of reference data Dref to generate a transformation matrix W. The projective transformation unit 314 generates image data D3 of a transformed image G3 from image data D1 of a performance image G1 by projective transformation using the transformation matrix W. The display control unit 40 causes the display device 14 to display the transformed image G3 represented by the image data D3.

[0119] 23 is a flowchart illustrating a specific procedure of a process (hereinafter, referred to as "first image process") executed by the control device 11 of the sixth embodiment. For example, the first image process is started in response to an instruction from a user via the operation device 13.

[0120] The user operates the operation device 13 to select one of a plurality of shooting conditions corresponding to different reference images Gref. The control device 11 (matrix generation unit 312) judges whether or not the selection of the shooting condition has been accepted from the user (S31). When the selection of the shooting condition has been accepted (S31: YES), the control device 11 (matrix generation unit 312) acquires reference data Dref (hereinafter referred to as "selected reference data Dref") corresponding to the shooting condition selected by the user from among the plurality of reference data Dref stored in the storage device 12 (S32). The selection of the shooting condition by the user corresponds to an operation of selecting one of a plurality of reference images Gref (reference data Dref) corresponding to different shooting conditions.

[0121] The control device 11 (matrix generation unit 312) executes a matrix generation process similar to that of the first embodiment using the selected reference data Dref (S33). Specifically, the control device 11 sets an initial matrix W0 by an initial setting process Sc1 using the selected reference data Dref. The control device 11 also generates a transformation matrix W by a matrix update process Sc2 that iteratively updates the initial matrix W0 so that the keyboard image g1 of the performance image G1 approaches the reference image Gref of the selected reference data Dref. On the other hand, if the selection of the shooting conditions is not accepted (S31: NO), the selection of the reference data Dref (S32) and the matrix generation process (S33) are not executed.

[0122] The control device 11 (projective transformation unit 314) generates a converted image G3 by executing a projective transformation process using the transformation matrix W on the performance image G1 (S34). The projective transformation process is the same as in the first embodiment. As a result of the projective transformation process, image data D3 representing the converted image G3 is generated. Specifically, a converted image G3 corresponding to shooting conditions equivalent to those of the reference image Gref of the selected reference data Dref is generated from the performance image G1. In other words, the converted image G3 is an image obtained by converting the shooting conditions of the performance image G1 to shooting conditions equivalent to those of the reference image Gref. As can be understood from the above explanation, according to the sixth embodiment, a converted image G3 corresponding to the shooting conditions selected by the user is generated.

[0123] The control device 11 (display control unit 40) causes the display device 14 to display the converted image G3 generated by the projective transformation process (S35). The control device 11 determines whether or not the end condition is satisfied (S36). For example, when the user instructs the end of the first image process by operating the operation device 13, the control device 11 determines that the end condition is satisfied. When the end condition is not satisfied (S36: NO), the control device 11 shifts the process to step S31. That is, the generation of the transformation matrix W (S32 to S33) and the generation and display of the converted image G3 (S34 to S35) are executed on the condition that the selection of the shooting conditions is accepted (S31: YES). On the other hand, when the end condition is satisfied (S36: YES), the control device 11 ends the first image process.

[0124] As described above, in the sixth embodiment, the transformation matrix W is generated so that the keyboard image g1 in the performance image G1 approaches the reference image Gref, and the projective transformation process using the transformation matrix W is executed on the performance image G1. Therefore, the performance image G1 of the keyboard instrument 200 played by the user can be transformed into a transformed image G3 that corresponds to the shooting conditions of the reference instrument in the reference image Gref.

[0125] In the sixth embodiment, one of the multiple reference data Drefs with different shooting conditions is selectively used in the matrix generation process. Therefore, a converted image G3 corresponding to various shooting conditions can be generated from a performance image G1 captured under a specific shooting condition. In particular, in the sixth embodiment, the reference data Dref corresponding to the shooting condition selected by the user among the multiple reference data Drefs is used in the matrix generation process, so that a converted image G3 corresponding to the shooting condition desired by the user can be generated. By changing the shooting condition of the performance image G1 as described above, a converted image G3 that can be used for various purposes can be generated. For example, a music instructor performs the first image process of the sixth embodiment on each of multiple performance images G1 captured by his / her own performance, and thus a plurality of converted images G3 with unified shooting conditions can be generated, for example, as teaching materials for music training.

[0126] 7: Seventh embodiment As exemplified in each of the above embodiments, the image extraction unit 311 extracts a specific area B including a keyboard image g1 and a fingering image g2 from the performance image G1. The seventh embodiment is an image processing system 700 that utilizes the above function of extracting a specific area B from the performance image G1. Note that the performance analysis systems 100 of the first to fifth embodiments can also be expressed as image processing systems 700 if attention is focused on the processing of the performance image G1 by the image extraction unit 311. Note that in the seventh embodiment, estimation of the user's fingering is not essential.

[0127] 24 is a block diagram illustrating the functional configuration of an image processing system 700 in the seventh embodiment. As with the performance analysis system 100 in the first embodiment, the image processing system 700 includes a control device 11, a storage device 12, an operation device 13, a display device 14, and a photographing device 15. The photographing device 15 photographs the keyboard instrument 200 under specific photographing conditions to generate a time series of image data D1 representing a performance image G1. As with the above-mentioned embodiments, the performance image G1 includes a keyboard image g1 and a fingering image g2.

[0128] The control device 11 executes a program stored in the storage device 12 to function as an image extraction unit 311 and a display control unit 40. The image extraction unit 311 generates image data D2 representing a performance image G2 obtained by extracting a partial area of ​​a performance image G1. Specifically, the image extraction unit 311 executes an area estimation process Sb1 for generating an image processing mask M, and an area extraction process Sb2 for applying the image processing mask M to the performance image G1, similar to the first embodiment. The display control unit 40 causes the display device 14 to display the performance image G2 represented by the image data D2.

[0129] In the first embodiment, a single estimation model 51 is illustrated. In the seventh embodiment, the estimation model 51 used in the region estimation process Sb1 includes a first model 511 and a second model 512. Each of the first model 511 and the second model 512 is configured with a deep neural network such as a convolutional neural network or a recurrent neural network.

[0130] The first model 511 is a statistical model for generating a first mask representing a first region of the performance image G1. The first region is a region of the performance image G1 that includes the keyboard image g1. The fingering image g2 is not included in the first region. The first mask is, for example, a binary mask in which each element in the first region is set to a numerical value "1" and each element in the region other than the first region is set to a numerical value "0". The image extraction unit 311 generates the first mask by inputting image data D1 representing the performance image G1 to the first model 511. In other words, the first model 511 is a learned model that has learned the relationship between the image data D1 and the first mask (first region) by machine learning.

[0131] The second model 512 is a statistical model for generating a second mask representing the second region of the performance image G1. The second region is a region of the performance image G1 that includes the fingering image g2. The keyboard image g1 is not included in the second region. The second mask is, for example, a binary mask in which each element in the second region is set to a numerical value of "1" and each element in the region other than the second region is set to a numerical value of "0". The image extraction unit 311 generates the second mask by inputting the image data D1 representing the performance image G1 to the second model 512. In other words, the second model 512 is a learned model that has learned the relationship between the image data D1 and the second mask (second region) by machine learning.

[0132] 25 is a flowchart illustrating a specific procedure of a process (hereinafter referred to as "second image process") executed by the control device 11 of the seventh embodiment. For example, the second image process is started in response to an instruction from a user via the operation device 13.

[0133] When the second image processing is started, the control device 11 (image extraction unit 311) executes a region estimation process Sb1 (S41 to S43). The region estimation process Sb1 of the seventh embodiment includes a first estimation process (S41), a second estimation process (S42), and a region combination process (S43).

[0134] The first estimation process is a process of estimating a first region of the performance image G1. Specifically, the control device 11 generates a first mask representing the first region by inputting image data D1 representing the performance image G1 to the first model 511 (S41). The second estimation process is a process of estimating a second region of the performance image G2. Specifically, the control device 11 generates a second mask representing the second region by inputting image data D1 representing the performance image G1 to the second model 512 (S42).

[0135] The area synthesis process is a process for generating an image processing mask M that represents a specific area B including a first area and a second area. Specifically, the specific area B represented by the image processing mask M corresponds to the sum of the first area and the second area. That is, the control device 11 generates the image processing mask M by synthesizing the first mask and the second mask (S43). As can be understood from the above description, the image processing mask M is a binary mask for extracting the specific area B including the keyboard image g1 and the fingering image g2 from the performance image G1, similar to the first embodiment.

[0136] The control device 11 (image extraction unit 311) executes the same area extraction process Sb2 as in the first embodiment by using the image processing mask M generated in the area estimation process Sb1 (S44). That is, the control device 11 extracts the specific area B from the performance image G1 represented by the image data D1 using the image processing mask M, thereby generating image data D2 representing the performance image G2.

[0137] The control device 11 (display control unit 40) causes the display device 14 to display the performance image G2 generated by the area extraction process Sb2 (S45). The control device 11 determines whether or not the end condition is met (S46). For example, when the user instructs the second image process to end by operating the operation device 13, the control device 11 determines that the end condition is met. When the end condition is not met (S46: NO), the control device 11 shifts the process to step S41. That is, the area estimation process Sb1 (S41 to S43), the area extraction process Sb2 (S44), and the display of the performance image G2 (S45) are executed. On the other hand, when the end condition is met (S46: YES), the control device 11 ends the second image process.

[0138] In the seventh embodiment, as in the first embodiment, a specific region B including the keyboard image g1 is extracted from the performance image G1. Therefore, it is possible to improve the usability of the performance image G1. In particular, in the seventh embodiment, a specific region B including the keyboard image g1 and the fingering image g2 is extracted from the performance image G1. Therefore, it is possible to generate a performance image G2 that allows the appearance of the keyboard 22 of the keyboard instrument 200 and the appearance of the user's fingers to be efficiently visually recognized.

[0139] According to the seventh embodiment, a first region including the keyboard image g1 in the performance image G1 is estimated by the first model 511, and a second region including the fingering image g2 in the performance image G1 is estimated by the second model 512. Therefore, compared with a configuration using a single estimation model 51 that collectively extracts both the keyboard image g1 and the fingering image g2, a specific region B including the keyboard image g1 and the fingering image g2 can be extracted with high accuracy. Also, since each of the first model 511 and the second model 512 is established by individual machine learning, the processing load related to the machine learning of the first model 511 and the second model 512 is reduced.

[0140] It is also assumed that the image extraction unit 311 can switch between the first mode and the second mode. The first mode is an operation mode in which both the keyboard image g1 and the fingering image g2 are extracted from the performance image G1. That is, in the first mode, the image extraction unit 311 executes both the first estimation process and the second estimation process. Therefore, as in the seventh embodiment, an image processing mask M representing the specific region B is generated. That is, in the first mode, the specific region B including both the keyboard image g1 and the fingering image g2 is extracted from the performance image G1.

[0141] The second mode is an operation mode in which the keyboard image g1 is extracted from the performance image G1. That is, in the second mode, the image extraction unit 311 executes the first estimation process but does not execute the second estimation process. That is, the first mask generated by the first estimation process is determined as the image processing mask M to be applied to the region extraction process Sb2. Therefore, in the second mode, the keyboard image g1 is extracted from the performance image G1.

[0142] As described above, according to the embodiment in which the first mode and the second mode can be switched, it is possible to easily switch the extraction target from the performance image G1. In the above description, the image extraction unit 311 executes the first estimation process in the second mode, but it is also possible to assume an embodiment in which the image extraction unit 311 executes the second estimation process while not executing the first estimation process in the second mode. In the above embodiment, the hand and finger image g2 is extracted from the performance image G1. As can be understood from the above example, the second mode is expressed as an operation mode in which one of the first estimation process and the second estimation process is executed.

[0143] 8: Variations Specific modified embodiments added to each of the above-mentioned embodiments are exemplified below. Two or more embodiments selected from the following examples may be appropriately combined as long as they are not mutually contradictory.

[0144] (1) In each of the above-described embodiments, the matrix generation process is executed on the performance image G2 processed by the image extraction process (FIG. 8), but the matrix generation process may be executed on the performance image G1 captured by the image capture device 15. In other words, the image extraction process (image extraction unit 311) that generates the performance image G2 from the performance image G1 may be omitted.

[0145] In the above-mentioned embodiments, the finger position estimation process is performed using the performance image G1, but the finger position estimation process may be performed using the performance image G2 after the image extraction process. That is, the position C[h,f] of each finger of the user may be estimated by analyzing the performance image G2. In addition, in the above-mentioned embodiments, the projective transformation process is performed on the performance image G1, but the projective transformation process may be performed on the performance image G2 after the image extraction process. That is, a transformed image may be generated by projective transformation of the performance image G2.

[0146] (2) In the above-described embodiments, the position c[h,f] of each finger of the user is converted to the position C[h,f] in the XY coordinate system by projective transformation processing, but finger position data F representing the position c[h,f] of each finger may be generated. In other words, the projective transformation processing (projective transformation unit 314) for converting the position c[h,f] to the position C[h,f] may be omitted.

[0147] (3) In the first to fifth embodiments, the transformation matrix W generated immediately after the start of the performance analysis process is continuously used in the subsequent processes, but the transformation matrix W may be updated at an appropriate time during the performance analysis process. For example, a form is envisaged in which the transformation matrix W is updated when the position of the photographing device 15 relative to the keyboard instrument 200 changes. Specifically, the transformation matrix W is updated when a change in the position of the photographing device 15 (hereinafter referred to as "position change") is detected by analysis of the performance image G1, or when a position change of the photographing device 15 is instructed by the user.

[0148] Specifically, the matrix generating unit 312 generates a transformation matrix δ that represents a position change (displacement) of the image capturing device 15. For example, the relationship expressed by the following formula (12) is assumed for the coordinates (x, y) in the performance image G (G1, G2) after the position change.

number

[0149] The matrix generation unit 312 generates a transformation matrix δ so that a coordinate x' / ε calculated from the x coordinate of a specific point after the position change using equation (12) approximates or matches the x coordinate of a point corresponding to the point in the performance image G before the position change, and a coordinate y' / ε calculated from the y coordinate of the specific point after the position change using equation (12) approximates or matches the y coordinate of a point corresponding to the point in the performance image G before the position change. The matrix generation unit 312 then generates a product Wδ of the transformation matrix W before the position change and the transformation matrix δ representing the position change as an initial matrix W0, and updates the initial matrix W0 using matrix update processing Sc2 to generate a transformation matrix W.

[0150] In the above configuration, the transformation matrix W after the position change is generated using the transformation matrix W calculated before the position change and the transformation matrix δ representing the position change. Therefore, it is possible to generate the transformation matrix W capable of identifying the position C[h,f] of each finger with high accuracy while reducing the load of the matrix generation process. Note that, although the above description assumes the first to fifth embodiments, the transformation matrix W may also be updated at an appropriate time during the execution of the first image process in the sixth embodiment.

[0151] (4) In each of the above embodiments, a keyboard instrument 200 having a keyboard 22 is exemplified, but the present disclosure can be applied to any type of instrument. For example, the above embodiments can be applied to any instrument that can be manually operated by a user, such as a string instrument, a wind instrument, or a percussion instrument. A typical example of an instrument is a type of instrument that a user plays with the fingers of one or both hands.

[0152] (5) The performance analysis system 100 may be realized by a server device that communicates with an information device such as a smartphone or a tablet terminal. For example, performance data P generated by a keyboard instrument 200 connected to the information device and image data D1 generated by a photographing device 15 mounted on or connected to the information device are transmitted from the information device to the performance analysis system 100. The performance analysis system 100 generates fingering data Q by executing a performance analysis process on the performance data P and image data D1 received from the information device, and transmits the fingering data Q to the information device. Similarly, the image processing system 700 exemplified in the sixth or seventh embodiment may be realized by a server device that communicates with the information device.

[0153] (6) The functions of the performance analysis system 100 according to the first to fifth embodiments or the image processing system 700 according to the sixth to seventh embodiments are realized by the cooperation of one or more processors constituting the control device 11 and the program stored in the storage device 12, as described above. The program according to the present disclosure can be provided in a form stored in a computer-readable recording medium and installed in a computer. The recording medium is, for example, a non-transitory recording medium, and a good example is an optical recording medium (optical disk) such as a CD-ROM, but also includes any known type of recording medium such as a semiconductor recording medium or a magnetic recording medium. Note that the non-transitory recording medium includes any recording medium except for a transient, propagating signal, and does not exclude volatile recording media. In addition, in a configuration in which a distribution device distributes a program via a communication network, the storage device 12 that stores the program in the distribution device corresponds to the non-transitory recording medium described above.

[0154] 9: Notes From the above-described exemplary embodiments, the following configurations can be understood, for example.

[0155] An image processing method according to one aspect (aspect 1) of the present disclosure estimates a specific region including an image of a musical instrument from a performance image including an image of the musical instrument and an image of the fingers of a user playing the musical instrument, and extracts the specific region from the performance image. In the above aspect, a specific region including an image of a musical instrument is extracted from a performance image including an image of a musical instrument and an image of the fingers of a user. Therefore, it is possible to improve the convenience of the performance image.

[0156] In a specific example (aspect 2) of aspect 1, the specific area is an area including an image of the musical instrument and an image of at least a part of the user's body. In the above aspect, a specific area including an image of the musical instrument and an image of the user's body is extracted. Therefore, an image that allows the appearance of the musical instrument and the appearance of the user's body to be efficiently visually recognized can be generated.

[0157] In a specific example (Aspect 3) of Aspect 2, in estimating the specific region, image data representing the performance image is input to a machine-learned estimation model to generate an image processing mask representing the specific region, and in extracting the specific region, the image processing mask is applied to the performance image to extract the specific region. In the above aspect, an image processing mask representing the specific region is generated by inputting image data of the performance image to a machine-learned estimation model. Therefore, it is possible to identify the specific region with high accuracy for unknown and diverse performance images.

[0158] In a specific example (aspect 4) of aspect 3, the estimation model includes a first model and a second model, and the estimation of the specific area includes a first estimation process for estimating a first area including the image of the musical instrument in the musical performance image by inputting image data representing the musical performance image into the first model, a second estimation process for estimating a second area including the image of the multiple fingers in the musical performance image by inputting image data representing the musical performance image into the second model, and an area synthesis process for generating the image processing mask representing the specific area including the first area and the second area. In the above aspect, the first area including the image of the musical instrument in the musical performance image is estimated by the first model, and the second area including the image of the user in the musical performance image is estimated by the second model. Therefore, the specific area including the image of the musical instrument and the image of the user can be extracted with high accuracy compared to a configuration using a single model that extracts both the image of the musical instrument and the image of the user at once. In addition, since each of the first model and the second model is established by individual machine learning, the processing load related to the machine learning of the first model and the second model is reduced.

[0159] In a specific example (Aspect 5) of Aspect 4, it is possible to switch between a first mode in which both the first estimation process and the second estimation process are executed, and a second mode in which one of the first estimation process and the second estimation process is executed. In the above aspect, in the first mode, a specific area including an image of the musical instrument and an image of the user is extracted from the performance image. On the other hand, in the second mode, a specific area including one of the image of the musical instrument and the image of the user is extracted from the performance image. As described above, it is possible to easily switch the extraction target from the performance image.

[0160] An image processing system according to one embodiment (embodiment 6) of the present disclosure includes an area estimation unit that estimates a specific area including an image of an instrument from a performance image including an image of an instrument and an image of multiple fingers of a user playing the instrument, and an area extraction unit that extracts the specific area from the performance image.

[0161] A program according to one embodiment (embodiment 7) of the present disclosure causes a computer system to function as an area estimation unit that estimates a specific area including an image of an instrument from a performance image including an image of an instrument and an image of the fingers of a user playing the instrument, and an area extraction unit that extracts the specific area from the performance image. [Explanation of symbols]

[0162] 100...performance analysis system, 11...control device, 12...storage device, 13...operation device, 14...display device, 15...imaging device, 200...keyboard instrument, 21...key, 22...keyboard, 30...performance analysis unit, 31...finger position data generation unit, 311...image extraction unit, 312...matrix generation unit, 313...finger position estimation unit, 314...projection transformation unit, 32...fingering data generation unit, 321...probability calculation unit, 322...fingering estimation unit, 323...control data generation unit, 40...display control unit, 51...estimation model, 51a...tentative model, 52[k]...estimation model, 700...image processing system.

Claims

1. A specific area is extracted from a performance image including a keyboard image of a keyboard instrument and an image of a plurality of fingers of a user playing the keyboard instrument. An image processing method implemented by a computer system, comprising: In the extraction of the specific region, a first mode in which the specific area including both the keyboard image and the hand image is extracted from the performance image; a second mode in which the specific area including the keyboard image and not including the fingering image is extracted from the performance image; It is switchable, Image processing methods.

2. In the extraction of the specific region, inputting image data representing the performance image into a machine-learned estimation model to generate an image processing mask representing the specific region; The specific region is extracted by applying the image processing mask to the performance image. The image processing method according to claim 1.

3. the estimation model includes a first model and a second model; In the first mode, a first estimation process for estimating a first region of the performance image including the keyboard image by inputting image data representing the performance image into the first model; a second estimation process for estimating a second region including the hand image in the performance image by inputting image data representing the performance image into the second model; generating a first image processing mask representing the specific region including the first region and the second region; extracting the specific region by applying the first image processing mask to the performance image; In the second mode, By executing the first estimation process, a second image processing mask is generated that represents the specific area including the keyboard image and not including the hand image; The specific region is extracted by applying the second image processing mask to the performance image. The image processing method according to claim 2.

4. An image extraction unit that extracts a specific area from a performance image including a keyboard image of a keyboard instrument and images of a plurality of fingers of a user playing the keyboard instrument, The image extraction unit a first mode in which the specific area including both the keyboard image and the hand image is extracted from the performance image; a second mode in which the specific area including the keyboard image and not including the fingering image is extracted from the performance image; It is switchable, Image processing system.

5. An image extraction unit that extracts a specific area from a performance image including a keyboard image of a keyboard instrument and images of a plurality of fingers of a user playing the keyboard instrument. A program for causing a computer system to function as The image extraction unit a first mode in which the specific area including both the keyboard image and the hand image is extracted from the performance image; a second mode in which the specific area including the keyboard image and not including the fingering image is extracted from the performance image; It is switchable, program.

Images