Information processing method, information processing system, and program

Abstract

This information processing system is provided with: an image acquisition unit 41 that acquires a plurality of captured images Gn obtained by imaging a to-be-captured object from different directions; a pitch acquisition unit 42 that acquires a played pitch P generated by the to-be-captured object; and an estimation unit 43 that estimates the position, in a three-dimensional space, of an estimation target related to the to-be-captured object through an analysis using the plurality of captured images Gn and the played pitch P.

Description

Information processing method, information processing system, and program

[0001] The present disclosure relates to a technique for estimating the position of a subject to be photographed.

[0002] Various techniques have been proposed for estimating the position of an object, such as a performer's hand, in three-dimensional space. For example, Patent Document 1 discloses a configuration for generating an image of a performer's fingering using a performance control recognized as black in an image captured by an infrared sensor. Furthermore, Patent Document 2 discloses a configuration for inputting a captured image of a performer's hand with markers added to a trained model to generate an image of the hand from which the markers have been deleted, and then generating three-dimensional posture data of the hand from the image after the markers have been deleted.

[0003] JP 2017-181850 A JP 2024-012764 A

[0004] However, the technology of Patent Document 1 requires a performance operator that is recognized as black in an image captured by an infrared sensor. Furthermore, the technology of Patent Document 2 requires a marker attached to the performer's hand. In other words, conventional technologies require an element (performance operator or marker) dedicated to analyzing the estimation target. Taking the above circumstances into consideration, one aspect of the present disclosure aims to estimate the position of an estimation target in three-dimensional space using a simple method that does not require an element dedicated to analyzing the estimation target.

[0005] In order to solve the above problems, an information processing method according to one aspect of the present disclosure acquires a plurality of captured images of a subject from different directions, acquires the pitch of a performance sound produced by the subject, and estimates the position in three-dimensional space of an estimation target related to the subject through an analysis process using the plurality of captured images and the pitch of the performance sound.

[0006] An information processing system according to one aspect of the present disclosure includes an image acquisition unit that acquires a plurality of captured images of a subject from different directions, a pitch acquisition unit that acquires the pitch of a performance sounded by the subject, and an estimation processing unit that estimates the position in three-dimensional space of an estimation target related to the subject through analysis processing using the plurality of captured images and the performance pitch.

[0007] A program according to one aspect of the present disclosure causes a computer system to function as an image acquisition unit that acquires multiple captured images of a subject from different directions, a pitch acquisition unit that acquires the pitch of a performance sounded by the subject, and an estimation processing unit that estimates the position in three-dimensional space of an estimation target related to the subject through analysis processing using the multiple captured images and the performance pitch.

[0008] FIG. 1 is a block diagram illustrating an example of the configuration of a performance analysis system in a first embodiment. FIG. 2 is a schematic diagram of a captured image. FIG. 3 is a block diagram illustrating an example of the configuration of an information processing system. FIG. 4 is a block diagram illustrating an example of the functional configuration of an information processing system. FIG. 5 is an explanatory diagram of an estimation result by an estimation processing unit. FIG. 6 is a flowchart of analysis processing. FIG. 7 is an explanatory diagram of analysis processing. FIG. 8 is a flowchart of candidate point identification processing. FIG. 9 is a block diagram illustrating an example of the configuration of a performance analysis system in a second embodiment.

[0009] A: First Embodiment Fig. 1 is a block diagram illustrating the configuration of a performance analysis system 100 according to the first embodiment. The performance analysis system 100 is a computer system for analyzing a performance by a performer U. The performance analysis system 100 includes a keyboard instrument 10, a photography system 20, and an information processing system 30.

[0010] The keyboard instrument 10 is an instrument that produces sounds in response to playing by a performer U. The keyboard instrument 10 includes a keyboard 11 on which a plurality of keys 12 are arranged horizontally. The plurality of keys 12 on the keyboard 11 include a plurality of white keys and a plurality of black keys. The keyboard instrument 10 may be either a natural musical instrument that produces musical sounds by vibration of a sound source such as a string, or an electronic musical instrument that electronically produces musical sounds using a sound source device.

[0011] The photographing system 20 photographs the situation in which the performer U plays the keyboard instrument 10. The photographing system 20 includes a plurality of photographing devices 21-n (n = 1, 2). For example, each photographing device 21-n includes a photographing optical system such as a photographing lens, an image sensor that receives incident light from the photographing optical system, and a processing circuit that generates image data according to the amount of light received by the image sensor. The electrical characteristics or optical characteristics (e.g., focal length or aberration characteristics) are common to the photographing devices 21-1 and 21-2. However, the characteristics may differ for each photographing device 21-n.

[0012] Each of the multiple image capturing devices 21-n is installed facing the keyboard instrument 10. The position and direction with respect to the keyboard instrument 10 (e.g., the keyboard 11) differs for each image capturing device 21-n. For example, the image capturing device 21-1 is installed diagonally above and to the left of the keyboard instrument 10. The image capturing device 21-2 is installed diagonally above and to the right of the keyboard instrument 10. Each image capturing device 21-n generates a captured image Gn by capturing an image of the keyboard instrument 10.

[0013] FIG. 2 is a schematic diagram of a captured image Gn. The captured image Gn is a planar image representing a performance of the keyboard instrument 10 by a performer U. Specifically, as illustrated in FIG. 2, each captured image Gn includes the keys 11 of the keyboard instrument 10 and the hands (right and left hands) of the performer U playing the keyboard instrument 10. The captured image G1 is an image of the keyboard instrument 10 and the performer U taken from an upper left angle, and the captured image G2 is an image of the keyboard instrument 10 and the performer U taken from an upper right angle. The captured image Gn is represented by image data in any format. The generation of the captured image Gn is repeated for each unit period (frame) of a predetermined length. A moving image of the performance by the performer U is constructed from a time series of multiple captured images Gn.

[0014] The multiple image capturing devices 21-n capture images of the performer U playing the keyboard instrument 10 in parallel with one another. As explained above, the multiple captured images Gn are images of the keyboard instrument 10 captured from different positions and directions. The captured images Gn captured by each image capturing device 21-n are transmitted from the image capturing device 21-n to the information processing system 30.

[0015] The information processing system 30 is a computer system that generates various information related to the performance of the keyboard instrument 10 by the performer U by analyzing multiple captured images Gn transmitted from different image capture devices 21-n. The information processing system 30 is realized by an information device such as a smartphone, a tablet terminal, or a personal computer.

[0016] 3 is a block diagram illustrating an example of the configuration of an information processing system 30. The information processing system 30 includes a control device 31, a storage device 32, a display device 34, an operation device 35, and a sound collection device 36. The information processing system 30 may be realized by a single device, or may be realized by multiple devices configured separately from each other.

[0017] The control device 31 is configured with one or more processors that control each element of the information processing system 30. Specifically, the control device 31 is configured with one or more types of processors, such as a central processing unit (CPU), a graphics processing unit (GPU), a sound processing unit (SPU), a digital signal processor (DSP), a field programmable gate array (FPGA), or an application specific integrated circuit (ASIC).

[0018] The storage device 32 is one or more memories that store programs executed by the control device 31 and data used by the control device 31. The storage device 32 is configured with a known storage medium such as a magnetic storage medium or a semiconductor storage medium. The storage device 32 may be configured with a combination of multiple types of storage media. Furthermore, a portable storage medium that is detachable from the information processing system 30, or a storage medium to which the control device 31 can write or read via a communication network (e.g., cloud storage) may be used as the storage device 32.

[0019] The communication device 33 communicates with an external device. Specifically, the communication device 33 receives the captured image Gn transmitted from each of the image capturing devices 21-n via a wired or wireless connection. Note that the communication device 33, which is separate from the information processing system 30, may be connected to the information processing system 30 via a wired or wireless connection.

[0020] The display device 34 displays an image under the control of the control device 31. The display device 34 is configured with a display panel such as a liquid crystal panel or an organic EL (Electroluminescence) panel. The operation device 35 is an input device that accepts operations from the performer U. Note that the display device 34 or the operation device 35, which are separate from the information processing system 30, may be connected to the information processing system 30 by wire or wirelessly.

[0021] The sound collection device 36 is one or more microphones that collect sounds produced by the keyboard instrument 10 in response to the performance of the performer U. The sound collection device 36 generates an audio signal A that represents the waveform of the sound produced by the keyboard instrument 10. For convenience, an amplifier that amplifies the audio signal A generated by the sound collection device 36 and an A / D converter that converts the audio signal A from analog to digital are not shown in the figure.

[0022] 4 is a block diagram illustrating an example of the functional configuration of the information processing system 30. The control device 31 executes a program stored in the storage device 32 to realize multiple functions (an image acquisition unit 41, a pitch acquisition unit 42, and an estimation processing unit 43) for analyzing multiple captured images Gn.

[0023] The image acquisition unit 41 acquires a plurality of photographed images Gn captured by different photographing devices 21-n. Specifically, the image acquisition unit 41 receives the photographed images Gn transmitted from each photographing device 21-n via the communication device 33.

[0024] The pitch acquisition unit 42 acquires the pitch of the note produced by the keyboard instrument 10 (hereinafter referred to as the "performance pitch P"). The performance pitch P is the pitch produced by the performer U on the keyboard instrument 10. Specifically, the pitch acquisition unit 42 identifies the performance pitch P by analyzing the audio signal A generated by the sound collection device 36. The estimation of the performance pitch P is repeated at a predetermined cycle. Any known analysis technique (pitch estimation process) may be used to estimate the performance pitch P. Furthermore, the pitch acquisition unit 42 of the first embodiment identifies the performance intensity β by analyzing the audio signal A. The performance intensity β is the intensity (volume) of the performance sound produced by the keyboard instrument 10.

[0025] The estimation processing unit 43 estimates the position of an element related to the keyboard instrument 10 (hereinafter referred to as an "estimation target") in the analysis space S through analysis processing using the multiple captured images Gn and the played pitch P. FIG. 5 is an explanatory diagram of the estimation result by the estimation processing unit 43. The analysis space S is a virtual three-dimensional space corresponding to real space. The analysis processing executed by the estimation processing unit 43 estimates the position (Zn, Dn, La, Lb, Q, J) of the estimation target in the analysis space S.

[0026] Specifically, the estimation processing unit 43 estimates, through analysis processing, the position Zn and direction Dn of each imaging device 21-n, the horizontal axis La and vertical axis Lb of the keyboard plane, multiple feature points Q on the keyboard plane, and each nodal point J (joint or fingertip) of the player U's right and left hands.

[0027] The keyboard plane corresponds to the surface of the keyboard 11 of the keyboard instrument 10. The horizontal axis La of the keyboard plane is an axis extending in the longitudinal direction of the keyboard 11 (i.e., the direction in which the multiple keys 12 are arranged), and the vertical axis Lb is an axis extending in the lateral direction of the keyboard 11. Multiple feature points Q are points located on the keyboard plane. Each node J corresponds to a joint or fingertip on the performer U's right and left hands. As described above, the objects to be estimated by the analysis process include the keyboard plane (horizontal axis La, vertical axis Lb) and multiple image capture devices 21-n (position Zn, direction Dn). The estimation processing unit 43 may display the results of the analysis process on the display device 34. For example, an image similar to that shown in FIG. 5 is displayed on the display device 34.

[0028] As described above, in the first embodiment, the position of the estimation target within the analysis space S is estimated by analysis processing that uses the multiple captured images Gn and the performance pitch P. Therefore, the position of the estimation target within the analysis space S can be estimated by a simple method that does not require an element dedicated to analyzing the estimation target, such as a performance operator that is recognized as black in an image captured by an infrared sensor (Patent Document 1) or a marker attached to the performer's hand (Patent Document 2).

[0029] Fig. 6 is a flowchart illustrating a specific procedure of the analysis process executed by the control device 31 (estimation processing unit 43). For example, the analysis process is started in response to an operation on the operation device 35. The analysis process is repeated for each unit period (for example, frame) during which a photographed image Gn is captured. Fig. 7 is an explanatory diagram of the analysis process.

[0030] When the analysis process begins, the control device 31 detects multiple observation points B for each of the multiple captured images Gn (S1). As illustrated in FIG. 7, the observation points B are points corresponding to each joint J (joint or fingertip) on the hand of the performer U in each captured image Gn. The process of detecting the multiple observation points B from each captured image Gn may employ any known analysis process, such as object detection using a machine-learned recognition model. Note that step S1 of the analysis process is an example of the "process of detecting multiple observation points."

[0031] The control device 31 executes a candidate point identification process (S2). The candidate point identification process is a process for identifying multiple candidate points C for each of the multiple captured images Gn. The candidate points C are points that are candidates for the feature point Q within the keyboard plane. Specifically, of the multiple observation points B detected from each captured image Gn, the observation points B that are likely to be close to the keyboard plane (i.e., points located on the surface of each key 12 of the keyboard instrument 10) are identified as candidate points C. Among the multiple fingers of the player U, the fingertips of the fingers used to play a specific pitch (performance pitch P) are likely to be close to the keyboard plane. Utilizing the above tendency, the control device 31 identifies multiple candidate points C from each captured image Gn and the performance pitch P.

[0032] FIG. 8 is a flowchart illustrating a specific procedure of the candidate point identification process. When the candidate point identification process is started, the control device 31 generates a plurality of (e.g., 88) processed images Rn[k] (k=1, 2, ...) corresponding to different pitches of the keyboard instrument 10, as illustrated in FIG. 7 (S21). A plurality of processed images Rn[k] are generated for each imaging device 21-n. That is, a plurality of processed images R1[k] corresponding to the captured image G1 and a plurality of processed images R2[k] corresponding to the captured image G2 are generated. For example, the control device 31 generates a plurality of processed images Rn[k] by duplicating each captured image Gn. Note that step S21 of the candidate point identification process is an example of a process of duplicating a captured image into a plurality of processed images.

[0033] 7, for each of the plurality of captured images Gn, the control device 31 selects, from among the plurality of processed images Rn[k] corresponding to the captured image Gn, a processed image Rn[k] (hereinafter referred to as a "processed performance image Rn[k]") that corresponds to the performance pitch P (S22). Specifically, from among the plurality of processed images Rn[k] corresponding to one unit period to be processed, the processed image Rn[k] that corresponds to the performance pitch P played by the performer U in that unit period is selected as the processed performance image Rn[k].

[0034] As illustrated in FIG. 7 , the control device 31 assigns a probability distribution N(μ,σ) to each of the observation points B in each performance processed image Rn[k] that corresponds to the fingertips of the player U (S23). The probability distribution N(μ,σ) is a numerical distribution within the plane of the performance processed image Rn[k]. For example, a Gaussian distribution is used as the probability distribution N(μ,σ). The probability distribution N(μ,σ) is a distribution with a mean μ of the fingertips of the player U in the performance processed image Rn[k] and a predetermined variance σ. Note that the probability distribution N(μ,σ) is not limited to a Gaussian distribution and may be arbitrarily changed.

[0035] The control device 31 calculates the probability distribution N(μ,σ) to be assigned to the observation point B of each fingertip of the player U by adjusting the initial or standard probability distribution N(μ,σ) according to the moving speed α and playing intensity β of each fingertip of the player U (N(μ,σ) = α β N(μ,σ)). Therefore, the greater the moving speed α or playing intensity β, the greater the numerical value (probability value) in the probability distribution N(μ,σ). In other words, a probability distribution N(μ,σ) with a large probability value is assigned to the observation point B of the fingertip that the player U actually moved to play. The fingertip used by the player U in playing is likely to be located on the surface of the keyboard 11 (i.e., likely to correspond to the feature point Q). Therefore, there is a tendency for a probability distribution N(μ,σ) with a large probability value to be assigned to the observation point B located on the keyboard plane.

[0036] The control device 31 selects multiple candidate points C from multiple observation points B in each performance processed image Rn[k] (S24). Specifically, an observation point B that corresponds to an extreme point of the probability value in the performance processed image Rn[k] and has a corresponding point among the multiple captured images Gn is selected as the candidate point C. The candidate point C selected from the performance processed image Rn[k] corresponds to one pitch assigned to the performance processed image Rn[k]. As can be understood from the above explanation, one candidate point C corresponds to the fingertip when the performer U plays the pitch corresponding to the candidate point C as the performance pitch P. Because the fingertip when playing the keyboard instrument 10 is close to the keyboard plane, the candidate point C can also be expressed as a point close to the keyboard plane. The specific steps of the candidate point identification process are as described above. Note that step S24 of the candidate point identification process is an example of the "processing of selecting a candidate point."

[0037] After the candidate point identification process is executed, the control device 31 executes an optimization process for estimating the position of the estimation target in the analysis space S (S3), as illustrated in Fig. 6. The optimization process is a process for estimating the position of the estimation target so as to minimize a plurality of objective functions F (Fa to Fd), the examples of which are shown below.

[0038] [Objective Function Fa] The objective function Fa is expressed by the following equation (1).

[0039] The symbol Bn_m in formula (1) means the m-th observation point B among the multiple observation points B detected from the captured image Gn. The function K is a function that represents the characteristics of the image capture device 21-n (for example, optical characteristics such as focal length or aberration characteristics). Therefore, the inverse function K of the function K -1 The function H functions as a function for removing the characteristics of the image capturing device 21-n from the captured image Gn in which the characteristics of the image capturing device 21-n are reflected. The function H is a function for converting any coordinate point in the analysis space S into a coordinate point on the imaging surface of the image capturing device 21-n. Therefore, the inverse function H of the function H -1 functions as a function that converts coordinate points on the imaging surface into coordinate points in the analysis space S. That is, the function H in Equation (1) -1 {K -1 (Bn_m)} means a calculation for converting the observation point Bn_m of the photographed image Gn to a point in the analytical space S.

[0040] The symbol W in formula (1) is a transformation matrix for rigid transformation including translation and rotation within the analysis space S. Specifically, the transformation matrix W means a process of translating and rotating the camera device 21-2 so that it matches the position and orientation of the camera device 21-1. Therefore, the symbol WH in formula (1) -1 {K -1 (B2_m)} is the coordinate point H in the analytical space S. -1 {K -1 (B2_m)} to a coordinate point as seen from the camera 21-1.

[0041] The symbol ||a-b||2 in formula (1) means the distance between element a and element b. Therefore, the objective function F in formula (1) is expressed as the distance between the coordinate point H in the analytical space S. -1 {K -1 (B1_m)} and coordinate point WH -1 {K -1 (B2_m)} for multiple observation points Bn_m. Observation point B1_m in captured image G1 and observation point B2_m in captured image G2 correspond to a common node J (joint or fingertip) of performer U, so the closer the function H, function K, and transformation matrix W are to the correct solution, the closer the objective function Fa is to 0.

[0042] In the optimization process (S2) of the analysis process, the control device 31 estimates the functions H, K, and transformation matrix W so as to minimize the objective function Fa in equation (1). From the estimation results of the optimization process (function H, function K, transformation matrix W), the control device 31 estimates the position Zn and direction Dn of each image capture device 21-n within the analysis space S. As described above, according to the first embodiment, the position Zn and direction Dn of each image capture device 21-n can be estimated using the captured images Gn captured by each of the multiple image capture devices 21-n.

[0043] [Objective Function Fb] The objective function Fb is expressed by the following equation (2).

[0044] The coordinates (xi, yi, zi) in formula (2) are the coordinate values ​​of a point corresponding to one of the multiple candidate points C (i-th candidate point C) in the analysis space S. The function (ax+by+cz+d) in formula (2) means an arbitrary three-dimensional plane in the analysis space S. The three-dimensional plane in the analysis space S corresponds to a provisional keyboard plane.

[0045] As explained above, the objective function Fb in Equation (2) is a numerical value obtained by adding up the distances between one candidate point C and an arbitrary three-dimensional plane in the analytical space S for multiple candidate points C. The control device 31 estimates the three-dimensional plane for which the objective function Fb is minimum as the keyboard plane. As described above, according to the first embodiment, the keyboard plane in the analytical space S can be estimated with high accuracy by estimating the three-dimensional plane for which the sum of the distances to each of the multiple candidate points C is minimum.

[0046] [Objective Function Fc] The objective function Fc is expressed by the following equation (3).

[0047] The symbol V in formula (3) is a vector (hereinafter referred to as a "horizontal vector") along the longitudinal direction of the keyboard 11 of the keyboard instrument 10 in the analysis space S. Also, the symbol [a, b, c] in formula (3) Ｔ means the normal vector of the three-dimensional plane (ax+by+cz+d) in the analytical space S. That is, the normal vector [a, b, c] Ｔ is a column vector orthogonal to the tentative keyboard plane.

[0048] As described above, the formula (3) expresses the horizontal vector V along the longitudinal direction of the keyboard 11 in the keyboard plane and the normal vector [a, b, c] of the keyboard plane. Ｔ The control device 31 estimates the horizontal vector V on the keyboard plane so that the inner product of equation (3) is minimized. The control device 31 determines the horizontal axis La on the keyboard plane so that it is aligned with the horizontal vector V. As described above, according to the first embodiment, the horizontal axis La along the longitudinal direction of the keyboard 11 can be estimated with high accuracy.

[0049] [Objective Function Fd] The objective function Fd is expressed by the following equation (4).

[0050] The symbol Ck (k = 1 to 4) in formula (4) means four candidate points C selected from multiple candidate points C. The sum Σ in formula (4) means the sum over multiple combinations of selecting four candidate points C from multiple candidate points C. Note that candidate point C1 is an example of a "first candidate point," candidate point C2 is an example of a "second candidate point," candidate point C3 is an example of a "third candidate point," and candidate point C3 is an example of a "fourth candidate point."

[0051] The distance |(C1-C2)·V| in formula (4) means the distance between candidate points C1 and C2 in the direction of horizontal vector V on the keyboard plane. In other words, the distance |(C1-C2)·V| means the magnitude (absolute value) of the vector (C1-C2) projected onto the horizontal vector V. Similarly, the distance |(C3-C4)·V| in formula (4) means the distance between candidate points C3 and C4 in the direction of horizontal vector V.

[0052] On the other hand, the symbol Pk in formula (4) is the pitch corresponding to candidate point Ck. Therefore, the part |P1-P2| in formula (4) means the pitch difference between pitch P1 corresponding to candidate point C1 and pitch P2 corresponding to candidate point C2. Similarly, the part |P3-P4| in formula (4) means the pitch difference between pitch P3 corresponding to candidate point C3 and pitch P4 corresponding to candidate point C4.

[0053] Because the arrangement of the multiple keys 12 (especially the white keys) on the keyboard instrument 10 is periodic, the distance between any two keys 12 on the keyboard instrument 10 is proportional to the pitch difference corresponding to each key 12. In other words, the distance |(Ck1-Ck2)V| between candidate points Ck1 and Ck2 in the direction of the horizontal vector V is proportional to the pitch difference |Pk1-Pk2| between the pitch Pk1 of candidate point Ck1 and the pitch Pk2 of candidate point Ck2. Therefore, the closer the horizontal vector V is to the correct answer, the closer the value in parentheses in equation (4) is to 0.

[0054] In the optimization process (S2) of the analysis process, the control device 31 estimates multiple combinations of candidate points Ck and horizontal vectors V so as to minimize the objective function Fd of equation (4). Minimizing the objective function Fd means bringing the ratio |(C1-C2)·V| / |(C3-C4)·V| of the distance |(C1-C2)·V| to the distance |(C3-C4)·V| closer to the ratio |P1-P2| / |P3-P4| of the pitch difference |P1-P2| to the pitch difference |P3-P4|. In other words, the control device 31 estimates combinations of candidate points Ck and horizontal vectors V so that the ratio of the distances |(C1-C2)·V| / |(C3-C4)·V| between the candidate points Ck approaches the ratio of the pitch differences |P1-P2| / |P3-P4|. The candidate points Ck estimated by the above process are determined as feature points Q on the keyboard plane. As described above, according to the first embodiment, the position on the keyboard plane, including the scale within the analysis space S, can be estimated with high accuracy.

[0055] The optimization process of the objective functions Fb, Fc, and Fd exemplified above is a process for estimating the keyboard plane using a plurality of candidate points C identified from a plurality of captured images Gn by the candidate point identification process. That is, according to the first embodiment, the keyboard plane of the keyboard instrument 10 can be estimated without requiring image recognition (object detection) of the keys 11 of the keyboard instrument 10 or processing such as analysis of the performer's fingering. Because the black keys and white keys on the keyboard 11 are periodically arranged in a predetermined pattern, it is actually difficult to achieve highly accurate image recognition of the keyboard 11. Considering the above circumstances, the first embodiment, which does not require image recognition of the keyboard 11, is particularly effective.

[0056] In the above explanation, each objective function F has been explained individually. However, in the actual optimization process (S3), the control device 31 estimates each position of the estimation target so as to minimize, for example, the sum of the objective functions Fa to Fd exemplified above (Fa+Fb+Fc+Fd).

[0057] B: Second Embodiment A second embodiment of the present disclosure will be described. Note that, for elements in the following exemplary aspects that have the same functions as those in the first embodiment, the same reference numerals as those in the first embodiment will be used, and detailed descriptions of each will be omitted as appropriate.

[0058] 9 is a block diagram illustrating the configuration of a performance analysis system 100 according to the second embodiment. The image capture system 20 according to the second embodiment includes an image capture device 21-3 in addition to the image capture devices 21-1 and 21-2 according to the first embodiment. The image capture device 21-3 is a camera similar to the image capture device 21-1 or 21-2, and generates a captured image G3 by capturing an image of the performer U playing the keyboard instrument 10. The image capture device 21-3 is installed, for example, in front of and above the performer U.

[0059] The control device 31 (image acquisition unit 41) acquires the photographed image G3 in addition to the photographed images G1 and G2. The control device 31 (estimation processing unit 43) executes the candidate point identification process described above for each of the three photographed images Gn (G1 to G3). The objective function Fa applied to the optimization process of the second embodiment is a numerical value obtained by summing the calculated values ​​of the above-described formula (1) for all combinations of selecting two photographing devices 21-n from the three photographing devices 21-n.

[0060] The operation of the control device 31 is basically the same as in the first embodiment, except for the points described above. Therefore, the second embodiment also achieves the same effects as in the first embodiment. Note that, although the above description has exemplified a configuration in which there are three image capturing devices 21-n, the image capturing system 20 may include four or more image capturing devices 21-n.

[0061] C: Modifications Specific modifications that can be added to each of the above-mentioned embodiments are exemplified below. Two or more embodiments arbitrarily selected from the following examples may be combined as appropriate within the scope of not mutually contradicting each other.

[0062] (1) In the above-described embodiments, the performed pitch P is identified by analyzing the audio signal A. However, the method by which the pitch acquisition unit 42 acquires the performed pitch P is not limited to the above examples. For example, in an embodiment in which performance data representing the content of a performance by the performer U is supplied from the keyboard instrument 10 to the information processing system 30, the pitch acquisition unit 42 may acquire the pitch (e.g., note number) specified by the performance data as the performed pitch P. The performance data is, for example, time-series data conforming to the MIDI (Musical Instrument Digital Interface) standard. As can be understood from the above examples, the acquisition of the performed pitch P by the pitch acquisition unit 42 includes identifying the performed pitch P by analyzing the audio signal A and receiving the performed pitch P specified by the performance data.

[0063] (2) In the above-described embodiments, the horizontal axis La of the keyboard plane is determined from the horizontal vector V estimated by the optimization process (S3), but the method for determining the horizontal axis La of the keyboard plane is not limited to the above examples. For example, the estimation processing unit 43 may determine the horizontal axis La of the keyboard plane from multiple feature points Q estimated by the optimization process. For example, the estimation processing unit 43 can determine the horizontal axis La by principal component analysis of the multiple feature points Q.

[0064] (3) In the above-described embodiments, multiple objective functions F (Fa to Fd) are exemplified, but application of some of the multiple objective functions F may be omitted in the optimization process. For example, one or more of the objective functions Fb to Fd may be omitted in the optimization process.

[0065] (4) In each of the above-described embodiments, the position within the analysis space S of an estimated object (e.g., the keyboard plane or the photographing device 21-n) related to the photographed object is estimated from multiple photographed images Gn of the keyboard instrument 10 and the performer U. However, the situations to which the present disclosure can be applied are not limited to the above examples.

[0066] For example, consider a scene in which multiple performers U are playing different instruments on stage. The image acquisition unit 41 acquires multiple captured images Gn of the multiple performers U captured from different directions by the camera devices 21-n. The pitch acquisition unit 42 acquires the performance pitch P produced by the performances of the multiple performers U. The estimation processing unit 43 estimates the position of an estimation target related to the performance target (multiple performers U) within the analysis space S through an analysis process using the multiple captured images Gn and the performance pitch P. For example, the position of each performer U, the horizontal axis La and vertical axis Lb along the top surface of the stage, the position Zn and direction Dn of each camera device 21-n, etc. are estimated through an analysis process similar to that of each of the above-described embodiments. As can be understood from the above examples, the target captured by the camera devices 21-n (i.e., the subject of the captured image Gn) and the target estimated by the analysis process can be arbitrarily changed.

[0067] (5) The information processing system 30 in each of the above-described embodiments may be realized by a server device that communicates with an information device such as a smartphone or a tablet terminal. The information processing system 30 receives multiple captured images Gn and an audio signal A (or performance data) from the information device via a communication network. The information processing system 30 operates in the same manner as in each of the above-described embodiments to estimate the position of the estimation target within the analysis space S and transmit the estimation result to the information device.

[0068] (6) As illustrated in the examples of the above embodiments, the functions of the information processing system 30 are realized through cooperation between one or more processors constituting the control device 31 and a program stored in the storage device 32. The program according to the present disclosure may be provided in a form stored on a computer-readable recording medium and installed on a computer. The recording medium may be, for example, a non-transitory recording medium, such as an optical recording medium (optical disk) such as a CD-ROM, but may also include any known type of recording medium, such as a semiconductor recording medium or a magnetic recording medium. Note that a non-transitory recording medium includes any recording medium other than a transient, propagating signal, and does not exclude volatile recording media. Furthermore, in a configuration in which a distribution device distributes a program via a communication network, the storage medium that stores the program in the distribution device corresponds to the non-transitory recording medium described above.

[0069] D: Supplementary Note From the above-described exemplary embodiments, the following configurations can be understood, for example.

[0070] An information processing method according to one aspect (aspect 1) of the present disclosure includes acquiring a plurality of captured images of a subject from different directions, acquiring a performance pitch produced by the subject, and estimating the position of an estimation target related to the subject in three-dimensional space through an analysis process using the plurality of captured images and the performance pitch. In the above aspect, the position of an estimation target related to the subject in three-dimensional space is estimated through an analysis process using the plurality of captured images and the performance pitch. Therefore, the position of the estimation target in three-dimensional space can be estimated using a simple method that does not require an element dedicated to analyzing the estimation target, such as a performance control recognized as black in an image captured by an infrared sensor (Patent Document 1) or a marker attached to the performer's hand (Patent Document 2).

[0071] The "photographed object" is an object that is photographed to obtain a photographed image to be used in the analysis process. Specifically, an object that operates to produce a specific musical pitch is exemplified as the "photographed object." For example, the fingers of a performer (both hands or one hand) or multiple performers playing musical instruments are exemplified as the "photographed object."

[0072] The "estimated object" is any object whose position in three-dimensional space is to be estimated. Specifically, an object located around the subject of image capture is exemplified as the "estimated object." Note that the "estimated object" does not need to be explicitly included in the multiple captured images. Note that the "estimated object" may be the subject of image capture itself.

[0073] In a specific example (Aspect 2) of Aspect 1, the photographed object includes the hands of a player playing a keyboard instrument, and the estimation object includes a keyboard plane corresponding to the surface of the keys of the keyboard instrument. In the above aspect, the position of the keyboard plane of the keyboard instrument in three-dimensional space is estimated by analyzing the photographed object including the player's hands. In other words, the keyboard plane of the keyboard instrument can be estimated without requiring processing such as image recognition (object detection) of the keys of the keyboard instrument or analysis of the player's fingering.

[0074] In a specific example (Aspect 3) of Aspect 2, the analysis process includes, for each of the plurality of captured images, detecting a plurality of observation points in the captured image corresponding to the captured subject, duplicating the captured image into a plurality of processed images corresponding to different pitches, and selecting candidate points on the keyboard plane from the plurality of observation points in the processed image corresponding to the played pitch among the plurality of processed images, and estimating the keyboard plane from the plurality of candidate points identified from the plurality of captured images. In the above aspect, candidate points on the keyboard plane are identified from observation points in the processed image corresponding to the played pitch among the plurality of processed images obtained by duplicating the captured image, and the keyboard plane is estimated using the plurality of candidate points. Therefore, the keyboard plane of a keyboard instrument can be estimated without requiring processes such as image recognition (object detection) of the keyboard of the keyboard instrument or analysis of the player's fingering.

[0075] In a specific example (Aspect 4) of Aspect 3, the analysis process estimates the three-dimensional plane that has the smallest total distance from each of the plurality of candidate points as the keyboard plane. Each candidate point is an observation point in a processed image corresponding to a played pitch among the plurality of processed images, and therefore is likely to be close to the keyboard plane. Therefore, by estimating the three-dimensional plane that has the smallest total distance from each of the plurality of candidate points, the keyboard plane in three-dimensional space can be estimated with high accuracy.

[0076] In a specific example (Aspect 5) of any of Aspects 2 to 4, the analysis process estimates the horizontal vector so that the dot product of a horizontal vector along the longitudinal direction of the key within the keyboard plane and a normal vector to the keyboard plane is minimized. In the above aspects, the horizontal vector is estimated so that the dot product of a horizontal vector along the longitudinal direction of the key and a normal vector to the keyboard plane is minimized. Therefore, the horizontal axis along the longitudinal direction of the key can be estimated with high accuracy.

[0077] In a specific example (Aspect 6) of Aspect 5, the analysis process estimates the horizontal vector in the direction of the horizontal vector so that the ratio of the distance between a first candidate point and a second candidate point among the plurality of candidate points to the distance between a third candidate point and a fourth candidate point among the plurality of candidate points approaches the ratio of the pitch difference between a first pitch corresponding to the first candidate point and a second pitch corresponding to the second candidate point to the pitch difference between a third pitch corresponding to the third candidate point and a fourth pitch corresponding to the fourth candidate point. In the above aspect, under the condition that the ratio of the distance between the candidate points and the pitch difference is constant, the feature points located on the keyboard plane and the horizontal vector of the keyboard plane are estimated. In other words, the position of the keyboard plane can be estimated, for example, including the scale in three-dimensional space.

[0078] In a specific example (Aspect 7) of any of Aspects 1 to 6, the estimation target includes a plurality of image capture devices that capture the plurality of images, and the analysis process estimates the position and orientation of each of the plurality of image capture devices within the three-dimensional space. According to the above aspect, the position and orientation of each image capture device can be estimated using the images captured by each of the plurality of image capture devices.

[0079] An information processing system according to one aspect (aspect 8) of the present disclosure includes an image acquisition unit that acquires a plurality of captured images of a subject from different directions, a pitch acquisition unit that acquires the pitch of a performance sounded by the subject, and an estimation processing unit that estimates the position in three-dimensional space of an estimation target related to the subject through analysis processing using the plurality of captured images and the performance pitch.

[0080] A program according to one aspect (aspect 9) of the present disclosure causes a computer system to function as an image acquisition unit that acquires multiple captured images of a subject from different directions, a pitch acquisition unit that acquires the pitch of a performance sounded by the subject, and an estimation processing unit that estimates the position in three-dimensional space of an estimation target related to the subject through analysis processing using the multiple captured images and the performance pitch.

[0081] 100...performance analysis system, 10...keyboard instrument, 11...keyboard, 12...key, 20...photography system, 21-n (21-1, 21-2, 21-3)...photography device, 30...information processing system, 31...control device, 32...storage device, 33...communication device, 34...display device, 35...operation device, 36...sound collection device, 41...image acquisition unit, 42...pitch acquisition unit, 43...estimation processing unit.

Claims

1. An information processing method implemented by a computer system that acquires multiple images of a subject taken from different directions, acquires the pitch of a musical performance produced by the subject, and estimates the position in three-dimensional space of an estimated object related to the subject through analysis processing using the multiple images and the pitch of the musical performance.

2. The information processing method according to claim 1, wherein the subject to be photographed includes the hands of a player playing a keyboard instrument, and the subject to be estimated includes a keyboard plane corresponding to the surface of a keyboard on the keyboard instrument.

3. The information processing method of claim 2, wherein the analysis process includes the steps of: detecting, for each of the plurality of captured images, a plurality of observation points in the captured image corresponding to the subject; duplicating the captured image into a plurality of processed images corresponding to different pitches; and selecting candidate points that are candidates for points on the keyboard plane from the plurality of observation points in the processed image corresponding to the played pitch among the plurality of processed images; and estimating the keyboard plane from the plurality of candidate points identified from the plurality of captured images.

4. The information processing method according to claim 3, wherein in the analysis process, the three-dimensional plane for which the sum of the distances to each of the plurality of candidate points is the smallest is estimated as the keyboard plane.

5. An information processing method according to any one of claims 2 to 4, wherein in the analysis process, the horizontal vector is estimated so that the dot product of a horizontal vector along the longitudinal direction of the key within the keyboard plane and a normal vector to the keyboard plane is minimized.

6. An information processing method according to claim 5, wherein in the analysis process, the horizontal vector is estimated so that the ratio of the distance between a first candidate point and a second candidate point among the plurality of candidate points to the distance between a third candidate point and a fourth candidate point among the plurality of candidate points in the direction of the horizontal vector approaches the ratio of the pitch difference between a first pitch corresponding to the first candidate point and a second pitch corresponding to the second candidate point to the pitch difference between a third pitch corresponding to the third candidate point and a fourth pitch corresponding to the fourth candidate point.

7. An information processing method according to claim 1, wherein the estimation target includes a plurality of image capturing devices that capture the plurality of images, and the analysis process estimates the position and direction within the three-dimensional space for each of the plurality of image capturing devices.

8. An information processing system comprising: an image acquisition unit that acquires a plurality of captured images of a subject from different directions; a pitch acquisition unit that acquires the pitch of a performance sounded by the subject; and an estimation processing unit that estimates the position in three-dimensional space of an estimation target related to the subject by performing an analysis process using the plurality of captured images and the performance pitch.

9. A program that causes a computer system to function as an image acquisition unit that acquires multiple images of a subject taken from different directions, a pitch acquisition unit that acquires the pitch of a performance sounded by the subject, and an estimation processing unit that estimates the position in three-dimensional space of an estimation target related to the subject by performing an analysis process using the multiple images and the performance pitch.

Images