Information processing methods, information processing systems, and programs

Abstract

The position of the object to be estimated in three-dimensional space is estimated using a simple method that does not require elements specifically for the analysis of the object to be estimated. [Solution] The information processing system comprises an image acquisition unit 41 that acquires multiple captured images Gn of the subject from different directions, a pitch acquisition unit 42 that acquires the pitch P of the sound produced by the subject, and an estimation processing unit 43 that estimates the position in three-dimensional space of an estimated target related to the subject by performing analysis processing using the multiple captured images Gn and the pitch P.

Description

[Technical Field]

[0001] This disclosure relates to a technique for estimating the position of a subject being photographed. [Background technology]

[0002] For example, various techniques have been proposed to estimate the position of objects such as a performer's hands in three-dimensional space. For instance, Patent Document 1 discloses a configuration that generates finger movements images by a performer using performance controls that are recognized as black when imaged by an infrared sensor. Patent Document 2 also discloses a configuration that generates an image of a hand with the markers removed by inputting a captured image of a performer's hand with markers attached into a trained model, and then generates three-dimensional pose data of the hand from the image after the markers have been removed. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2017-181850 [Patent Document 2] Japanese Patent Publication No. 2024-012764 [Overview of the project] [Problems that the invention aims to solve]

[0004] However, the technology described in Patent Document 1 requires a performance control that is recognized as black by imaging with an infrared sensor. Furthermore, the technology described in Patent Document 2 requires a marker attached to the performer's hand. In other words, conventional technologies require elements (performance control or marker) that are dedicated to the analysis of the target to be estimated. Taking these circumstances into consideration, one aspect of this disclosure aims to estimate the position of a target to be estimated in three-dimensional space by a simple method that does not require elements dedicated to the analysis of the target to be estimated. [Means for solving the problem]

[0005] To solve the above problems, an information processing method according to one aspect of the present disclosure acquires a plurality of images of a subject taken from different directions, acquires the pitch of a musical sound produced by the subject, and estimates the position in three-dimensional space of an estimated object related to the subject by performing an analysis process using the plurality of images and the musical pitch.

[0006] An information processing system according to one aspect of this disclosure comprises: an image acquisition unit that acquires a plurality of images of a subject taken from different directions; a pitch acquisition unit that acquires the pitch of a musical sound produced by the subject; and an estimation processing unit that estimates the position in three-dimensional space of an estimated object related to the subject by performing an analysis process using the plurality of images and the musical pitch.

[0007] A program according to one aspect of this disclosure causes a computer system to function as an image acquisition unit that acquires a plurality of images of a subject taken from different directions, a pitch acquisition unit that acquires the pitch of a musical sound produced by the subject, and an estimation processing unit that estimates the position in three-dimensional space of an estimated object related to the subject by performing an analysis process using the plurality of images and the musical pitch. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram illustrating the configuration of the performance analysis system in the first embodiment. [Figure 2] This is a schematic diagram of the captured image. [Figure 3] This is a block diagram illustrating the configuration of an information processing system. [Figure 4] This is a block diagram illustrating the functional configuration of an information processing system. [Figure 5] This is an explanatory diagram of the estimation results performed by the estimation processing unit. [Figure 6] This is a flowchart of the analysis process. [Figure 7] This is an explanatory diagram of the analysis process. [Figure 8]It is a flowchart of candidate point identification processing. [Figure 9] It is a block diagram illustrating the configuration of the performance analysis system in the second embodiment.

Embodiments for Carrying Out the Invention

[0009] A: First Embodiment FIG. 1 is a block diagram illustrating the configuration of a performance analysis system 100 in the first embodiment. The performance analysis system 100 is a computer system for analyzing the performance by a performer U. The performance analysis system 100 includes a keyboard instrument 10, a photographing system 20, and an information processing system 30.

[0010] The keyboard instrument 10 is an instrument that produces sound in response to the performance by the performer U. The keyboard instrument 10 includes a keyboard 11 in which a plurality of keys 12 are arranged in a horizontal direction. The plurality of keys 12 of the keyboard 11 include a plurality of white keys and a plurality of black keys. The keyboard instrument 10 may be either a natural instrument that generates musical sounds by the vibration of a sound source such as strings or the like, or an electronic instrument that electronically generates musical sounds by a sound source device.

[0011] The photographing system 20 photographs the situation where the performer U performs the keyboard instrument 10. The photographing system 20 includes a plurality of photographing devices 21-n (n = 1, 2). For example, each photographing device 21-n includes a photographing optical system such as a photographing lens, an image sensor that receives incident light from the photographing optical system, and a processing circuit that generates image data according to the amount of light received by the image sensor. The electrical characteristics or optical characteristics (e.g., focal length or aberration characteristics, etc.) are common to the photographing device 21-1 and the photographing device 21-2. However, the characteristics may be different for each photographing device 21-n.

[0012] Each of the plurality of imaging devices 21-n is installed facing the keyboard instrument 10. The position and direction with respect to the keyboard instrument 10 (e.g., the keyboard 11) are different for each imaging device 21-n. For example, the imaging device 21-1 is installed diagonally above and to the left of the keyboard instrument 10. The imaging device 21-2 is installed diagonally above and to the right of the keyboard instrument 10. Each imaging device 21-n generates a captured image Gn by capturing the keyboard instrument 10.

[0013] FIG. 2 is a schematic diagram of the captured image Gn. The captured image Gn is a planar image representing the performance of the keyboard instrument 10 by the performer U. Specifically, as illustrated in FIG. 2, each captured image Gn includes the keyboard 11 of the keyboard instrument 10 and the hands (right hand and left hand) of the performer U who performs the keyboard instrument 10. The captured image G1 is an image captured from diagonally above and to the left of the keyboard instrument 10 and the performer U, and the captured image G2 is an image captured from diagonally above and to the right of the keyboard instrument 10 and the performer U. The captured image Gn is represented by image data in any format. The generation of the captured image Gn is repeated for each unit period (frame) of a predetermined length. A moving image of the performance by the performer U is constituted by the time series of the plurality of captured images Gn.

[0014] The plurality of imaging devices 21-n capture the performance of the keyboard instrument 10 by the performer U in parallel with each other. As described above, the plurality of captured images Gn are images captured of the keyboard instrument 10 from different positions and directions. The captured image Gn captured by each imaging device 21-n is transmitted from the imaging device 21-n to the information processing system 30.

[0015] The information processing system 30 is a computer system that generates various types of information regarding the performance of the keyboard instrument 10 by the performer U by analyzing the plurality of captured images Gn transmitted from different imaging devices 21-n. The information processing system 30 is realized by an information device such as a smartphone, a tablet terminal, or a personal computer.

[0016] Figure 3 is a block diagram illustrating the configuration of the information processing system 30. The information processing system 30 comprises a control device 31, a storage device 32, a display device 34, an operating device 35, and a sound pickup device 36. The information processing system 30 can be implemented as a single device or as a group of devices configured separately from each other.

[0017] The control device 31 consists of one or more processors that control each element of the information processing system 30. Specifically, the control device 31 is composed of one or more types of processors, such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), SPU (Sound Processing Unit), DSP (Digital Signal Processor), FPGA (Field Programmable Gate Array), or ASIC (Application Specific Integrated Circuit).

[0018] The storage device 32 is one or more memories that store programs executed by the control device 31 and data used by the control device 31. The storage device 32 is composed of known recording media, such as magnetic recording media or semiconductor recording media. The storage device 32 may be composed of a combination of multiple types of recording media. Alternatively, a portable recording media that can be attached to and detached from the information processing system 30, or a recording media that can be written to or read by the control device 31 via a communication network (e.g., cloud storage), may be used as the storage device 32.

[0019] The communication device 33 communicates with external devices. Specifically, the communication device 33 receives captured images Gn transmitted from each imaging device 21-n via wired or wireless connection. Alternatively, a separate communication device 33 may be connected to the information processing system 30 via wired or wireless connection.

[0020] The display device 34 displays an image under the control of the control device 31. The display device 34 is composed of a display panel such as a liquid crystal panel or an organic EL (electroluminescence) panel. The operating device 35 is an input device that receives operations from the performer U. Note that the display device 34 or the operating device 35, which are separate from the information processing system 30, may be connected to the information processing system 30 by wire or wireless connection.

[0021] The sound-collecting device 36 is one or more microphones that capture the sound produced by the keyboard instrument 10 in response to the performance by the performer U. The sound-collecting device 36 generates an acoustic signal A that represents the waveform of the sound produced by the keyboard instrument 10. For convenience, the amplifier that amplifies the acoustic signal A generated by the sound-collecting device 36 and the A / D converter that converts the acoustic signal A from analog to digital are omitted from the illustration.

[0022] Figure 4 is a block diagram illustrating the functional configuration of the information processing system 30. The control device 31 executes a program stored in the storage device 32 to realize multiple functions (image acquisition unit 41, pitch acquisition unit 42, estimation processing unit 43) for analyzing multiple captured images Gn.

[0023] The image acquisition unit 41 acquires multiple captured images Gn taken by different imaging devices 21-n. Specifically, the image acquisition unit 41 receives the captured images Gn transmitted from each imaging device 21-n via the communication device 33.

[0024] The pitch acquisition unit 42 acquires the pitch (hereinafter referred to as "played pitch P") produced by the keyboard instrument 10. The played pitch P is the pitch played by the performer U on the keyboard instrument 10. Specifically, the pitch acquisition unit 42 identifies the played pitch P by analyzing the acoustic signal A generated by the sound collection device 36. The estimation of the played pitch P is repeated at a predetermined period. Known analysis techniques (pitch estimation processing) can be arbitrarily employed for the estimation of the played pitch P. In addition, the pitch acquisition unit 42 of the first embodiment identifies the performance intensity β by analyzing the acoustic signal A. The performance intensity β is the intensity (volume) of the sound produced by the keyboard instrument 10.

[0025] The estimation processing unit 43 estimates the position of elements related to the keyboard instrument 10 (hereinafter referred to as "estimated objects") in the analysis space S by performing an analysis process using multiple captured images Gn and the played pitch P. Figure 5 is an explanatory diagram of the estimation results by the estimation processing unit 43. The analysis space S is a virtual three-dimensional space corresponding to the real space. The analysis process performed by the estimation processing unit 43 estimates the position of the estimated objects (Zn, Dn, La, Lb, Q, J) in the analysis space S.

[0026] Specifically, the estimation processing unit 43 estimates the position Zn and direction Dn of each imaging device 21-n, the horizontal axis La and vertical axis Lb of the keyboard plane, multiple feature points Q on the keyboard plane, and each node J (joint or fingertip) of the performer U's right and left hands through analysis processing.

[0027] The keyboard plane is a plane corresponding to the surface of the keys 11 of the keyboard instrument 10. The horizontal axis La of the keyboard plane is an axis extending in the longitudinal direction of the keys 11 (i.e., the direction of arrangement of the multiple keys 12), and the vertical axis Lb is an axis extending in the short direction of the keys 11. Multiple feature points Q are locations located on the keyboard plane. Each node J is a location corresponding to a joint or fingertip on the right and left hands of the performer U. As described above, the estimation target by the analysis process includes the keyboard plane (horizontal axis La, vertical axis Lb) and multiple imaging devices 21-n (position Zn, direction Dn). The estimation processing unit 43 may also display the results of the analysis process on the display device 34. For example, an image similar to that in Figure 5 may be displayed on the display device 34.

[0028] As described above, in the first embodiment, the position of the target to be estimated within the analysis space S is estimated by an analysis process that utilizes multiple captured images Gn and the pitch of the played sound P. Therefore, the position of the target to be estimated within the analysis space S can be estimated by a simple method that does not require elements dedicated to the analysis of the target to be estimated, such as a performance control that is recognized as black by imaging with an infrared sensor (Patent Document 1) or a marker attached to the performer's hand (Patent Document 2).

[0029] Figure 6 is a flowchart illustrating the specific steps of the analysis process performed by the control device 31 (estimation processing unit 43). For example, the analysis process is initiated by an operation on the operating device 35. The analysis process is repeated for each unit period (e.g., frame) in which the captured image Gn is captured. Figure 7 is an explanatory diagram of the analysis process.

[0030] When the analysis process begins, the control device 31 detects multiple observation points B for each of the multiple captured images Gn (S1). As illustrated in Figure 7, the observation points B correspond to each node J (joint or fingertip) on the performer U's hand in each captured image Gn. The process of detecting multiple observation points B from each captured image Gn can be any known analysis process, such as object detection using a machine learning-based recognition model. Step S1 of the analysis process is an example of "the process of detecting multiple observation points".

[0031] The control device 31 performs candidate point identification processing (S2). Candidate point identification processing is the process of identifying multiple candidate points C for each of the multiple captured images Gn. Candidate points C are locations that are candidates for feature points Q within the keyboard plane. Specifically, among the multiple observation points B detected from each captured image Gn, observation points B that are likely to be close to the keyboard plane (i.e., locations located on the surface of each key 12 of the keyboard instrument 10) are identified as candidate points C. The fingertips of the fingers used to play a specific pitch (played pitch P) among the multiple fingers of the performer U are likely to be close to the keyboard plane. Utilizing this tendency, the control device 31 identifies multiple candidate points C from each captured image Gn and the played pitch P.

[0032] Figure 8 is a flowchart illustrating the specific steps of the candidate point identification process. When the candidate point identification process begins, the control device 31 generates multiple (e.g., 88) processed images Rn[k] (k=1,2,…) corresponding to different pitches of the keyboard instrument 10, as illustrated in Figure 7 (S21). Multiple processed images Rn[k] are generated for each imaging device 21-n. That is, multiple processed images R1[k] corresponding to the captured image G1 and multiple processed images R2[k] corresponding to the captured image G2 are generated. For example, the control device 31 generates multiple processed images Rn[k] by duplicating each captured image Gn. Step S21 of the candidate point identification process is an example of "the process of duplicating the captured image into multiple processed images".

[0033] As illustrated in Figure 7, the control device 31 selects, for each of the multiple captured images Gn, a processed image Rn[k] corresponding to the performance pitch P from among the multiple processed images Rn[k] corresponding to the captured image Gn (hereinafter referred to as "performance processing image Rn[k]") (S22). Specifically, from among the multiple processed images Rn[k] corresponding to one unit period to be processed, the processed image Rn[k] corresponding to the performance pitch P played by performer U during that unit period is selected as the performance processing image Rn[k].

[0034] As illustrated in Figure 7, the control device 31 adds a probability distribution N(μ,σ) to each observation point B corresponding to the fingertips of the performer U among the multiple observation points B in each performance processing image Rn[k] (S23). The probability distribution N(μ,σ) is a numerical distribution in the plane of the performance processing image Rn[k]. For example, a Gaussian distribution is used as the probability distribution N(μ,σ). The probability distribution N(μ,σ) is a distribution in the performance processing image Rn[k] with the fingertips of the performer U as the mean μ and a predetermined variance σ. Note that the probability distribution N(μ,σ) is not limited to a Gaussian distribution and may be changed arbitrarily.

[0035] The control device 31 adjusts the initial or standard probability distribution N0(μ,σ) according to the movement speed α and performance intensity β of each fingertip of the performer U, thereby calculating the probability distribution N(μ,σ) to be attached to the observation point B of that fingertip (N(μ,σ) = α·β·N0(μ,σ)). Therefore, the larger the movement speed α or performance intensity β, the larger the numerical value (probability value) in the probability distribution N(μ,σ). In other words, the observation point B of the fingertip that the performer U actually moved for performance is attached to a probability distribution N(μ,σ) with a large probability value. The fingertips used by the performer U for performance are highly likely to be located on the surface of the keyboard 11 (i.e., likely to correspond to feature point Q). Therefore, there is a tendency for the observation point B located on the keyboard plane to be attached to a probability distribution N(μ,σ) with a large probability value.

[0036] The control device 31 selects multiple candidate points C from multiple observation points B in each performance processing image Rn[k] (S24). Specifically, observation points B that correspond to the extreme value points of the probability value in the performance processing image Rn[k] and for which there are corresponding points among the multiple captured images Gn are selected as candidate points C. The candidate points C selected from the performance processing image Rn[k] correspond to one pitch assigned to the performance processing image Rn[k]. As can be understood from the above explanation, one candidate point C corresponds to the fingertip when the performer U plays the pitch P corresponding to that candidate point C. Since the fingertips are close to the keyboard plane when playing the keyboard instrument 10, candidate point C can also be described as a point close to the keyboard plane. The specific procedure for the candidate point identification process is as described above. Step S24 of the candidate point identification process is an example of the "process of selecting candidate points".

[0037] When the candidate point identification process is executed, the control device 31 performs an optimization process to estimate the position of the target in the analysis space S, as illustrated in Figure 6 (S3). The optimization process estimates the position of the target so as to minimize several objective functions F(Fa~Fd) as illustrated below.

[0038] [Objective function Fa] The objective function Fa is expressed by the following equation (1). [Number]

[0039] The symbol Bn_m in Equation (1) means the m-th observation point B among the plurality of observation points B detected from the captured image Gn. The function K is a function representing the characteristics of the imaging device 21-n (for example, optical characteristics such as focal length or aberration characteristics, etc.). Therefore, the inverse function K -1 functions as a function for removing such characteristics from the captured image Gn in which the characteristics of the imaging device 21-n are reflected. Also, the function H is a function for converting an arbitrary coordinate point in the analysis space S into a coordinate point on the imaging surface of the imaging device 21-n. Therefore, the inverse function H -1 functions as a function for converting a coordinate point on the imaging surface into a coordinate point in the analysis space S. That is, the function H -1 {K -1 (Bn_m)} means an operation for converting the observation point Bn_m of the captured image Gn into a point in the analysis space S.

[0040] The symbol W in Equation (1) is a transformation matrix for rigid transformation including translation and rotation in the analysis space S. Specifically, the transformation matrix W means a process of moving and rotating the imaging device 21-2 so as to coincide with the position and orientation of the imaging device 21-1. Therefore, the symbol WH -1 {K -1 (B2_m)} is the coordinate point H -1 {K -1 (B2_m)} means an operation for converting it into a coordinate point as seen from the imaging device 21-1.

[0041] The symbol ||a-b||2 in Equation (1) means the distance between element a and element b. Therefore, the objective function Fa in Equation (1) is the coordinate point H -1 {K -1 (B_{1m})} and the coordinate point WH -1 {K -1This is the sum of the distances to (B2_m) across multiple observation points Bn_m. Since observation point B1_m in captured image G1 and observation point B2_m in captured image G2 correspond to a common node J (joint or fingertip) of performer U, the closer function H, function K, and transformation matrix W are to the correct answer, the closer the objective function Fa approaches 0.

[0042] In the optimization process (S2) of the analysis process, the control device 31 estimates the function H, the function K, and the transformation matrix W so as to minimize the objective function Fa of equation (1). From the estimation results (function H, function K, transformation matrix W) obtained by the optimization process, the control device 31 estimates the position Zn and direction Dn of each imaging device 21-n in the analysis space S. As described above, according to the first embodiment, the position Zn and direction Dn of each imaging device 21-n can be estimated using the captured image Gn taken by each of the multiple imaging devices 21-n.

[0043] [Objective function Fb] The objective function Fb is expressed by the following equation (2).

number

[0044] The coordinates (xi, yi, zi) in equation (2) are the coordinate values ​​of a point in the analysis space S corresponding to one of the candidate points C (the i-th candidate point C) among multiple candidate points C. The function (ax + by + cz + d) in equation (2) represents an arbitrary three-dimensional plane in the analysis space S. The three-dimensional plane in the analysis space S corresponds to a provisional keyboard plane.

[0045] As explained above, the objective function Fb in equation (2) is the sum of the distances between one candidate point C in the analysis space S and an arbitrary three-dimensional plane, for multiple candidate points C. The control device 31 estimates the three-dimensional plane that minimizes the objective function Fb as the keyboard plane. As described above, according to the first embodiment, the keyboard plane in the analysis space S can be estimated with high accuracy by estimating the three-dimensional plane that minimizes the sum of the distances to each of the multiple candidate points C.

[0046] [Objective function Fc] The objective function Fc is expressed by the following equation (3).

number

[0047] The symbol V in equation (3) is a vector (hereinafter referred to as the "horizontal vector") that lies along the longitudinal direction of the keys 11 of the keyboard instrument 10 in the analytical space S. Also, the symbol [a,b,c] in equation (3) T This represents the normal vector of the 3D plane (ax+by+cz+d) in the analytical space S. That is, the normal vector [a,b,c] T This is a column vector orthogonal to the provisional keyboard plane.

[0048] As described above, equation (3) is the horizontal vector V along the longitudinal direction of the key 11 within the keyboard plane and the normal vector [a,b,c] of the keyboard plane. T This represents the dot product of the two. The control device 31 estimates the horizontal vector V of the keyboard plane such that the dot product of equation (3) is minimized. The control device 31 determines the horizontal axis La of the keyboard plane along the horizontal vector V. As described above, according to the first embodiment, the horizontal axis La along the longitudinal direction of the keyboard 11 can be estimated with high accuracy.

[0049] [Objective function Fd] The objective function Fd is expressed by the following equation (4).

number

[0050] In formula (4), the symbol Ck (k=1~4) represents four candidate points C selected from multiple candidate points C. The sum Σ in formula (4) represents the sum over multiple combinations of selecting four candidate points C from multiple candidate points C. Note that candidate point C1 is an example of the "first candidate point", candidate point C2 is an example of the "second candidate point", candidate point C3 is an example of the "third candidate point", and candidate point C4 is an example of the "fourth candidate point".

[0051] In equation (4), the distance |(C1-C2)·V| represents the distance between candidate point C1 and candidate point C2 in the direction of the horizontal vector V on the keyboard plane. That is, the distance |(C1-C2)·V| represents the magnitude (absolute value) when the vector (C1-C2) is projected onto the horizontal vector V. Similarly, in equation (4), the distance |(C3-C4)·V| represents the distance between candidate point C3 and candidate point C4 in the direction of the horizontal vector V.

[0052] On the other hand, the symbol Pk in equation (4) is the pitch corresponding to candidate point Ck. Therefore, part |P1-P2| of equation (4) means the pitch difference between pitch P1 corresponding to candidate point C1 and pitch P2 corresponding to candidate point C2. Similarly, part |P3-P4| of equation (4) means the pitch difference between pitch P3 corresponding to candidate point C3 and pitch P4 corresponding to candidate point C4.

[0053] Since the period of the arrangement of multiple keys 12 (especially the white keys) in the keyboard instrument 10 is constant, the distance between two keys 12 in the keyboard instrument 10 is proportional to the pitch difference corresponding to each key 12. That is, the distance between candidate points Ck1 and Ck2 in the direction of the horizontal vector V, |(Ck1-Ck2)·V|, is proportional to the pitch difference |Pk1-Pk2| between the pitch Pk1 of candidate point Ck1 and the pitch Pk2 of candidate point Ck2. Therefore, the closer the horizontal vector V is to the correct answer, the closer the value in parentheses in equation (4) approaches 0.

[0054] In the optimization process (S2) of the analysis process, the control device 31 estimates multiple combinations of candidate points Ck and the lateral vector V such that the objective function Fd of equation (4) is minimized. Minimizing the objective function Fd means bringing the ratio of the distance |(C1-C2)·V| to the distance |(C3-C4)·V| |(C1-C2)·V| / |(C3-C4)·V| close to the ratio of the pitch difference |P1-P2| to the pitch difference |P3-P4| |P1-P2| / |P3-P4|. In other words, the control device 31 estimates combinations of candidate points Ck and the lateral vector V such that the ratio of the distances between candidate points Ck |(C1-C2)·V| / |(C3-C4)·V| closes to the ratio of the pitch differences |P1-P2| / |P3-P4|. The candidate point Ck estimated through the above process is determined as the feature point Q on the keyboard plane. As described above, according to the first embodiment, the position of the keyboard plane, including the scale (reduced size) within the analysis space S, can be estimated with high accuracy.

[0055] The optimization process for the objective functions Fb, Fc, and Fd exemplified above is a process that estimates the keyboard plane using multiple candidate points C identified from multiple captured images Gn through a candidate point identification process. In other words, according to the first embodiment, the keyboard plane of the keyboard instrument 10 can be estimated without requiring image recognition (object detection) of the keys 11 of the keyboard instrument 10 or analysis of the performer's fingerings. Since the black keys and white keys on the keyboard 11 are arranged periodically in a predetermined pattern, it is actually difficult to achieve high-precision image recognition of the keys 11. Considering these circumstances, the first embodiment, which does not require image recognition of the keys 11, is particularly effective.

[0056] In the above explanation, each objective function F was described individually, but in the actual optimization process (S3), the control device 31 estimates each position of the target to be estimated in such a way that the sum of the objective functions Fa to Fd (Fa + Fb + Fc + Fd) is minimized.

[0057] B: Second Embodiment A second embodiment of this disclosure will now be described. For elements whose function is the same as in the first embodiment in each of the embodiments described below, the same reference numerals as in the first embodiment will be used, and detailed descriptions of each will be omitted as appropriate.

[0058] Figure 9 is a block diagram illustrating the configuration of the performance analysis system 100 in the second embodiment. The shooting system 20 of the second embodiment includes a shooting device 21-3 in addition to the shooting devices 21-1 and 21-2 of the first embodiment. The shooting device 21-3 is a camera similar to the shooting device 21-1 or shooting device 21-2, and generates a captured image G3 by shooting the situation in which the performer U plays the keyboard instrument 10. The shooting device 21-3 is installed, for example, in front of and above the performer U.

[0059] The control device 31 (image acquisition unit 41) acquires captured images G1 and G2, as well as captured image G3. The control device 31 (estimation processing unit 43) performs the aforementioned candidate point identification process for each of the three captured images Gn (G1 to G3). Furthermore, the objective function Fa applied to the optimization process of the second embodiment is the sum of the calculated values ​​of the aforementioned formula (1) for all combinations of selecting two imaging devices 21-n from three imaging devices 21-n.

[0060] The operation of the control device 31 is basically the same as in the first embodiment, except for the points described above. Therefore, the same effects as in the first embodiment are achieved in the second embodiment as well. In the above description, a configuration with three imaging devices 21-n has been given as an example, but the imaging system 20 may include four or more imaging devices 21-n.

[0061] C: Variant The following are examples of specific modifications that may be added to each of the embodiments exemplified above. Two or more embodiments may be arbitrarily selected from the following examples and merged as appropriate, provided they do not contradict each other.

[0062] (1) In each of the above-described embodiments, the performance pitch P is identified by analyzing the acoustic signal A, but the method by which the pitch acquisition unit 42 acquires the performance pitch P is not limited to the above examples. For example, in an embodiment in which performance data representing the content of the performance by performer U is supplied from the keyboard instrument 10 to the information processing system 30, the pitch acquisition unit 42 may acquire the pitch (e.g., note number) specified by the performance data as the performance pitch P. The performance data is, for example, time-series data conforming to the MIDI (Musical Instrument Digital Interface) standard. As can be understood from the above examples, the acquisition of the performance pitch P by the pitch acquisition unit 42 includes identifying the performance pitch P by analyzing the acoustic signal A and receiving the performance pitch P specified by the performance data.

[0063] (2) In each of the above embodiments, the horizontal axis La of the keyboard plane was determined from the horizontal vector V estimated by the optimization process (S3), but the method for determining the horizontal axis La of the keyboard plane is not limited to the above examples. For example, the estimation processing unit 43 may determine the horizontal axis La of the keyboard plane from a plurality of feature points Q estimated by the optimization process. For example, the estimation processing unit 43 can determine the horizontal axis La by principal component analysis of the plurality of feature points Q.

[0064] (3) In each of the above forms, multiple objective functions F(Fa~Fd) have been given as examples, but the application of some of the multiple objective functions F may be omitted in the optimization process. For example, one or more of the objective functions Fb~Fd may be omitted in the optimization process.

[0065] (4) In each of the embodiments described above, the position in the analysis space S of an estimated object (e.g., the keyboard plane or the imaging device 21-n) related to the object being photographed was estimated from a plurality of photographed images Gn of the keyboard instrument 10 and the performer U. However, the applications of this disclosure are not limited to the examples given above.

[0066] For example, consider a scenario where multiple performers U play different instruments on a stage. The image acquisition unit 41 acquires multiple captured images Gn taken by the camera 21-n from different directions of the multiple performers U. The pitch acquisition unit 42 acquires the pitch P produced by the performance by the multiple performers U. The estimation processing unit 43 estimates the position of estimation targets related to the performance targets (multiple performers U) in the analysis space S by performing an analysis using the multiple captured images Gn and the performance pitch P. For example, the position of each performer U, the horizontal axis La and vertical axis Lb along the top surface of the stage, the position Zn and direction Dn of each camera 21-n, etc., are estimated by the same analysis processing as in each of the above-described forms. As can be understood from the above examples, the subjects captured by the camera 21-n (i.e., the subjects of the captured images Gn) and the estimation targets by the analysis processing can be arbitrarily changed.

[0067] (5) For example, the information processing system 30 in each of the above-described forms may be realized by a server device that communicates with an information device such as a smartphone or tablet terminal. The information processing system 30 receives multiple captured images Gn and acoustic signals A (or performance data) from the information device via a communication network. The information processing system 30 operates in the same manner as in each of the above-described forms to estimate the position of the object to be estimated in the analysis space S and transmits the result of the estimation to the information device.

[0068] (6) As illustrated in the examples of each embodiment described above, the functions of the information processing system 30 are realized through the cooperation of one or more processors constituting the control device 31 and a program stored in the storage device 32. The program according to this disclosure can be provided in a form stored on a computer-readable recording medium and installed on a computer. The recording medium is, for example, a non-transitory recording medium, such as an optical recording medium (optical disc) like a CD-ROM, but also includes any known form of recording medium such as a semiconductor recording medium or a magnetic recording medium. A non-transitory recording medium includes any recording medium except for transient propagation signals, and volatile recording media are not excluded. Furthermore, in a configuration in which a distribution device distributes a program via a communication network, the storage medium in which the distribution device stores the program corresponds to the non-transitory recording medium described above.

[0069] D: Note From the forms exemplified above, the following configuration can be understood, for example.

[0070] An information processing method according to one aspect of this disclosure (Aspect 1) acquires multiple images of a subject taken from different directions, acquires the pitch of a musical sound produced by the subject, and estimates the position in three-dimensional space of an estimated object related to the subject by performing an analysis using the multiple images and the musical pitch. In this aspect, the position in three-dimensional space of an estimated object related to the subject is estimated by an analysis using multiple images and the musical pitch. Therefore, the position of the estimated object in three-dimensional space can be estimated by a simple method that does not require elements dedicated to the analysis of the estimated object, such as a musical control element that is recognized as black by imaging with an infrared sensor (Patent Document 1) or a marker attached to the performer's hand (Patent Document 2).

[0071] A "subject to be photographed" is an object photographed in order to acquire images used for analysis processing. Specifically, an object that performs actions to produce a specific musical pitch is an example of a "subject to be photographed." For example, multiple fingers of a performer (both hands or one hand), or multiple performers playing a musical instrument are examples of "subjects to be photographed."

[0072] An "estimated object" is any object that serves as the target for estimating its position in three-dimensional space. Specifically, objects located around the subject being photographed are examples of "estimated objects." Note that an "estimated object" does not need to be explicitly included in multiple images. Furthermore, the "estimated object" can be the subject being photographed itself.

[0073] In a specific example of Embodiment 1 (Embodiment 2), the object to be photographed includes the hands of a performer playing a keyboard instrument, and the object to be estimated includes the keyboard plane corresponding to the surface of the keys on the keyboard instrument. In the above embodiment, the position of the keyboard plane of the keyboard instrument in three-dimensional space is estimated by analyzing the object to be photographed, including the performer's hands. In other words, the keyboard plane of the keyboard instrument can be estimated without requiring image recognition (object detection) of the keys on the keyboard instrument or analysis of the performer's fingerings.

[0074] In a specific example of Embodiment 2 (Embodiment 3), the analysis process includes, for each of the multiple captured images, a process to detect multiple observation points corresponding to the object being photographed in the captured image; a process to duplicate the captured image into multiple processed images corresponding to different pitches; and a process to select candidate points that are candidates for locations on the keyboard plane from the multiple observation points in the processed image corresponding to the pitch being played among the multiple processed images, and the keyboard plane is estimated from the multiple candidate points identified from the multiple captured images. In the above embodiment, candidate points on the keyboard plane are identified from the observation points in the processed image corresponding to the pitch being played among the multiple processed images obtained by duplicating the captured image, and the keyboard plane is estimated using the multiple candidate points. Therefore, the keyboard plane of a keyboard instrument can be estimated without requiring image recognition (object detection) of the keys on the keyboard instrument or analysis of the performer's fingerings.

[0075] In a specific example of Embodiment 3 (Embodiment 4), the analysis process estimates the keyboard plane as the three-dimensional plane that minimizes the sum of the distances to each of the multiple candidate points. Since each candidate point is an observation point in the processing image corresponding to the pitch of the played note among the multiple processing images, there is a high probability that it is close to the keyboard plane. Therefore, by estimating the three-dimensional plane that minimizes the sum of the distances to each of the multiple candidate points, the keyboard plane in three-dimensional space can be estimated with high accuracy.

[0076] In any specific example of Embodiments 2 to 4 (Embodiment 5), the analysis process estimates the horizontal vector such that the dot product of the horizontal vector along the longitudinal direction of the keyboard and the normal vector of the keyboard plane is minimized within the keyboard plane. In the above embodiment, the horizontal vector is estimated such that the dot product of the horizontal vector along the longitudinal direction of the keyboard and the normal vector of the keyboard plane is minimized. Therefore, the horizontal axis along the longitudinal direction of the keyboard can be estimated with high accuracy.

[0077] In a specific example of Embodiment 5 (Embodiment 6), the analysis process estimates the horizontal vector such that, in the direction of the horizontal vector, the ratio of the distance between the first candidate point and the second candidate point among the plurality of candidate points, and the ratio of the distance between the third candidate point and the fourth candidate point among the plurality of candidate points, approaches the ratio of the pitch difference between the first pitch corresponding to the first candidate point and the second pitch corresponding to the second candidate point, and the pitch difference between the third pitch corresponding to the third candidate point and the fourth pitch corresponding to the fourth candidate point. In the above embodiment, under the condition that the ratio of the distance between candidate points to the pitch difference is constant, feature points located on the keyboard plane and the horizontal vector of the keyboard plane are estimated. That is, for example, the position of the keyboard plane can be estimated including the scale in three-dimensional space.

[0078] In any specific example of Embodiments 1 to 6 (Embodiment 7), the estimation target includes a plurality of imaging devices that each capture the plurality of images, and in the analysis process, the position and orientation in the three-dimensional space are estimated for each of the plurality of imaging devices. According to the above embodiments, the position and orientation of each imaging device can be estimated using the images captured by each of the plurality of imaging devices.

[0079] An information processing system according to one aspect of the present disclosure (Aspect 8) comprises an image acquisition unit that acquires a plurality of images of a subject taken from different directions, a pitch acquisition unit that acquires the pitch of a musical sound produced by the subject, and an estimation processing unit that estimates the position in three-dimensional space of an estimated object related to the subject by performing an analysis process using the plurality of images and the musical pitch.

[0080] A program according to one aspect of the present disclosure (Aspect 9) causes a computer system to function as an image acquisition unit that acquires a plurality of images of a subject taken from different directions, a pitch acquisition unit that acquires the pitch of a musical sound produced by the subject, and an estimation processing unit that estimates the position in three-dimensional space of an estimated object related to the subject by performing an analysis process using the plurality of images and the musical pitch. [Explanation of symbols]

[0081] 100...Performance analysis system, 10...Keyboard instrument, 11...Keyboard, 12...Key, 20...Photography system, 21-n(21-1,21-2,21-3)...Photography device, 30...Information processing system, 31...Control device, 32...Storage device, 33...Communication device, 34...Display device, 35...Operating device, 36...Sound pickup device, 41...Image acquisition unit, 42...Pitch acquisition unit, 43...Estimation processing unit.

Claims

1. By acquiring multiple images of the subject taken from different directions, The pitch of the musical notes produced by the aforementioned subject is obtained. By performing an analysis process using the aforementioned multiple captured images and the aforementioned musical pitch, the position of the estimated object related to the captured object is estimated in three-dimensional space. Information processing methods implemented by computer systems.

2. The aforementioned subject of filming includes the hands of a performer playing a keyboard instrument. The aforementioned estimation target includes the keyboard plane corresponding to the surface of the keys in the keyboard instrument. The information processing method of claim 1.

3. In the aforementioned analysis process, For each of the aforementioned multiple captured images, A process for detecting multiple observation points corresponding to the object being photographed in the captured image, The process involves duplicating the captured image into multiple processed images corresponding to different pitches, A process to select candidate points on the keyboard plane from among the plurality of observation points in the processing image corresponding to the pitch of the played sound among the plurality of processing images. Execute, The keyboard plane is estimated from a plurality of candidate points identified from the plurality of captured images. The information processing method of claim 2.

4. In the aforementioned analysis process, The three-dimensional plane that minimizes the sum of the distances to each of the aforementioned candidate points is estimated as the keyboard plane. The information processing method of claim 3.

5. In the aforementioned analysis process, The horizontal vector is estimated such that the dot product of the horizontal vector along the longitudinal direction of the key and the normal vector of the key plane is minimized within the keyboard plane. An information processing method according to any one of claims 2 to 4.

6. In the aforementioned analysis process, The horizontal vector is estimated such that, in the direction of the horizontal vector, the ratio of the distance between the first candidate point and the second candidate point among the plurality of candidate points, and the ratio of the distance between the third candidate point and the fourth candidate point among the plurality of candidate points, approaches the ratio of the pitch difference between the first pitch corresponding to the first candidate point and the second pitch corresponding to the second candidate point, and the pitch difference between the third pitch corresponding to the third candidate point and the fourth pitch corresponding to the fourth candidate point. The information processing method of claim 5.

7. The aforementioned target of estimation includes a plurality of imaging devices that capture the plurality of images, In the aforementioned analysis process, For each of the plurality of imaging devices, the position and orientation within the three-dimensional space are estimated. The information processing method of claim 1.

8. An image acquisition unit that acquires multiple images of the subject taken from different directions, A pitch acquisition unit that acquires the pitch of the musical notes produced by the subject being photographed, An estimation processing unit estimates the position in three-dimensional space of an estimated object related to the captured object through analysis processing using the aforementioned multiple captured images and the pitch of the played sound. An information processing system equipped with the following features.

9. Image acquisition unit that acquires multiple images of the subject taken from different directions. A pitch acquisition unit that acquires the pitch of the musical notes produced by the subject being photographed, and An estimation processing unit estimates the position in three-dimensional space of an estimated object related to the captured object through analysis processing using the aforementioned multiple captured images and the aforementioned pitch of the played sound. A program that makes a computer system function.

Images