Information processing system, information processing method, and program

Abstract

To enable a player to play a keyboard instrument while visually referring to performance of another player.SOLUTION: An information processing system 100 includes: an image acquisition unit 41 that acquires, from an imaging device 21, an observation image including keyboards of a keyboard instrument 10 in a real space; and a display control unit 42 that displays, on a display device 22, a reference image which is a moving image including fingers of a reference player who performs a performance operation such that the fingers of the reference player overlap the keyboards in the observation image.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates to technology for assisting playing of keyboard instruments. [Background technology]

[0002] Various techniques have been proposed for supporting the playing of musical instruments such as keyboard instruments. For example, Patent Document 1 discloses a technique for estimating time-series information about the posture of fingers by analyzing images of the fingers of a user playing a keyboard instrument. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2021 / 157691 Summary of the Invention [Problem to be solved by the invention]

[0004] For example, when practicing playing a keyboard instrument, it is desirable for an instructor to be present near the learner. However, it is practically difficult for an instructor to always be present near the learner. While the above description has been based on the assumption that a learner practices a keyboard instrument, similar problems arise in situations where multiple users play keyboard instruments together. In consideration of the above circumstances, one aspect of the present disclosure aims to provide a technology that enables a player to play a keyboard instrument while visually referring to the performances of other players. [Means for solving the problem]

[0005] In order to solve the above problems, an information processing system according to one aspect of the present disclosure includes an image acquisition unit that acquires an observation image including the keys of a keyboard instrument in real space from an imaging device, and a display control unit that displays a reference image, which is a video including the fingers of a reference performer performing a performance action, on a display device so that the fingers of the reference performer overlap the keys in the observation image.

[0006] An information processing method according to one aspect of the present disclosure includes obtaining an observation image including the keys of a keyboard instrument in real space from an imaging device, and displaying a reference image, which is a video including the fingers of a reference performer performing a performance action, on a display device so that the fingers of the reference performer overlap the keys in the observation image.

[0007] A program according to one aspect of the present disclosure causes a computer system to function as an image acquisition unit that acquires an observation image including the keys of a keyboard instrument in real space from an imaging device, and a display control unit that displays a reference image, which is a video including the fingers of a reference performer performing a performance action, on a display device so that the fingers of the reference performer overlap the keys in the observation image. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a configuration diagram of a performance system. [Figure 2] FIG. 1 is a block diagram illustrating a configuration of a video system. [Figure 3] FIG. 2 is an explanatory diagram of the operation of the information processing system. [Figure 4] FIG. 1 is a block diagram illustrating a configuration of an information processing system. [Figure 5] FIG. 2 is a block diagram illustrating an example of a functional configuration of an information processing system. [Figure 6] 10 is a flowchart of an image generation process. [Figure 7] 10 is a flowchart of a control process. [Figure 8] FIG. 10 is a schematic diagram of an observation image in the second embodiment. [Figure 9] 10 is a flowchart of an image generation process in the fourth embodiment. [Figure 10] 13 is a flowchart of an image generation process in the fifth embodiment. [Figure 11] FIG. 13 is a block diagram illustrating the configuration of a performance system according to a sixth embodiment. [Figure 12]FIG. 13 is a block diagram illustrating an example of the functional configuration of an information processing system according to a sixth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] A: First embodiment FIG. 1 is a block diagram illustrating the configuration of a performance system 100 in the first embodiment. The performance system 100 is a computer system that assists a user U in playing a keyboard instrument 10. The user U uses the performance system 100 to practice playing the keyboard instrument 10. The performance system 100 includes the keyboard instrument 10, a video system 20, and an information processing system 30. Each of the keyboard instrument 10 and the video system 20 can communicate with the information processing system 30 via wired or wireless communication.

[0010] The keyboard instrument 10 is an electronic musical instrument installed in the real space where the user U is located, and includes a keyboard Bx and a housing 11. The keyboard Bx is composed of a plurality of keys Kx (white keys and black keys) corresponding to different pitches. The user U plays a specific piece of music (hereinafter referred to as a "target piece of music") by sequentially operating each key Kx on the keyboard instrument 10. The keyboard instrument 10 supplies performance data P representing the performance to the information processing system 30 in parallel with the performance by the user U. The performance data P is data representing the notes played by the user U. The performance data P is sequentially output from the keyboard instrument 10 each time the user U operates the keyboard Bx (e.g., each time a key is pressed). For example, the performance data P specifies the pitch and key pressing strength corresponding to the key Kx pressed by the user U. The performance data P is, for example, event data conforming to the MIDI (Musical Instrument Digital Interface) standard.

[0011] The video system 20 is a video device (HMD: Head Mounted Display) that is worn on the head of the user U. For example, a goggle-type or eyeglass-type HMD is used as the video system 20.

[0012] 2 is a block diagram illustrating the configuration of the video system 20. The video system 20 includes an imaging device 21 and a display device 22. That is, the imaging device 21 and the display device 22 are worn on the head of a user U. Therefore, the positions and directions of the imaging device 21 and the display device 22 change according to the movement of the user U's head.

[0013] The imaging device 21 captures an image of a predetermined range in real space. Specifically, the imaging device 21 captures an image of a predetermined range located in front of the head of the user U (i.e., in the direction of the line of sight). The user U playing the keyboard instrument 10 visually recognizes the user U's fingers Hx and the keyboard Bx. Therefore, as illustrated in FIG. 3, the imaging device 21 captures a moving image (hereinafter referred to as an "observation image") X including the user U's left and right fingers Hx and the keyboard Bx of the keyboard instrument 10. The range captured by the imaging device 21 changes depending on the movement of the user U's head. The imaging device 21 generates observed image data Dx representing the observed image X in parallel with the user U's performance of the keyboard instrument 10. The imaging device 21 includes, for example, an optical system such as a photographing lens, an imaging element that receives incident light from the optical system, and a processing circuit that generates observed image data Dx according to the amount of light received by the imaging element. The imaging device 21 automatically adjusts the exposure so that the boundaries between the white and black keys of the keyboard Bx and the fingers Hx of the user U are clearly visible, for example.

[0014] The display device 22 in Fig. 2 displays various images under the control of the information processing system 30. For example, various display panels such as a liquid crystal display panel or an organic EL (Electroluminescence) panel are used as the display device 22. The display device 22 is installed in front of both eyes of the user U. The display device 22 of the first embodiment is a non-transmissive display panel that does not transmit light arriving from real space.

[0015] The display device 22 of the first embodiment displays a guide image Z of FIG. 3. The guide image Z is a moving image including an observation image X and a reference image Y. The reference image Y is a three-dimensional moving image including the fingers Hy of a virtual performer (hereinafter referred to as the "reference performer") performing a performance action and the keyboard By on which the reference performer plays. The reference performer is a virtual instructor who instructs the user U on performance. Note that another performer who plays duets with the user U may also be the reference performer. The keyboard By includes multiple keys Ky arranged horizontally, similar to the keyboard Bx in the real space. The reference image Y is a moving image that shows the reference performer performing the target piece of music by operating the keys By with the fingers Hy. In other words, the reference image Y shows exemplary fingering for the target piece of music.

[0016] In the guide image Z, the reference image Y is displayed so as to overlap the keyboard Bx in the observation image X. That is, the guide image Z is displayed on the display device 22 so that the fingers Hy and keys By of the reference performer in the reference image Y overlap the keys Bx in the observation image X. Specifically, the size and orientation of the fingers Hy and keys By in the reference image Y are set according to the size and orientation of the keyboard Bx in the observation image X. For example, the reference image Y is enlarged or reduced so that each key Ky on the keyboard By and each key Kx on the keyboard Bx in the observation image X are the same size. Furthermore, the angles of the fingers Hy and keys By in the reference image Y are controlled so that the surface of the keyboard By and the surface of the keyboard Bx in the observation image X are parallel to each other. Therefore, the user U can play the keyboard instrument 10 with the sensation that the reference performer is actually present in real space.

[0017] 4 is a block diagram illustrating an example of the configuration of the information processing system 30. The information processing system 30 is a computer system for controlling the video system 20. Specifically, the information processing system 30 displays a guide image Z on the display device 22.

[0018] The information processing system 30 includes a control device 31, a storage device 32, an operation device 33, and a sound emitting device 34. The information processing system 30 is realized by a portable information device such as a smartphone or a tablet terminal, or a portable or stationary information device such as a personal computer. The information processing system 30 may be realized as a single device, or may be realized as a plurality of devices configured separately from each other. The information processing system 30 may also be mounted on the keyboard instrument 10 or the video system 20.

[0019] The control device 31 is one or more processors that control the operation of the information processing system 30. Specifically, the control device 31 is configured by one or more types of processors, such as a central processing unit (CPU), a graphics processing unit (GPU), a sound processing unit (SPU), a digital signal processor (DSP), a field programmable gate array (FPGA), or an application specific integrated circuit (ASIC).

[0020] The storage device 32 is one or more memories that store programs executed by the control device 31 and various data used by the control device 31. For example, a known recording medium such as a semiconductor recording medium or a magnetic recording medium, or a combination of multiple types of recording media, is used as the storage device 32. Note that, for example, a portable recording medium that is detachable from the information processing system 30, or a recording medium that the control device 31 can access via a communication network (for example, cloud storage) may also be used as the storage device 32.

[0021] The storage device 32 of the first embodiment stores music data M and reference image data Dy for each of a plurality of music pieces. The music data M for each music piece is data representing an exemplary or standard performance of that music piece. Specifically, the music data M for each music piece is time-series data that specifies the time sequence of the notes that make up that music piece. For example, music files that comply with the MIDI standard are exemplified as the music data M.

[0022] The reference image data Dy for each piece of music is video data representing the reference image Y for that piece of music. That is, the reference image Y for each piece of music is a three-dimensional video representing the appearance of a reference performer performing that piece of music. The reference image Y for each piece of music can also be expressed as a video representing the fingering for that piece of music.

[0023] The reference image data Dy is 3DCG (three-dimensional computer graphics) data in any format. For example, point cloud data that represents the fingers Hy and the keyboard By of the reference performer as a point cloud is used as the reference image data Dy. Alternatively, skeleton data that represents the bones or joints of the fingers Hy and the keyboard By may be used as the reference image data Dy. For example, the reference image data Dy is generated based on the results of capturing images of the fingers Hy and the keyboard By from multiple directions in parallel with the performance of a musical piece by a real performer. Therefore, each of the multiple keys Ky of the keyboard By in the reference image Y displaces in conjunction with the performance by the reference performer. In other words, a user U who views the reference image Y can confirm how the keys Ky displace when the reference performer presses the keys.

[0024] The operation device 33 is an input device that accepts operations by the user U. For example, the user U selects a target song from a plurality of songs by operating the operation device 33. For example, an operator operated by the user U or a touch panel that detects contact by the user U is used as the operation device 33. Note that the operation device 33, which is separate from the information processing system 30, may be connected to the information processing system 30 by wire or wirelessly.

[0025] The sound emitting device 34 emits sound under the control of the control device 31. The sound emitting device 34 is, for example, a speaker or headphones. For example, an audio signal A representing a waveform of a musical sound corresponding to a performance action by the user U is supplied to the sound emitting device 34. For convenience, a D / A converter that converts the audio signal A from digital to analog and an amplifier that amplifies the audio signal A are not shown in the figure. Furthermore, the sound emitting device 34, which is separate from the information processing system 30, may be connected to the information processing system 30 by wire or wirelessly.

[0026] 5 is a block diagram illustrating an example of the functional configuration of the information processing system 30. The control device 31 executes a program stored in the storage device 32 to realize a plurality of functions (an image acquisition unit 41, a display control unit 42, a reference identification unit 43, a sound source processing unit 44, and a performance analysis unit 45) for displaying the guide image Z on the display device 22.

[0027] The image acquisition unit 41 acquires the observation image X from the imaging device 21. Specifically, the image acquisition unit 41 receives the observation image data Dx generated by the imaging device 21 from the video system 20.

[0028] The display control unit 42 displays the guide image Z on the display device 22. Specifically, the display control unit 42 outputs guide image data Dz representing the guide image Z to the display device 22, thereby displaying the guide image Z on the display device 22. The display control unit 42 generates the guide image Z by combining the observation image X acquired by the image acquisition unit 41 with the reference image Y of the target song stored in the storage device 32.

[0029] As described above, the display control unit 42 generates the guide image Z by combining the reference image Y with the observation image X so that the reference performer's fingers Hy and keys By overlap the keys Bx in the observation image X. The size and orientation of the fingers Hy and keys By in the reference image Y are set according to the size and orientation of the keys Bx in the observation image X. Specifically, the display control unit 42 enlarges or reduces the reference image Y so that each key Ky on the keyboard By and each key Kx on the keyboard Bx in the observation image X are the same size. The display control unit 42 also controls the angles of the fingers Hy and keys By in the reference image Y so that the surface of the keyboard By and the surface of the keyboard Bx in the observation image X are parallel to each other.

[0030] The reference specifying unit 43 specifies a position R in the observed image X that serves as a reference for the reference image Y (hereinafter referred to as the "reference position"). The reference image Y is placed at a position in the observed image X that corresponds to the reference position R, thereby generating a guide image Z. In other words, the reference image Y is synthesized with the observed image X so as to have a predetermined positional relationship with the reference position R.

[0031] The reference identification unit 43 identifies the reference position R by analyzing the observed image X. Specifically, as illustrated in FIG. 3 , the reference identification unit 43 detects the edge 12 of the keyboard Bx in the observed image X and identifies the position of the edge 12 as the reference position R. One or both of the left edge 12 and the right edge 12 of the keyboard Bx are detected. Any known object detection technology can be used to detect the edge 12 of the keyboard Bx. For example, an object detection process using a trained model such as a deep neural network is used for the reference identification unit 43 to detect the edge 12 (i.e., identify the reference position R). Note that the edge 12 of the keyboard instrument 10 and the portions of the housing 11 of the keyboard instrument 10 located on both sides of the keyboard Bx have significantly different optical characteristics, such as color or material. Therefore, the edge 12 of the keyboard Bx can be accurately and easily detected from the observed image X.

[0032] The sound source processing unit 44 in Fig. 5 generates an audio signal A in response to the operation of the keyboard Bx by the user U. As described above, performance data P specifying the pitch corresponding to the key Kx operated by the user U is supplied from the keyboard instrument 10 to the information processing system 30. The sound source processing unit 44 is a MIDI sound source that generates an audio signal A corresponding to the pitch specified by the performance data P. In other words, the audio signal A is a signal representing a musical sound waveform of the pitch corresponding to one or more keys Kx operated by the user U. Note that, although a software sound source realized by the control device 31 is exemplified as the sound source processing unit 44 in the first embodiment, a hardware sound source dedicated to generating the audio signal A may also be used as the sound source processing unit 44.

[0033] The performance analysis unit 45 analyzes the performance of the keyboard instrument 10 by the user U. The performance analysis unit 45 of the first embodiment analyzes the performance of the user U using performance data P supplied from the keyboard instrument 10. The performance analysis unit 45 of the first embodiment identifies a performance point Q1, a performance speed Q2, and an operation position Q3. The analysis by the performance analysis unit 45 is repeated at a predetermined cycle.

[0034] A performance point Q1 is a point on the time axis at which the user U is performing within the target song. Any known technique may be employed to identify the performance point Q1. For example, the performance analysis unit 45 identifies the performance point Q1 by comparing the time series of performance data P supplied from the keyboard instrument 10 with the song data M of the target song. The performance speed Q2 is the speed (tempo) at which the user U performs the target song. The performance analysis unit 45 identifies the performance speed Q2 from the change in the performance point Q1 over time. For example, the amount of movement of the performance point Q1 within a unit time is calculated as the performance speed Q2.

[0035] The operation position Q3 is a spatial position on the keyboard Bx of the keyboard instrument 10 where the user U is playing. As described above, the performance data P specifies the pitch played by the user U. The performance analysis unit 45 identifies the pitch specified by the performance data P as the operation position Q3.

[0036] The display control unit 42 generates a guide image Z by using the reference position R identified by the reference identification unit 43 and the analysis results (Q1 to Q3) by the performance analysis unit 45. Fig. 6 is a flowchart of a process S5 in which the display control unit 42 generates a guide image Z (hereinafter referred to as "image generation process").

[0037] When the image generation process S5 starts, the display control unit 42 controls the playback of the reference image Y in accordance with the performance point Q1 and the performance speed Q2 (S51). The display control unit 42 sequentially acquires from the storage device 32 a portion of the reference image Y of the target song that corresponds to the performance point Q1. For example, if the performance point Q1 is the time when a specific note of the target song is played, the display control unit 42 acquires an image of the reference image Y at the time when the reference performer played that note. The display control unit 42 also controls the playback speed of the reference image Y in accordance with the performance speed Q2. For example, the higher the performance speed Q2, the faster the playback speed of the reference image Y. That is, the display control unit 42 of the first embodiment progresses the playback of the reference image Y in conjunction with the performance of the target song by the user U. Specifically, the display control unit 42 controls the playback of the reference image Y so that the performance of the target song by the user U and the playback of the reference image Y are synchronized with each other.

[0038] The display control unit 42 sets the position of the reference image Y relative to the observed image X according to the operation position Q3 and the reference position R (S52). Specifically, the display control unit 42 identifies the operation position Q3 within the keyboard Bx using the reference position R (i.e., the end 12 of the keyboard Bx) as a reference, and identifies a position having a predetermined relationship with the operation position Q3 as the position of the reference image Y. For example, the display control unit 42 positions the reference image Y within a range on the high note side or low note side of a predetermined range of the keyboard Bx that includes the operation position Q3. The predetermined range that includes the operation position Q3 is, for example, a range that spans one octave on the high note side and one octave on the low note side centered on the operation position Q3. Therefore, for example, the reference image Y is positioned at a position one octave higher or lower than the pitch (operation position Q3) played by the user U. As can be understood from the above explanation, the position of the reference image Y is set so as not to overlap with the fingers Hx of the user U himself in the observed image X. Therefore, the user U can easily check the reference image Y while playing the keyboard instrument 10.

[0039] The display control unit 42 generates a guide image Z by combining the reference image Y with the observation image X (S53). Specifically, a portion of the reference image Y corresponding to the performance point Q1 and the performance speed Q2 is combined at a position in the observation image X that is set according to the operation position Q3 and the reference position R. The display control unit 42 controls the size and angle of the reference image Y so that it overlaps with the keyboard Bx in the observation image X.

[0040] The position or angle of the keyboard Bx in the observation image X changes in response to the movement of the user U's head. Because the reference image Y is positioned so as to overlap the keyboard Bx in the observation image X, the position and angle of the reference performer's fingers Hy and keyboard By displayed by the display device 22 also change in response to the movement of the user U's head. As can be understood from the above explanation, the user U can view the reference performer's fingers Hy and keyboard By from a desired position and angle by appropriately changing the position and direction of his or her head. Also, as described above, each key Ky of the keyboard By in the reference image Y moves in conjunction with the performance movement of the reference performer. Therefore, in the guide image Z displayed by the display device 22, each key Ky of the keyboard By also moves in response to the performance movement of the fingers Hy. With the above configuration, the user U can visually view the reference performer's fingers Hy, as if the reference performer were actually operating the keyboard Bx of the keyboard instrument 10.

[0041] 7 is a flowchart of the process (hereinafter referred to as "control process") executed by the control device 31. The control process exemplified below is repeated at a predetermined cycle.

[0042] When the control process starts, the control device 31 (image acquisition unit 41) acquires an observed image X (S1). The reference identification unit 43 identifies a reference position R by analyzing the observed image X (S2). The control device 31 (sound source processing unit 44) generates an audio signal A according to performance data P supplied from the keyboard instrument 10 (S3). The control device 31 (performance analysis unit 45) analyzes the performance of the keyboard instrument 10 by the user U, thereby identifying an operation position Q3, a performance point Q1, and a performance speed Q2 (S4). The order of the processes (S1 to S4) illustrated above may be changed as desired.

[0043] The control device 31 (display control unit 42) generates the guide image Z by executing the image generation process S5 illustrated in Fig. 6. That is, as described above, the guide image Z is generated by combining the observed image X and the reference image Y. The control device 31 (display control unit 42) outputs the guide image data Dz representing the guide image Z to the display device 22, thereby displaying the guide image Z on the display device 22 (S6).

[0044] As described above, according to the first embodiment, the fingers Hy of the reference performer in the reference image Y are displayed so as to overlap the keys Bx in the observation image X. Therefore, the user U can enjoy an effective customer experience of playing the keyboard instrument 10 while visually referring to the fingers Hy of the reference performer displayed so as to overlap the keys Bx of the keyboard instrument 10. In particular, in the first embodiment, the positions and angles of the imaging device 21 and the display device 22 change according to the position and angle of the user U's head. Therefore, the position and angle of the keys By in the reference image Y displayed on the display device 22 change in conjunction with the movement (e.g., movement or rotation) of the user U's head. Therefore, the user can enjoy an effective customer experience of playing the keyboard instrument 10 as if the reference performer were present in real space.

[0045] Furthermore, in the first embodiment, a reference image Y is displayed at a position in the observed image X that corresponds to the reference position R of the keyboard instrument 10. Therefore, the fingers Hy of the reference performer can be placed at appropriate positions relative to the keyboard instrument 10 in real space.

[0046] In the first embodiment, the playback of the reference image Y is controlled according to the performance points Q1 and the performance speed Q2. That is, the playback of the reference image Y progresses in conjunction with the performance of the music piece by the user U. Therefore, the user U can play the target music piece with the feeling that he or she is playing the keyboard instrument 10 in cooperation with the reference performer.

[0047] B: Second embodiment A second embodiment will be described. Note that for elements whose functions are similar to those of the first embodiment in each of the following exemplary aspects, the same reference numerals as those in the description of the first embodiment will be used, and detailed descriptions of each will be omitted as appropriate. In the first embodiment, the end 12 of the keyboard Bx in the observation image X was identified as the reference position R. In the second and third embodiments, the method of identifying the reference position R differs from that of the first embodiment. The configuration and operation other than identifying the reference position R are similar to those of the first embodiment.

[0048] FIG. 8 is a schematic diagram of an observation image X in the second embodiment. A mark 13 is added to the keyboard instrument 10 of the second embodiment. FIG. 8 illustrates a star-shaped mark 13. The mark 13 is formed on the exterior of the keyboard instrument 10. For example, the mark 13 is formed in the horizontal center of the housing 11 of the keyboard instrument 10. The mark 13 is a sign such as a figure or symbol for visually identifying the product name or provider (manufacturer or seller) of the keyboard instrument 10. The mark 13 is typically a trademark. The mark 13 is formed, for example, by painting or printing on the housing 11. It is also possible to imagine a configuration in which a plate-shaped member with the mark 13 formed on its surface is fixed to the housing 11, or a configuration in which a plate-shaped member molded in the shape of the mark 13 is fixed to the housing 11.

[0049] The reference identification unit 43 identifies the reference position R by analyzing the observed image X (S2). The reference identification unit 43 of the second embodiment detects the mark 13 of the keyboard instrument 10 in the observed image X and identifies the position of the mark 13 as the reference position R. Any known object detection process may be used to detect the mark 13. For example, an object detection process using a trained model such as a deep neural network is used to detect the mark 13 by the reference identification unit 43 (i.e., to identify the reference position R). The process of setting the position of the reference image Y with respect to the observed image X using the reference position R as a reference (S52) is the same as in the first embodiment.

[0050] The second embodiment also achieves the same effects as the first embodiment. Note that the mark 13 of the keyboard instrument 10 is often formed at a specific position on the keyboard instrument 10. According to the second embodiment, in which the position of the mark 13 of the keyboard instrument 10 is identified as the reference position R, the reference position R can be appropriately identified even in a situation in which the end 12 of the keyboard Bx cannot be detected. For example, the reference position R can be identified even if the end 12 of the keyboard Bx is outside the imaging range of the imaging device 21 (the end 12 is not included in the observed image X).

[0051] C: Third embodiment The reference identification unit 43 of the third embodiment identifies the position of a predetermined specific key (hereinafter referred to as a "specific key") Kx among the keys Bx of the keyboard instrument 10 as the reference position R.

[0052] Prior to the display of the guide image Z, the user U operates a specific key Kx on the keyboard Bx. The reference identification unit 43 identifies the reference position R by analyzing the observed image X (S2). The reference identification unit 43 of the third embodiment analyzes the observed image X to detect the key Kx operated by the user U in the observed image X (i.e., the specific key Kx), and identifies the position of the key Kx as the reference position R. For example, among the multiple keys Kx in the observed image X, the key Kx whose gradation has changed due to the operation by the user U is detected as the specific key Kx. The process (S52) of setting the position of the reference image Y with respect to the observed image X using the reference position R as a reference is the same as in the first embodiment.

[0053] The third embodiment also achieves the same effects as the first embodiment. Furthermore, in the third embodiment, the position of the specific key Kx operated by the user U is identified as the reference position R. Therefore, even in a state where the end 12 or the mark 13 of the keyboard instrument 10 cannot be detected, the reference position R can be appropriately identified. For example, even if the end 12 and the mark 13 of the keyboard instrument 10 are outside the imaging range of the imaging device 21, it is possible to identify the reference position R.

[0054] D: Fourth embodiment In the fourth embodiment, the operation of the performance analysis unit 45 and the display control unit 42 differs from that of the first embodiment. The performance analysis unit 45 of the fourth embodiment generates the same information (Q1 to Q3) as in the first embodiment, and also determines whether or not the user U is playing the keyboard instrument 10. Specifically, the performance analysis unit 45 determines that the keyboard instrument 10 is being played when performance data P is supplied from the keyboard instrument 10, and determines that the keyboard instrument 10 is not being played when performance data P is not supplied from the keyboard instrument 10. Note that the method of determination by the performance analysis unit 45 is not limited to the above example. For example, the performance analysis unit 45 may determine whether or not the keyboard instrument 10 is being played by analyzing the observed image X.

[0055] As in the first embodiment, the display control unit 42 generates a guide image Z by combining the observation image X and the reference image Y. The display control unit 42 of the fourth embodiment controls the value of a transparency parameter α for the reference image Y in generating the guide image Z. The transparency parameter α is an index (alpha value) related to the degree to which elements behind the reference image Y are transparent to the reference image Y. The transparency parameter α in the fourth embodiment is transparency. That is, the larger the value of the transparency parameter α, the higher the degree to which elements behind the reference image Y are transparent to the reference image Y. For example, when the transparency parameter α is set to 0%, the reference image Y is displayed as an opaque image. On the other hand, when the transparency parameter α is set to 50%, the reference image Y is displayed as a translucent image. When the reference image Y is displayed translucently, the user U can see the part of the observation image X that is located behind the reference image Y.

[0056] The display control unit 42 controls the value of the transparency parameter α depending on whether or not a performance is being performed on the keyboard instrument 10. Fig. 9 is a flowchart of the image generation process S5 in the fourth embodiment. The process of controlling the reproduction of the reference image Y (S51) and the process of controlling the position of the reference image Y (S52) are the same as those in the first embodiment.

[0057] The control device 31 (performance analysis unit 45) determines whether or not the keyboard instrument 10 is being played (S54). If the keyboard instrument 10 is not being played (S54: NO), the control device 31 (display control unit 42) sets the transparency parameter α to a value α1 (S55). On the other hand, if the keyboard instrument 10 is being played (S54: YES), the control device 31 (display control unit 42) sets the transparency parameter α to a value α2 (S56). The control device 31 (display control unit 42) generates a guide image Z by combining the reference image Y with the observation image X based on the transparency parameter α set by the above processing (S53). The operations other than the image generation processing S5 are the same as those in the first embodiment.

[0058] The value α2 of the transparency parameter α indicates a higher degree of transparency compared to the value α1. As described above, since the transparency parameter α in the fourth embodiment represents transparency, the value α2 is set to a value greater than the value α1. For example, the value α2 is set to 50% (semi-transparent), and the value α1 is set to 0% (opaque). As described above, when the keyboard instrument 10 is not being played, the reference image Y is displayed opaque (α=α1), and when the keyboard instrument 10 is being played, the reference image Y is displayed semi-transparent (α=α2). In other words, when the user U is playing the keyboard instrument 10, the degree of transparency of the reference image Y is higher compared to when the keyboard instrument 10 is not being played. Note that the value α1 is an example of a "first value," and the value α2 is an example of a "second value."

[0059] The fourth embodiment also achieves the same effects as the first embodiment. In particular, in the fourth embodiment, the value of the transmission parameter α of the reference image Y is controlled. Therefore, compared to a configuration in which the value of the transmission parameter α is fixed, it is possible to switch between a usage mode in which the reference image Y is preferentially viewed and a usage mode in which the user U's own fingers Hx are preferentially viewed.

[0060] For example, in the fourth embodiment, when the user U is playing the keyboard instrument 10, the transparency parameter α is set to a value α2 that is more transparent than the value α1. Therefore, compared to a configuration in which the transparency parameter α is fixed to the value α1 (a configuration in which the reference image Y is always displayed opaque), the user U can preferentially view his or her own fingers Hx. On the other hand, when the user U is not playing the keyboard instrument 10, the transparency parameter α is set to a value α1 that is less transparent than the value α2. Therefore, compared to a configuration in which the transparency parameter α is fixed to the value α2 (a configuration in which the reference image Y is always displayed semi-transparent), the user U can easily view the reference image Y. Note that the second or third embodiment is also applicable to the fourth embodiment.

[0061] E: Fifth embodiment In the fifth embodiment, as in the fourth embodiment, the value of the transparency parameter α applied to the reference image Y is variably controlled. In the fourth embodiment, the transparency parameter α was controlled depending on whether or not the keyboard instrument 10 was being played, but in the fifth embodiment, the transparency parameter α is controlled depending on the operation mode. The transparency parameter α in the fifth embodiment is the degree of transparency, as in the fourth embodiment. In other words, the larger the value of the transparency parameter α, the greater the degree to which elements behind the reference image Y are transparent to the reference image Y.

[0062] The display control unit 42 sets either a first operation mode or a second operation mode as the operation mode. The first operation mode is a reference mode that prioritizes viewing of the reference image Y, and the second operation mode is a practice mode that prioritizes viewing of the fingers Hx of the user U. For example, the display control unit 42 sets the operation mode in response to an instruction from the user U to the operation device 33. For example, every time the user U instructs to switch the operation mode, the display control unit 42 changes the operation mode from one of the first operation mode and the second operation mode to the other.

[0063] 10 is a flowchart of the image generation process S5 in the fifth embodiment. The process of controlling the reproduction of the reference image Y (S51) and the process of controlling the position of the reference image Y (S52) are the same as those in the first embodiment.

[0064] The control device 31 (display control unit 42) determines whether the operation mode is the first operation mode (S57). If the operation mode is the first operation mode (S57: YES), the control device 31 (display control unit 42) sets the transmission parameter α to a value α1 (S58). On the other hand, if the operation mode is the second operation mode (S57: NO), the control device 31 (display control unit 42) sets the transmission parameter α to a value α2 (S59). The control device 31 (display control unit 42) generates a guide image Z by combining the reference image Y with the observation image X based on the transmission parameter α set by the above processing (S53). The operations other than the image generation processing S5 are the same as those in the first embodiment.

[0065] As in the fourth embodiment, the value α2 of the transparency parameter α means a higher degree of transparency compared to the value α1. Specifically, the value α2 is set to a value greater than the value α1. For example, the value α2 is set to 50% (semi-transparent), and the value α1 is set to 0% (opaque). As described above, in the first operation mode, the reference image Y is displayed opaque (α=α1), and in the second operation mode, the reference image Y is displayed semi-transparent (α=α2). That is, in the first operation mode, the degree of transparency of the reference image Y is higher compared to the second operation mode. Note that the value α1 is an example of a "first value," and the value α2 is an example of a "second value."

[0066] The fifth embodiment also achieves the same effects as the first embodiment. In particular, the fifth embodiment controls the value of the transmission parameter α of the reference image Y. Therefore, similar to the fourth embodiment, it is possible to switch between a usage mode in which the reference image Y is preferentially viewed and a usage mode in which the user U's own fingers Hx are preferentially viewed.

[0067] For example, in the fifth embodiment, in the second operation mode, the transparency parameter α is set to a value α2 that is more transparent than the value α1. Therefore, compared to a configuration in which the transparency parameter α is fixed to the value α1 (a configuration in which the reference image Y is always displayed opaque), the user U can preferentially view his or her own fingers Hx. On the other hand, in the first operation mode, the transparency parameter α is set to a value α1 that is less transparent than the value α2. Therefore, compared to a configuration in which the transparency parameter α is fixed to the value α2 (a configuration in which the reference image Y is always displayed semi-transparently), the user U can easily view the reference image Y. Note that the second or third embodiment is also applicable to the fifth embodiment.

[0068] F: Sixth embodiment FIG. 11 is a block diagram illustrating the configuration of a performance system 100 in the sixth embodiment. The information processing system 30 in the performance system 100 of the sixth embodiment includes a communication device 35 in addition to the same elements as in the first embodiment (a control device 31, a storage device 32, an operation device 33, and a sound emitting device 34). The communication device 35 communicates with an external device via a communication network 200 such as the Internet. Note that communication between the communication device 35 and the communication network 200 may be either wired or wireless. Furthermore, the communication device 35, which is separate from the information processing system 30, may be connected to the information processing system 30 by wire or wireless.

[0069] The recording system 60 in FIG. 11 is a computer system that records a reference image Y, and is used by an instructor T to instruct a user U on how to play a keyboard instrument 10. The instructor T corresponds to a reference performer. That is, the reference image Y in the sixth embodiment is a video image showing the instructor T playing a keyboard instrument 63. The keyboard instrument 63 is, for example, an acoustic or electronic instrument equipped with a keyboard By. The recording system 60 includes an imaging device 61 and a communication device 62.

[0070] The imaging device 61 captures an image of an instructor T playing a keyboard instrument 63. That is, the imaging device 61 captures a reference image Y including the left and right fingers Hy of the instructor T and the keys By of the keyboard instrument 63. The communication device 62 is, for example, a portable information device such as a smartphone or a tablet terminal, or a portable or stationary information device such as a personal computer. The imaging device 61 may be mounted on the communication device 62.

[0071] The communication device 62 transmits reference image data Dy representing the reference image Y to the information processing system 30. The reference image data Dy is video data representing a performance action by the instructor T (reference performer). For example, as in the first embodiment, point cloud data or skeleton data representing the instructor T's fingers Hy and the keyboard By is exemplified as the reference image data Dy. In parallel with the instructor T's performance of the keyboard instrument 63, the reference image data Dy is transmitted from the recording system 60 to the performance system 100. The reference image data Dy is received by the communication device 35 of the information processing system 30 via the communication network 200.

[0072] 12 is a block diagram illustrating the functional configuration of an information processing system 30 according to the sixth embodiment. The control device 31 according to the sixth embodiment functions as a reception control unit 46 in addition to the same elements as those in the first embodiment (an image acquisition unit 41, a display control unit 42, a reference identification unit 43, a sound source processing unit 44, and a performance analysis unit 45). The reception control unit 46 receives the reference image data Dy transmitted from the recording system 60 via the communication device 35.

[0073] The display control unit 42 in the first embodiment generates a guide image Z by combining a reference image Y of the reference image data Dy stored in the storage device 32 with an observed image X. The display control unit 42 in the sixth embodiment generates a guide image Z by combining a reference image Y of the reference image data Dy acquired by the reception control unit 46 with an observed image X (S5). That is, the display control unit 42 displays the reference image Y represented by the reference image data Dy on the display device 22. As described above, the guide image Z in the sixth embodiment is composed of an observed image X representing the user U playing the keyboard instrument 10 and a reference image Y representing the instructor T playing the keyboard instrument 63. The guide image Z including the reference image Y of the performance is displayed on the display device 22 in real time in parallel with the instructor T's performance of the keyboard instrument 63.

[0074] The sixth embodiment also achieves the same effects as the first embodiment. Furthermore, in the sixth embodiment, while the performance action of the instructor T (reference performer) is being imaged, a reference image Y of the performance action can be displayed on the display device 22 used by the user U. Therefore, the user U can play the keyboard instrument 10 with the feeling that the instructor T, who is actually located far away, is actually nearby.

[0075] G: Variation Specific modified embodiments that can be added to each of the above-described embodiments are exemplified below. Multiple embodiments arbitrarily selected from the above-described embodiments and the modified embodiments exemplified below may be combined as appropriate within the scope of not mutually contradicting each other.

[0076] (1) In the above-described embodiments, a non-transmissive display panel is exemplified as the display device 22. However, a transmissive display panel that transmits incoming light from real space may be used as the display device 22. In an embodiment in which a transmissive display device 22 is used, it is not necessary to display the observation image X on the display device 22. That is, the display control unit 42 displays the reference image Y as a guide image Z on the display device 22. The user U can view the keyboard Bx of the keyboard instrument 10 through transmitted light from real space and the reference image Y superimposed on the keyboard Bx. Even in an embodiment in which a transmissive display device 22 is used, the observation image X is used to identify the reference position R of the reference image Y. As can be understood from the above explanation, the display of the observation image X may be omitted. The display device 22 displays the reference image Y using augmented reality (AR) or mixed reality (MR).

[0077] (2) In the fourth and fifth embodiments, the transparency parameter α is transparency. However, the transparency parameter α may be opacity. In the embodiment where the transparency parameter α is opacity, the larger the value of the transparency parameter α, the lower the degree to which elements behind the reference image Y are transparent to the reference image Y. In the above embodiments, similar to the fourth embodiment, the transparency parameter α is set to a value α1 when the keyboard instrument 10 is not being played, and is set to a value α2 when the keyboard instrument 10 is being played. Also, similar to the fifth embodiment, the transparency parameter α is set to a value α1 in the first operation mode, and is set to a value α2 in the second operation mode. However, when the transparency parameter α is opacity, the value α2 is set to a value smaller than the value α1. For example, the value α2 is set to 50% (semi-transparent), and the value α1 is set to 100% (opaque). As can be seen from the above examples, regardless of whether the transmission parameter α is transparency or opacity, the value α2 is comprehensively expressed as a value that has a higher degree of transparency compared to the value α1. The relationship between the magnitude of the transmission parameter α and the degree of transparency of the reference image Y is arbitrary.

[0078] (3) In the above-described embodiments, an electronic musical instrument capable of outputting performance data P has been exemplified as the keyboard instrument 10. However, a natural musical instrument that does not output performance data P may also be used as the keyboard instrument 10. An audio signal A is generated by collecting musical sounds emitted from the natural musical instrument keyboard 10 with a microphone. The performance analysis unit 45 uses the audio signal A to analyze the performance of the keyboard instrument 10 by the user U. For example, the performance analysis unit 45 generates a time series of performance data P by analyzing the audio signal A, and identifies a performance point Q1 by comparing the time series of performance data P with music data M of the target music piece. The performance speed Q2 and operation position Q3 are similarly identified by analyzing the audio signal A.

[0079] (4) In the above-described embodiments, the performance analysis unit 45 analyzes the performance of the user U using the performance data P. However, the performance analysis unit 45 may also analyze the performance of the user U using the observed image X. For example, the performance analysis unit 45 identifies the operation position Q3 by analyzing the observed image X. For example, the performance analysis unit 45 detects the fingers Hx of the user U from the observed image X, and identifies the operation position Q3 according to the positional relationship of the fingers Hx with respect to the reference position R.

[0080] (5) In the above-described embodiments, the imaging device 21 and the display device 22 are attached to the head of the user U, but it is not essential that the imaging device 21 and the display device 22 are attached to the head of the user U. For example, an embodiment in which one or both of the imaging device 21 and the display device 22 are installed near the keyboard instrument 10 is also conceivable. However, in an embodiment in which the imaging device 21 and the display device 22 are attached to the head of the user U, the position and angle of the imaging device 21 and the display device 22 change according to the position and angle of the user U's head. In other words, the observed image X and the reference image Y displayed on the display device 22 change in conjunction with the movement of the user U's head. Therefore, the user U can play the keyboard instrument 10 with the sensation that the reference performer is actually present in real space.

[0081] (6) In the above-described embodiments, the fingers Hy of the reference performer (instructor T) are displayed on the display device 22, but the upper body or the entire body of the reference performer may be displayed on the display device 22 as the reference image Y. In addition, in the above-described embodiments, the reference image Y includes the fingers Hy of the reference performer and the keyboard By, but the keyboard By may be omitted from the reference image Y.

[0082] (7) In each of the above-described embodiments, it is assumed that the user U and the reference performer play a common target piece of music. However, the performance content of the user U and the performance content of the reference performer may differ. For example, of multiple performance parts of the target piece of music, the user U may play a first part and the reference performer may play a second part. Each of the first part and the second part is one or more performance parts of the target piece of music.

[0083] (8) The information processing system 30 may be realized by a server device that communicates with a terminal device such as a smartphone or a tablet terminal. The imaging device 21 is mounted on or connected to the terminal device. The terminal device transmits the observation image data Dx generated by the imaging device 21 and the performance data P generated by the keyboard instrument 10 to the information processing system 30. As in the above-mentioned embodiments, the information processing system 30 generates guide image data Dz from the observation image data Dx and the performance data P and transmits the guide image data Dz to the terminal device. The terminal device displays the guide image Z represented by the guide image data Dz on the display device 22. The display device 22 may be mounted on the terminal device, for example.

[0084] (9) As described above, the functions of the information processing system 30 according to each of the above embodiments are realized through cooperation between one or more processors constituting the control device 31 and a program stored in the storage device 32. The programs exemplified above can be provided in a form stored on a computer-readable recording medium and installed on a computer. The recording medium is, for example, a non-transitory recording medium, such as an optical recording medium (optical disk) such as a CD-ROM, but also includes any known form of recording medium, such as a semiconductor recording medium or a magnetic recording medium. Note that a non-transitory recording medium includes any recording medium other than a transitory, propagating signal, and does not exclude volatile recording media. Furthermore, in a configuration in which a distribution device distributes a program via a communication network, the recording medium storing the program in the distribution device corresponds to the non-transitory recording medium described above.

[0085] H: Notes From the above-described exemplary embodiments, the following configurations can be understood, for example.

[0086] An information processing system according to one aspect (aspect 1) of the present disclosure includes an image acquisition unit that acquires an observation image including the keys of a keyboard instrument in real space from an imaging device, and a display control unit that displays, on a display device, a reference image that is a moving image including the fingers of a reference performer performing a performance action, so that the fingers of the reference performer are superimposed on the keys in the observation image. With the above configuration, the fingers of the reference performer are displayed in the reference image so as to be superimposed on the keys in the observation image. Therefore, a user can play the keyboard instrument while visually referring to the fingers of the reference performer displayed superimposed on the keys of the keyboard instrument.

[0087] In a specific example (Aspect 2) of Aspect 1, the imaging device and the display device are worn on the user's head. In the above aspect, the position and angle of the imaging device and the display device change according to the position and angle of the user's head. That is, the position and angle of the keyboard in the reference image displayed on the display device change in conjunction with the movement of the user's head. Therefore, the user can play a keyboard instrument with the sensation that the reference performer is actually present in real space.

[0088] In a specific example (Aspect 3) of Aspect 1 or Aspect 2, the reference image includes a reference keyboard played by the reference performer, the reference keyboard including a plurality of keys that shift in response to the performance actions of the reference performer, and the display control unit displays the reference image on the display device so that the reference keyboard is superimposed on the keyboard in the observed image. According to the above aspect, the reference keyboard, each key of which shifts in response to the performance actions of the reference performer, is displayed superimposed on the keyboard in real space. Therefore, the user can visually see the fingers of the reference performer, as if the reference performer were actually operating the keyboard.

[0089] In a specific example (Aspect 4) of any one of Aspects 1 to 3, a reference specifying unit is further provided that specifies a reference position relative to the keyboard instrument in the observed image by analyzing the observed image, and the display control unit displays the reference image at a position corresponding to the reference position. According to the above aspect, the reference image is displayed at a position corresponding to the reference position relative to the keyboard instrument. Therefore, the fingers of the reference performer can be positioned appropriately relative to the keyboard instrument in real space.

[0090] In a specific example (Aspect 5) of Aspect 4, the reference identification unit detects the edge of the keyboard in the observed image and identifies the position of the edge as the reference position. The edge (left or right edge) of a keyboard instrument and the parts (clappers or arms) located on both sides of the keyboard of the keyboard instrument have significantly different optical characteristics, such as color. Therefore, the edge of the keyboard can be accurately and easily detected from the image of the keyboard. In other words, by configuring the position of the edge of the keyboard to be identified as the reference position, the reference position can be identified accurately and easily.

[0091] In a specific example (Aspect 6) of Aspect 4, the reference identification unit detects a mark of the keyboard instrument in the observed image and identifies the position of the mark as the reference position. Marks such as trademarks of keyboard instruments are often displayed in specific positions on the keyboard instrument (for example, the center of the casing in the horizontal direction). Therefore, by using a configuration that identifies the position of the mark of the keyboard instrument as the reference position, the reference position can be appropriately identified even in a state where, for example, the edge of the keyboard cannot be detected.

[0092] In a specific example (Aspect 7) of Aspect 4, the reference identification unit detects an operated key from among the multiple keys of the keyboard in the observed image and identifies the position of that key as the reference position. According to the above aspect, the position of the key operated by the user is identified as the reference position. Therefore, the reference position can be appropriately identified even in a state where the edge or mark of the keyboard instrument cannot be detected.

[0093] In a specific example (Aspect 8) of any of Aspects 1 to 7, the display control unit controls the value of the transmission parameter of the reference image. In the above aspects, the value of the transmission parameter of the reference image is controlled. Therefore, compared to a configuration in which the value of the transmission parameter of the reference image is fixed, it is possible to switch between a usage mode in which the reference performer's fingers are prioritized for viewing and a usage mode in which the user himself is prioritized for viewing.

[0094] In a specific example (aspect 9) of aspect 8, the display control unit further includes a performance analysis unit that determines whether the keyboard instrument is being played, and when it is determined that the keyboard instrument is not being played, the display control unit sets the transparency parameter to a first value, and when it is determined that the keyboard instrument is being played, the display control unit sets the transparency parameter to a second value that is more transparent than the first value. In the above aspect, when a user is playing a keyboard instrument, the transparency parameter is set to the second value that is more transparent than the first value. Therefore, compared to a configuration in which the transparency parameter is fixed to the first value, the user can preferentially view their own fingers. On the other hand, when the user is not playing a keyboard instrument, the transparency parameter is set to the first value that is less transparent than the second value. Therefore, compared to a configuration in which the transparency parameter is fixed to the second value, the user can more easily view the reference image.

[0095] In a specific example (Aspect 10) of Aspect 8, the display control unit sets the transparency parameter to a first value in the first operating mode, and sets the transparency parameter to a second value that is more transparent than the first value in the second operating mode. In the above aspect, in the second operating mode, the transparency parameter is set to the second value that is more transparent than the first value. Therefore, compared to a configuration in which the transparency parameter is fixed to the first value, the user can preferentially view his or her own fingers. On the other hand, in the first operating mode, the transparency parameter is set to the first value that is less transparent than the second value. Therefore, compared to a configuration in which the transparency parameter is fixed to the second value, the user can more easily view the reference image.

[0096] In a specific example (Aspect 11) of any of Aspects 1 to 10, the keyboard instrument further includes a performance analysis unit that identifies an operation position on the keyboard of the keyboard, and the display control unit displays the reference image within a range on the high-pitched or low-pitched side of a range on the keyboard that includes the operation position. According to the above aspect, the reference image is displayed within a range on the high-pitched or low-pitched side of the operation position of the user. In other words, the reference image is displayed so as not to overlap with the user's fingers. Therefore, the user can easily check the reference image while playing the keyboard instrument.

[0097] In a specific example (Aspect 12) of any one of Aspects 1 to 11, a performance analysis unit that identifies performance points within the musical piece is further provided, and the display control unit displays, on the display device, portions of the reference image corresponding to the musical piece that correspond to the performance points. According to the above aspect, the playback of the reference image progresses in conjunction with the user's performance of the musical piece. Therefore, the user can play the musical piece with the feeling that they are playing a keyboard instrument in cooperation with the reference performer.

[0098] In a specific example (aspect 13) of aspect 12, the performance analysis unit further identifies a performance speed, and the display control unit controls the playback speed of the reference image in accordance with the performance speed. According to the above aspect, the playback of the reference image progresses in conjunction with the user's performance of the music piece. Therefore, the user can play the music piece with the feeling that they are playing a keyboard instrument in cooperation with the reference performer.

[0099] In a specific example (Aspect 14) of any of Aspects 1 to 13, the system further includes a reception control unit that receives video data representing performance actions by the reference performer via a communication device, and the display control unit displays the reference image represented by the video data on the display device. According to the above aspects, the reference image of the performance actions by the reference performer can be displayed on the display device used by the user in parallel with the capture of the performance actions by the reference performer. Therefore, the user can play a keyboard instrument with the feeling that the reference performer, who is actually located far away, is actually nearby.

[0100] An information processing method according to one aspect of the present disclosure includes acquiring an observation image including the keys of a keyboard instrument in real space from an imaging device, and displaying a reference image, which is a video including the fingers of a reference performer performing a performance action, on a display device so that the fingers of the reference performer overlap the keys in the observation image.

[0101] A program according to one aspect of the present disclosure causes a computer system to function as an image acquisition unit that acquires an observation image including the keys of a keyboard instrument in real space from an imaging device, and a display control unit that displays a reference image, which is a video including the fingers of a reference performer performing a performance action, on a display device so that the fingers of the reference performer overlap the keys in the observation image. [Explanation of symbols]

[0102] 100...performance system, 10...keyboard instrument, 20...video system, 21...imaging device, 22...display device, 30...information processing system, 31...control device, 32...storage device, 33...operation device, 34...sound emission device, 35...communication device, 41...image acquisition unit, 42...display control unit, 43...reference identification unit, 44...sound source processing unit, 45...performance analysis unit, 46...reception control unit, 60...recording system, 61...imaging device, 62...communication device, 63...keyboard instrument, 200...communication network.

Claims

1. an image acquisition unit that acquires an observation image including the keyboard of a keyboard instrument in real space from an imaging device; a display control unit that displays, on a display device, a reference image that is a moving image including the fingers of a reference performer performing a performance action, so that the fingers of the reference performer overlap the keyboard in the observed image; a performance analysis unit that identifies performance points within a piece of music and a performance speed at which a user plays the piece of music on the keyboard instrument; Equipped with The display control unit displays a portion of the reference image corresponding to the musical piece that corresponds to the performance point on the display device, and controls a playback speed of the reference image in accordance with the performance speed. Information processing system.

2. The imaging device and the display device are worn on the head of the user. The information processing system of claim 1.

3. the reference image includes a reference keyboard played by the reference performer; the reference keyboard includes a plurality of keys that are displaced in response to a performance action by the reference performer; The display control unit displays the reference image on the display device so that the reference keyboard is superimposed on the keyboard in the observed image. The information processing system of claim 1.

4. a reference specifying unit that specifies a reference position for the keyboard instrument in the observation image by analyzing the observation image; The display control unit displays the reference image at a position corresponding to the reference position. The information processing system of claim 1.

5. The reference identification unit detects an edge of the keyboard in the observed image and identifies the position of the edge as the reference position.

5. The information processing system of claim 4.

6. The reference identification unit detects a mark of the keyboard instrument in the observed image and identifies the position of the mark as the reference position.

5. The information processing system of claim 4.

7. The reference identification unit detects an operated key among the plurality of keys on the keyboard in the observed image and identifies the position of the operated key as the reference position.

5. The information processing system of claim 4.

8. The display control unit controls a value of a transmission parameter of the reference image. The information processing system of claim 1.

9. a performance analysis unit that determines whether the keyboard instrument is being played; The display control unit setting the transparency parameter to a first value when it is determined that the keyboard instrument is not being played; When it is determined that the keyboard instrument is being played, the transparency parameter is set to a second value that is higher in transparency than the first value. The information processing system of claim 8.

10. The display control unit in a first mode of operation, setting the transmission parameter to a first value; In a second mode of operation, the transmission parameter is set to a second value that is more transmissive than the first value. The information processing system of claim 8.

11. a performance analysis unit that identifies an operation position on the keyboard of the keyboard instrument; The display control unit displays the reference image within a range on the keyboard that is either higher or lower than the range that includes the operation position. The information processing system of claim 1.

12. a reception control unit that receives, via a communication device, video data representing the performance actions of the reference performer; The display control unit displays the reference image represented by the video data on the display device. The information processing system of claim 1.

13. acquiring an observation image including the keyboard of a keyboard instrument in real space from an imaging device; displaying a reference image, which is a moving image including the fingers of a reference performer performing a performance action, on a display device so that the fingers of the reference performer overlap the keys in the observed image; Identifying a performance point within the piece of music and a performance speed at which the user plays the piece of music on the keyboard instrument; An information processing method implemented by a computer system, comprising: In displaying the reference image, a portion of the reference image corresponding to the musical piece that corresponds to the performance point is displayed on the display device, and a playback speed of the reference image is controlled in accordance with the performance speed. Information processing methods.

14. an image acquisition unit that acquires an observation image including the keyboard of the keyboard instrument in real space from an imaging device; a display control unit that displays, on a display device, a reference image that is a moving image including the fingers of a reference performer performing a performance action, so that the fingers of the reference performer overlap the keyboard in the observation image; and a performance analysis unit that identifies performance points within a piece of music and a performance speed at which the user plays the piece of music on the keyboard instrument; A program that causes a computer system to function as The display control unit displays a portion of the reference image corresponding to the musical piece that corresponds to the performance point on the display device, and controls a playback speed of the reference image in accordance with the performance speed. program.

Images