Information processing systems and information processing methods
By overlaying images of a virtual performer and keyboard instruments on a display device using an information processing system, the problem of instructors being unable to provide prolonged assistance to keyboard instrument playing is solved. This enables synchronized performance assistance between the learner and the virtual instructor, thereby improving the practice effect.
Patent Information
- Application Number
- CN202310327288.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2022-04-06
- Filing Date
- 2023-03-30
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2043-03-30
AI Technical Summary
In existing technologies, it is difficult for instructors to stay by the learner's side for extended periods to assist with keyboard instrument playing, especially when multiple people are playing together, as they cannot effectively provide real-time guidance.
The information processing system uses a camera to capture images of keyboard instruments in the real world, and then overlays the animation of a virtual performer onto a display device to achieve synchronized display with the keyboard instruments, thus assisting the learner in playing.
Practitioners can visually reference a virtual performer while playing, enhancing practice effectiveness as if an instructor were present, thus improving coordination and efficiency.
Smart Images

Figure CN116893735B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to techniques for assisting in the playing of keyboard instruments. Background Technology
[0002] Various techniques have been proposed to assist in playing instruments such as keyboard instruments. For example, Patent Document 1 discloses a technique for estimating time-series information related to finger posture by analyzing images of the fingers of a user playing a keyboard instrument.
[0003] Patent Document 1: International Publication No. 2021 / 157691 Summary of the Invention
[0004] For example, when practicing playing a keyboard instrument, it is preferable for the instructor to be near the learner. However, in reality, it is difficult for the instructor to always be near the learner. Furthermore, the above description assumes a learner practicing a keyboard instrument, but the same problem occurs when multiple users play keyboard instruments together. Considering the above, one object of the present invention is to provide a technique that allows one to play a keyboard instrument while visually referencing the performance of other players.
[0005] To address the above issues, one aspect of the present invention relates to an information processing system comprising: an image acquisition unit that acquires an observation image of a real space, including a keyboard of a keyboard instrument, from an imaging device; and a display control unit that displays an animation, i.e., a reference image, including the fingers of a reference performer performing a playing action, on a display device such that the fingers of the reference performer overlap with the keyboard in the observation image.
[0006] One aspect of the present invention relates to an information processing method comprising the steps of: acquiring an observation image of a real space, including a keyboard of a keyboard instrument, from an imaging device; and displaying an animation, i.e. a reference image, including the fingers of a reference performer performing a playing action, on a display device such that the fingers of the reference performer overlap with the keyboard in the observation image.
[0007] One aspect of the present invention relates to a program that enables a computer system to function as follows: an image acquisition unit that acquires an observation image of real space, including a keyboard of a keyboard instrument, from an imaging device; and a display control unit that displays an animation, i.e., a reference image, including the fingers of a reference performer performing a playing action, on a display device such that the fingers of the reference performer overlap with the keyboard in the observation image. Attached Figure Description
[0008] Figure 1This is a structural diagram of the performance system.
[0009] Figure 2 This is a block diagram illustrating the structure of an imaging system.
[0010] Figure 3 This is an explanatory diagram of the operation of an information processing system.
[0011] Figure 4 This is a block diagram illustrating the structure of an information processing system.
[0012] Figure 5 This is a block diagram illustrating the functional structure of an information processing system.
[0013] Figure 6 This is a flowchart of the image generation and processing.
[0014] Figure 7 It is a flowchart of the control process.
[0015] Figure 8 This is a schematic diagram of the observation image in the second embodiment.
[0016] Figure 9 This is a flowchart of the image generation process in the fourth embodiment.
[0017] Figure 10 This is a flowchart of the image generation process in the fifth embodiment.
[0018] Figure 11 This is a block diagram illustrating the structure of the performance system in the sixth embodiment.
[0019] Figure 12 This is a block diagram illustrating the functional structure of the information processing system in the sixth embodiment. Detailed Implementation
[0020] A: Implementation Method 1
[0021] Figure 1 This is a block diagram illustrating the structure of the performance system 100 in the first embodiment. The performance system 100 is a computer system that assists a user U in playing a keyboard instrument 10. The user U uses the performance system 100 to practice playing the keyboard instrument 10. The performance system 100 includes a keyboard instrument 10, an image system 20, and an information processing system 30. The keyboard instrument 10 and the image system 20 are each capable of communicating with the information processing system 30 via wired or wireless means.
[0022] The keyboard instrument 10 is an electronic musical instrument located in the physical space where the user U is situated. It has a keyboard Bx and a housing 11. The keyboard Bx consists of multiple keys Kx (white and black keys) corresponding to different pitches. The user U plays a specific piece of music (hereinafter referred to as the "target piece") by sequentially operating each key Kx of the keyboard instrument 10. The keyboard instrument 10 supplies performance data P, representing the performance, to the information processing system 30 in parallel with the performance performed by the user U. The performance data P represents the notes played by the user U. Each time the user U operates the keyboard Bx (e.g., each time a key is pressed), the performance data P is sequentially output from the keyboard instrument 10. For example, the performance data P specifies the pitch and intensity of the key Kx operated by the user U. The performance data P is, for example, event data according to the MIDI (Musical Instrument Digital Interface) standard.
[0023] The imaging system 20 is an imaging device (HMD) installed on the head of the user U. For example, a protective or eyeglass-type HMD is used as the imaging system 20.
[0024] Figure 2 This is a block diagram illustrating the structure of the imaging system 20. The imaging system 20 includes a capturing device 21 and a display device 22. That is, the capturing device 21 and the display device 22 are mounted on the head of the user U. Therefore, the position and orientation of the capturing device 21 and the display device 22 change in correspondence with the movement of the user U's head.
[0025] The imaging device 21 captures images of a defined area within the real space. Specifically, the imaging device 21 captures images of a defined area located in front of the user U's head (i.e., in the direction of their gaze). The user U, who is playing the keyboard instrument 10, visually confirms the user U's fingers Hx and the keyboard Bx. Therefore, the imaging device 21, as... Figure 3 As illustrated, an animation (hereinafter referred to as "observation image") X, including the left and right fingers Hx of the user U and the keyboard Bx of the keyboard instrument 10, is captured. The range captured by the capturing device 21 varies in correspondence with the movement of the user U's head. The capturing device 21 generates observation image data Dx representing the observation image X in parallel with the playing of the keyboard instrument 10 by the user U. The capturing device 21 includes, for example, an optical system such as a capturing lens; a capturing element that receives incident light from the optical system; and a processing circuit that generates observation image data Dx corresponding to the amount of light received by the capturing element. The capturing device 21 automatically adjusts the exposure in a manner that makes the boundary between the white and black keys of the keyboard Bx and the user U's fingers Hx clear.
[0026] Figure 2 The display device 22 displays various images based on control performed by the information processing system 30. For example, various display panels such as liquid crystal display panels or organic EL (electroluminescence) panels are used as the display device 22. The display device 22 is positioned in front of the user U's eyes. The display device 22 of the first embodiment is a non-transparent type display panel that does not allow light from real space to pass through.
[0027] The display device 22 of the first embodiment Figure 3 The guide image Z is displayed. The guide image Z is an animation that includes the observation image X and the reference image Y. The reference image Y is a stereoscopic animation that includes the fingers Hy of a virtual performer (hereinafter referred to as the "reference performer") performing the playing actions and the keyboard By played by the reference performer. The reference performer is a virtual instructor guiding the user U in playing. In addition, other performers playing together with the user U can also be used as reference performers. The keyboard By, like the keyboard Bx in real space, includes multiple keys Ky arranged in the width direction. The reference image Y is an animation showing the target piece of music being played by the reference performer using the fingers Hy to operate the keyboard By. That is, the reference image Y represents the model fingering associated with the target piece of music.
[0028] In the guide image Z, the reference image Y is displayed in a manner that overlaps with the keyboard Bx of the observation image X. That is, the guide image Z is displayed on the display device 22 in a manner that overlaps with the reference player's fingers Hy and keyboard By in the reference image Y with the keyboard Bx in the observation image X. Specifically, the size and orientation of the fingers Hy and keyboard By in the reference image Y are set to correspond to the size and orientation of the keyboard Bx in the observation image X. For example, the reference image Y is enlarged or reduced in a manner that the size of each key Ky of keyboard By is equal to the size of each key Kx of keyboard Bx in the observation image X. In addition, the angle of the fingers Hy and keyboard By in the reference image Y is controlled in a manner that the surface of keyboard By and the surface of keyboard Bx in the observation image X are parallel to each other. Therefore, the user U can play the keyboard instrument 10 with the feeling that the reference player is exactly present in real space.
[0029] Figure 4 This is a block diagram illustrating the structure of the information processing system 30. The information processing system 30 is a computer system used to control the image system 20. Specifically, the information processing system 30 displays the guide image Z on the display device 22.
[0030] The information processing system 30 includes a control device 31, a storage device 32, an operation device 33, and a sound playback device 34. The information processing system 30 can be implemented, for example, via a mobile information device such as a smartphone or tablet terminal, or a mobile or fixed information device such as a personal computer. Furthermore, the information processing system 30 can be implemented as a single device or as multiple devices configured separately. Additionally, the information processing system 30 can be mounted on a keyboard instrument 10 or an image system 20.
[0031] The control device 31 is a single or multiple processors that control the operation of the information processing system 30. Specifically, the control device 31 may be composed of one or more processors such as CPU (Central Processing Unit), GPU (Graphics Processing Unit), SPU (Sound Processing Unit), DSP (Digital Signal Processor), FPGA (Field Programmable Gate Array), or ASIC (Application Specific Integrated Circuit).
[0032] Storage device 32 is a single or multiple memory devices that store the program executed by control device 31 and various data used by control device 31. For example, known recording media such as semiconductor recording media and magnetic recording media, or a combination of multiple recording media, may be used as storage device 32. Furthermore, for example, removable recording media that can be detached from information processing system 30, or recording media that control device 31 can access via a communication network (e.g., cloud storage) may be used as storage device 32.
[0033] The storage device 32 of the first embodiment stores music data M and reference image data Dy for each of multiple music pieces. The music data M for each music piece is data representing a model or standard performance of that music piece. Specifically, the music data M for each music piece is time sequence data that specifies the time sequence of the notes constituting that music piece. For example, a music file according to the MIDI standard is shown as music data M.
[0034] The reference image data Dy for each piece of music represents the animation data of the reference image Y for that piece. That is, the reference image Y for each piece of music is a 3D animation representing the performance of that piece by a reference performer. The reference image Y for each piece of music also represents the animation of the fingering techniques related to that piece of music.
[0035] The reference image data Dy is any form of 3DCG (3 Dimensional Computer Graphics) data. For example, dot group data representing the fingers Hy and keyboard By of a reference performer can be used as reference image data Dy. Alternatively, skeletal data representing the bones or joints of the fingers Hy and keyboard By can also be used as reference image data Dy. For example, the fingers Hy and keyboard By can be photographed from multiple directions in parallel with a performance of music by a real performer, and reference image data Dy can be generated corresponding to the results of these photographs. Therefore, the multiple keys Ky of the keyboard By in reference image Y are each displaced in conjunction with the performance by the reference performer. That is, the user U, who visually confirms the reference image Y, can confirm the displacement of key Ky by the key presses of the reference performer.
[0036] The operating device 33 is an input device that accepts operations performed by the user U. For example, the user U selects a target song from multiple songs by operating the operating device 33. For example, the operating device 33 may be a control element operated by the user U or a touch panel that detects the user U's touch. Furthermore, the operating device 33, which is separate from the information processing system 30, can be connected to the information processing system 30 via wired or wireless means.
[0037] The sound playback device 34 plays sound based on the control of the control device 31. The sound playback device 34 is, for example, a speaker or headphones. For example, an audio signal A, representing the waveform of a musical note corresponding to a playing action performed by the user U, is supplied to the sound playback device 34. Furthermore, for convenience, the D / A converter that converts the audio signal A from digital to analog and the amplifier that amplifies the audio signal A are omitted from the illustration. Additionally, the sound playback device 34, which is separate from the information processing system 30, can be connected to the information processing system 30 in a wired or wireless manner.
[0038] Figure 5 This is a block diagram illustrating the functional structure of the information processing system 30. The control device 31 performs multiple functions (image acquisition unit 41, display control unit 42, reference determination unit 43, sound source processing unit 44, and performance analysis unit 45) for displaying the guide image Z on the display device 22 by executing the program stored in the storage device 32.
[0039] The image acquisition unit 41 acquires the observation image X from the imaging device 21. Specifically, the image acquisition unit 41 receives the observation image data Dx generated by the imaging device 21 from the imaging system 20.
[0040] The display control unit 42 displays the guide image Z on the display device 22. Specifically, the display control unit 42 displays the guide image Z on the display device 22 by outputting guide image data Dz representing the guide image Z to the display device 22. The display control unit 42 generates the guide image Z by combining the observation image X acquired by the image acquisition unit 41 and the reference image Y of the target music stored in the storage device 32.
[0041] As described above, the display control unit 42 composites the reference image Y onto the observation image X in such a way that the reference performer's fingers Hy and keyboard By overlap with the keyboard Bx of the observation image X, thereby generating a guide image Z. The size and orientation of the fingers Hy and keyboard By in the reference image Y are set corresponding to the size and orientation of the keyboard Bx in the observation image X. Specifically, the display control unit 42 enlarges or reduces the reference image Y such that each key Ky of keyboard By and each key Kx of keyboard Bx in the observation image X are of equal size. In addition, the display control unit 42 controls the angle of the fingers Hy and keyboard By in the reference image Y such that the surface of keyboard By and the surface of keyboard Bx in the observation image X are parallel to each other.
[0042] The reference determination unit 43 determines the position (hereinafter referred to as the "reference position") R in the observation image X that serves as the reference for the reference image Y. A guide image Z is generated by placing the reference image Y in the observation image X at the position corresponding to the reference position R. That is, the reference image Y is composited onto the observation image X in a predetermined positional relationship relative to the reference position R.
[0043] The reference determination unit 43 determines the reference position R by analyzing the observed image X. Specifically, as... Figure 3 As illustrated, the reference determination unit 43 detects the end 12 of the keyboard Bx in the observed image X and determines the position of the end 12 as a reference position R. It detects one or both of the left and right end 12 of the keyboard Bx. For the detection of the end 12 of the keyboard Bx, any known object detection technique is employed. For example, object detection processing using a model trained with a deep neural network is used for the detection of the end 12 (i.e., the determination of the reference position R) performed by the reference determination unit 43. Furthermore, the end 12 of the keyboard instrument 10 and the portions of the frame 11 of the keyboard instrument 10 located on either side of the keyboard Bx have significantly different optical characteristics, such as color or material. Therefore, the end 12 of the keyboard Bx can be detected accurately and easily from the observed image X.
[0044] Figure 5The sound source processing unit 44 generates an audio signal A corresponding to the user U's operation on the keyboard Bx. As described above, performance data P, specifying the pitch corresponding to the key Kx operated by the user U, is supplied from the keyboard instrument 10 to the information processing system 30. The sound source processing unit 44 is a MIDI sound source that generates the audio signal A corresponding to the pitch specified by the performance data P. That is, the audio signal A is a signal representing a musical tone waveform corresponding to the pitch of one or more keys Kx operated by the user U. Furthermore, in the first embodiment, a software sound source implemented by the control device 31 is exemplified as the sound source processing unit 44, but a hardware sound source dedicated to generating the audio signal A may also be used as the sound source processing unit 44.
[0045] The performance analysis unit 45 analyzes the performance of the keyboard instrument 10 by the user U. In this first embodiment, the performance analysis unit 45 analyzes the performance by the user U using performance data P supplied from the keyboard instrument 10. The performance analysis unit 45 determines the performance point Q1, performance speed Q2, and operation position Q3. The analysis is performed by the performance analysis unit 45 repeatedly at a predetermined cycle.
[0046] Performance point Q1 is a point in time on the timeline of the user U's performance within the target piece of music. Known techniques are arbitrarily employed to determine performance point Q1. For example, the performance analysis unit 45 determines performance point Q1 by comparing the time sequence of performance data P supplied from the keyboard instrument 10 with the music data M of the target piece. Furthermore, performance speed Q2 is the speed (rhythm) at which the user U performs the target piece of music. The performance analysis unit 45 determines performance speed Q2 based on the temporal changes of performance point Q1. For example, the amount of movement of performance point Q1 per unit time is calculated as performance speed Q2.
[0047] Operation position Q3 is the position of user U in the space of the keyboard Bx of the keyboard instrument 10 where he / she is playing. As mentioned above, the performance data P specifies the pitch played by user U. The performance analysis unit 45 determines the pitch specified by the performance data P as operation position Q3.
[0048] The display control unit 42 generates a guide image Z using the reference position R determined by the reference determination unit 43 and the analysis results (Q1 to Q3) obtained by the performance analysis unit 45. Figure 6 This is a flowchart of the process (hereinafter referred to as "image generation process") S5 in which the display control unit 42 generates the guide image Z.
[0049] If image generation processing S5 begins, the display control unit 42 controls the playback of the reference image Y in accordance with the performance point Q1 and performance speed Q2 (S51). The display control unit 42 sequentially retrieves the portion of the reference image Y of the target music corresponding to the performance point Q1 from the storage device 32. For example, if the performance point Q1 is the time point when a specific note of the target music is played, the display control unit 42 retrieves the image of the time point in the reference image Y when the reference performer played that note. In addition, the display control unit 42 controls the playback speed of the reference image Y in accordance with the performance speed Q2. For example, the higher the performance speed Q2, the higher the playback speed of the reference image Y. That is, the display control unit 42 of the first embodiment promotes the playback of the reference image Y in conjunction with the performance of the target music by the user U. Specifically, the display control unit 42 controls the playback of the reference image Y in a way that synchronizes the performance of the target music by the user U and the playback of the reference image Y.
[0050] The display control unit 42 sets the position of the reference image Y relative to the observed image X in accordance with the operation position Q3 and the reference position R (S52). Specifically, the display control unit 42 uses the reference position R (i.e., the end 12 of the keyboard Bx) as a reference to determine the operation position Q3 within the keyboard Bx, and determines the position of the reference image Y as a position that is in a predetermined relationship with the operation position Q3. For example, the display control unit 42 positions the reference image Y within the keyboard Bx within a range that includes the operation position Q3, on the high-pitched side or the low-pitched side. The predetermined range that includes the operation position Q3 is, for example, a range that covers one octave between the high-pitched side and the low-pitched side with the operation position Q3 as the center. Therefore, for example, the reference image Y is positioned at a position one octave above the high-pitched side or the low-pitched side relative to the pitch (operation position Q3) played by the user U. As understood from the above description, the position of the reference image Y is set in a way that does not overlap with the user U's own finger Hx in the observed image X. Therefore, the user U can easily check the reference image Y while playing the keyboard instrument 10.
[0051] The display control unit 42 generates a guide image Z by superimposing the reference image Y onto the observation image X (S53). Specifically, the portion of the reference image Y corresponding to the playing point Q1 and the playing speed Q2 is superimposed onto the position in the observation image X that corresponds to the operation position Q3 and the reference position R. The display control unit 42 controls the size and angle of the reference image Y by overlapping it with the keyboard Bx in the observation image X.
[0052] The position or angle of the keyboard Bx in the observation image X changes in response to the movement of the user U's head. Referring to the image Y, which is arranged to overlap with the keyboard Bx in the observation image X, the position and angle of the reference performer's fingers Hy and the keyboard By displayed on the display device 22 also change in response to the movement of the user U's head. As understood from the above description, by appropriately changing the position and direction of the user U, the user U can confirm the reference performer's fingers Hy and the keyboard By from a desired position and angle. Furthermore, as mentioned above, the keys Ky of the keyboard By in the image Y are displaced in conjunction with the playing action performed by the reference performer. Therefore, in the guide image Z displayed on the display device 22, the keys Ky of the keyboard By are also displaced in response to the playing action of the fingers Hy. Based on this structure, the user U can visually reference the reference performer's fingers Hy with the feeling that the reference performer is actually operating the keyboard Bx of the keyboard instrument 10.
[0053] Figure 7 This is a flowchart of the process (hereinafter referred to as "control process") performed by the control device 31. The control process illustrated below is repeated at a predetermined cycle.
[0054] If control processing begins, the control device 31 (image acquisition unit 41) acquires the observation image X (S1). The reference determination unit 43 determines the reference position R by analyzing the observation image X (S2). The control device 31 (sound source processing unit 44) generates an audio signal A corresponding to the performance data P supplied from the keyboard instrument 10 (S3). The control device 31 (performance analysis unit 45) determines the operation position Q3, performance point Q1, and performance speed Q2 by analyzing the performance of the keyboard instrument 10 by the user U (S4). The order of each process (S1 to S4) shown above can be arbitrarily changed.
[0055] Control device 31 (display control unit 42) executes Figure 6 The illustrated image generation process S5 generates a guide image Z. That is, as described above, the guide image Z is generated by combining the observed image X and the reference image Y. The control device 31 (display control unit 42) outputs the guide image data Dz representing the guide image Z to the display device 22, and displays the guide image Z on the display device 22 (S6).
[0056] As explained above, according to the first embodiment, the fingers Hy of the reference performer in the reference image Y are displayed in a manner that overlaps with the keyboard Bx in the observed image X. Therefore, the user U can enjoy an effective customer experience whereby they can play the keyboard instrument 10 while visually referencing the fingers Hy of the reference performer, which are displayed overlapping with the keyboard Bx of the keyboard instrument 10. In the first embodiment, in particular, the positions and angles of the imaging device 21 and the display device 22 change in relation to the position and angle of the user U's head. Therefore, the position and angle of the keyboard By displayed on the reference image Y on the display device 22 change in conjunction with the user U's head movements (e.g., movement or rotation). Thus, the user can enjoy a customer experience where they feel as if the reference performer is actually present in real space while playing the keyboard instrument 10.
[0057] Furthermore, in the first embodiment, a reference image Y is displayed at a position in the observation image X corresponding to the reference position R of the keyboard instrument 10. Therefore, it is possible to position the reference player's finger Hy at an appropriate position relative to the keyboard instrument 10 in real space.
[0058] In the first embodiment, the playback of the reference image Y is controlled in accordance with the performance point Q1 and the performance speed Q2. That is, the playback of the reference image Y is advanced in conjunction with the performance of the music by the user U. Therefore, the user U can perform the target music with the feeling of playing the keyboard instrument 10 in coordination with a reference performer.
[0059] B: Implementation Method 2
[0060] The second embodiment will be described. Furthermore, in the embodiments illustrated below, elements that function the same as in the first embodiment will be appropriately omitted with the same reference numerals as in the first embodiment. In the first embodiment, the end 12 of the keyboard Bx in the observed image X is determined as the reference position R. The methods for determining the reference position R in the second and third embodiments differ from those in the first embodiment. The structure and operation other than the determination of the reference position R are the same as in the first embodiment.
[0061] Figure 8 This is a schematic diagram of the observed image X in the second embodiment. A label 13 is attached to the keyboard instrument 10 in the second embodiment. Figure 8In the example shown, a star-shaped label 13 is formed on the exterior of the keyboard instrument 10. Label 13 is formed, for example, in the center of the width direction of the frame 11 of the keyboard instrument 10. Label 13 is a graphic or symbol used to visually identify the trade name or provider (manufacturer or seller) of the keyboard instrument 10. Label 13 is, in some cases, a trademark. Label 13 is formed, for example, by painting or printing on the frame 11. Furthermore, it is envisioned that a plate-like component with the label 13 formed on its surface be fixed to the frame 11, or that a plate-like component shaped like the label 13 be fixed to the frame 11.
[0062] The reference determination unit 43 determines the reference position R by analyzing the observed image X (S2). In the second embodiment, the reference determination unit 43 detects the sign 13 of the keyboard instrument 10 in the observed image X and determines the position of the sign 13 as the reference position R. For the detection of the sign 13, a known object detection process is arbitrarily employed. For example, object detection processing using a model trained with a deep neural network is used for the detection of the sign 13 performed by the reference determination unit 43 (i.e., the determination of the reference position R). The process of setting the position of the reference image Y relative to the observed image X using the reference position R as a reference (S52) is the same as in the first embodiment.
[0063] In the second embodiment, the same effect as in the first embodiment is achieved. Furthermore, the label 13 of the keyboard instrument 10 is mostly formed at a predetermined position on the keyboard instrument 10. According to the second embodiment, which determines the position of the label 13 of the keyboard instrument 10 as the reference position R, the reference position R can be appropriately determined, for example, even when the end 12 of the keyboard Bx cannot be detected. For example, the reference position R can also be determined when the end 12 of the keyboard Bx is outside the shooting range of the shooting device 21 (when the observed image X does not include the end 12).
[0064] C: Third Implementation Method
[0065] The reference determination unit 43 of the third embodiment determines the position of a predetermined key (hereinafter referred to as "determined key") Kx in the keyboard Bx of the keyboard instrument 10 as the reference position R.
[0066] Before the display of the guide image Z, the user U operates the confirm key Kx on the keyboard Bx. The reference determination unit 43 determines the reference position R by analyzing the observed image X (S2). The reference determination unit 43 of the third embodiment detects the key Kx (i.e., the confirm key Kx) operated by the user U in the observed image X by analyzing the observed image X, and determines the position of the key Kx as the reference position R. For example, among the multiple keys Kx in the observed image X, the key Kx whose grayscale changed due to the operation performed by the user U is detected as the confirm key Kx. The process of setting the position of the reference image Y relative to the observed image X based on the reference position R (S52) is the same as in the first embodiment.
[0067] In the third embodiment, the same effect as in the first embodiment is achieved. Furthermore, in the third embodiment, the position of the selection key Kx operated by the user U is determined as the reference position R. Therefore, even when the end 12 or the sign 13 of the keyboard instrument 10 cannot be detected, the reference position R can still be appropriately determined. For example, the reference position R can be determined even when the end 12 of the keyboard instrument 10 and the sign 13 are outside the shooting range of the shooting device 21.
[0068] D: Fourth Implementation Method
[0069] In the fourth embodiment, the operation of the performance analysis unit 45 and the display control unit 42 differs from that in the first embodiment. In addition to generating the same information (Q1 to Q3) as in the first embodiment, the performance analysis unit 45 in the fourth embodiment also determines whether the user U is playing the keyboard instrument 10. Specifically, the performance analysis unit 45 determines that the keyboard instrument 10 is being played if performance data P is supplied from the keyboard instrument 10, and determines that the keyboard instrument 10 is not being played if performance data P is not supplied from the keyboard instrument 10. Furthermore, the determination method performed by the performance analysis unit 45 is not limited to the examples described above. For example, the performance analysis unit 45 can determine whether or not the keyboard instrument 10 is being played by observing the analysis of the image X.
[0070] Similar to the first embodiment, the display control unit 42 generates a guide image Z by combining the observed image X and the reference image Y. In the generation of the guide image Z, the display control unit 42 of the fourth embodiment controls the value of the transmission parameter α related to the reference image Y. The transmission parameter α is an index (α value) related to the degree to which elements behind the reference image Y are visible through the reference image Y. In the fourth embodiment, the transmission parameter α is transparency. That is, the larger the value of the transmission parameter α, the higher the degree to which elements behind the reference image Y are visible through the reference image Y. For example, when the transmission parameter α is set to 0%, the reference image Y is displayed as an opaque image. On the other hand, when the transmission parameter α is set to 50%, the reference image Y is displayed as a semi-transparent image. When the reference image Y is displayed in a semi-transparent manner, the user U can visually confirm the portion of the observed image X located behind the reference image Y.
[0071] The display control unit 42 controls the value of parameter α in accordance with whether or not the keyboard instrument 10 is being played. Figure 9 This is a flowchart of the image generation process S5 in the fourth embodiment. The process of controlling the playback of the reference image Y (S51) and the process of controlling the position of the reference image Y (S52) are the same as in the first embodiment.
[0072] The control device 31 (performance analysis unit 45) determines whether the keyboard instrument 10 is being played (S54). If the keyboard instrument 10 is not being played (S54: NO), the control device 31 (display control unit 42) sets the transmission parameter α to the value α1 (S55). On the other hand, if the keyboard instrument 10 is being played (S54: YES), the control device 31 (display control unit 42) sets the transmission parameter α to the value α2 (S56). Based on the transmission parameter α set by the above processing, the control device 31 (display control unit 42) composites the reference image Y onto the observation image X, thereby generating a guide image Z (S53). The operations other than the image generation process S5 are the same as in the first embodiment.
[0073] The value α2 of the transmission parameter α refers to a higher degree of transmission compared to the value α1. As mentioned above, the transmission parameter α in the fourth embodiment is transparency, therefore the value α2 is set to a value greater than the value α1. For example, the value α2 is set to 50% (semi-transparent), and the value α1 is set to 0% (opaque). As described above, when the keyboard instrument 10 is not being played, the image Y is displayed in an opaque manner (α = α1), and when the keyboard instrument 10 is being played, the image Y is displayed in a semi-transparent manner (α = α2). That is, when the user U is playing the keyboard instrument 10, the transmission degree of the image Y is higher than when the keyboard instrument 10 is not being played. Furthermore, the value α1 is an example of the "first value," and the value α2 is an example of the "second value."
[0074] In the fourth embodiment, the same effect as in the first embodiment is achieved. In particular, the value of the transmission parameter α of the reference image Y is controlled. Therefore, compared with the structure where the value of the transmission parameter α is fixed, it is possible to switch between the mode of using the reference image Y for priority visual confirmation and the mode of using the user U's own finger Hx for priority visual confirmation.
[0075] For example, in the fourth embodiment, when the user U is playing the keyboard instrument 10, the transmission parameter α is set to a value α2, which has a higher degree of transparency compared to the value α1. Therefore, compared to the structure where the transmission parameter α is fixed at the value α1 (referring to the structure where image Y is always displayed in an opaque manner), the user U can visually confirm their own finger Hx first. On the other hand, when the user U is not playing the keyboard instrument 10, the transmission parameter α is set to a value α1, which has a lower degree of transparency compared to the value α2. Therefore, compared to the structure where the transmission parameter α is fixed at the value α2 (referring to the structure where image Y is always displayed in a semi-transparent manner), the user U can easily visually confirm the image Y. Furthermore, the second or third embodiment can also be applied to the fourth embodiment in the same way.
[0076] E: Fifth Implementation
[0077] In the fifth embodiment, similarly to the fourth embodiment, the value of the transmission parameter α applied to the reference image Y is variably controlled. In the fourth embodiment, the transmission parameter α was controlled in relation to the presence or absence of playing of the keyboard instrument 10, but in the fifth embodiment, the transmission parameter α is controlled in relation to the action mode. The transmission parameter α in the fifth embodiment, like in the fourth embodiment, is transparency. That is, the larger the value of the transmission parameter α, the greater the degree to which elements behind the reference image Y are visible through the reference image Y.
[0078] The display control unit 42 sets either the first operation mode or the second operation mode as the operation mode. The first operation mode is a reference mode that prioritizes visual confirmation with reference image Y, and the second operation mode is a practice mode that prioritizes visual confirmation with the user U's finger Hx. For example, the display control unit 42 sets the operation mode in correspondence with the instructions from the user U to the operating device 33. For example, each time the user U instructs a switch of operation mode, the display control unit 42 changes the operation mode from one of the first operation mode and the second operation mode to the other.
[0079] Figure 10 This is a flowchart of the image generation process S5 in the fifth embodiment. The process of controlling the playback of the reference image Y (S51) and the process of controlling the position of the reference image Y (S52) are the same as in the first embodiment.
[0080] The control device 31 (display control unit 42) determines whether the operation mode is the first operation mode (S57). If the operation mode is the first operation mode (S57: YES), the control device 31 (display control unit 42) sets the transmission parameter α to the value α1 (S58). On the other hand, if the operation mode is the second operation mode (S57: NO), the control device 31 (display control unit 42) sets the transmission parameter α to the value α2 (S59). Based on the transmission parameter α set by the above processing, the control device 31 (display control unit 42) composites the reference image Y onto the observation image X, thereby generating a guide image Z (S53). The operations other than the image generation process S5 are the same as in the first embodiment.
[0081] Similar to the fourth embodiment, the value α2 of the transmittance parameter α refers to a higher degree of transmittance compared to the value α1. Specifically, the value α2 is set to a value greater than the value α1. For example, the value α2 is set to 50% (semi-transparent), and the value α1 is set to 0% (opaque). As described above, in the first operating mode, the image Y is displayed in an opaque manner (α = α1), and in the second operating mode, the image Y is displayed in a semi-transparent manner (α = α2). That is, in the first operating mode, the transmittance of the image Y is higher compared to the second operating mode. Furthermore, the value α1 is an example of a "first value," and the value α2 is an example of a "second value."
[0082] In the fifth embodiment, the same effect as in the first embodiment is achieved. In particular, the value of the transmission parameter α of the reference image Y is controlled. Therefore, similar to the fourth embodiment, it is possible to switch between the mode of using the reference image Y for priority visual confirmation and the mode of using the user U's own finger Hx for priority visual confirmation.
[0083] For example, in the fifth embodiment, in the second operating mode, the transmission parameter α is set to a value α2, which has a higher degree of transparency compared to the value α1. Therefore, compared to the structure where the transmission parameter α is fixed at the value α1 (referring to the structure where image Y is always displayed in an opaque manner), the user U can visually confirm their own finger Hx first. On the other hand, in the first operating mode, the transmission parameter α is set to a value α1, which has a lower degree of transparency compared to the value α2. Therefore, compared to the structure where the transmission parameter α is fixed at the value α2 (referring to the structure where image Y is always displayed in a semi-transparent manner), the user U can easily visually confirm the image Y. Furthermore, the second or third embodiment can also be applied to the fifth embodiment in the same way.
[0084] F: Implementation Method 6
[0085] Figure 11 This is a block diagram illustrating the structure of the performance system 100 in the sixth embodiment. In addition to the same elements as in the first embodiment (control device 31, storage device 32, operation device 33, and sound playback device 34), the information processing system 30 in the performance system 100 of the sixth embodiment also includes a communication device 35. The communication device 35 communicates with external devices, for example, via a communication network 200 such as the Internet. Furthermore, communication between the communication device 35 and the communication network 200 can be either wired or wireless. Additionally, the communication device 35, which is separate from the information processing system 30, is connected to the information processing system 30 via either a wired or wireless connection.
[0086] Figure 11 The recording system 60 is a computer system that records reference images Y for use in order for an instructor T to guide a user U in playing the keyboard instrument 10. The instructor T is equivalent to a reference performer. That is, the reference image Y in the sixth embodiment is an animation showing the instructor T playing the keyboard instrument 63. The keyboard instrument 63 is, for example, a natural instrument or an electronic instrument with a keyboard. The recording system 60 includes a recording device 61 and a communication device 62.
[0087] The camera device 61 captures images of the instructor T playing the keyboard instrument 63. Specifically, the camera device 61 captures an image Y showing the instructor T's left and right fingers (Hy) and the keyboard (By) of the instrument 63. The communication device 62 is, for example, a mobile information device such as a smartphone or tablet, or a mobile or fixed information device such as a personal computer. The camera device 61 can be mounted on the communication device 62.
[0088] The communication device 62 transmits reference image data Dy, representing reference image Y, to the information processing system 30. Reference image data Dy is animation data representing the playing actions performed by the instructor T (referring to the performer). For example, similar to the first embodiment, dot group data or skeleton data representing the instructor T's fingers Hy and keyboard By are exemplified as reference image data Dy. Reference image data Dy is transmitted from the recording system 60 to the performance system 100 in parallel with the playing of the keyboard instrument 63 by the instructor T. Reference image data Dy is received by the communication device 35 of the information processing system 30 via the communication network 200.
[0089] Figure 12 This is a block diagram illustrating the functional structure of the information processing system 30 in the sixth embodiment. In addition to the same elements as in the first embodiment (image acquisition unit 41, display control unit 42, reference determination unit 43, sound source processing unit 44, and performance analysis unit 45), the control device 31 in the sixth embodiment also functions as a receiving control unit 46. The receiving control unit 46 receives reference image data Dy sent from the recording system 60 via the communication device 35.
[0090] The display control unit 42 of the first embodiment combines the reference image Y of the reference image data Dy stored in the storage device 32 into the observation image X, thereby generating a guide image Z. The display control unit 42 of the sixth embodiment combines the reference image Y of the reference image data Dy obtained by the receiving control unit 46 into the observation image X, thereby generating a guide image Z (S5). That is, the display control unit 42 displays the reference image Y represented by the reference image data Dy on the display device 22. As described above, the guide image Z of the sixth embodiment is composed of the observation image X representing the user U playing the keyboard instrument 10 and the reference image Y representing the instructor T playing the keyboard instrument 63. In parallel with the playing of the keyboard instrument 63 by the instructor T, the guide image Z including the reference image Y of the playing is displayed on the display device 22 in real time.
[0091] In the sixth embodiment, the same effect as in the first embodiment is achieved. Furthermore, in the sixth embodiment, the reference image Y of the performance action performed by the instructor T (referencing the performer) can be displayed on the display device 22 used by the user U, in parallel with the recording of the performance action. Therefore, the user U can play the keyboard instrument 10 with the feeling that the instructor T, who is actually at a distance, is nearby.
[0092] G: Variation
[0093] The following examples illustrate specific variations of the methods illustrated above. Multiple methods arbitrarily selected from the foregoing embodiments and the variations illustrated below can be appropriately combined without contradiction.
[0094] (1) In the aforementioned embodiments, a non-transparent display panel was exemplified as the display device 22, but a transmissive display panel through which light from the real space passes can also be used as the display device 22. In the embodiment using the transmissive display device 22, it is not necessary to display the observation image X on the display device 22. That is, the display control unit 42 displays the reference image Y as a guide image Z on the display device 22. The user U can visually confirm the keyboard Bx of the keyboard instrument 10 through the light transmitted from the real space, and visually confirm the reference image Y that overlaps with the keyboard Bx. In the embodiment using the transmissive display device 22, the observation image X is also used to determine the reference position R of the reference image Y. As understood from the above description, the display of the observation image X can be omitted. The display device 22 displays the reference image Y through augmented reality (AR) or mixed reality (MR).
[0095] (2) In the fourth and fifth embodiments, the case where the transparency parameter α is set to transparency is illustrated, but it is also conceivable that the transparency parameter α is set to opacity. In the case where the transparency parameter α is opaque, the larger the value of the transparency parameter α, the less the elements behind the reference image Y are visible through the reference image Y. In the above method, similar to the fourth embodiment, the transparency parameter α is set to the value α1 when the keyboard instrument 10 is not being played, and the transparency parameter α is set to the value α2 when the keyboard instrument 10 is being played. In addition, similar to the fifth embodiment, the transparency parameter α is set to the value α1 in the first operating mode and to the value α2 in the second operating mode. However, when the transparency parameter α is opaque, the value α2 is set to a value smaller than the value α1. For example, the value α2 is set to 50% (semi-transparent), and the value α1 is set to 100% (opaque). As understood from the examples above, regardless of whether the transmission parameter α represents transparency or opacity, the value α2 generally represents a higher degree of transmission compared to the value α1. The relationship between the magnitude of the transmission parameter α and the level of transmission of the reference image Y is arbitrary.
[0096] (3) In the aforementioned methods, an electronic musical instrument capable of outputting performance data P was exemplified as a keyboard instrument 10, but a natural musical instrument that does not output performance data P can also be used as a keyboard instrument 10. The musical sounds emitted from the keyboard instrument 10 of the natural musical instrument are picked up by a microphone, thereby generating an audio signal A. The performance analysis unit 45 analyzes the performance of the keyboard instrument 10 by the user U using the audio signal A. For example, the performance analysis unit 45 generates a time sequence of performance data P by analyzing the audio signal A, compares the time sequence of performance data P with the music data M of the target piece, and thereby determines the performance point Q1. The performance speed Q2 and the operation position Q3 are also determined by analyzing the audio signal A.
[0097] (4) Among the aforementioned methods, the performance analysis unit 45 analyzes the performance performed by the user U using performance data P. However, the performance analysis unit 45 can also analyze the performance performed by the user U using the observation image X. For example, the performance analysis unit 45 determines the operation position Q3 by analyzing the observation image X. For example, the performance analysis unit 45 detects the user U's finger Hx from the observation image X and determines the operation position Q3 corresponding to the positional relationship of the finger Hx relative to the reference position R.
[0098] (5) Among the foregoing embodiments, the example shown is that the imaging device 21 and the display device 22 are mounted on the user U's head. However, the imaging device 21 and the display device 22 are not necessarily mounted on the user U's head. For example, it is also conceivable that one or both of the imaging device 21 and the display device 22 are placed near the keyboard instrument 10. However, in the case where the imaging device 21 and the display device 22 are mounted on the user U's head, the position and angle of the imaging device 21 and the display device 22 change in accordance with the position and angle of the user U's head. That is, the observed image X and the reference image Y displayed on the display device 22 change in conjunction with the movement of the user U's head. Therefore, the user U can play the keyboard instrument 10 with the feeling of being in real space, just like a reference performer.
[0099] (6) In the aforementioned methods, an example is shown where the finger Hy of the reference performer (instructor T) is displayed on the display device 2, but the upper body or the whole body of the reference performer can also be displayed on the display device 22 as a reference image Y. In addition, in the aforementioned methods, an example is shown where the reference image Y includes the finger Hy of the reference performer and the keyboard By, but the keyboard By can also be omitted from the reference image Y.
[0100] (7) In the aforementioned methods, a scenario is envisioned where the user U and the reference performer perform a common target piece of music, but the content of the user U's performance and the content of the reference performer's performance can be different. For example, the user U can play the first part of the multiple performance parts of the target piece of music, and the reference performer can play the second part. The first part and the second part are each one or more performance parts of the target piece of music.
[0101] (8) For example, the information processing system 30 can be implemented by a server device that enables communication between terminal devices such as smartphones or tablets. The imaging device 21 is mounted on or connected to the terminal device. The terminal device sends the observation image data Dx generated by the imaging device 21 and the performance data P generated by the keyboard instrument 10 to the information processing system 30. Similarly to the aforementioned methods, the information processing system 30 generates guide image data Dz based on the observation image data Dx and the performance data P, and sends the guide image data Dz to the terminal device. The terminal device displays the guide image Z represented by the guide image data Dz on the display device 22. The display device 22 can, for example, be mounted on the terminal device.
[0102] (9) The functions of the information processing system 30 involved in the aforementioned methods are realized, as described above, through the coordinated operation of one or more processors constituting the control device 31 and the program stored in the storage device 32. The program exemplified above can be provided and installed in a computer in the form of a computer-readable recording medium. The recording medium is, for example, a non-transitory recording medium, preferably an optical recording medium (optical disc) such as a CD-ROM, and also includes any known form of recording medium such as semiconductor recording media or magnetic recording media. Furthermore, as a non-transitory recording medium, it includes any recording medium other than a transient propagating signal, and may also include volatile recording media. In addition, in the structure where the transmission device transmits the program via a communication network, the storage device storing the program in the transmission device is equivalent to the aforementioned non-transitory recording medium.
[0103] H: Appendix
[0104] Based on the examples above, for instance, understand the following structure.
[0105] One aspect (Aspect 1) of the present invention relates to an information processing system comprising: an image acquisition unit that acquires an observation image of a real space, including the keyboard of a keyboard instrument, from an imaging device; and a display control unit that displays an animation, i.e., a reference image, including the fingers of a reference performer performing a playing action, on a display device such that the fingers of the reference performer overlap with the keyboard in the observation image. According to this structure, the fingers of the reference performer in the reference image are displayed in a manner that overlaps with the keyboard in the observation image. Therefore, the user can play the keyboard instrument while visually referring to the fingers of the reference performer displayed overlapping with the keyboard of the keyboard instrument.
[0106] In a specific example of Method 1 (Method 2), the shooting device and the display device are mounted on the user's head. In the above methods, the position and angle of the shooting device and the display device change in accordance with the position and angle of the user's head. That is, the position and angle of the keyboard in the reference image displayed on the display device change in conjunction with the movement of the user's head. Therefore, the user can play the keyboard instrument with the feeling of being in real space, just like a reference performer.
[0107] In a specific example of Method 1 or Method 2 (Method 3), the reference image includes a reference keyboard played by the reference performer. The reference keyboard includes multiple keys that are displaced corresponding to the playing actions performed by the reference performer. The display control unit displays the reference image on the display device in a manner that overlaps the reference keyboard with the keyboard in the observed image. According to the above method, the reference keyboard, with each key displaced corresponding to the playing actions performed by the reference performer, is displayed superimposed on a keyboard in real space. Therefore, the user can visually reference the fingers of the reference performer as if the reference performer were actually operating the keyboard.
[0108] In a specific example (method 4) of any of methods 1 to 3, the information processing system further includes a reference determination unit that determines a reference position for the keyboard instrument in the observed image through analysis of the observed image, and the display control unit displays the reference image at a position corresponding to the reference position. According to the above method, by displaying the reference image at a position corresponding to the reference position relative to the keyboard instrument, it is possible to position the fingers of a reference player at an appropriate position relative to the keyboard instrument in real space.
[0109] In a specific example of Method 4 (Method 5), the reference determination unit detects the end of the keyboard in the observed image and determines the position of that end as the reference position. The end (left or right end) of a keyboard instrument has significantly different optical characteristics, such as color, from the components (clappers or trusses) located on either side of the keyboard in the keyboard instrument. Therefore, the end of the keyboard can be detected accurately and easily based on the image of the keyboard. That is, by using a structure that determines the position of the end of the keyboard as the reference position, the reference position can be determined accurately and easily.
[0110] In a specific example of Method 4 (Method 6), the reference determination unit detects the label of the keyboard instrument in the observed image and determines the position of the label as the reference position. The label of a keyboard instrument, such as a trademark, is usually displayed at a predetermined position on the keyboard instrument (e.g., the center of the frame in the width direction). Therefore, according to the structure that determines the position of the keyboard instrument's label as the reference position, the reference position can be appropriately determined even when the end of the keyboard cannot be detected.
[0111] In a specific example of Method 4 (Method 7), the reference determination unit detects the key that has been operated among the multiple keys of the keyboard in the observed image and determines the position of that key as the reference position. According to the above method, the position of the key operated by the user is determined as the reference position. Therefore, even when the end of the keyboard instrument or its label cannot be detected, the reference position can be appropriately determined.
[0112] In a specific example (method 8) of any of methods 1 to 7, the display control unit controls the value of the transmission parameter of the reference image. In the above methods, the value of the transmission parameter of the reference image is controlled. Therefore, compared with the structure where the value of the transmission parameter of the reference image is fixed, it is possible to switch between the method of using the performer's fingers as a reference for priority visual confirmation and the method of using the user's own method as a reference for priority visual confirmation.
[0113] In a specific example of Method 8 (Method 9), the information processing system further includes a performance analysis unit that determines whether the keyboard instrument is being played. If the display control unit determines that the keyboard instrument is not being played, it sets the transmission parameter to a first value; if it determines that the keyboard instrument is being played, it sets the transmission parameter to a second value, which has a higher degree of transmission than the first value. In the above method, when the user is playing the keyboard instrument, the transmission parameter is set to a second value, which has a higher degree of transmission than the first value. Therefore, compared to a structure where the transmission parameter is fixed at the first value, the user can visually confirm their own fingers first. On the other hand, when the user is not playing the keyboard instrument, the transmission parameter is set to a first value, which has a lower degree of transmission than the second value. Therefore, compared to a structure where the transmission parameter is fixed at the second value, the user can easily visually confirm the reference image.
[0114] In a specific example of Method 8 (Method 10), the display control unit sets the transmission parameter to a first value in the first operating mode and to a second value in the second operating mode, where the second value has a higher degree of transmission than the first value. In the above method, in the second operating mode, the transmission parameter is set to a second value, which has a higher degree of transmission than the first value. Therefore, compared to a structure where the transmission parameter is fixed at the first value, the user can visually confirm their own finger first. On the other hand, in the first operating mode, the transmission parameter is set to a first value, which has a lower degree of transmission than the second value. Therefore, compared to a structure where the transmission parameter is fixed at the second value, the user can easily visually confirm the reference image.
[0115] In a specific example (method 11) of any of methods 1 to 10, the information processing system further includes a performance analysis unit that determines the operation position on the keyboard of the keyboard instrument, and the display control unit displays the reference image within a range on the high-pitched or low-pitched side of the keyboard relative to the range containing the operation position. According to the above method, the reference image is displayed within a range on the high-pitched or low-pitched side relative to the user's operation position. That is, the reference image is displayed in a state that does not overlap with the user's fingers. Therefore, it is easy for the user to check the reference image while playing the keyboard instrument.
[0116] In a specific example (method 12) of any of methods 1 to 11, the information processing system further includes a performance analysis unit that determines the performance points within the music piece, and the display control unit displays the portion of the reference image corresponding to the performance point on the display device. According to the above method, the playback of the reference image is advanced in conjunction with the user's performance of the music. Therefore, the user performs the music with the feeling of playing a keyboard instrument in coordination with a reference performer.
[0117] In a specific example of method 12 (method 13), the performance analysis unit further determines the performance speed, and the display control unit controls the playback speed of the reference image in accordance with the performance speed. According to the above method, the playback of the reference image is advanced in conjunction with the user's performance of the music. Therefore, the user can perform the music with the feeling of playing a keyboard instrument in coordination with a reference performer.
[0118] In a specific example (method 14) of any of methods 1 to 13, the information processing system further includes a receiving control unit that receives animation data representing the playing actions performed by the reference performer via a communication device, and a display control unit displays the reference image represented by the animation data on the display device. According to the above method, a reference image of the playing actions performed by the reference performer can be displayed on the display device used by the user in parallel with the recording of the playing actions performed by the reference performer. Therefore, the user can play the keyboard instrument with the feeling that the reference performer, who is actually at a distance, is nearby.
[0119] One aspect of the present invention relates to an information processing method comprising: acquiring an observation image of a real space, including a keyboard of a keyboard instrument, from a photographing device; and displaying an animation, i.e., a reference image, including the fingers of a reference performer performing a playing action, on a display device such that the fingers of the reference performer overlap with the keyboard in the observation image.
[0120] One aspect of the present invention relates to a program that enables a computer system to function as follows: an image acquisition unit that acquires an observation image of real space, including a keyboard of a keyboard instrument, from an imaging device; and a display control unit that displays an animation, i.e., a reference image, including the fingers of a reference performer performing a playing action, on a display device such that the fingers of the reference performer overlap with the keyboard in the observation image.
[0121] Explanation of the label
[0122] 100…Playing system, 10…Keyboard instrument, 20…Image system, 21…Shooting device, 22…Display device, 30…Information processing system, 31…Control device, 32…Storage device, 33…Operating device, 34…Sound playback device, 35…Communication device, 41…Image acquisition unit, 42…Display control unit, 43…Reference determination unit, 44…Sound source processing unit, 45…Performance analysis unit, 46…Receiving control unit, 60…Recording system, 61…Shooting device, 62…Communication device, 63…Keyboard instrument, 200…Communication network.
Claims
1. An information processing system, comprising: The image acquisition unit acquires observational images of the real space, including the keyboard of a keyboard musical instrument, from the imaging device; The display control unit displays an animation, i.e., a reference image, including the fingers of a reference performer performing a playing action, on a display device such that the fingers of the reference performer overlap with the keyboard in the observed image; and The performance analysis unit analyzes the keyboard instrument performances made by the user. The performance analysis unit determines the user's operation position on the keyboard, the user's playing points within the music, and the user's playing speed. The display control unit is located on the high-pitched or low-pitched side of the keyboard relative to the range including the operation position, and displays the portion of the reference image corresponding to the playing point in relation to the music. The playback speed of the reference image is controlled in accordance with the playing speed.
2. The information processing system according to claim 1, wherein, The shooting device and the display device are mounted on the user's head.
3. The information processing system according to claim 1, wherein, The reference image includes a reference keyboard played by the reference performer. The reference keyboard includes multiple keys that are displaced corresponding to the playing actions performed by the reference performer. The display control unit displays the reference image on the display device in such a way that the reference keyboard overlaps with the keyboard in the observed image.
4. The information processing system according to claim 1, wherein, It also includes a reference determination unit that determines the reference position of the keyboard instrument in the observed image through analysis of the observed image. The display control unit displays the reference image at a position corresponding to the reference position.
5. The information processing system according to claim 4, wherein, The reference determination unit detects the end of the keyboard in the observed image and determines the position of the end as the reference position.
6. The information processing system according to claim 4, wherein, The reference determination unit detects the label of the keyboard instrument in the observed image and determines the position of the label as the reference position.
7. The information processing system according to claim 4, wherein, The reference determination unit detects the key that is operated among the multiple keys of the keyboard in the observed image and determines the position of the key as the reference position.
8. The information processing system according to claim 1, wherein, The display control unit controls the value of the transmission parameter of the reference image.
9. The information processing system according to claim 8, wherein, It also includes a performance analysis unit that determines whether the keyboard instrument is being played. The display control unit, If it is determined that the keyboard instrument is not being played, the transmission parameter is set to the first value. If it is determined that the keyboard instrument is being played, the transmission parameter is set to a second value, which has a higher degree of transmission compared to the first value.
10. The information processing system according to claim 8, wherein, The display control unit, In the first operation mode, the transmission parameter is set to the first value. In the second operating mode, the transmission parameter is set to a second value, which has a higher degree of transmission compared to the first value.
11. The information processing system according to claim 1, wherein, It also includes a receiving control unit that receives, via a communication device, animation data representing the performance actions performed by the reference performer. The display control unit displays the reference image represented by the animation data on the display device.
12. An information processing method, implemented by a computer system, comprising the following steps: The imaging device captures observational images of the real space, including the keyboard of a keyboard instrument. Animation, i.e., reference image, including the fingers of a reference performer performing a playing action, is displayed on a display device such that the fingers of the reference performer overlap with the keyboard in the observed image; and The keyboard instrument performances by the user are analyzed. When analyzing the performance, the user's operation position on the keyboard, the user's playing points within the music, and the user's playing speed are determined. When displaying the reference image, within the range of the keyboard relative to the area including the operation position, which is on the high-pitched or low-pitched side, the portion of the reference image corresponding to the playing point corresponding to the music is displayed on the display device. The playback speed of the reference image is controlled in accordance with the playing speed.
13. The information processing method according to claim 12, wherein, The observed image was acquired from the image acquisition unit.
14. The information processing method according to claim 13, wherein, The image acquisition unit is installed on the head of the user of the information processing system.
Citation Information
Patent Citations
Information processing device, information processing method, and information processing program
WO2021157691A1
Method for realizing unmarked augmented reality piano teaching system
CN102737534A
Playing apparatus, motion capture method for fingering, and performance support system
JP2017181850A
