Information processing method, information processing system, and program

Abstract

This information processing system is provided with: a performance analysis unit that analyzes the state of performance by a user in a real space; and a display control unit that displays a performance character played in the virtual space on a display device, and controls the action of the performance character in accordance with the state of the performance.

Description

Technical Field

[0001] The present invention relates to a technology for assisting a user in playing a musical instrument. Background Art

[0002] Various technologies for assisting users in playing musical instruments have been proposed. For example, Patent Document 1 discloses a performance agent that automatically plays a piece of music in accordance with the user's performance.

[0003] Patent Document 1: International Publication No. 2021 / 193032 Summary of the Invention

[0004] According to the technology of Patent Document 1, the user's performance is assisted by the automatic performance of a performance agent. However, with the recent development of virtual reality technology and information and communication technology, further development of technologies that intuitively or interactively assist the user's performance is desired. In view of this, one embodiment of the present invention is to intuitively or interactively assist the user's performance.

[0005] In order to solve the above problems, one embodiment of the present invention involves an information processing method that analyzes the performance status of a user in a real space, displays a performance character that performs performance actions in a virtual space on a display device, and controls the guiding actions of the performance character according to the performance status performed by the user.

[0006] An information processing system according to one embodiment of the present invention comprises: a performance analysis unit that analyzes the state of a user's performance in a real space; and a display control unit that displays a performance character performing in a virtual space on a display device and controls the movement of the performance character according to the state of the performance performed by the user.

[0007] One embodiment of the present invention relates to a program that causes a computer system to function as the following functional units: a performance analysis unit that analyzes the state of a user's performance in a real space; and a display control unit that displays a performance character performing in a virtual space on a display device and controls the movements of the performance character according to the state of the performance of the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 This is a block diagram of the performance system according to the first embodiment.

[0009] Figure 2 This is a block diagram illustrating the functional structure of an information processing system.

[0010] Figure 3 is a schematic diagram of the guide image.

[0011] Figure 4 It is a schematic diagram of the playing role.

[0012] Figure 5 This is a block diagram of the display control unit.

[0013] Figure 6 This is an illustration of the guided action.

[0014] Figure 7 This is a flowchart of the motion control process.

[0015] Figure 8 This is a block diagram of a performance system according to the second embodiment. DETAILED DESCRIPTION

[0016] A: First embodiment

[0017] Figure 1 This is a block diagram illustrating the structure of a performance system 100A according to the first embodiment. Performance system 100A according to the first embodiment is a computer system that assists a user U in performing a piece of music (hereinafter referred to as a "target piece of music"). Performance system 100A includes a performance device 10 and an information processing system 20. Performance device 10 is connected to information processing system 20 via a wired or wireless connection. Information processing system 20 may be mounted on performance device 10. Performance device 10 may be interpreted as a component of information processing system 20.

[0018] The performance device 10 is an input device for receiving a performance of a target music piece by the user U. The performance device 10 is, for example, a MIDI (Musical Instrument Digital

[0019] The keyboard 11 is composed of a plurality of keys 12 corresponding to different pitches. The user U plays a target piece of music by sequentially operating the keys 12.

[0020] The target music piece in the first embodiment includes a melody part and an accompaniment part. The melody part is the performance part consisting of the melody sound of the target music piece, and the accompaniment part is the performance part consisting of the accompaniment sound of the target music piece. The user U performs the melody part using the performance device 10. The melody part is an example of a "first part," and the accompaniment part is an example of a "second part."

[0021] The musical performance device 10 outputs performance data P representing the performance performed by the user U to the information processing system 20. The performance data P is, for example, event data compliant with the MIDI standard. Specifically, the performance data P specifies the pitch corresponding to the key 12 operated by the user U. The performance data P is sequentially transmitted from the musical performance device 10 for each operation by the user U.

[0022] The information processing system 20 is a computer system that analyzes and assists the user U in performing a target musical piece. The information processing system 20 is implemented as an information device such as a smartphone, tablet terminal, or personal computer. The information processing system 20 includes a control device 21, a storage device 22, a display device 23, an operating device 24, a sound source device 25, and a sound playback device 26. Furthermore, the information processing system 20 can be implemented as a single device or as multiple devices configured separately from each other.

[0023] The control device 21 is a single processor or multiple processors that control the various elements of the information processing system 20. Specifically, the control device 21 is composed of one or more processors such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an SPU (Sound Processing Unit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or an ASIC (Application Specific Integrated Circuit).

[0024] The storage device 22 is a single memory or multiple memories that store programs executed by the control device 21 and various data used by the control device 21. For example, a well-known recording medium such as a semiconductor recording medium or a magnetic recording medium, or a combination of multiple recording media can be used as the storage device 22. In addition, for example, a removable recording medium that can be attached to and detached from the information processing system 20, or a recording medium that can be accessed by the control device 21 via a communication network (for example, a network hard disk) can also be used as the storage device 22.

[0025] The storage device 22 stores music data M. The music data M specifies the time series of notes that make up the target music (i.e., the musical score). The music data M in the first embodiment consists of melody data M1 that specifies each note of the melody part, and accompaniment data M2 that specifies each note of the accompaniment part. Melody data M1 and accompaniment data M2 are time-series data consisting of a sequence of note data N that specifies the notes and instructs them to be played, and time data that specifies the timing of reading each note data N. The music data M is, for example, a file format that complies with the MIDI standard (SMF: Standard MIDI File).

[0026] The display device 23 displays images under the control of the control device 21. For example, various display panels such as a liquid crystal display panel or an organic EL (electroluminescence) panel are used as the display device 23. The operating device 24 is an instruction input device that receives instructions from the user U. For example, an operating element operated by the user U or a touch panel integrally formed with the display device 23 is used as the operating device 24. In addition, the display device 23 or the operating device 24, which is separate from the information processing system 20, can be connected to the information processing system 20 via a wired or wireless connection.

[0027] The sound source device 25 is sequentially supplied with the performance data P and the note data N of the accompaniment data M2 generated by the musical performance device 10. The sound source device 25 generates an acoustic signal V representing the waveform of the musical sound specified by the performance data P and the note data N. In other words, the sound source device 25 generates an acoustic signal V representing a mixed sound of the musical sound of the melody part played by the user U and the musical sound of the accompaniment part of the target music. The functions of the sound source device 25 can be realized by the control device 21 executing a program.

[0028] The sound emitting device 26 plays the musical sound represented by the acoustic signal V. For example, a speaker or headphones can be used as the sound emitting device 26. The D / A converter that converts the acoustic signal V from digital to analog and the amplifier that amplifies the acoustic signal V are not shown. The sound emitting device 26, which is separate from the information processing system 20, can be connected to the information processing system 20 via a wired or wireless connection.

[0029] Figure 2 This is a block diagram illustrating the functional configuration of the information processing system 20. The control device 21 executes programs stored in the storage device 22 to implement multiple functions (performance control unit 31, display control unit 32, and performance analysis unit 33) for analyzing and assisting the performance of the user U.

[0030] The performance control unit 31 is a sequencer that sequentially outputs the note data N of the accompaniment data M2 in the music data M to the sound source device 25. Specifically, the performance control unit 31 instructs the sound source device 25 to generate the various musical tones of the accompaniment part specified by the accompaniment data M2. In parallel with the supply of the note data N by the performance control unit 31, the sound source device 25 is sequentially supplied with the performance data P generated by the performance device 10. Consequently, the tones of the melody part played by the user U and the tones of the accompaniment part of the target music are simultaneously reproduced by the sound reproduction device 26.

[0031] The performance control section 31 controls the output timing of the note data N to the sound source device 25 to be variable so that the reproduction of the musical tones of the accompaniment part follows the performance of the user U. The timing at which the user U plays each note of the melody part of the target music piece changes dynamically due to the intended musical expression of the user U or the shortcomings of the performance technique of the user U. Therefore, the timing at which the performance control section 31 outputs each note data N of the accompaniment data M2 to the sound source device 25 is also variable.

[0032] Specifically, the performance control unit 31 estimates the time point (hereinafter referred to as the "performance time point") that the user U is currently playing in the target music by analyzing the time series of the performance data P. For example, the performance control unit 31 estimates the performance time point by comparing the melody data M1 of the music data M and the time series of the performance data P. The estimation of the performance time point is performed sequentially in parallel with the performance of the melody part by the user U. For the estimation of the performance time point, for example, the well-known sound analysis technology (score alignment) described in Japanese Patent Application Laid-Open No. 2015-79183 can be arbitrarily adopted. The performance control unit 31 outputs each note data N to the sound source device 25 in a manner that synchronizes the generation of the accompaniment part performed by the sound source device 25 with the advancement of the performance time point. Therefore, the playback of the musical sound of the melody part played by the user U and the playback of the musical sound of the accompaniment part of the target music are synchronized with each other.

[0033] Figure 2 The display control unit 32 displays an image (hereinafter referred to as “guide image G”) for assisting the user U in playing the target music piece on the display device 23 . Figure 3 is a schematic diagram of the guide image G. The guide image G is an animation video including the virtual musical instrument 50 and the performance character 60 .

[0034] The virtual instrument 50 and the player character 60 are virtual objects located within a virtual space. The virtual instrument 50 is a virtual keyboard instrument. Meanwhile, the player character 60 is a humanoid object (agent) that simulates the appearance of a human player playing the virtual instrument 50 within the virtual space. Specifically, the player character 60 of the first embodiment is a virtual player that plays the accompaniment part of a target musical piece within the virtual space. The player character 60 is located in the virtual space, while the user U is located in the real space.

[0035] Figure 4 60 is an explanatory diagram of a player character 60. The player character 60 is a three-dimensional object including arms, a chest, and a head. Eyes 61 (right and left) are provided on the head of the player character 60. Therefore, a line of sight L is defined for the player character 60.

[0036] The skeleton of the character 60 is composed of a plurality of control points 63 and a plurality of links 64. Each control point 63 corresponds to a joint of the character 60 and can move within the virtual space. The links 64 are line segments connecting the control points 63. The movement of the character 60 is controlled by changing the position of each control point 63 and the angle of the links 64 relative to the control points 63.

[0037] An observation point Q is set in the virtual space. The observation point Q is a location from which the virtual space is observed. Specifically, a virtual camera is set at the observation point Q in the virtual space. The virtual camera is a virtual shooting device that shoots the virtual space. The guide image G is a video obtained by shooting the virtual space by the virtual camera. The display control unit 32 generates image data of the guide image G by, for example, using well-known image processing such as 3D rendering, and outputs the image data to the display device 23. Therefore, the guide image G, which shows the state of the performance character 60 playing the virtual musical instrument 50 as viewed from the observation point Q, is displayed on the display device 23. As can be understood from the above description, the observation point Q corresponds to the perspective of the user U in the virtual space.

[0038] Figure 5 2 is a block diagram illustrating a functional configuration of the display control unit 32. The display control unit 32 of the first embodiment includes a first motion control unit 321 and a second motion control unit 322.

[0039] The first motion control unit 321 controls the motions (hereinafter referred to as "performance motions") performed by the character 60 playing the virtual musical instrument 50. Specifically, the first motion control unit 321 links the performance motions of the character 60 with the playback of the accompaniment part. Specifically, the character 60 sways its body to the beat of the accompaniment part, and while the musical tones corresponding to the notes of the accompaniment part are played, the character 60 performs key presses.

[0040] As described above, the playback of the musical sound of the accompaniment part follows the performance performed by the user U. Therefore, the first action control unit 321 controls the performance action of the performance character 60 in playing the accompaniment part so that it follows the performance of the melody part by the user U. According to the above structure, the user U can feel as if the performance character 60 is playing the accompaniment part of the target music. As described above, the playback of the musical sound of the melody part played by the user U and the playback of the musical sound of the accompaniment part are synchronized with each other. Therefore, according to the first embodiment, the user U can feel as if he is playing the target music in an ensemble or in a duet with the performance character 60.

[0041] The first motion control unit 321 controls the control points 63 and the connecting parts 64 of the character 60 based on the note data N output by the performance control unit 31, thereby enabling the character 60's performance movements to be linked to the playback of the accompaniment part. Specifically, the first motion control unit 321 generates control data corresponding to the note data N output by the performance control unit 31. The control data specifies the position of each control point 63 and the angle of the connecting part 64 relative to each control point 63.

[0042] For the generation of control data corresponding to the note data N, a trained model is used, for example. The trained model is a statistical prediction model obtained by learning (training) the relationship between the note data N and the control data through machine learning. The first action control unit 321 processes the note data N sequentially output by the performance control unit 31 through the trained model, thereby generating control data, and controlling the performance action of the performance character 60 according to the control data. The trained model is composed of a deep neural network such as a convolutional neural network or a recursive neural network. For the generation of control data corresponding to the note data N, for example, the known technology recorded in Japanese Patent Laid-Open No. 2019-139294 is used.

[0043] Figure 5 The second action control unit 322 controls the action of guiding the performance of the user U by the performance character 60 (hereinafter referred to as "guiding action"). The guiding action is, for example, the movement of the body of the performance character 60 or the change of the expression of the performance character 60. The guiding action is an action related to the performance of the melody part by the user U. In other words, the guiding action is an action that is not directly related to the performance of the accompaniment part (performance action) of the performance character 60 itself. The guiding action is also manifested as an action with less change in the control point 63 or the connection part 64 compared to the performance action.

[0044] Figure 2 The performance analysis unit 33 analyzes the performance status of the user U (hereinafter referred to as "performance status"). Specifically, the performance analysis unit 33 analyzes the performance status by processing the performance data P output by the performance device 10. For example, the performance analysis unit 33 analyzes the performance status by comparing the time series of the melody data M1 of the target music piece with the performance data P.

[0045] The second motion control unit 322 controls the guiding motion of the playing character 60 based on the playing state of the user U. For example, when a condition specified in relation to the playing state (hereinafter referred to as an "action condition") is satisfied, the second motion control unit 322 causes the playing character 60 to perform a specific guiding motion. As described above, a playing motion is a motion controlled based on the accompaniment part, and a guiding motion is a motion controlled based on the playing state of the user U. Specific examples of playing states and guiding motions are given below.

[0046] [Action Example 1a]

[0047] The operating condition of Example Action 1a is that the user U has stopped playing the melody part for a predetermined period of time. Specifically, the performance analysis unit 33 determines whether the performance of the user U has stopped for the predetermined period of time. Specifically, the performance analysis unit 33 determines that the performance of the user U has stopped if, after a predetermined period of time has passed since the melody data M1 indicated the sounding of a specific note, the performance data P corresponding to the note has not been output from the performance device 10.

[0048] When the performance of the user U stops for a predetermined period of time, the second motion control unit 322 causes the player character 60 to perform a guiding motion of moving the sight line L toward the observation point Q in the virtual space. Specifically, the guiding motion of the motion example 1a is as follows: Figure 6 As illustrated, the action is to change the line of sight L of the character playing the instrument 60 from state A, which is directed toward the virtual musical instrument 50, to state B, which is directed toward the observation point Q. State B is a state in which the character playing the instrument 60 turns its head and line of sight L toward the virtual user U in the virtual space. The second motion control unit 322 implements the above-mentioned guiding action by controlling the direction of the head of the character playing the instrument 60.

[0049] The guiding action of Action Example 1a is an action in which the player character 60 urges the performance to proceed by making eye contact with the user U. The user U who visually recognizes the guiding action of Action Example 1a can intuitively or interactively understand the state in which he or she should proceed with the performance of the melody part.

[0050] [Action Example 1b]

[0051] The operation condition of the operation example 1b is that the user U has finished playing the melody part. Specifically, the performance analysis unit 33 determines whether the user U has played the last note specified by the melody data M1.

[0052] When the performance of the melody part ends, the second action control unit 322 causes the player character 60 to perform a guidance action, similar to Action Example 1a, to move the line of sight L toward the observation point Q in the virtual space. Specifically, the guidance action in Action Example 1b is an action by which the player character 60 notifies the user U of the end of the ensemble. By visually recognizing the guidance action in Action Example 1b, the user U can intuitively or interactively grasp the end of the performance of the melody part.

[0053] As can be understood from the description of Action Examples 1a and 1b, the guidance action of moving the sight line L of the playing character 60 toward the observation point Q is recognized by the user U as an action of the playing character 60 making eye contact with the user U. Therefore, the user U who visually recognizes the guidance action can intuitively or interactively grasp what to pay attention to in his or her own performance.

[0054] [Action Example 1c]

[0055] In action example 1c, the second action control unit 322 causes the playing character 60 to perform a guiding action that prompts the user U to start playing the melody part. The second action control unit 322 determines the time point at which the performance starts by referring to the melody data M1. Then, the second action control unit 322 causes the playing character 60 to perform a guiding action of moving the head up and down, for example, at the time point at which the performance of the melody part starts at the beginning of the target music. In addition, the second action control unit 322 causes the playing character 60 to perform a guiding action of moving the head up and down, at the time point at which the performance of the melody part is restarted from a rest whose duration exceeds a specified value in the middle of the target music. Therefore, the user U can intuitively or interactively grasp the time point at which the performance of the melody part should start.

[0056] [Action Example 1d]

[0057] The action condition for action example 1d is that the user U's performance rhythm of the melody part fluctuates unstably. Specifically, the performance analysis unit 33 periodically determines the user U's performance rhythm and determines whether the performance rhythm fluctuates over time. For example, if the distribution of the performance rhythm exceeds a predetermined value, the performance analysis unit 33 determines that the performance rhythm fluctuates unstably. If the distribution of the performance rhythm is less than the predetermined value, the performance analysis unit 33 determines that the performance rhythm is stable.

[0058] When the performance rhythm fluctuates, the second motion control unit 322 causes the character 60 to perform a conducting motion as a guiding motion. Specifically, the conducting motion is a guiding motion in which one (or both) arms of the character 60 swing in a cycle corresponding to the standard performance rhythm specified by the music data M. Therefore, the user U, who visually recognizes the guiding motion, can intuitively or interactively grasp the instability of their own performance rhythm and the standard performance rhythm of the target music piece.

[0059] [Action Example 1e]

[0060] The action condition of action example 1e is that the user U is hesitant in playing the melody part. The situation in which the user U is hesitant in playing refers to, for example, a situation in which the user U is hesitant in playing or a situation in which the user U cannot accurately grasp which part of the target music piece should be played. For example, the performance analysis unit 33 determines that the user U is hesitant in playing when (1) the playing tempo of the user U is slower than the standard playing tempo, (2) the timing of playing the note is delayed relative to the timing of the note specified by the melody data M1, or (3) the playing of the user U is stopped for a predetermined period of time.

[0061] If the user U is hesitant about playing, the second motion control unit 322 causes the player character 60 to perform a guiding motion to encourage the performance of the user U. For example, the second motion control unit 322 causes the player character 60 to swing its head or arms at a cycle corresponding to the standard performance tempo specified by the music data M. Therefore, the user U, who visually recognizes the guiding motion, can intuitively or interactively grasp the need to play more actively and confidently.

[0062] Figure 7 1 is a flowchart of a process (hereinafter referred to as "motion control process") executed by the control device 21 to control the performance character 60. The motion control process is started in response to an instruction from the user U via the operating device 24.

[0063] When the motion control process begins, the control device 21 (performance control unit 31) outputs the note data N of the accompaniment data M2 to the sound source device 25 (S1). The sound source device 25 generates an acoustic signal V representing the musical sound specified by the note data N and the musical sound specified by the performance data P. As a result, the musical sound of the melody part played by the user U and the musical sound of the accompaniment part of the target music piece are played simultaneously from the sound player 26. Furthermore, the control device 21 (display control unit 32) advances the performance motion of the character 60 based on the note data N output to the sound source device 25 (S2).

[0064] The control device 21 (performance analysis unit 33) analyzes the performance status of the user U (S3) and determines whether the action condition is met with respect to the performance status (S4). If the action condition is met (S4: YES), the control device 21 (display control unit 32) causes the player character 60 to perform the guidance action (S5). On the other hand, if the action condition is not met (S4: NO), the player character 60 does not perform the guidance action.

[0065] The control device 21 (display control unit 32) determines whether the time to start the melody part performance is approaching (S6). Specifically, the control device 21 determines whether the time has arrived a predetermined time before the first note of the target music piece or a note following a long rest. If the time to start the melody part performance is approaching (S6: YES), the control device 21 (display control unit 32) causes the playing character 60 to perform a guidance action prompting the start of the melody part performance (S7).

[0066] The control device 21 determines whether a predetermined end condition has been met (S8). The end condition may be, for example, a predetermined time having passed since the end of the performance of the target music piece, or an instruction to end the performance has been given by operating the operating device 24. If the end condition is not met (S8: NO), the control device 21 proceeds to step S1. That is, the above-described exemplary processes (S1 to S7) are repeated until the end condition is met. If the end condition is met (S8: YES), the control device 21 terminates the motion control process.

[0067] As described above, in the first embodiment, the movement of the performance character 60 that performs in the virtual space is controlled according to the performance status of the user U in the real space. Therefore, the user U can smoothly advance the performance while confirming the movement of the performance character 60. That is, the performance of the user U can be assisted intuitively or interactively. In the first embodiment, in particular, when the action condition is satisfied with respect to the performance of the user U, the performance character 60 performs a guiding action. Therefore, the user U can intuitively or interactively grasp the satisfaction of the action condition with respect to the performance of the user U. As described above, according to the first embodiment, the user U can be provided with a special customer experience of interactive performance with the performance character 60.

[0068] B: Second embodiment

[0069] In each embodiment described below, elements having the same functions as those in the first embodiment are denoted by the same reference numerals as those in the first embodiment, and detailed descriptions thereof are omitted as appropriate.

[0070] Figure 8 1 is a block diagram of a musical performance system 100B according to the second embodiment. The musical performance system 100B according to the second embodiment includes a display unit 70 in addition to the musical performance device 10 and the information processing system 20 similar to those of the first embodiment.

[0071] The display unit 70 is an image device (HMD: Head Mounted Display) worn on the head of the user U. For example, a goggle-type or glasses-type HMD is used as the display unit 70. Figure 8 As illustrated, the display unit 70 includes a detection device 71 and a display device 72 .

[0072] The detection device 71 is a sensor that outputs a detection signal D corresponding to the orientation of the display unit 70. Specifically, the detection device 71 is composed of a sensor such as a gyro sensor that detects angular velocity or an accelerometer that detects acceleration. As mentioned above, the display unit 70 is worn on the head of the user U. Therefore, the detection signal D generated by the detection device 71 also represents the orientation of the user U's head.

[0073] The display device 72 displays images under the control of the information processing system 20. For example, various display panels such as a liquid crystal display panel and an organic EL panel are used as the display device 72. The display device 72 is a non-transparent display panel that does not transmit incoming light from the real space, and is installed in front of the eyes of the user U.

[0074] The display control unit 32 of the second embodiment displays a guide image G on the display device 72 of the display unit 70. The guide image G of the second embodiment is a stereoscopic video composed of a right-eye image and a left-eye image. Therefore, the user U perceives a three-dimensional effect on the virtual instrument 50 and the character 60. As described above, in the second embodiment, the guide image G is displayed on the display device 72, so the display device 23 can be omitted.

[0075] The display control unit 32 controls the position and orientation of the virtual camera (observation point Q) within the virtual space based on the detection signal D supplied from the detection device 71. Therefore, the virtual line of sight of the user U within the virtual space (hereinafter referred to as the "observation line of sight") is controlled based on the orientation of the user U's head detected by the detection device 71. The observation line of sight corresponds to the optical axis of the virtual camera.

[0076] The display control unit 32 of the second embodiment controls the movements of the player character 60 displayed on the display device 72, similarly to the first embodiment. For example, the display control unit 32 controls the guiding movements of the player character 60, similarly to the aforementioned movement examples 1a to 1e. Therefore, the second embodiment also achieves the same effects as the first embodiment.

[0077] Furthermore, the display control unit 32 (second motion control unit 322) of the second embodiment causes the player character 60 to perform an action (hereinafter referred to as a "response action") in response to the actions of the user U. Specifically, the display control unit 32 controls the response action of the player character 60 based on the line of sight of the user U. A specific example of the response action is described below.

[0078] [Action Example 2a]

[0079] If the observation line of sight is directed toward the head of the character 60 for a predetermined period of time, the second motion control unit 322 causes the character 60 to perform a response action of moving the line of sight L toward the observation point Q in the virtual space. The observation line of sight being directed toward the head of the character 60 means that the user U is looking at the character 60. Therefore, the response action in action example 2a corresponds to the character 60 looking back at the user U in the virtual space, or the character 60 turning its head to look at the user U.

[0080] [Action Example 2b]

[0081] When the observation sight line swings up and down toward the player character 60, the second motion control unit 322 causes the player character 60 to perform an action of swinging its head up and down. The swinging of the observation sight line refers to the state in which the user U swings his or her head. Therefore, the response action in Action Example 2b corresponds to the player character 60 swinging its head in sync with the user U in the virtual space.

[0082] The performance control unit 31 can control the advancement speed (i.e., the performance rhythm) of the accompaniment part based on the period of the observation line's swing. For example, the shorter the period of the observation line's swing, the faster the performance control unit 31 increases the performance rhythm. Specifically, the performance control unit 31 controls the performance rhythm of the accompaniment part so that the beat of the accompaniment part falls exactly at the end point of the observation line's swing. According to action example 2b, the user U can feel as if the performance character 60 is playing in sync with the swing of the user U's head.

[0083] C: Modification

[0084] Specific variations of the above-described embodiments are exemplified below. Two or more embodiments arbitrarily selected from the following examples may be combined as appropriate within the scope of non-contradiction.

[0085] (1) The content of the guidance action performed by the player character 60 is not limited to the examples above. Furthermore, the combination of the action conditions that trigger the guidance action and the guidance action performed by the player character 60 is arbitrary and not limited to the examples above. For example, the guidance action exemplified below can be performed by the player character 60.

[0086] As exemplified in the following modes A1 to A4 , the second motion control unit 322 causes the musical instrument character 60 to perform a guidance motion for providing the user U with advice regarding the musical performance.

[0087] [Method A1]

[0088] When the performance of user U stops, the second motion control unit 322 causes the character 60 to perform a guiding action, informing user U of the pitch to be played. Specifically, the second motion control unit 322 causes the character 60 to perform a guiding action, speaking while the line of sight L is directed toward the observation point Q. In parallel with the guiding action of the character 60, the second motion control unit 322 determines the pitch that user U should next play based on the melody data M1 and causes the sound playback device 26 to play a sound informing of the pitch. For example, a sound such as "Next, play 'do'" is played from the sound playback device 26. This gives user U the feeling that the character 60 is speaking and informing them of the next pitch to be played. In other words, the user U can be provided with a unique customer experience of being guided by the character 60.

[0089] [Method A2]

[0090] When the user U stops playing, the second motion control unit 322 causes the musical instrument character 60 to perform a guiding action to encourage the user U to play. Specifically, the second motion control unit 322 causes the musical instrument character 60 to perform a guiding action in which the musical instrument character 60 speaks while the line of sight L is directed toward the observation point Q. In parallel with the guiding action, the second motion control unit 322 causes the sound emitting device 26 to play a sound such as "Play more happily." In other words, the musical instrument character 60 can provide the user U with a unique customer experience in which the musical instrument character 60 motivates the user to play more.

[0091] [Method A3]

[0092] If the user U plays a note that differs from the pitch specified by the melody data M1, the second motion control unit 322 causes the player character 60 to perform a guidance action notifying the user of the performance error. Specifically, the second motion control unit 322 causes the player character 60 to perform a guidance action in which the player character 60 speaks while keeping its line of sight L directed toward the observation point Q. In parallel with the guidance action, the second motion control unit 322 causes the sound player 26 to play a sound notifying the user U of the performance error. For example, the sound player 26 may play a sound such as, "Where I just played 'do', I actually played 're'." This allows the user U to experience a unique performance guided by the player character 60.

[0093] [Mode A4]

[0094] When user U finishes playing the melody part, the second motion control unit 322 causes the player character 60 to perform a guidance action that informs the user of the evaluation of the previous performance. Specifically, the second motion control unit 322 causes the player character 60 to perform a guidance action in which the player character 60 speaks while keeping its line of sight L directed toward the observation point Q. In parallel with the player character 60's guidance action, the second motion control unit 322 evaluates user U's previous performance and causes the sound playback device 26 to play a voice informing user U of the evaluation result. For example, the sound playback device 26 may play a voice such as "This performance is 50 points!" Furthermore, the second motion control unit 322 may use the sound playback device 26 to play a voice prompting user U of the next piece of music to be played. For example, the sound playback device 26 may prompt user U of a piece of music with a similar melody to the target piece previously played by user U. In this way, user U can be provided with a unique customer experience of enjoying a music performance with the player character 60.

[0095] As exemplified in the following modes B1 and B2, the second motion control unit 322 causes the player character 60 to perform a response motion in response to the user U. A response motion is an action that changes the expression of the player character 60 to one corresponding to various emotions, such as "happiness" or "sadness." A response motion is an example of a "guiding motion."

[0096] [Method B1]

[0097] The second motion control unit 322 causes the character 60 to perform a response motion corresponding to the length of time the user U can play the melody part without errors (hereinafter referred to as the "performance duration"). For example, as the performance duration increases, the second motion control unit 322 causes the character 60's expression to become closer to "happy." This approach provides the user U with a unique customer experience in which the character 60 motivates them to reduce errors.

[0098] [Method B2]

[0099] The second motion control unit 322 causes the player character 60 to perform a response motion corresponding to the number of playing errors made by the user U. For example, as the number of playing errors increases, the second motion control unit 322 causes the player character 60's expression to become closer to "sadness" or "anger." This approach, similar to approach B1, allows the user U to be provided with a unique customer experience in which the player character 60 motivates them to reduce playing errors.

[0100] (2) The content of the performance status and action conditions is not limited to the examples of the aforementioned methods. For example, the performance analysis unit 33 detects the performance position within the target music as the performance status of the user U, and the display control unit 32 can use the performance position being within a specific interval of the target music (hereinafter referred to as the "specific interval") as an action condition to cause the performance character 60 to perform a guiding action. That is, the display control unit 32 can cause the performance character 60 to perform a guiding action when the user U is playing a specific interval of the target music. The content of the guiding action is arbitrary. For example, the performance character 60 performs any of the guiding actions exemplified above during the performance within the specific interval.

[0101] A specific interval can be any of a plurality of structural intervals obtained by dividing the target music piece on the timeline according to its musical meaning. Examples of structural intervals include the intro, verse, bridge, chorus, and outro. Furthermore, specific intervals can be set based on user U's past performances. For example, a section where user U made mistakes in a previous performance or a section where user U performed smoothly can be set as a specific interval. A section specified by user U through operation of the operating device 24 can also be set as a specific interval.

[0102] In the above configuration, when the user U is playing a specific section of the target music piece, the instrument character 60 performs a guiding action, thereby drawing the user U's attention to the specific section of the target music piece. In other words, the instrument character 60's guiding action can provide the user U with a unique customer experience in which the user can understand that the specific section of the target music piece is being played.

[0103] (3) In the aforementioned embodiments, the information processing system 20 is shown as including the sound source device 25 and the sound emitting device 26. However, the sound source device 25 and the sound emitting device 26 may also be incorporated into the musical instrument 10. In other words, an electronic musical instrument including the sound source device 25 and the sound emitting device 26 may be utilized as the musical instrument 10.

[0104] (4) In each of the aforementioned embodiments, the performance status of the user U is analyzed based on the time series of the performance data P generated by the performance device 10. However, the structure and method of analyzing the performance status by the performance analysis unit 33 are arbitrary and are not limited to the examples in the aforementioned embodiments. For example, in an embodiment in which a sound pickup device picks up the sound produced by the instrument played by the user U, the performance analysis unit 33 analyzes the performance status using the acoustic signal generated by the sound pickup device. Alternatively, in an embodiment in which a camera captures the performance of the user U, the performance analysis unit 33 analyzes the performance status using the image signal generated by the camera. Alternatively, various sensors that detect the movements of the user U may be used to analyze the performance status.

[0105] (5) In the aforementioned embodiments, a configuration is illustrated in which a single instrument character 60 is displayed on the display device 23 when a single user U is performing. However, the display control unit 32 may also display a plurality of instrument characters 60 corresponding to different instrument parts of the target music piece on the display device 23. For example, each instrument character 60 may play a different instrument. The display control unit 32 controls the movements of each of the plurality of instrument characters 60. For example, the display control unit 32 controls the direction of the line of sight L of each instrument character 60.

[0106] For example, the display control unit 32 controls the line of sight L of each of the multiple playing characters 60 to be directed toward the observation point Q in the virtual space. That is, the line of sight L of the multiple playing characters 60 is directed toward the virtual user U in the virtual space. In addition, the direction of the line of sight L of the multiple playing characters 60 is not limited to the direction of the observation point Q. The direction of the line of sight L of each playing character 60 (for example, the playing character 60 that should be directed toward the line of sight L) can be set in advance for each playing character 60 according to a predetermined rule. For example, the line of sight L of another playing character 60 is directed toward the playing character 60 corresponding to the solo playing part among the multiple playing parts of the target music piece.

[0107] Furthermore, the display control unit 32 (second motion control unit 322) can control the motion of each of the multiple playing characters 60, for example, based on whether motion conditions related to the performance state are met. For example, if the performance of the user U stops for a predetermined period of time, the display control unit 32 controls the line of sight L of the multiple playing characters 60 to be directed toward the observation point Q. In other words, if the user U is hesitant about the performance, the display control unit 32 can reproduce a situation where the multiple playing characters 60 are simultaneously looking at the user U.

[0108] (6) While the aforementioned embodiments illustrate a configuration in which a single performance character 60 is displayed on the display device 23 when a single user U is performing, the aforementioned embodiments are also applicable when multiple users U are performing in parallel. For example, when multiple users U are performing different performance parts of a target piece of music, a single performance character 60 is displayed on the display device 23 used by each user U. For example, multiple observation points Q corresponding to different users U are set in the virtual space, and a guide image G captured from the observation point Q corresponding to each user U into the virtual space is displayed on the display device 23 of each user U.

[0109] The display control unit 32 (second motion control unit 322) sets the direction of the line of sight L of the character 60 to the direction of a reference point set in the virtual space. Furthermore, the display control unit 32 controls the direction of the line of sight L of the character 60 to be the direction of one of a plurality of observation points Q in the virtual space. In other words, the display control unit 32 reproduces a state in which the character 60 directs its head and line of sight L toward one of a plurality of virtual users U in the virtual space.

[0110] Specifically, the display control unit 32 controls the character 60 so that its line of sight L is directed toward the observation point Q of a user U among the multiple users U whose performance status satisfies a predetermined action condition. An example of an action condition is a solo performance. Specifically, the display control unit 32 directs the character 60's line of sight L toward the observation point Q of the user U corresponding to a solo performance part among the multiple performance parts. In other words, the character 60 performs the action of directing its head and line of sight L toward the user U responsible for a specific performance part among the multiple users U.

[0111] Furthermore, the display control unit 32 can control the movement of the playing character 60 based on the performance quality of each user U. For example, the performance analysis unit 33 compares the music data M of the target music piece with the performance data P of each user U to calculate an evaluation score for each user U. The evaluation score is an indicator of the performance quality of the user U. The display control unit 32 directs the line of sight L of the playing character 60 toward the observation point Q corresponding to the user U with the lowest or highest evaluation score among the multiple users U.

[0112] The method of selecting the observation point Q to which the line of sight L of the playing character 60 is directed from among the plurality of observation points Q is not limited to the above examples and is arbitrary. Furthermore, in a situation where a plurality of users U corresponding to different performance parts are performing in parallel, the plurality of playing characters 60 corresponding to the different performance parts can be displayed on the display device 23.

[0113] (7) In each of the aforementioned embodiments, the virtual instrument 50 played by the character 60 is exemplified as a keyboard instrument. However, the type of virtual instrument 50 is not limited to a keyboard instrument. For example, any type of instrument, such as a string instrument, a wind instrument, or a percussion instrument, may be displayed as the virtual instrument 50 played by the character 60. Furthermore, the display of the virtual instrument 50 may be omitted.

[0114] (8) In the second embodiment, a camera that captures the front of the user U's head (i.e., the direction of their line of sight) in real space may be mounted on the display unit 70. The display control unit 32 displays the virtual musical instrument 50 and the performing character 60 on the display device 72 using the image of the real space captured by the camera as a background. In other words, the display control unit 32 displays the virtual musical instrument 50 and the performing character 60 using augmented reality (AR) or mixed reality (MR).

[0115] (9) In the second embodiment, a translucent display panel that transmits incoming light from the real space can be used as the display device 72 of the display unit 70. That is, the virtual musical instrument 50 and the performing character 60 can be displayed against the background of an optical image formed by incoming light from the real space.

[0116] (10) The information processing system 20 can be implemented by a server device that communicates with the performance device 10 via a communication network. The information processing system 20 generates a guide image G based on the performance data P supplied from the performance device 10, and transmits the image data of the guide image G to the information device of the user U. The information device is, for example, a smartphone or a personal computer, and displays the guide image G based on the image data received from the information processing system 20. In addition, the display device 23 of the display unit 70 worn on the head of the user U can display the guide image G based on the image data received from the information processing system 20.

[0117] (11) The functions of the information processing system 20 illustrated above are realized by the coordinated action of a single or multiple processors constituting the control device 21 and the program stored in the storage device 22, as described above. The program involved in the present invention can be provided in the form of a computer-readable recording medium and installed in the computer. The recording medium is, for example, a non-transitory recording medium, preferably an optical recording medium (optical disc) such as a CD-ROM, and also includes any known recording medium such as a semiconductor recording medium or a magnetic recording medium. In addition, as a non-transitory recording medium, any recording medium other than a temporary transmission signal (transitory, propagating signal) is included, and volatile recording media are not excluded. In addition, in a structure in which a transmission device transmits a program via a communication network, the storage device that stores the program in the transmission device is equivalent to the aforementioned non-transitory recording medium.

[0118] D: Appendix

[0119] According to the above-exemplified embodiment, for example, the following configuration can be understood.

[0120] One embodiment of the present invention (method 1) involves an information processing method that analyzes the state of a user's performance in a real space, displays a performance character that performs performance actions in a virtual space on a display device, and controls the guiding actions of the performance character based on the state of the performance performed by the user. In the above embodiment, the guiding actions of the performance character that performs performance actions in a virtual space are controlled based on the state of the user's performance in the real space. Therefore, the user can smoothly advance the performance while visually confirming the guiding actions of the performance character. That is, the user's performance can be assisted intuitively or interactively.

[0121] The "performance status" is the status related to the user's performance of the instrument. For example, whether the user can smoothly progress the performance, the number of performance errors, and the progress / stop status of the user's performance are analyzed as "performance status".

[0122] "Virtual space" is a virtual space created through various information processing methods such as image processing. On the other hand, "real space" is a space that actually exists in the real world and is the opposite of virtual space.

[0123] A "performer character" is a virtual object (agent) that plays a musical instrument in a virtual space. A specific example of a performer character is a virtual creature such as a human, but non-living objects such as robots can also be included in the "performer character" category. For example, a "performer character" is a virtual user who plays other parts in parallel with (i.e., in conjunction with) a user's performance of a specific part. However, a virtual user who plays music independently of the user's performance is also included in the "performer character" category.

[0124] The "guiding action" of the playing character is an action used to guide the user's performance. Specifically, the guiding action is the movement of the playing character's body (such as posture) or the change of expression. The playing character's "expression" is the presentation (external expression) of the state of emotion, mood, intention or thinking. "Body movement" and "change of expression" are not necessarily concepts that can be clearly distinguished. In other words, it is also conceivable that the playing character's "action" corresponds to both "body movement" and "change of expression".

[0125] "Control corresponding to the state of the performance" means that the presence or content of the guiding action of the performance character is linked to the state of the performance performed by the user. For example, the performance character performs the guiding action when the conditions specified in the state of the performance performed by the user are met. As can be understood from the above examples, the actions of the performance character do not always need to be linked to the state of the performance. For example, when a specific condition regarding the state of the performance is met, the processing of temporarily changing the actions of the performance character is also included in the "control corresponding to the state of the performance".

[0126] In a specific example of method 1 (method 2), in controlling the guiding action, the playing character is caused to perform the guiding action when a condition specified regarding the user's performance is satisfied. In the above method, the playing character performs the guiding action when a condition specified regarding the user's performance is satisfied. Therefore, the user can intuitively or interactively understand whether the condition specified regarding their own performance has been satisfied.

[0127] The "predetermined conditions" are various conditions related to the state of the performance. For example, the user's performance has stopped, the performance of the music has ended, the performance is progressing smoothly, and there are many performance errors.

[0128] In a specific example of method 2 (method 3), the predetermined condition includes the predetermined duration of the performance pause. In the above method, when the user pauses the performance, the performing character's line of sight moves toward the observation point. Therefore, the user can intuitively or interactively grasp the state in which the performance should be advanced.

[0129] In the specific example of mode 2 (mode 4), the predetermined condition includes the end of the performance. In the above mode, when the user's performance ends, the performance character performs the guiding action. Therefore, the user can intuitively or interactively grasp the end of the performance.

[0130] In a specific example of any of Methods 1 to 4 (Method 5), the guiding action includes the player character moving their gaze toward an observation point within the virtual space. In these methods, when a condition specified regarding the performance status is met, the player character's gaze moves toward the observation point. In other words, the player character makes eye contact with the user. Thus, the user can intuitively or interactively grasp what they should pay attention to regarding the performance.

[0131] An "observation point" is a location within a virtual space where a viewpoint is set. Specifically, an image of the virtual space viewed from the observation point is displayed on a display device. Specifically, the location within the virtual space where a virtual camera is set is called an "observation point."

[0132] In the case where the character has one or more eyes, the character's "line of sight" is a straight line extending in the direction of the eye. When the character's line of sight is directed toward the observation point, the viewer (e.g., user) of the displayed image perceives the character as looking directly at the viewer. The movement of the character's line of sight is an example of processing to control the character's expression.

[0133] In a specific example of any one of Methods 1 to 5 (Method 6), the guiding action includes prompting the user to start playing the music. In the above methods, the performance of the music is prompted by the performance character. Therefore, the user can intuitively or interactively grasp the time point when the performance should start. In addition, "start of performance" includes not only starting the performance at the beginning of the music, but also restarting the performance from a long rest in the middle of the music.

[0134] In a specific example (mode 7) of any one of modes 1 to 6, the performance action is an action in which the character plays a second part of the music, other than the first part, in a manner that follows the user's performance of the first part of the music. In the above mode, the character plays the second part in a manner that follows the user's performance of the first part. Therefore, the user can feel as if they are playing an ensemble or a duet with the character.

[0135] In a specific example (mode 8) of any one of modes 1 to 7, in controlling the guiding action, the playing character is caused to perform the guiding action when the user is playing a specific section of the music. According to the above mode, since the playing character performs a specific action when the specific section of the music is being played, the user's attention can be called to the specific section of the music.

[0136] A "specific section" can be any of a plurality of structural sections, for example, obtained by dividing a piece of music on a timeline based on its musical meaning. Examples of structural sections include the introduction, verse, bridge, chorus, and outro. Furthermore, "specific sections" can be set based on the user's past performances. For example, sections where the user made mistakes in previous performances or sections where the user performed smoothly can be set as "specific sections."

[0137] In a specific example (mode 9) of any one of modes 1 to 8, in the control of the guiding action, the guiding action of the player character is further controlled based on the line of sight of the user in the virtual space.

[0138] In a specific example of method 9 (method 10), in the control of the guiding action, when the state in which the user's line of sight in the virtual space is directed toward the playing character continues for a predetermined period of time, the playing character is caused to perform an action of moving the playing character's line of sight toward the user in the virtual space as the guiding action.

[0139] In a specific example of method 9 or method 10 (method 11), the playing character includes a head, and in the control of the guiding action, when the user's line of sight swings, the playing character is caused to perform an action of swinging the head as the guiding action.

[0140] In a specific example of method 11 (method 12), the performance performed by the user is the performance of the first part of the music, and the playback of the musical sound of the second part of the music other than the first part is further controlled. In the control of the playback, the advancement speed of the playback of the second part is controlled according to the cycle of the user's eye movement.

[0141] An information processing system according to one embodiment of the present invention (embodiment 13) comprises: a performance analysis unit that analyzes the state of a user's performance in a real space; and a display control unit that displays a performance character performing in a virtual space on a display device and controls the movement of the performance character according to the state of the performance of the user.

[0142] One embodiment of the present invention (embodiment 14) involves a program that enables a computer system to function as the following functional units: a performance analysis unit that analyzes the state of a user's performance in a real space; and a display control unit that displays a performance character performing in a virtual space on a display device and controls the movements of the performance character according to the state of the performance of the user.

[0143] Description of the label

[0144] 100A, 100B…performance system, 10…performance device, 11…keyboard, 12…keys, 20…information processing system, 21…control device, 22…storage device, 23…display device, 24…operating device, 25…sound source device, 26…playback device, 31…performance control unit, 32…display control unit, 321…first action control unit, 322…second action control unit, 33…performance analysis unit, 50…virtual musical instrument, 60…performance character, 61…eyes, 63…control points, 64…connecting unit, 70…display unit, 71…detection device, 72…display device.

Claims

1. An information processing method, which is implemented by a computer system, Analyze the performance status of the user in the real space, The performance character performing the performance action in the virtual space is displayed on the display device, The guiding action of the playing character is controlled according to the state of the performance performed by the user.

2. The information processing method according to claim 1, wherein: In the control of the guidance action, when a predetermined condition regarding the state of the performance by the user is satisfied, the performance character is caused to execute the guidance action.

3. The information processing method according to claim 2, wherein: The prescribed condition includes stopping the performance for a prescribed period of time.

4. The information processing method according to claim 2, wherein: The predetermined condition includes the end of the performance.

5. The information processing method according to any one of claims 1 to 4, wherein: The guiding action includes an action of the playing character moving the line of sight toward the observation point in the virtual space.

6. The information processing method according to any one of claims 1 to 5, wherein: The guiding operation includes an operation of prompting the user to start playing a piece of music.

7. The information processing method according to any one of claims 1 to 6, wherein: The performance action is an action in which the performance character performs a second part of the music excluding the first part so as to follow the performance of the first part of the music by the user.

8. The information processing method according to any one of claims 1 to 7, wherein: In the control of the guidance action, when the user is playing a specific section of music, the performance character is caused to perform the guidance action.

9. The information processing method according to any one of claims 1 to 8, wherein: In the control of the guiding action, the guiding action of the playing character is further controlled according to the line of sight of the user in the virtual space.

10. The information processing method according to claim 9, wherein: In the control of the guiding action, when the state in which the user's line of sight in the virtual space is directed toward the playing character continues for a specified period of time, the playing character is caused to perform an action of moving the playing character's line of sight toward the user in the virtual space as the guiding action.

11. The information processing method according to claim 9 or 10, wherein: The playing character includes a head, In the control of the guiding action, when the user's line of sight is moved, the playing character is caused to perform an action of moving the head as the guiding action.

12. The information processing method according to claim 11, wherein: The performance performed by the user is the performance of the first part of the music. Further controlling the playing of the second part of the music except the first part, In the playback control, the advancement speed of the playback of the second part is controlled according to the cycle of the user's eye movement.

13. An information processing system comprising: a performance analysis unit that analyzes the state of a user's performance in real space; and The display control unit displays a musical instrument character performing a performance in a virtual space on a display device, and controls the movement of the musical instrument character according to the state of the performance performed by the user.

14. A program that causes a computer system to function as the following functional unit: a performance analysis unit that analyzes the state of a user's performance in real space; and The display control unit displays a musical instrument character performing a performance in a virtual space on a display device, and controls the movement of the musical instrument character according to the state of the performance performed by the user.

Images