Information processing methods, information processing systems, and programs

Abstract

The actions of a user playing a musical instrument in the real world are appropriately reflected in a video in a virtual space where a virtual user uses a virtual musical instrument. [Solution] The information processing system 10 comprises a posture identification unit 41 that identifies the posture X of the instrument from a first detection signal D1 generated by a first sensor installed on the instrument used by the user, an action identification unit 42 that identifies the user's actions Y from a second detection signal D2 generated by a second sensor that detects the user's actions on the instrument, and a video control unit 43 that controls the posture of the virtual instrument according to the result of identifying the posture X of the instrument and controls the actions of the virtual user according to the result of identifying the user's actions Y for a video in which a virtual user uses a virtual instrument.

Description

Technical Field

[0001] This disclosure relates to a technique for controlling videos.

Background Art

[0002] Instruments equipped with various sensors for detecting performances by users have been proposed in the past. For example, Patent Document 1 discloses a stringed instrument in which detection sensors are installed on the instrument body. According to the detection signal output by the detection sensors, an acoustic effect is imparted to the string vibration signal.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Assume a scenario where the actions of a virtual user (hereinafter referred to as "virtual user") in a virtual space are controlled according to the actions of a user in the real space. For example, an action where a user plays an instrument in the real space is reflected as an action where a virtual user plays a virtual instrument in the virtual space. However, in a configuration where only the detection results by the detection sensors installed on the instrument body in the real space are reflected in the virtual user and the virtual instrument, it is actually difficult to appropriately reflect the performance actions by the user in the actions of the virtual user. Considering the above circumstances, one aspect of this disclosure aims to appropriately reflect an action where a user plays an instrument in the real space in a video where a virtual user uses a virtual instrument in the virtual space.

Means for Solving the Problems

[0005] An information processing method according to one aspect of the present disclosure includes: identifying the posture of a musical instrument from a first detection signal generated by a first sensor installed on a musical instrument used by a user; identifying the user's actions from a second detection signal generated by a second sensor that detects the user's actions on the musical instrument; controlling the posture of a virtual musical instrument in a video in which a virtual user uses a virtual musical instrument, according to the result of identifying the posture of the musical instrument; and controlling the actions of a virtual user in accordance with the result of identifying the user's actions.

[0006] An information processing system according to one aspect of the present disclosure comprises: a posture identification unit that identifies the posture of a musical instrument from a first detection signal generated by a first sensor installed on the musical instrument used by a user; an action identification unit that identifies the user's actions from a second detection signal generated by a second sensor that detects the user's actions on the musical instrument; and a video control unit that controls the posture of a virtual musical instrument in accordance with the result of identifying the posture of the musical instrument and controls the actions of a virtual user in accordance with the result of identifying the user's actions, for a video in which a virtual user uses a virtual musical instrument.

[0007] A program according to one aspect of this disclosure causes a computer system to function as follows: an attitude identification unit that identifies the attitude of an instrument from a first detection signal generated by a first sensor installed on an instrument used by a user; an action identification unit that identifies the user's actions from a second detection signal generated by a second sensor that detects the user's actions on the instrument; and a video control unit that controls the attitude of a virtual instrument in a video in which a virtual user uses a virtual instrument, according to the result of identifying the attitude of the instrument, and controls the actions of the virtual user according to the result of identifying the user's actions. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram illustrating the configuration of the information processing system in the first embodiment. [Figure 2] This is a block diagram illustrating the internal structure of a wind instrument. [Figure 3] This is a schematic diagram of a virtual performance image. [Figure 4] This is a block diagram illustrating the functional configuration of an information processing system. [Figure 5] This is an explanatory diagram illustrating the relationship between users and wind instruments in the real world and virtual users and virtual wind instruments in the virtual world. [Figure 6] This is a flowchart of the video control process. [Figure 7] This is a flowchart of the first control process in the second embodiment. [Figure 8] This is a flowchart of the second control process in the third embodiment. [Figure 9] This is a block diagram illustrating the configuration of the video control system in the fourth embodiment. [Figure 10] This is a flowchart of the second control process in the fourth embodiment. [Figure 11] This is a flowchart of the correction process in the fifth embodiment. [Figure 12] This is a flowchart of the video control process in the modified example. [Modes for carrying out the invention]

[0009] A: First Embodiment Figure 1 is a block diagram illustrating the configuration of the video control system 100 in the first embodiment. The video control system 100 comprises a wind instrument M and an information processing system 10. The user U of the video control system 100 plays the wind instrument M. The video control system 100 is a computer system that generates video data Z of a video corresponding to the user U's performance of the wind instrument M.

[0010] The wind instrument M is an electronic instrument with an appearance similar to a natural wind instrument such as a saxophone or clarinet, and comprises a blowing section 21, a main body section 22, and a sound-emitting section 23. The blowing section 21 has a mouthpiece for the user U to blow air into when playing the wind instrument M. The sound-emitting section 23 is a cylindrical part that widens in a curved shape towards the tip and emits the sound produced by the wind instrument M. The main body section 22 is located between the blowing section 21 and the sound-emitting section 23. The main body section 22 includes an operating section 24 that receives operations from the user U. The operating section 24 includes, for example, a plurality of controls 25 (keys) for the user U to specify the pitch when playing the wind instrument M. The user U can operate each of the plurality of controls 25 as they wish.

[0011] Figure 2 is a block diagram showing the internal configuration of the wind instrument M. The wind instrument M comprises a first sensor 31, a second sensor 32, a sound source device 33, a sound emission device 34, and a communication device 35. The sound source device 33 generates an acoustic signal A in response to the playing actions of the user U. Specifically, when the user U blows into the blowing section 21, the sound source device 33 generates an acoustic signal A representing the pitch of the sound played at the pitch instructed by the user U through an operation of the control section 24. The sound emission device 34 is a speaker that outputs sound waves corresponding to the acoustic signal A. The sound waves output from the sound emission device 34 are radiated into the outside space via the sound emission section 23.

[0012] The first sensor 31 is a sensor for detecting the posture (position and orientation in real space) of the wind instrument M. Specifically, the first sensor 31 includes an accelerometer 311 that detects acceleration in each of the three orthogonal axes, and a gyro sensor 312 that detects angular velocity of rotation (yaw, pitch, and roll directions) around each of the three orthogonal axes. The first sensor 31 generates a first detection signal D1. The first detection signal D1 is a signal corresponding to the posture of the wind instrument M. Specifically, the first detection signal D1 represents the acceleration in the three orthogonal axes detected by the accelerometer 311 and the angular velocity of each axis detected by the gyro sensor 312.

[0013] The second sensor 32 is a sensor for detecting the operation of the user U with respect to the wind instrument M. That is, the second sensor 32 detects the playing operation of the wind instrument M by the user U. Specifically, the second sensor 32 includes a blowing sensor 321 and an operation sensor 322. The blowing sensor 321 is a sensor for detecting the operation of the user U blowing air into the blowing portion 21. For example, a pressure sensor that detects the pressure of the exhalation by the user U, or a flow rate sensor that detects the flow rate of the exhalation by the user U, etc. are used as the blowing sensor 321. The operation sensor 322 detects an operation from the user U with respect to the operation portion 24. For example, the operation sensor 322 includes a key sensor capable of detecting the presence or absence of an operation for each of the plurality of operators 25.

[0014] The second sensor 32 generates a second detection signal D2. The second detection signal D2 is a signal corresponding to the operation (playing operation) of the user U with respect to the wind instrument M. Specifically, the second detection signal D2 represents the blowing characteristics (for example, pressure) detected by the blowing sensor 321 and the content of the operation (for example, the operator 25 operated by the user U) detected by the operation sensor 322. As can be understood from the above description, in the first embodiment, the blowing into the blowing portion 21 and the operation with respect to the operation portion 24 are detected by the second sensor 32 as the playing operation of the user U with respect to the wind instrument M.

[0015] The communication device 35 communicates with the information processing system 10. Specifically, the communication device 35 transmits the acoustic signal A generated by the sound source device 33, the first detection signal D1 generated by the first sensor 31, and the second detection signal D2 generated by the second sensor 32 to the information processing system 10.

[0016] As illustrated in FIG. 1, the information processing system 10 includes a control device 11, a storage device 12, a communication device 13, an operation device 14, a display device 15, and a sound playback device 16. The information processing system 10 is realized by an information device such as a smartphone, a tablet terminal, or a personal computer. Note that the information processing system 10 is realized not only as a single device but also as a plurality of devices configured separately from each other.

[0017] The control device 11 is composed of one or more processors that control each element of the information processing system 10. For example, the control device 11 is composed of one or more types of processors such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), SPU (Sound Processing Unit), DSP (Digital Signal Processor), FPGA (Field Programmable Gate Array), or ASIC (Application Specific Integrated Circuit).

[0018] The storage device 12 is one or more memories that store the programs executed by the control device 11 and various data used by the control device 11. The storage device 12 is composed of a recording medium such as a magnetic recording medium or a semiconductor recording medium, for example. The storage device 12 may be composed of a combination of multiple types of recording media. Also, a portable recording medium that can be attached to and detached from the information processing system 10, or a recording medium (e.g., cloud storage) that the control device 11 can write to or read from via a communication network may be used as the storage device 12.

[0019] The communication device 13 communicates with the wind instrument M. Specifically, the communication device 13 receives the acoustic signal A, the first detection signal D1, and the second detection signal D2 from the wind instrument M. Note that the communication between the wind instrument M and the information processing system 10 may be either wired communication or wireless communication.

[0020] The operation device 14 is an input device that receives operations by the user U. The operation device 14 is, for example, an operation button operated by the user U or a touch panel that detects contact by the user U. The display device 15 displays various images under the control of the control device 11. The display device 15 is composed of a display panel such as a liquid crystal panel or an organic EL (Electroluminescence) panel, for example.

[0021] The sound emission device 16 emits sound waves under the control of the control device 11. For example, a speaker or earphones are exemplified as the sound emission device 16. For example, sound waves represented by acoustic signal A are emitted from the sound emission device 16.

[0022] In the above configuration, the control device 11 generates video data Z of a video (hereinafter referred to as "virtual performance video V") corresponding to the user U playing the wind instrument M. The video data Z generated by the control device 11 is stored in the storage device 12. Figure 3 is a schematic diagram of the virtual performance video V. The control device 11 displays the virtual performance video V shown in Figure 3, represented by the video data Z, on the display device 15.

[0023] The virtual performance video V is a video that depicts a situation in which a virtual performer (hereinafter referred to as "virtual user Uv") plays a virtual wind instrument M (hereinafter referred to as "virtual wind instrument Mv") in a virtual space. The virtual space is a virtual three-dimensional space realized by calculations performed by the control device 11.

[0024] A virtual user Uv is a display object that surrogately represents a real-world user U. Specifically, a human-shaped character (i.e., an avatar) representing the performer of a virtual wind instrument Mv is exemplified as a virtual user Uv. That is, a virtual user Uv includes a head, torso, and arms. A virtual wind instrument Mv is a display object that surrogately represents a real-world wind instrument M. A virtual wind instrument Mv with the same appearance as wind instrument M is placed in the virtual space.

[0025] In the virtual performance video V, the performance of the virtual wind instrument Mv by the virtual user Uv is controlled to follow the performance of the wind instrument M by the real-world user U. In other words, the virtual user Uv in the virtual space performs the same actions as the performance of the wind instrument M by user U.

[0026] Figure 4 is a block diagram illustrating the functional configuration of the information processing system 10. The control device 11 executes a program stored in the storage device 12 to realize multiple functions (attitude identification unit 41, motion identification unit 42, video control unit 43, playback control unit 44) for generating video data Z.

[0027] The attitude determination unit 41 determines the attitude X of the wind instrument M from the first detection signal D1. For the analysis of the first detection signal D1, an analysis process such as a Madgwick filter is used. Attitude X is expressed, for example, by the position on each of the three orthogonal axes and the rotation angles (yaw angle, pitch angle, roll angle) around each of the three orthogonal axes.

[0028] The operation identification unit 42 identifies the user U's operation Y from the second detection signal D2. Specifically, the operation identification unit 42 identifies whether or not there is blowing into the blowing part 21 of the wind instrument M, and whether or not there is operation on each of the multiple control elements 25.

[0029] The video control unit 43 generates video data Z representing a virtual performance video V. For the generation of video data Z, for example, display data and control data of the virtual user Uv and display data of the virtual wind instrument Mv are stored in the storage device 12. The display data of the virtual user Uv is model data representing the three-dimensional appearance and skeleton of the virtual user Uv. The control data of the virtual user Uv is motion data that defines the operation of the virtual user Uv. The display data of the virtual wind instrument Mv is model data representing the three-dimensional appearance of the virtual wind instrument Mv.

[0030] The video control unit 43 generates video data Z of the virtual performance video V by executing the following processes: placing the virtual user Uv and the virtual wind instrument Mv in the virtual space using the display data of the virtual user Uv and the display data of the virtual wind instrument Mv; operating the virtual user Uv using control data; and capturing (rendering) the virtual user Uv and the virtual wind instrument Mv with a virtual camera in the virtual space. The generation of video data Z by the video control unit 43 is performed in parallel with the performance of the wind instrument M by the user U. The video data Z generated by the video control unit 43 is stored in the storage device 12. The format of the video data Z is arbitrary.

[0031] In generating the virtual performance video V, the video control unit 43 controls the posture of the virtual wind instrument Mv according to the posture X of the wind instrument M identified by the posture identification unit 41, and controls the movement of the virtual user Uv according to the movement Y of the user U identified by the movement identification unit 42. Figure 5 is an explanatory diagram of the relationship between the user U and wind instrument M in the real space and the virtual user Uv and virtual wind instrument Mv in the virtual space.

[0032] The video control unit 43 controls the virtual wind instrument Mv in the virtual space to have the same posture X as the wind instrument M in the real space. The video control unit 43 also causes the virtual user Uv in the virtual space to perform the same actions Y as the user U in the real space.

[0033] For example, in Figure 5, State 1 represents a state where user U is facing forward and playing a wind instrument M. If user U swings the wind instrument M to the right from State 1 (State 2), virtual user Uv also rotates their body to the right while swinging the virtual wind instrument Mv to the right. Similarly, if user U swings the wind instrument M to the left from State 1 (State 3), virtual user Uv also rotates their body to the left while swinging the virtual wind instrument Mv to the left.

[0034] Furthermore, for example, if user U performs a blowing motion into the blowing section 21, the video control unit 43 causes virtual user Uv to perform the action of putting the blowing section 21 of the virtual wind instrument Mv into its mouth and blowing air into it. Also, if user U operates any of the multiple control elements 25, the video control unit 43 controls the angle or shape of virtual user Uv's fingers so that virtual user Uv operates the corresponding control element 25 of the virtual wind instrument Mv.

[0035] In the first embodiment, the video control unit 43 synchronizes the movements of the virtual user Uv with the posture of the virtual wind instrument Mv. Specifically, the video control unit 43 controls the posture of the virtual user Uv in conjunction with changes in the posture of the virtual wind instrument Mv. For example, user U in real space plays the wind instrument M while swinging their body (head, torso, and arms) in any direction. Therefore, the posture X of the wind instrument M changes over time in conjunction with the swinging of user U's body. As illustrated in Figure 5, the video control unit 43 swings the body (head, torso, and arms) of the virtual user Uv in conjunction with the swinging of the virtual wind instrument Mv corresponding to the posture X of the wind instrument M.

[0036] Furthermore, the video control unit 43 controls the movements of the virtual user Uv so that the head (especially the mouth) of the virtual wind instrument Mv does not move away from the blowing part 21, and the fingers of the virtual user Uv do not move away from the controls of the virtual wind instrument Mv. For example, forward kinematics (FK) or inverse kinematics (IK) can be used to control the relationship between the posture of the virtual wind instrument Mv and the movements of the virtual user Uv.

[0037] The playback control unit 44 in Figure 4 plays the virtual performance video V. Specifically, the playback control unit 44 displays the virtual performance video V on the display device 15 by supplying video data Z to the display device 15. In parallel with the display of the virtual performance video V on the display device 15, the playback control unit 44 also causes the sound emitter 16 to emit sound waves represented by the acoustic signal A. Therefore, it is possible to make the viewer perceive that the virtual user Uv is playing the virtual wind instrument Mv.

[0038] Figure 6 is a flowchart of the process by which the information processing system 10 generates video data Z (hereinafter referred to as "video control processing"). For example, video control processing is initiated by an operation from user U to the control device 14.

[0039] When the video control process starts, the control device 11 (attitude identification unit 41) acquires the first detection signal D1 received by the communication device 13 from the wind instrument M (Sa1), and identifies the attitude X of the wind instrument M from the first detection signal D1 (Sa2). The control device 11 (operation identification unit 42) also acquires the second detection signal D2 received by the communication device 13 from the wind instrument M (Sa3), and identifies the operation Y of user U from the second detection signal D2 (Sa4). Note that the order of the processes for identifying the attitude X of the wind instrument M (Sa1, Sa2) and identifying the operation Y of user U (Sa3, Sa4) may be reversed.

[0040] The control device 11 (video control unit 43) generates video data Z of a virtual performance video V in which the virtual wind instrument Mv is controlled to a posture X and the virtual user Uv performs an action Y (Sa5). The control device 11 (playback control unit 44) displays the virtual performance video V represented by the video data Z on the display device 15 (Sa6), and emits sound waves represented by the acoustic signal A using the sound emission device 16 (Sa7).

[0041] The control device 11 determines whether a predetermined termination condition has been met (Sa8). The termination condition is, for example, that the termination of the video control process has been instructed by an operation on the operating device 14. If the termination condition is not met (Sa8: NO), the control device 11 moves the process to step Sa1. That is, the identification of the posture X of the wind instrument M (Sa1, Sa2), the identification of the user U's actions Y (Sa3, Sa4), the generation of video data Z (Sa5), and the playback of the virtual performance video V and sound signal A (Sa6, Sa7) are repeated until the termination condition is met. If the termination condition is met (Sa8: YES), the control device 11 terminates the video control process.

[0042] As described above, in the first embodiment, the posture of the virtual wind instrument Mv is controlled according to the result of identifying the posture X of the wind instrument M in real space, and the movements of the virtual user Uv are controlled according to the result of identifying the movements of the user U in real space. Therefore, it is possible to appropriately reflect the movements of user U playing the wind instrument M in real space in the virtual performance video V in which the virtual user Uv uses the virtual wind instrument Mv in the virtual space. In particular, in the first embodiment, since the movements of the virtual user Uv and the posture of the virtual wind instrument Mv are linked to each other, it is possible to generate a natural video that appropriately reflects the movements of user U playing the wind instrument M.

[0043] Furthermore, in the first embodiment, the first sensor 31 and the second sensor 32 installed on the wind instrument M are used to detect the playing by the user U.Therefore, there is the advantage that it is not necessary to attach sensors to the user U's body to detect the user U's movements Y.As a method for detecting the posture X of the wind instrument M and the user U's movements Y, for example, a method in which the user U's playing of the wind instrument M is captured by an imaging device and the captured results are analyzed is also conceivable.In the first embodiment, since the first sensor 31 and the second sensor 32 installed on the wind instrument M are used to detect the playing, there is the advantage that the installation of an imaging device and the analysis of captured results are unnecessary.However, a configuration in which the posture X of the wind instrument M and the user U's movements Y are identified from the detection results of sensors attached to the user U's body, or a configuration in which the posture X of the wind instrument M and the user U's movements Y are identified from the captured results of the user U by an imaging device, may also be included in the scope of this disclosure.

[0044] B: Second Embodiment A second embodiment will now be described. For elements whose function is the same as in the first embodiment in each of the embodiments described below, the same reference numerals as in the first embodiment will be used, and detailed descriptions of each will be omitted as appropriate.

[0045] Figure 7 is a flowchart of the process (hereinafter referred to as "first control process") executed by the control device 11 of the second embodiment in the video control process. The first control process is executed in the process (Sa5) that generates video data Z within the video control process.

[0046] When the first control process is started, the control device 11 (video control unit 43) determines whether or not user U is in a state of playing the wind instrument M (hereinafter referred to as "playing state") (Sb1). In the following description, the states of user U are assumed to be the playing state, in which user U is holding the wind instrument M and actually playing it, and the state in which user U is holding the wind instrument M but not actually playing it (hereinafter referred to as "standby state"). The playing state is the state in which user U is blowing air into the blowing unit 21 while operating the operation unit 24. On the other hand, the standby state is, for example, the state in which user U is waiting to play the wind instrument M, or the state in which user U is moving the wind instrument M.

[0047] Specifically, the control device 11 determines whether user U is in a playing state by analyzing at least one of the first detection signal D1 and the second detection signal D2. For example, in a playing state, user U may swing the wind instrument M in a predetermined pattern. Therefore, for example, if the pattern of change in the posture X of the wind instrument M represented by the first detection signal D1 approximates or matches a pattern that is likely to be observed in a playing state, the control device 11 determines that user U is in a playing state.

[0048] As mentioned above, the playing state is when the user U blows air into the blowing section 21 while operating the control unit 24. Therefore, the control device 11 determines, for example, whether the blowing pressure to the blowing section 21, represented by the second detection signal D2, exceeds a predetermined threshold, and determines that the user U is in the playing state if the pressure exceeds the threshold, and determines that the user U is in the standby state if the pressure falls below the threshold. The control device 11 also determines whether the second detection signal D2 represents an operation on the control unit 24, and determines that the user U is in the playing state if the control unit 24 is operated, and determines that the user U is in the standby state if the control unit 24 is not operated.

[0049] If it is determined that user U is in a performance state (Sb1:YES), the control device 11 (video control unit 43) causes the virtual user Uv to perform the first action (Sb2). On the other hand, if user U is not in a performance state (Sb1:NO), the control device 11 (video control unit 43) causes the virtual user Uv to perform a second action different from the first action (Sb3).

[0050] The first action, similar to that of the first embodiment, is the action of the virtual user Uv playing the virtual wind instrument Mv. That is, the control device 11 controls the virtual wind instrument Mv to the posture X of the wind instrument M, while causing the virtual user Uv to perform the action Y of user U. In other words, in the first action, the action of the virtual user Uv and the posture of the virtual wind instrument Mv are interconnected.

[0051] On the other hand, the second action is the action of the virtual user Uv holding the virtual wind instrument Mv without playing it. Specifically, the control device 11 is the action of keeping the body still with the mouthpiece 21 of the virtual wind instrument Mv separated from the head of the virtual user Uv. In other words, in the second action, the actions of the virtual user Uv and the posture of the virtual wind instrument Mv are not linked to each other.

[0052] The video control processing, excluding the first control processing, is the same as in the first embodiment. Therefore, the same effects as in the first embodiment are achieved in the second embodiment. In addition, in the second embodiment, the virtual user Uv performs either the first or second action depending on whether user U is in a performance state or not. Therefore, it is possible to reflect the real-world tendency that the actions performed by user U in a performance state (for example, user U shaking their body while holding the wind instrument M) and the actions performed by user U not in a performance state (for example, waiting for the start of performance, or moving the wind instrument M) are different in the virtual user Uv and virtual wind instrument Mv.

[0053] In the second embodiment, the virtual user Uv performs the actions of playing the virtual wind instrument Mv and holding the virtual wind instrument Mv. Therefore, it is possible to have the virtual user Uv perform natural actions similar to those that a user U would perform in real space when playing a wind instrument M.

[0054] Furthermore, in the second embodiment, whether or not user U is in a playing state is determined by analyzing at least one of the first detection signal D1 and the second detection signal D2. That is, the first detection signal D1 or the second detection signal D2 is used for both identifying the posture X of the wind instrument M or the movement Y of user U, and determining whether or not user U is in a playing state. Therefore, compared to a configuration that determines whether or not user U is in a playing state using a method that does not utilize the first detection signal D1 and the second detection signal D2, the processing load required for controlling the virtual user Uv and the virtual wind instrument Mv can be reduced.

[0055] C: Third Embodiment

[0056] Figure 8 is a flowchart of the process (hereinafter referred to as "second control process") executed by the control device 11 of the third embodiment in the video control process. The second control process is executed in the process (Sa5) that generates video data Z in the video control process.

[0057] When the second control process is initiated, the control device 11 (video control unit 43) determines whether the wind instrument M is in a state where it is being held by the user U (hereinafter referred to as the "held state") (Sc1). In the following description, the state of the user U is assumed to be either the held state where the wind instrument M is being held by the user U, or the state where the wind instrument M is not being held by the user U (hereinafter referred to as the "storage state"). The held state includes the playing state and the standby state described above in the second embodiment. On the other hand, the storage state is a state where the wind instrument M is stationary, separated from the user U's body.

[0058] Specifically, the control device 11 determines whether the wind instrument M is in a held state by analyzing at least one of the first detection signal D1 and the second detection signal D2. For example, the control device 11 determines whether the posture X of the wind instrument M, identified from the first detection signal D1, changes over time. If the posture X changes, it determines that the wind instrument M is in a held state; if the posture X does not change, it determines that the wind instrument M is in a stored state. The control device 11 also determines whether the angle representing the posture X of the wind instrument M, identified from the first detection signal D1, is within a predetermined range. If the angle of the wind instrument M is within a predetermined range (for example, an angle close to the horizontal), it determines that the wind instrument M is in a stored state. Furthermore, the control device 11 determines whether the user U is in a playing state by analyzing the second detection signal D2. If the user U is in a playing state, it determines that the wind instrument M is in a held state. The determination of whether the user U is in a playing state is the same as in the second embodiment.

[0059] If it is determined that the wind instrument M is in a holding state (Sc1: YES), the control device 11 (video control unit 43) links the movements of the virtual user Uv and the movements of the virtual wind instrument Mv together, as in the first embodiment (Sc2). Specifically, the control device 11 controls the posture of the virtual user Uv in conjunction with the changes in the posture of the virtual wind instrument Mv, as in the first embodiment.

[0060] On the other hand, if it is determined that the wind instrument M is not in a held state (Sc1:NO), the control device 11 (video control unit 43) independently controls the movement of the virtual user Uv and the posture of the virtual wind instrument Mv (Sc3). Specifically, the control device 11 independently controls the movement of the virtual user Uv and the posture of the virtual wind instrument Mv. For example, it keeps the virtual wind instrument Mv stationary and causes the virtual user Uv to perform a predetermined standby operation.

[0061] The video control processing, excluding the second control processing, is the same as in the first embodiment. Therefore, the same effects as in the first embodiment are achieved in the third embodiment. In addition, in the third embodiment, the linkage / independence between the operation of the virtual user Uv and the posture of the virtual wind instrument Mv is switched depending on whether the wind instrument M is in a holding state or not. Therefore, it is possible to reflect the real-world tendency in the virtual user Uv and virtual wind instrument Mv, such that when the wind instrument M is in a holding state, the user U and the wind instrument M are linked, and when the wind instrument M is not in a holding state (for example, when it is in storage), the user U and the wind instrument M are not linked.

[0062] In the third embodiment, in particular, whether or not the wind instrument M is in a holding state is determined by analyzing at least one of the first detection signal D1 and the second detection signal D2. That is, the first detection signal D1 or the second detection signal D2 is used for both identifying the posture X of the wind instrument M or the movement Y of user U, and determining whether or not the wind instrument M is in a holding state. Therefore, compared to a configuration that determines whether or not the wind instrument M is in a holding state using a method that does not utilize the first detection signal D1 and the second detection signal D2, the processing load required for controlling the virtual user Uv and the virtual wind instrument Mv can be reduced.

[0063] D: Fourth Embodiment Figure 9 is a block diagram illustrating the configuration of the video control system 100 in the fourth embodiment. The video control system 100 of the fourth embodiment includes a support 50 in addition to the wind instrument M and the information processing system 10. The support 50 is a structure that supports the wind instrument M when it is not being played by the user U. Specifically, a storage case in which the wind instrument M is housed or a support stand that holds the wind instrument M are exemplified as the support 50.

[0064] A third sensor 51 is installed on the support 50. The third sensor 51 is a sensor for detecting whether or not the wind instrument M is supported by the support 50. Specifically, the third sensor 51 is an optical sensor that optically detects the presence or absence of the wind instrument M, or a mechanical sensor that switches on / off depending on the presence or absence of the wind instrument M. The third sensor 51 generates a third detection signal D3. The third detection signal D3 is a signal that indicates the presence or absence of the wind instrument M on the support 50. The third detection signal D3 generated by the third sensor 51 is transmitted to the information processing system 10. The communication device 13 receives the third detection signal D3 from the support 50.

[0065] Figure 10 is a flowchart of the second control process in the fourth embodiment. Similar to the third embodiment, the second control process shown in Figure 10 is executed in the process of generating video data Z (Sa5) within the video control process. When the second control process starts, the control device 11 (video control unit 43) acquires the third detection signal D3 received by the communication device 13 from the support 50 (Sc0).

[0066] In the third embodiment, the presence or absence of the wind instrument M was determined by analyzing the first detection signal D1 or the second detection signal D2. In the fourth embodiment, the control device 11 (video control unit 43) determines whether or not the wind instrument M is in a held state by analyzing the third detection signal D3 (Sc1). Specifically, the control device 11 determines that the wind instrument M is not in a held state (for example, in a storage state) if the third detection signal D3 indicates that the wind instrument M is supported by the support 50. On the other hand, if the third detection signal D3 indicates that the wind instrument M is not supported by the support 50, the control device 11 determines that the wind instrument M is in a held state.

[0067] The operation of linking the actions of virtual user Uv and virtual wind instrument Mv when wind instrument M is in a holding state (Sc1:YES) (Sc2), and making the actions of virtual user Uv and the posture of virtual wind instrument Mv independent of each other when wind instrument M is not in a holding state (Sc1:NO) (Sc3), is the same as in the third embodiment. Therefore, the same effects as in the third embodiment are achieved in the fourth embodiment as well.

[0068] Furthermore, in the third embodiment, whether or not the wind instrument M is in a holding state is determined by analyzing the third detection signal D3 generated by the third sensor 51 installed on the support 50 that supports the wind instrument M. Therefore, compared to the embodiment in which the first detection signal D1 or the second detection signal D2 is used for determination, it is possible to accurately determine whether or not the wind instrument M is in a holding state.

[0069] In the second embodiment, it was determined whether the wind instrument M was in a holding state by analyzing at least one of the first detection signal D1 and the second detection signal D2. As can be seen from the example of the third embodiment, it is not essential to use the first detection signal D1 or the second detection signal D2 in determining whether the wind instrument M is in a holding state.

[0070] E: Fifth Embodiment The attitude X of the wind instrument M, determined by the analysis of the first detection signal D1, is accompanied by an error component (drift component). The error component is an error that accumulates over time due to various errors, such as detection errors in the first sensor 31 (especially the gyro sensor 312), calculation errors related to the analysis of the first detection signal D1 (for example, errors during integration to calculate the angle from angular velocity), and measurement errors caused by temporal or temporary deformation of the first sensor 31. The control device 11 (attitude determination unit 41) of the fifth embodiment performs a correction process to reduce the error component related to the attitude X of the wind instrument M.

[0071] Figure 11 is a flowchart of the correction process performed by the control device 11 of the fifth embodiment during the video control process. The correction process shown in Figure 11 is performed during the process (Sa2) of identifying the posture X of the wind instrument M from the first detection signal D1 in the video control process. That is, the identification of the posture X of the wind instrument M (Sa2) includes a correction process to reduce error components.

[0072] When the correction process is started, the control device 11 (attitude identification unit 41) calculates the rotation angle θa(t) by analyzing the first detection signal D1 (Sd1). The rotation angle θa(t) is a measured value corresponding to the angular velocity measured by the gyro sensor 312, which constitutes the first sensor 31, for the rotation of the wind instrument M in the yaw direction. The symbol t represents any point in time on the time axis.

[0073] The control device 11 calculates the angular velocity ωa(t) as the difference (θa(t)-θa(t-1)) between the current rotation angle θa(t) and the previous rotation angle θa(t-1) (Sd2). The angular velocity ωa(t) is the angular velocity of the wind instrument M in the yaw direction. In a configuration where the angular velocity ωa(t) is directly calculated by analyzing the first detection signal D1, the process of calculating the angular velocity ωa(t) from the rotation angle θa(t) (Sd1, Sd2) may be omitted.

[0074] In playing the wind instrument M, it is assumed that the user U repeatedly performs an action of oscillating the wind instrument M evenly from side to side (yaw direction). Assuming the above, the average value of the angular velocity ωa(t) over a sufficiently long time period compared to the oscillation period of the wind instrument M will be close to zero when there is no error component attached to the angular velocity ωa(t). That is, the average value of the angular velocity ωa(t) over a predetermined time period corresponds to the error component attached to the angular velocity ωa(t). Taking these circumstances into consideration, the control device 11 calculates the error component E(t) by averaging the past angular velocity ωa(t) over a predetermined period (Sd3). In the fifth embodiment, a configuration in which the error component E(t) is calculated at each time t is illustrated, but the error component E(t) may also be calculated at intervals of a predetermined length longer than the interval of time t.

[0075] The control device 11 calculates the corrected angular velocity ωb(t) by subtracting the error component E(t) from the current angular velocity ωa(t) (Sd4). The control device 11 calculates the rotation angle θb(t) by integrating the corrected angular velocity ωb(t). The rotation angle θb(t) is the angle in the yaw direction of the wind instrument M. As described above, the correction process of the fifth embodiment includes a process of subtracting the average value of the measured values ​​(angular velocity ωa(t)) related to the attitude X of the wind instrument M, which is identified from the first detection signal D1, from each measured value on the time axis. The rotation angle θb(t) after the correction process is applied to the generation of video data Z (Sa5) as information representing the attitude X of the wind instrument M.

[0076] The same effects as in the first embodiment are achieved in the fifth embodiment. Furthermore, in the fifth embodiment, the error component E(t) accumulated in the result of determining the posture X of the wind instrument M is reduced, making it possible to accurately reflect the posture X of the wind instrument M in real space to the virtual wind instrument Mv. In particular, in the fifth embodiment, given the tendency that the average value of the measured values ​​(angular velocity ωa(t)) related to the posture X of the wind instrument M is linked to the error component E(t), the error component E(t) can be appropriately reduced by the correction process. In addition, since the measured values ​​(angular velocity ωa(t)) from the first sensor 31 are used in the correction process, there is also the advantage that a sensor dedicated to the correction process (for example, a geomagnetic sensor) is not required.

[0077] F: Variant The following are examples of specific modifications that may be added to each of the embodiments exemplified above. Two or more embodiments may be arbitrarily selected from the following examples and merged as appropriate, provided they do not contradict each other.

[0078] (1) In the second embodiment, whether or not user U is in a playing state is determined by analyzing at least one of the first detection signal D1 and the second detection signal D2, but the method for determining whether or not user U is in a playing state is not limited to the above examples. For example, the control device 11 may determine that user U is in a playing state if the volume of the playing sound represented by the acoustic signal A exceeds a predetermined threshold. Alternatively, whether or not user U is in a playing state may be determined based on the detection results from sensors other than the first sensor 31 and the second sensor 32.

[0079] (2) The structure and detection targets of the first sensor 31 and the second sensor 32 are not limited to the examples of each embodiment described above. For example, the second sensor 32 may include an embouchure sensor that detects the embouchure of the user U. The embouchure sensor is installed, for example, on the reed or mouthpiece that constitutes the blowing part 21 of the wind instrument M. The detection results from any sensor constituting the first sensor 31 or the second sensor 32 can be used for video control processing (identification of the posture X of the wind instrument M, identification of the movement Y of the user U, determination of whether the user U is in a playing state or not, determination of whether the wind instrument M is in a holding state or not).

[0080] Furthermore, in the above-described embodiments, an example was given in which the operation sensor 322 of the second sensor 32 determines whether or not an operation is performed on each operator 25. However, the operation sensor 322 may also detect, for example, the amount of operation of each operator 25 (for example, the amount of displacement of the operator 25). The control device 11 controls the fingers of the virtual user Uv corresponding to each operator 25 to an angle or shape corresponding to the amount of operation of the operator 25.

[0081] (3) In each of the above-described embodiments, the posture X of the wind instrument M is identified by analyzing the first detection signal D1 generated by the first sensor 31, and the movement Y of user U is identified by analyzing the second detection signal D2 generated by the second sensor 32. However, the relationship between the first detection signal D1 and the second detection signal D2 and the object identified by the control device 11 is not limited to the above examples. For example, the control device 11 may use the first detection signal D1 to identify both the posture X of the wind instrument M and the movement Y of user U. The control device 11 may also use the second detection signal D2 to identify both the posture X of the wind instrument M and the movement Y of user U.

[0082] (4) In each of the above-described embodiments, the acoustic signal A generated by the sound source device 33 of the wind instrument M was reproduced, but the method of generating the acoustic signal A is not limited to the above examples. For example, in a configuration in which the information processing system 10 is equipped with a sound pickup device (microphone), an embodiment is envisioned in which the acoustic signal A generated by the sound pickup device, based on the sound output of the performance sound from the sound emission device 34 of the wind instrument M, is reproduced together with a virtual performance video V. In addition, in a configuration in which time-series data (e.g., MIDI data) representing the performance content of the wind instrument M is transmitted from the wind instrument M to the information processing system 10, the control device 11 (or sound source circuit) of the information processing system 10 may generate the acoustic signal A from the time-series data.

[0083] (5) In the above-described embodiments, the example given is when user U plays a wind instrument M, but the instrument played by user U is not limited to the wind instrument M. For example, the above-described embodiments can be applied in situations where user U plays any type of instrument, such as a keyboard instrument, string instrument, or percussion instrument, in addition to the wind instrument M. However, since the identification of posture X by the posture identification unit 41 is premised on the posture X of the instrument changing, a typical example of an instrument to which this disclosure applies is a portable instrument whose posture X changes in accordance with the playing movements of user U. The first sensor 31 installed on the instrument is a sensor for detecting the posture X of the instrument, and the second sensor 32 is a sensor for detecting the movements Y of user U.

[0084] For example, when user U plays a keyboard instrument (e.g., a portable shoulder keyboard), a first sensor 31 is installed on the keyboard instrument to detect its posture X, and a second sensor 32 detects the key-pressing conditions for each key that makes up the keyboard (e.g., whether or not a key is pressed, the speed, and the amount of the key press). When user U plays a stringed instrument, a first sensor 31 is installed on the stringed instrument to detect its posture X, and a second sensor 32 detects user U's actions Y (pressing, plucking, or bowing) on ​​each string of the stringed instrument. Also, when user U plays a percussion instrument, a first sensor 31 is installed on the percussion instrument to detect its posture X, and a second sensor 32 detects user U's striking action on the percussion instrument (whether or not a strike is made or its intensity).

[0085] (6) The specific method for the correction process to reduce the error component E(t) related to the posture X of the wind instrument M is not limited to the method described above in the fifth embodiment. For example, the control device 11 (posture identification unit 41) may perform a calculation to bring the error component E(t) closer to zero at predetermined intervals (for example, every time t).

[0086] (7) In each of the above embodiments, an example was given in which the video control unit 43 generates video data Z representing a virtual performance video V. However, the video control unit 43 may also generate control data representing the posture of the virtual wind instrument Mv and the movements of the virtual user Uv. The control data is, for example, motion data representing the changes in the posture of the virtual wind instrument Mv over time and the changes in the movements of the virtual user Uv over time. For example, the control data includes a time series of control values ​​for controlling the posture of the virtual wind instrument Mv and a time series of control values ​​for controlling the movements of the virtual user Uv.

[0087] Figure 12 is a flowchart of the video control process according to a modified example. Identifying the posture X of the wind instrument M (Sa1, Sa2) and identifying the actions Y of the user U (Sa3, Sa4) are the same as in the previously described forms. The control device 11 (video control unit 43) generates control data (Sa9) that represents the change in the posture of the virtual wind instrument Mv according to the posture X of the wind instrument M, and the change in the actions of the virtual user Uv according to the actions Y of the user U. In other words, the posture X of the wind instrument M and the actions Y of the user U are reflected in the control data. The method for controlling the posture of the virtual wind instrument Mv and the actions of the virtual user Uv in the generation of the control data is the same as in the previously described forms. The control data generated by the video control unit 43 is stored in, for example, the storage device 12. In the video control process, the identification of the posture X of the wind instrument M (Sa1, Sa2), the identification of the actions Y of the user U (Sa3, Sa4), and the generation of control data (Sa9) are repeated until the termination condition is met (Sa8: YES). The control device 11 (playback control unit 44) uses the control data to display the virtual performance video V on the display device 15. In the virtual wind instrument Mv, as in the forms described above, the posture of the virtual wind instrument Mv is controlled according to the posture X of the wind instrument M, and the movement of the virtual user Uv is controlled according to the movement Y of the user U.

[0088] The control device 11 (video control unit 43) may edit the control data generated by the video control processing. Specifically, the control device 11 edits the control data in response to operations on the operating device 14, for example. For example, the control device 11 adds the operation specified by the operation of the operating device 14 to the posture of the virtual wind instrument Mv or the operation of the virtual user Uv represented by the control data. The control device 11 also modifies the posture of the virtual wind instrument Mv or the operation of the virtual user Uv represented by the control data in response to operations on the operating device 14.

[0089] As can be understood from the above explanation, the video control unit 43 is comprehensively represented as an element that controls the posture of the virtual wind instrument Mv in accordance with the result of identifying the posture X of the wind instrument M, and controls the actions of the virtual user Uv in accordance with the result of identifying the actions Y of user U, for a virtual performance video V in which the virtual user Uv uses the virtual wind instrument Mv. The specific method of utilizing the results of the above control is arbitrary in this disclosure, and the generation of video data and the generation of control data are examples of methods of utilizing the control results.

[0090] (8) The scope of application of this disclosure is not limited to situations in which musical instruments are used. For example, the above-described embodiments may be applied to situations in which user U uses any type of device, not limited to musical instruments. Examples of devices used by user U include information devices such as smartphones or tablet terminals, work tools used for various tasks, and electrical equipment used for various purposes. The first sensor 31 is installed on the device and generates a first detection signal D1 corresponding to the orientation X of the device. The second sensor 32 detects the movement Y of user U using the device.

[0091] (9) For example, the information processing system 10 in each of the above embodiments may be realized by a server device that communicates with an information device such as a smartphone or tablet terminal. The information processing system 10 receives the acoustic signal A, the first detection signal D1, and the second detection signal D2 from the information device via a communication network. The information processing system 10 may also receive the third detection signal D3, as exemplified in the third embodiment, from the information device. The information processing system 10 generates video data Z by operating in the same manner as in each of the above embodiments and transmits the video data Z to the information device. The information device plays back the video data Z received from the information processing system 10. Therefore, the playback control unit 44 may be omitted from the information processing system 10. The video data Z may also be transmitted from the information processing system 10 to a video distribution system and stored therein. The video distribution system is a system for a distribution service that distributes video data Z to a large number of information devices.

[0092] In the configuration in which the posture identification unit 41 is mounted on the information device, the posture identification unit 41 may be omitted from the information processing system 10. That is, the information processing system 10 may receive the posture X of the wind instrument M from the information device. Similarly, in the configuration in which the operation identification unit 42 is mounted on the information device, the operation identification unit 42 may be omitted from the information processing system 10. That is, the information processing system 10 may receive the operation Y of the user U from the information device.

[0093] Furthermore, the information processing system 10 that generates control data through the video control processing illustrated in Figure 12 may be implemented by a server device that communicates with an information device. For example, the control data generated by the video control processing (or control data edited after generation) is transmitted to an information device such as a smartphone or tablet terminal. The information device uses the control data received from the information processing system 10 to display a virtual performance video V representing the performance of a virtual wind instrument Mv by a virtual user Uv. In the virtual wind instrument Mv, as in the forms described above, the posture of the virtual wind instrument Mv is controlled according to the posture X of the wind instrument M, and the movements of the virtual user Uv are controlled according to the movements Y of user U.

[0094] (10) The functions of the information processing system 10 exemplified above are realized through the cooperation of one or more processors constituting the control device 11 and a program stored in the storage device 12, as described above. The program according to this disclosure can be provided in a form stored on a computer-readable recording medium and installed on a computer. The recording medium is, for example, a non-transitory recording medium, such as an optical recording medium (optical disc) like a CD-ROM, but also includes any form of recording medium such as a semiconductor recording medium or a magnetic recording medium. A non-transitory recording medium includes any recording medium except for transient propagation signals, and volatile recording media are not excluded. Furthermore, in a configuration in which a distribution device distributes a program via a communication network, the storage medium in which the distribution device stores the program corresponds to the non-transitory recording medium described above.

[0095] G: Note From the forms exemplified above, the following configuration can be understood, for example.

[0096] An information processing method according to one aspect of this disclosure (Aspect 1) identifies the posture of a musical instrument from a first detection signal generated by a first sensor installed on the musical instrument used by the user, identifies the user's actions from a second detection signal generated by a second sensor that detects the user's actions on the musical instrument, controls the posture of the virtual musical instrument in accordance with the result of identifying the posture of the musical instrument, and controls the actions of the virtual user in accordance with the result of identifying the user's actions, for a video in which a virtual user uses a virtual musical instrument. In this aspect, the posture of the virtual musical instrument is controlled in accordance with the result of identifying the posture of the musical instrument in the real space, and the actions of the virtual user are controlled in accordance with the result of identifying the user's actions in the real space. Therefore, it is possible to appropriately reflect the actions of a user playing a musical instrument in the real space in a video in which a virtual user uses a virtual musical instrument in the virtual space.

[0097] In the specific example of Embodiment 1 (Embodiment 2), the control of the posture of the virtual instrument and the movements of the virtual user involves linking the movements of the virtual user with the posture of the virtual instrument. In the above embodiments, since the movements of the virtual user and the posture of the virtual instrument are linked with each other, it is possible to generate natural videos that appropriately reflect the movements of the user playing the instrument. It is also conceivable that the movements of the virtual user are controlled according to the results of identifying the movements of the user in the real world, and that the movements of the virtual instrument, according to the results of identifying the posture of the instrument in the real world, are linked with the movements of the virtual user. Furthermore, it is conceivable that the posture of the virtual instrument is controlled according to the results of identifying the posture of the instrument in the real world, and that the movements of the virtual user, according to the results of identifying the movements of the user in the real world, are linked with the posture of the virtual instrument.

[0098] In a specific example of Embodiment 1 or Embodiment 2 (Embodiment 3), the control of the virtual user's actions involves determining whether the user is in a playing state of the instrument. If it is determined that the user is in a playing state, the virtual user is made to perform a first action. If it is determined that the user is not in a playing state, the virtual user is made to perform a second action different from the first action. In the above embodiments, the virtual user performs either the first or second action depending on whether the user is in a playing state of the instrument. Therefore, it is possible to reflect the real-world tendency that actions performed by a user in a playing state (e.g., swaying the body) differ from actions performed by a user not in a playing state (e.g., waiting to play, or carrying the instrument) in the virtual user and the virtual instrument.

[0099] In a specific example of Embodiment 3 (Embodiment 4), the first action is the action of the virtual user playing the virtual instrument, and the second action is the action of the virtual user holding the virtual instrument without playing it. According to the above embodiments, the virtual user performs both the action of playing the virtual instrument and the action of holding the virtual instrument (for example, a waiting action). Therefore, it is possible to have the virtual user perform natural actions similar to those that a user performs in real space when it comes to playing a musical instrument.

[0100] In a specific example of Embodiment 3 or Embodiment 4 (Embodiment 5), the determination of whether or not the user is in a playing state is made by analyzing at least one of the first detection signal and the second detection signal. In the above embodiments, at least one of the first detection signal and the second detection signal is used for both identifying the posture of the instrument or the user's movements and determining whether or not the user is in a playing state. Therefore, compared to a configuration that determines whether or not the user is in a playing state using a method that does not utilize the first detection signal and the second detection signal, the processing load required for controlling the virtual user and virtual instrument can be reduced.

[0101] In any specific example of Embodiments 1 to 5 (Embodiment 6), in controlling the posture of the virtual instrument and the actions of the virtual user, it is determined whether or not the instrument is being held by the user. If it is determined that the instrument is being held, the actions of the virtual user and the posture of the virtual instrument are linked together. If it is determined that the instrument is not being held, the actions of the virtual user and the posture of the virtual instrument are controlled independently of each other. In the above embodiments, the linkage / independence between the actions of the virtual user and the posture of the virtual instrument is switched depending on whether or not the instrument is being held. Therefore, it is possible to reflect the real-world tendency in the virtual user and virtual instrument that the user and the instrument are linked when the instrument is being held, and that the user and the instrument are not linked when the instrument is not being held.

[0102] In a specific example of Embodiment 6 (Embodiment 7), the determination of whether or not the instrument is in a holding state is made by analyzing at least one of the first detection signal and the second detection signal. In the above embodiments, at least one of the first detection signal and the second detection signal is used for both identifying the posture of the instrument or the user's actions and determining whether or not the instrument is in a holding state. Therefore, compared to a configuration that determines whether or not the instrument is in a holding state using a method that does not utilize the first detection signal and the second detection signal, the processing load required for controlling the virtual user and virtual instrument can be reduced.

[0103] In a specific example of Embodiment 6 or Embodiment 7 (Embodiment 8), a third detection signal generated by a third sensor installed on a support that supports the musical instrument is acquired, and in determining whether the musical instrument is in a holding state, the determination of whether the musical instrument is in a holding state is made by analyzing the third detection signal. In the above embodiments, whether the musical instrument is in a holding state is determined by analyzing the third detection signal generated by a third sensor installed on a support that supports the musical instrument. Therefore, compared to embodiments in which a first detection signal or a second detection signal is used for determination, it is possible to accurately determine whether the musical instrument is in a holding state.

[0104] In any specific example of Embodiments 1 to 8 (Embodiment 9), the identification of the instrument's posture includes a correction process to reduce the error component accumulated in the result of the identification. In the above embodiments, since the error component (drift) accumulated in the result of identifying the instrument's posture is reduced, it is possible to accurately reflect the instrument's posture in real space onto the virtual instrument.

[0105] In a specific example of Embodiment 9 (Embodiment 10), the correction process includes subtracting the average value of the measured values ​​relating to the posture of the instrument, which are identified from the first detection signal, from each measured value on the time axis. In the above embodiments, given the tendency for the average value of the measured values ​​relating to the posture of the instrument to be linked to the error component, the error component can be appropriately reduced by the correction process. Furthermore, since the measured values ​​from the first sensor are used in the correction process, there is also the advantage that a sensor dedicated to the correction process (e.g., a geomagnetic sensor) is not required.

[0106] In any specific example of Embodiments 1 to 10 (Embodiment 11), video data is generated that represents the video in which the control of the posture of the virtual instrument and the movements of the virtual user are reflected. In the above embodiments, since video data is generated that represents a video in which the virtual user uses the virtual instrument, the video represented by the video data can be displayed without requiring additional processing, compared to, for example, a form in which control data representing the posture of the virtual instrument and the movements of the virtual user is generated.

[0107] In any specific example of Embodiments 1 to 10 (Embodiment 12), control data representing the posture of the virtual instrument and the actions of the virtual user are generated. In the above embodiments, the control data can be used to display a video of the virtual user using the virtual instrument. There is also the advantage that the control data can be edited through editing work, etc., before being used in the video.

[0108] An information processing system according to one aspect of the present disclosure (Aspect 13) comprises: an attitude identification unit that identifies the attitude of an instrument from a first detection signal generated by a first sensor installed on an instrument used by a user; an action identification unit that identifies the user's actions from a second detection signal generated by a second sensor that detects the user's actions on the instrument; and a video control unit that controls the attitude of a virtual instrument in accordance with the result of identifying the attitude of the instrument and controls the actions of a virtual user in accordance with the result of identifying the user's actions, for a video in which a virtual user uses a virtual instrument.

[0109] A program according to one aspect of the present disclosure (Aspect 14) causes a computer system to function as follows: an attitude identification unit that identifies the attitude of an instrument from a first detection signal generated by a first sensor installed on an instrument used by a user; an action identification unit that identifies the user's actions from a second detection signal generated by a second sensor that detects the user's actions on the instrument; and a video control unit that controls the attitude of a virtual instrument in a video in which a virtual user uses a virtual instrument, according to the result of identifying the attitude of the instrument, and controls the actions of the virtual user according to the result of identifying the user's actions. [Explanation of symbols]

[0110] 100...Video control system, 10...Information processing system, 11...Control device, 12...Storage device, 13...Communication device, 14...Operation device, 15...Display device, 16...Sound emission device, M...Wind instrument, 21...Blowing unit, 22...Main unit, 23...Sound emission unit, 24...Operation unit, 25...Operator, 31...First sensor, 311...Accelerometer, 312...Gyro sensor, 32...Second sensor, 321...Blowing sensor, 322...Operation sensor, 33...Sound source device, 34...Sound emission device, 35...Communication device, 41...Attitude identification unit, 42...Motion identification unit, 43...Video control unit, 44...Playback control unit, 50...Support, 51...Third sensor.

Claims

1. The orientation of the instrument is determined from the first detection signal generated by the first sensor installed on the instrument used by the user. The user's actions are identified from the second detection signal generated by the second sensor that detects the user's actions toward the instrument. Regarding videos in which virtual users use virtual musical instruments, The posture of the virtual instrument is controlled according to the result of determining the posture of the instrument. The actions of the virtual user are controlled according to the results of identifying the actions of the aforementioned user. Information processing methods implemented by computer systems.

2. In controlling the posture of the virtual instrument and the actions of the virtual user, The actions of the virtual user and the posture of the virtual musical instrument are linked to each other. The information processing method of claim 1.

3. In controlling the actions of the virtual user, Determine whether the user is playing the instrument, If it is determined that the user is in a performance state, the virtual user is instructed to perform the first action. If it is determined that the user is not in a playing state, the system will cause the virtual user to perform a second action, which is different from the first action. The information processing method of claim 1.

4. The first operation is the operation in which the virtual user plays the virtual musical instrument, The second operation is the operation in which the virtual user holds the virtual instrument without playing it. The information processing method of claim 3.

5. In determining whether the user is in a playing state, By analyzing at least one of the first detection signal and the second detection signal, it is determined whether or not the user is in the performance state. The information processing method of claim 3.

6. In controlling the posture of the virtual instrument and the actions of the virtual user, Determine whether the instrument is being held by the user. If it is determined that the instrument is in a holding state, the actions of the virtual user and the posture of the virtual instrument are linked together. If it is determined that the instrument is not in a holding state, the actions of the virtual user and the posture of the virtual instrument are controlled independently of each other. The information processing method of claim 1.

7. In determining whether the instrument is in a holding state, By analyzing at least one of the first detection signal and the second detection signal, it is determined whether or not the instrument is in the holding state. The information processing method of claim 6.

8. moreover, A third detection signal generated by a third sensor installed on a support structure that supports the aforementioned musical instrument is acquired. In determining whether the instrument is in a holding state, Analysis of the third detection signal determines whether the instrument is in the holding state. The information processing method of claim 6 or claim 7.

9. The determination of the position of the aforementioned instrument is This includes a correction process to reduce the error component accumulated in the specific result. The information processing method of claim 1.

10. The aforementioned correction process is performed as follows: This process includes subtracting the average of the measurements related to the posture of the instrument, which are identified from the first detection signal, from each measurement on the time axis. The information processing method of claim 9.

11. This generates video data representing the video that reflects the control of the virtual instrument's posture and the virtual user's movements. The information processing method of claim 1.

12. This generates control data representing the posture of the virtual musical instrument and the actions of the virtual user. The information processing method of claim 1.

13. A posture identification unit that identifies the posture of the instrument from a first detection signal generated by a first sensor installed on the instrument used by the user, An action identification unit identifies the user's actions from a second detection signal generated by a second sensor that detects the user's actions toward the musical instrument, A video control unit controls the posture of a virtual instrument in a video in which a virtual user uses a virtual instrument, based on the results of identifying the posture of the instrument, and controls the actions of the virtual user based on the results of identifying the user's actions. An information processing system equipped with the following features.

14. A posture identification unit that identifies the posture of a musical instrument from a first detection signal generated by a first sensor installed on the instrument used by the user. An action identification unit that identifies the user's actions from a second detection signal generated by a second sensor that detects the user's actions toward the instrument, and A video control unit controls the posture of a virtual instrument in a video in which a virtual user uses a virtual instrument, according to the result of identifying the posture of the instrument, and controls the actions of the virtual user according to the result of identifying the user's actions. A program that makes a computer system function.

Images