Model data control method, model data control device, and model data control program

Abstract

To provide a model data control method capable of expressing model data of a player and a musical instrument more naturally.SOLUTION: A model data control method includes: receiving first model data on a player and second model data on a musical instrument; receiving calibration data relating to a size of the player or the musical instrument; receiving motion data on the player; using the calibration data to correct a size of the first model data or the second model data; rendering the first model data or the second model data after the correction; and using the motion data to control the first model data.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] One embodiment of the present invention relates to a method for controlling model data 、 and a model data control device and model data control program .

Background Art

[0002] Patent Document 1 discloses a configuration that adjusts musical timing by temporally compressing or stretching motion data based on MIDI data.

[0003] Patent Document 2 discloses a configuration that controls the time length of mouth motion data based on sound data.

[0004] Patent Document 3 discloses a configuration in which when blending first motion data and second motion data, the individual blending rate of the second motion data is increased for the M-th bone of the skeleton and decreased for the N-th bone.

[0005] Patent Document 4 discloses a configuration that applies an effect to a virtual image of a performer's hand based on the performer's motion and body information.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Summary of the Invention

Problems to be Solved by the Invention

[0007] None of the configurations disclosed in Patent Documents 1 to 4 correct the size of the instrument model data and the performer model data. When the creators of the instrument model data and the performer model data are different, the sizes of the instrument model data and the performer model data may differ. As a result, viewers who see the performer model data and instrument model data in the virtual space may feel a sense of incongruity.

[0008] One embodiment of the present invention aims to provide a model data control method that can more naturally represent model data of performers and instruments. [Means for solving the problem]

[0009] A model data control method according to one embodiment of the present invention receives first model data of a performer and second model data of an instrument, receives calibration data related to the size of the performer or the instrument, receives motion data of the performer, corrects the size of the first model data or the second model data with the calibration data, renders the corrected first model data and the second model data, and controls the first model data using the motion data. [Effects of the Invention]

[0010] According to one embodiment of the present invention, model data of performers and instruments can be represented more naturally. [Brief explanation of the drawing]

[0011] [Figure 1] This block diagram shows the configuration of Model Data Control System 1. [Figure 2] This is a block diagram showing the configuration of guitar amplifier 11. [Figure 3] This is a block diagram showing the configuration of user terminal 12. [Figure 4]This is a functional block diagram of an application program. [Figure 5] This flowchart shows the operation of the model data control method. [Figure 6] This is a schematic diagram of the model data. [Figure 7] This is a block diagram of user terminal 12A related to modified example 2. [Modes for carrying out the invention]

[0012] Figure 1 is an external view showing an example of a model data control system 1. The model data control system 1 includes an electric guitar 10, a guitar amplifier 11, a user terminal 12, and a motion sensor 15. Note that the guitar amplifier 11 is not essential in this invention. The electric guitar 10 may be connected to the user terminal 12 without going through the guitar amplifier 11.

[0013] The electric guitar 10 is an example of a stringed instrument. In this embodiment, the electric guitar 10 is shown as an example of an instrument, but the instruments of the present invention also include other instruments such as electric basses and acoustic instruments such as violins. Furthermore, the instruments of the present invention are not limited to stringed instruments. The instruments of the present invention also include other types of instruments such as keyboard instruments and wind instruments.

[0014] The guitar amplifier 11 is connected to the electric guitar 10 via an audio cable. Alternatively, the guitar amplifier 11 may be connected to the user terminal 12 via wireless communication such as Bluetooth® or Wi-Fi. Of course, the guitar amplifier 11 may also be connected to the user terminal 12 via a communication cable. The electric guitar 10 outputs analog sound signals related to the sound being played to the guitar amplifier 11. If the stringed instrument is an acoustic instrument, the sound signal is input to the guitar amplifier 11 using a microphone or magnetic pickup.

[0015] FIG. 2 is a block diagram showing the configuration of the guitar amplifier 11. The guitar amplifier 11 includes a display 101, a user interface (I / F) 102, a flash memory 103, a CPU 104, a RAM 105, a DSP 106, a communication I / F 107, an audio I / F 108, an A / D converter 109, a D / A converter 110, an amplifier 111, and a speaker 112.

[0016] The display 101 is composed of, for example, an LED, an LCD (Liquid Crystal Display), or an OLED (Organic Light-Emitting Diode), etc., and displays the state of the guitar amplifier 11 and the like.

[0017] The user I / F 102 is composed of a knob, a switch, or a button, etc., and accepts the user's operation. Also, the user I / F 102 may be a touch panel laminated on the LCD of the display 101.

[0018] The CPU 104 reads out various programs stored in the flash memory 103, which is a storage medium, to the RAM 105 and controls the guitar amplifier 11. For example, the CPU 104 accepts parameters related to signal processing via the user I / F 102 and controls the DSP 106 and the amplifier 111.

[0019] The communication I / F 107 connects to other devices such as the user terminal 12 via Bluetooth (registered trademark) or a wireless LAN, etc.

[0020] The audio I / F 108 has an analog audio terminal. The audio I / F 108 accepts an analog audio signal from the electric guitar 10 via an audio cable.

[0021] The A / D converter 109 converts the analog audio signal received by the audio I / F 108 into a digital audio signal.

[0022] The DSP106 applies various signal processing, such as effects, to the digital audio signal. Parameters related to signal processing are received via the user I / F102. The DSP106 outputs the processed digital audio signal to the D / A converter 110.

[0023] The CPU 104 transmits the digital audio signal, either after signal processing by the DSP 106 or before signal processing, to the user terminal 12 via the communication I / F 107.

[0024] The D / A converter 110 converts the digital audio signal received from the DSP 106 into an analog audio signal. The amplifier 111 amplifies the analog audio signal. Amplification parameters are received via the user interface 102.

[0025] The speaker 112 outputs the sound of the electric guitar 10 being played, based on the analog sound signal amplified by the amplifier 111.

[0026] Figure 3 is a block diagram showing the configuration of the user terminal 12. The user terminal 12 is an information processing device such as a personal computer or a smartphone. The user terminal 12 is equipped with a display 201, a user interface 202, flash memory 203, a CPU 204, RAM 205, and a communication interface 206.

[0027] In this embodiment, the user of the user terminal 12 is the player of the electric guitar 10. The player of the electric guitar 10 uses the user terminal 12 to distribute their own playing sound and to operate a 3D model that is a virtual representation of themselves performing in a virtual space. The user terminal 12 generates motion data to control the movement of the 3D model. The user terminal 12 reads application programs for distributing the playing sound, generating motion data, and controlling the 3D model from the flash memory 203 into the RAM 205 and executes them.

[0028] The display unit 201 consists of, for example, an LED, LCD, or OLED, and displays various information. The user interface 202 is a touch panel stacked on the LCD or OLED of the display unit 201. Alternatively, the user interface 202 may be a keyboard or mouse. If the user interface 202 is a touch panel, it, together with the display unit 201, constitutes a GUI (Graphical User Interface).

[0029] The communication interface 206 is connected to the guitar amplifier 11 and the motion sensor 15 via wireless communication such as Bluetooth® or Wi-Fi.

[0030] The CPU 204 is a control unit that controls the operation of the user terminal 12. The CPU 204 performs various operations by reading a predetermined program, such as an application program, stored in the flash memory 203 (a storage medium) into the RAM 205 and executing it. The program may also be stored on a server (not shown). The CPU 204 may download and execute a program from the server via a network. The CPU 204 executes a method for controlling model data by executing the application program.

[0031] Figure 4 is a functional block diagram of the application program read by the CPU 204. Figure 5 is a flowchart showing the operation of the model data control method. The CPU 204 configures the reception unit 51 and the control unit 52 based on the read application program.

[0032] The reception unit 51 receives model data, motion data, and calibration data (S11, S12, S13). The model data consists of image data and the like for constructing a 3D model.

[0033] Figure 6 is a schematic diagram of the model data. The model data includes a first model data 501 for the performer and a second model data 502 for the instrument.

[0034] The first model data 501 is downloaded from another device, such as a server (not shown) (for example, a computer used by the creator), and stored in the flash memory 203.

[0035] The first model data 501 has multiple polygon data and bone data to constitute, for example, the performer's face, torso, arms, fingers, and legs. The multiple bone data have a link structure connected by multiple joint data.

[0036] The positional information of each bone data in the first model data 501 is defined by the motion data.

[0037] The second model data 502 is downloaded from another device, such as a server (not shown) (for example, a computer used by a musical instrument manufacturer), and stored in the flash memory 203.

[0038] The second model data 502 includes multiple polygon data and bone data for constructing, for example, the body, neck, and strings of a guitar. The multiple bone data have a link structure connected by multiple joint data. However, since a guitar does not have a range of motion other than the strings, the link structure is not essential. In the case of a musical instrument with a range of motion, it is preferable that the second model data 502 includes a link structure.

[0039] Motion data is input from the motion sensor 15. The motion sensor 15 is a sensor for capturing the performer's motion, and is, for example, an optical, inertial, or image-based sensor. The control unit 52 updates the position information of the bone data of the first model data 501 based on the motion data received from the motion sensor 15.

[0040] The control unit 52 corrects the size of the first model data 501 or the second model data 502 using calibration data (S14). As described above, the first model data 501 is downloaded, for example, from a computer used by the creator, and the second model data 502 is downloaded from a computer used by the instrument manufacturer. In this way, if the creators of the first model data 501 and the second model data 502 are different, the sizes of the first model data 501 and the second model data 502 may differ. Therefore, viewers who see the 3D model of the performer and the 3D model of the instrument in the virtual space may feel a sense of incongruity.

[0041] Therefore, the control unit 52 corrects the size of the first model data 501 or the second model data 502 using the calibration data. The calibration data is used by the performer or This is data related to the size of the instrument. For example, a user of user terminal 12 inputs their height and the neck length of the electric guitar 10 via user I / F 202. The control unit 52 inputs first model data 501 according to the input height and neck length. or The size of the second model data 502 is corrected. For example, the control unit 52 corrects the length of each bone data of the first model data 501 by the ratio of the input height to the height of the first model data 501. The control unit 52 also corrects the size of each bone data of the second model data 502 by the ratio of the neck length of the second model data 502 to the neck length of the input electric guitar 10.

[0042] Alternatively, the user may simply input the length of the instrument's neck. In this case, the control unit 52 accepts the user's height as a general height (for example, the average height of an adult male or adult female).

[0043] Finally, the control unit 52 renders the corrected first model data 501 and second model data 502 (S15), and controls each bone data of the first model data 501 using the motion data (S16).

[0044] As described above, the model data control method of this embodiment ensures that even when the sizes of the first model data 501 and the second model data 502 are different, the first model data 501 or By correcting the size of the second model data 502, the model data of the performer and instrument can be represented more naturally. Therefore, users of the model data control method of this embodiment can enjoy a customer experience in which virtual performances in a virtual space are provided with a greater sense of realism.

[0045] (Variation 1) In the above embodiment, the user of the user terminal 12 input the length of the neck of the electric guitar 10 via the user interface 202. In the modified example 1, the user of the user terminal 12 inputs information regarding the product name of the electric guitar 10 via the user interface 202. The control unit 52 receives the information regarding the product name of the instrument input by the user. Based on the information regarding the product name, the control unit 52 obtains calibration data related to the size of the instrument. For example, the control unit 52 transmits the information regarding the product name to another device, such as a server (a computer used by the instrument manufacturer) not shown. The server has a database that associates the product name of each instrument with size information of parts including the neck. The control unit 52 receives data related to the size of the instrument corresponding to the product name (for example, information regarding the length of the neck) from the database of the server.

[0046] This allows users to input calibration data without having to measure and input the length of the electric guitar's neck.

[0047] (Modification 2) Figure 7 is a block diagram of the user terminal 12A according to modified example 2. Components common to the user terminal 12 in Figure 3 are denoted by the same reference numerals and their descriptions are omitted.

[0048] User terminal 12A is further equipped with a camera 207. The user is connected to user terminal 12 A Use camera 207 to photograph electric guitar 10.

[0049] The control unit 52 obtains information about the product name of a corresponding instrument based on the image data of the instrument captured by the camera 207. For example, the control unit 52 obtains information about the product name of an instrument based on a trained model that has been trained using a DNN (Deep Neural Network) to understand the relationship between the image data of the instrument and the information about the product name of the instrument. Training is performed in advance by a program that is executed on a computer (server) used by the instrument manufacturer, for example.

[0050] Then, the control unit 52 obtains calibration data related to the size of the instrument based on the product name information obtained using the trained model, as shown in Modification 1.

[0051] This eliminates the need for users to manually enter instrument names; calibration data can be entered simply by photographing the instrument with camera 207.

[0052] (Variation 3) The user terminal 12 in the modified example 3 obtains calibration data based on the sound signal related to the performance. Specifically, the user, who is the performer, plays the electric guitar 10 using a specific playing technique. This specific playing technique is, for example, pressing the first fret of the sixth string with the index finger of the left hand, and also pressing the fret of the first string within reach of the little finger of the left hand, while plucking the string with the right hand.

[0053] The user terminal 12 receives special performance sounds when the instrument is played using the specified playing technique. Based on the received special performance sounds, the user terminal 12 obtains calibration data related to the performer's size. The control unit 52 obtains calibration data based on a trained model, for example, which is trained using a DNN to understand the relationship between special performance sounds and calibration data related to the performer's size. Training is performed in advance by a program, for example, on a computer (server) used by the instrument manufacturer.

[0054] The fret position that can be reached with the little finger of the left hand varies depending on the size of the player's hand. For example, a player who can reach the third fret of the first string has larger hands than a player who can reach the first fret of the first string. The size of the hand corresponding to the sound produced when playing the first fret of the first string (pitch F4) and the size of the hand corresponding to the sound produced when playing the third fret of the first string (pitch G4) can be uniquely correlated. In other words, there is a correlation between the special sound produced when playing with a particular playing technique and the size of the player's hand. Therefore, a computer can be trained to recognize the relationship between the special sound produced when playing with a particular playing technique and the size of the player's hand (calibration data related to the player's size), and generate a trained model.

[0055] The control unit 52 of the user terminal 12 obtains the performer's hand size, which is determined using a trained model, as calibration data. The control unit 52 corrects the size of the first model data 501 or the second model data 502 using the performer's hand size information. For example, the control unit 52 corrects the length of each bone data of the first model data 501 by the ratio of the performer's hand size, which is determined using a trained model, to the hand size of the first model data 501.

[0056] This allows users to input calibration data simply by performing using a special playing technique.

[0057] (Modification 4) In this embodiment, the model data control method does not necessarily require correcting the size of the first or second model data with calibration data. For example, in this embodiment, the position of the first or second model data may be corrected based on calibration data. In the modified example 4, the user terminal 12 corrects the relative positional relationship between the first model data 501 and the second model data 502 based on calibration data. Specifically, the user terminal 12 sets constraints on the relative positional relationship between the first model data 501 and the second model data 502. For example, the user terminal 12 always places the first model data 501 of the performer on the back side of the second model data 502, which is a guitar. The user terminal 12 also places the position of the left hand bone of the first model data 501 on the neck portion of the second model data 502. Furthermore, the user terminal 12 places the part of the right hand bone of the first model data 501 closest to the elbow on the upper part of the body of the second model data 502.

[0058] More specifically, for example, the user terminal 12 places the fingertips of the left-hand bone of the first model data onto the frets on the neck of the second model data 502. For example, if the length of the bone's fingers is completely different from the actual finger length of the performer, even if the performer places their fingertips on the fret position of the instrument, the fingertips defined by the motion data may be off-center from the frets. However, by setting a constraint that places the fingertips of the left-hand bone of the first model data 501 onto the frets on the neck of the second model data 502, the user terminal 12 can position the first model data 501 and the second model data 502 in a more natural position.

[0059] This allows the user terminal 12 to position the second model data 502 of the instrument in a more natural position relative to the first model data 501 of the performer.

[0060] As mentioned above, the positional information of each bone data in the first model data 501 is defined by motion data. The user terminal 12 in the modified example 4 corrects the relative positional relationship between the first model data 501 and the second model data 502. Therefore, the positional information of the second model data 502 can also be defined by motion data.

[0061] Therefore, the user terminal 12 according to the modified example 4 can also determine the position and orientation of the second model data 502 without using a motion sensor to detect the position or orientation of the instrument.

[0062] (Variation 5) In the modified example 5, the user terminal 12 estimates the sound emission position and direction of the instrument based on the second model data 502, receives the sound played by the instrument, and performs sound signal processing on the sound based on the estimated sound emission position and direction.

[0063] As shown in Modification 4, the user terminal 12 can determine the position and orientation of the second model data 502 based on the position information of the first model data 501, for example. Based on the position of the second model data 502, the user terminal 12 can perform sound localization processing such as panning on the sound signal related to the performance sound.

[0064] For example, the user terminal 12 distributes the sound signal of the performance to the L channel and R channel, and adjusts the volume balance of the L channel and R channel based on the position of the instrument. For instance, if the position information of the second model data 502 is on the right side in the virtual space (to the right from the listener's perspective), the user terminal 12 increases the volume of the R channel and decreases the volume of the L channel. As a result, the performance sound is localized to the right from the listener's perspective. Therefore, the listener of the performance sound can perceive the sound image as being localized at the position of the second model data 502 in the virtual space.

[0065] Alternatively, for example, the user terminal 12 may perform sound localization processing by convolving a head-related transfer function (HRTF) into the sound signal related to the performance sound. The head-related transfer function is a transfer function from the position of the sound source to the listener's right and left ears. The user terminal 12 reads a head-related transfer function corresponding to the position information of the instrument from a server or the like and convolves it into the sound signal related to the performance sound. As a result, the listener can perceive the sound image as being localized at the position of the second model data 502 in the virtual space.

[0066] Furthermore, the head-related transfer function may differ depending on the orientation of the instrument. The user terminal 12 reads the head-related transfer function corresponding to the instrument's orientation information from the server or the like and convolves it into the sound signal related to the performance sound. As a result, the listener of the performance sound can perceive that the sound image is localized at the position of the second model data 502 in the virtual space, and can also perceive the change in sound due to the orientation of the instrument.

[0067] This allows listeners to experience more realistic sound localization and obtain a customer experience that was previously unattainable.

[0068] (Experimental variation 6) The motion data may include data detected by a motion sensor that detects the position or orientation of the instrument. The user terminal 12 may control the second model data 502 using the data detected by the motion sensor. The motion sensor includes a 3-axis accelerometer and a position sensor. The 3-axis accelerometer detects the orientation of the instrument. The position sensor detects the position of the instrument. The position sensor may be an indoor positioning sensor such as a BLE (Bluetooth® Low Energy) beacon, or an outdoor positioning sensor such as a GPS (Global Positioning System).

[0069] The user terminal 12 may receive the orientation of the instrument detected by the 3-axis accelerometer and the position of the instrument detected by the position sensor, and control the position and orientation of the second model data 502. In this case as well, it is preferable for the user terminal 12 to set constraints on the relative positional relationship between the first model data 501 and the second model data 502, as shown in Modification 4. The user terminal 12 can further position the first model data 501 of the performer and the second model data 502 of the instrument in a more natural position.

[0070] Furthermore, as shown in Modification 5, the user terminal 12 may receive the orientation of the instrument detected by the 3-axis accelerometer and the position of the instrument detected by the position sensor, and perform sound signal processing on the played sound according to the position and orientation of the instrument.

[0071] The description of this embodiment should be considered in all respects to be illustrative and not restrictive. The scope of the invention is indicated by the claims, rather than by the embodiments described above. Furthermore, the scope of the invention includes the scope equivalent to the claims. [Explanation of Symbols]

[0072] 1: Model Data Control System 10: Electric guitar 11: Guitar Amplifier 12: User terminal 12A: User terminal 15: Motion Sensor 51: Reception Department 52: Control Unit 101:Display unit 102: User Interface 103: Flash memory 104:CPU 105: RAM 106: DSP 107: Communication I / F 108: Audio Interface 109: A / D converter 110: D / A converter 111: Amplifier 112: Speaker 201:Display unit 202: User Interface 203: Flash memory 204:CPU 205: RAM 206: Communication I / F 207: Camera 501: First Model Data 502: Second Model Data

Claims

1. We accept the first model data of the performer and the second model data of the instrument. The system receives calibration data related to the size of the performer or the instrument, Upon receiving the motion data of the aforementioned performer, The first model data or the second model data is corrected with the calibration data, the corrected first model data and the second model data are rendered, and the first model data is controlled using the motion data. A model data control method, We accept information regarding the product name of the aforementioned musical instrument, Based on the information regarding the product name, calibration data related to the size of the instrument is obtained. Model data control method.

2. The system receives first model data of the performer and second model data of the instrument, The system receives calibration data related to the size of the performer or the instrument, Upon receiving the motion data of the aforementioned performer, The first model data or the second model data is corrected with the calibration data, the corrected first model data and the second model data are rendered, and the first model data is controlled using the motion data. A model data control method, The system receives image data of the aforementioned instrument, Based on the aforementioned image data, calibration data related to the size of the instrument is obtained. Model data control method.

3. Receiving first model data of a performer and second model data of an instrument, The system receives calibration data related to the size of the performer or the instrument, Upon receiving the motion data of the aforementioned performer, The first model data or the second model data is corrected with the calibration data, the corrected first model data and the second model data are rendered, and the first model data is controlled using the motion data. A model data control method, The instrument receives the sound of the aforementioned instrument being played, Based on the aforementioned sound of performance, calibration data related to the size of the performer is obtained. Model data control method.

4. The aforementioned performance sound accepts special performance sounds produced when the performer plays the instrument using a specific playing technique. The model data control method according to claim 3.

5. Based on the information input from the aforementioned performer, calibration data is obtained. A model data control method according to any one of claims 1 to 4.

6. The first model data is data created by the first creator, The second model data is data created by a second creator who is different from the first creator. A model data control method according to any one of claims 1 to 4.

7. The first model data is received from the first terminal, The second model data is received from a second terminal different from the first terminal. A model data control method according to any one of claims 1 to 4.

8. Based on the calibration data, the size of the first model data or the second model data, or the relative positional relationship between the first model data and the second model data is corrected. A model data control method according to any one of claims 1 to 4.

9. Based on the second model data, the sound emission position and direction of the instrument are estimated. The instrument receives the sound of the aforementioned instrument being played, Based on the estimated sound emission position and direction, sound signal processing is performed on the sound being played. A model data control method according to any one of claims 1 to 4.

10. The sound signal processing includes localization processing to localize the performance sound to the estimated sound emission position. The model data control method according to claim 9.

11. The motion data includes data on the position or orientation of the instrument. The second model data is controlled using the aforementioned motion data. A model data control method according to any one of claims 1 to 4.

12. A receiving unit that receives first model data of the performer, second model data of the instrument, calibration data related to the size of the performer or the instrument, and motion data of the performer. A control unit that corrects the first model data or the second model data with the calibration data, renders the corrected first model data and the second model data, and controls the first model data using the motion data, Equipped with, The reception unit receives information regarding the product name of the musical instrument, The control unit obtains calibration data related to the size of the instrument based on the information regarding the product name. Model data control device.

13. A receiving unit that receives first model data of a performer, second model data of an instrument, calibration data related to the size of the performer or the instrument, and motion data of the performer. A control unit that corrects the first model data or the second model data with the calibration data, renders the corrected first model data and the second model data, and controls the first model data using the motion data, Equipped with, The reception unit receives image data of the instrument, The control unit obtains calibration data related to the size of the instrument based on the image data. Model data control device.

14. A receiving unit that receives first model data of a performer, second model data of an instrument, calibration data related to the size of the performer or the instrument, and motion data of the performer. A control unit that corrects the first model data or the second model data with the calibration data, renders the corrected first model data and the second model data, and controls the first model data using the motion data, Equipped with, The reception unit receives the sound of the instrument being played, The control unit obtains calibration data related to the size of the performer based on the sound of the performance. Model data control device.

15. We accept the first model data of the performer and the second model data of the instrument. The system receives calibration data related to the size of the performer or the instrument, Upon receiving the motion data of the aforementioned performer, The first model data or the second model data is corrected with the calibration data, the corrected first model data and the second model data are rendered, and the first model data is controlled using the motion data. Let the computer perform the process, We accept information regarding the product name of the aforementioned musical instrument, Based on the information regarding the product name, calibration data related to the size of the instrument is obtained. A model data control program that causes the computer to perform further processing.

16. Receiving first model data of a performer and second model data of an instrument, The system receives calibration data related to the size of the performer or the instrument, Upon receiving the motion data of the aforementioned performer, The first model data or the second model data is corrected with the calibration data, the corrected first model data and the second model data are rendered, and the first model data is controlled using the motion data. Let the computer perform the process, The system receives image data of the aforementioned instrument, Based on the aforementioned image data, calibration data related to the size of the instrument is obtained. A model data control program that causes the computer to perform further processing.

17. Receiving first model data of a performer and second model data of an instrument, The system receives calibration data related to the size of the performer or the instrument, Upon receiving the motion data of the aforementioned performer, The first model data or the second model data is corrected with the calibration data, the corrected first model data and the second model data are rendered, and the first model data is controlled using the motion data. Let the computer perform the process, The instrument receives the sound of the aforementioned instrument being played, Based on the aforementioned sound of performance, calibration data related to the size of the performer is obtained. A model data control program that causes the computer to perform further processing.

Images