Terminal device, system operation method and program

Abstract

To allow the sense of realism in the remote playing of musical instruments together to be improved.SOLUTION: A terminal device including an imaging unit for capturing images of a user and a control unit for generating images to be displayed on a display unit further includes a communication unit for communicating with another terminal device used by another user. The control unit transmits a first captured image of the user to the other terminal device via the communication unit, and based on the first captured image, a second captured image of the other user, and a display position on the display unit, the control unit generates, for display on the display unit, a composite image that includes the user and the other user, with the other user's line of sight facing the position of the other user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a terminal device, a system operation method, and a program. [Background technology]

[0002] For example, Patent Document 1 discloses a musical instrument and a musical instrument system in which a display that shows the hands of the other player placed on the keyboard is placed in front of the keyboard. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2006-078863 Summary of the Invention [Problem to be solved by the invention]

[0004] There is room to improve the realism of long-distance duets.

[0005] The purpose of the present disclosure, made in consideration of the above circumstances, is to enable an improvement in the sense of realism when playing musical instruments remotely in duets. [Means for solving the problem]

[0006] A terminal device according to an embodiment of the present disclosure is a terminal device having an imaging unit that images a user and a control unit that generates an image to be displayed on a display unit, and further includes a communication unit that communicates with another terminal device used by another user, and the control unit transmits a first captured image of the user to the other terminal device via the communication unit, and generates a display image of the other user including the user and the other user, the display image of the other user being displayed on the display unit based on the first captured image, a second captured image of the other user, and a display position of the display unit. Kiyu A composite image facing the user's position is generated for display on the display unit. [Effects of the Invention]

[0007] According to one embodiment of the present disclosure, it is possible to improve the sense of realism when playing musical instruments remotely duet-style. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram illustrating an example configuration of a system according to an embodiment of the present disclosure. [Figure 2] FIG. 1 is a diagram illustrating an example of how the system is used. [Figure 3] 10 is a flowchart illustrating an example of the operation of the terminal device. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0010] <Outline of the embodiment> With reference to FIG. 1, an example configuration of a system 1 according to an embodiment of the present disclosure will be described. The system 1 includes at least one pair of terminal devices 10. The terminal devices 10 are connected to each other so as to be able to communicate information with each other via a network 30 including, for example, the Internet and a mobile communication network. The terminal devices 10 exchange information related to musical instrument duets with each other. Users of the terminal devices 10 use the terminal devices 10 to play musical instrument duets (hereinafter also referred to as "remote duets") via a network 20. In other words, in this embodiment, the users are musical instrument players. The terminal devices 10 may communicate information with each other via a server connected to the network 20. Such a server may belong to, for example, a cloud computing system or other computing system and implement various functions. The number of terminal devices 10 included in the system 1 is two in this embodiment, but is not limited to this and may be determined arbitrarily depending on, for example, the number of players playing the duet.

[0011] First, the terminal device 10 of this embodiment has an imaging unit 11 that captures an image of a musical instrument player, and a control unit 18 that generates an image to be displayed on the display unit 13. The terminal device 10 further includes a communication unit 14 that communicates with another terminal device 10 used by another player. The control unit 18 of the terminal device 10 transmits a first captured image of the player to the other terminal device 10 via the communication unit 14, and further generates a composite image that includes the player and the other player, with the line of sight of the other player directed toward the player's position, based on the first captured image of the player, the second captured image of the other player, and the display positions on the display unit 13, to be displayed on the display unit 13.

[0012] In this way, according to this embodiment, a composite image is generated that includes the player and another player, with the other player's gaze directed toward the player. For example, by displaying the composite image on the display unit 13 of the terminal device 10 while the player is playing, the player can experience eye contact with the other player playing next to them as if through a mirror. This makes it easier to enhance the sense of realism of a duet.

[0013] Next, each component of the system 1 will be described in detail.

[0014] <Configuration of terminal device> As shown in FIG. 2, the terminal device 10 includes an imaging unit 11, a projection unit 12, a display unit 13, a communication unit 14, a memory unit 15, an audio input / output 16, a detection unit 17, and a control unit 18.

[0015] The imaging unit 11 includes a camera that captures images of a subject using visible light and a distance measurement sensor that measures the distance to the subject to obtain a distance image. The camera captures images of the subject at, for example, 30 frames per second to generate a video consisting of consecutive captured images. The distance measurement sensor includes a ToF (Time of Flight) camera, a LiDAR (Light Detection and Ranging) sensor, and a stereo camera, and generates a distance image of the subject containing distance information. The distance measurement sensor can acquire information such as the position (spatial coordinates, etc.) of the subject captured by the imaging unit 11 and the subject's position relative to the display unit 13. The imaging unit 11 sends the captured image and distance image to the control unit 18. In one embodiment, the terminal device 10 uses the imaging unit 11 to capture an image of a player of a musical instrument using the terminal device 10 and obtains the captured image of the player. For example, the control unit 18 can acquire information such as the position of the subject's pupil, gaze point, and line of sight by analyzing the acquired captured image.

[0016] The projection unit 12 is a projector of any type, such as an LCD (liquid crystal display) projector, an LCOS (liquid crystal on silicon) projector, or a DLP (digital light processing) projector. The projector includes an image display element, a light source, and a projection lens. For example, an LCD or LCOS projector uses a liquid crystal as the image display element, while a DLP projector uses a DMD (digital micromirror device). The light source irradiates the image display element with projection light, and projects an image onto a projection surface through the projection unit 12. The light source may also irradiate illumination light in addition to the projection light. The illumination range of the projection light and the illumination range of the illumination light may be the same or different. The projection lens enlarges or reduces the image on the image display element at any magnification and displays it on the projection surface. In one embodiment, the terminal device 10 uses the projection unit 12 to project an image onto an instrument played by a player using the terminal device 10.

[0017] The display unit 13 is disposed at an arbitrary position that is easily visible to the user of the terminal device 10. For example, it is an LCD or an organic EL (Electro Luminescence) display. The display unit 13 displays data obtained by the operation of the terminal device 10, for example, images acquired by the imaging unit 11. The display unit 13 may be connected to the terminal device 10 as an external output device instead of being provided in the terminal device 10. For example, an interface compatible with standards such as USB (Universal Serial Bus) or Bluetooth (registered trademark) can be used as the connection interface. The display unit 13 can be disposed at an arbitrary position that is easily visible to the user of the terminal device 10. In one embodiment, the terminal device 10 displays, on the display unit 13, captured images of a musical instrument player and a composite image generated based on the captured images.

[0018] The communication unit 14 includes one or more communication interfaces. The communication interfaces include an interface compatible with a wired or wireless LAN standard, an interface compatible with a mobile communication standard such as LTE (Long Term Evolution), 4G (4th Generation), or 5G (5th Generation), etc. The communication unit 14 receives information used in the operation of the terminal device 10 and transmits information obtained by the operation of the terminal device 10. The terminal device 10 is connected to a network 20 via a nearby router device or a mobile communication base station by the communication unit 14, and communicates with external devices such as another terminal device 10 via the network 20. In one embodiment, the terminal device 10 uses the communication unit 14 to communicate with another terminal device 10 used by another player.

[0019] The storage unit 15 includes, for example, one or more semiconductor memories, one or more magnetic memories, one or more optical memories, or a combination of at least two of them, that function as a main storage device, an auxiliary storage device, or a cache memory. The semiconductor memory is, for example, a random access memory (RAM) or a read only memory (ROM). The RAM is, for example, a static RAM (SRAM) or a dynamic RAM (DRAM). The ROM is, for example, an electrically erasable programmable read only memory (EEPROM). The storage unit 15 stores information used in the operation of the terminal device 10 and information obtained by the operation of the terminal device 10.

[0020] The audio input / output 16 includes a microphone for receiving acoustic and voice inputs and a speaker. In one embodiment, the audio input / output 16 provides an audio interface between a musical instrument player and the terminal device 10. The audio input / output 16 receives audio data from the musical instrument played by the player, converts the audio data into electrical signals, and transmits the electrical signals to a speaker. The speaker converts the electrical signals into sound waves audible to humans. The audio input / output 16 may also include a headset jack. The headset jack provides an interface between a detachable audio input / output peripheral, such as a headset with headphones and / or a microphone.

[0021] The detection unit 17 includes an optional sensor module, such as an eye-tracking camera, capable of acquiring data related to human eye movements. The detection unit 17 sends information indicating the detection results of the sensor module to the control unit 18. The sensor module may include, for example, a laser sensor, a photoelectric sensor, an ultrasonic sensor, a near-infrared LED (Infrared Emitting Diode), or a combination thereof. However, the detection unit 17 is not limited to these examples and may include any other sensor module that detects the state or operation of each component of the terminal device 10, or an interface with each sensor module. The detection unit 17 sends information indicating the detection results of the sensor module to the control unit 18. In one embodiment, the control unit 18 may acquire information such as the pupil position, gaze point, and line of sight of the subject by processing the data information sent from the detection unit 17, in addition to or instead of image analysis of the captured image acquired using the imaging unit 11.

[0022] The control unit 18 includes one or more processors, one or more programmable circuits, one or more dedicated circuits, or a combination thereof. The processor may be, for example, but is not limited to, a general-purpose processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), or a dedicated processor specialized for a specific process. The programmable circuit may be, for example, but is not limited to, an FPGA (Field-Programmable Gate Array). The dedicated circuit may be, for example, but is not limited to, an ASIC (Application Specific Integrated Circuit). The control unit 18 controls the overall operation of the terminal device 10. In one embodiment, the terminal device 10 generates an image to be displayed on the display unit 13 using the control unit 18.

[0023] The functions of the terminal device 10 are realized by a control program being executed by a processor included in the control unit 18. The control program is a program for causing a computer to function as the terminal device 10. Some or all of the functions of the terminal device 10 may also be realized by a dedicated circuit included in the control unit 18. The control program may also be stored in a non-transitory recording or storage medium readable by the terminal device 10, and read by the terminal device 10 from the medium.

[0024] FIG. 2 is a diagram showing an example of how the system 1 is used. As shown in FIG. 2, the system 1 is used for remote duet playing of musical instruments 100. In the example shown in FIG. 2, the musical instrument 100 is a piano and includes a keyboard 101. However, the musical instrument is not limited to a piano, and may be any instrument in which one player plays together with another player, such as another keyboard instrument (such as an organ) or a percussion instrument (such as a drum). The musical instrument may also be an electronic instrument that generates electronic sounds. FIG. 2 shows the situation on the player U1's side when a player U1 of the musical instrument 100 using a terminal device 10 and another player of another musical instrument 100 using a different terminal device 10 (hereinafter also referred to as "player U2") are playing remote duet playing using their respective terminal devices 10. In the following, in order to distinguish between the terminal device 10 and the musical instrument 100 on the player U1 side and the terminal device 10 and the musical instrument 100 on the player U2 side, they will be referred to as the terminal device 10A and the musical instrument 100A, and the terminal device 10B and the musical instrument 100B, respectively. Similarly, the components of each terminal device will be distinguished by adding a sub-number "A" or "B" to the reference number as necessary.

[0025] On the player U1 side, the image of player U1 playing musical instrument 100A is captured by the camera of imaging unit 11A on the terminal device 10A side. This captured image (hereinafter also referred to as the "first captured image") is transmitted to terminal device 10B of player U2 via network 20. At the same time, on the player U2 side, the image of player U2 playing musical instrument 100B is captured by the camera of imaging unit 11B on the terminal device 10B side. This captured image (hereinafter also referred to as the "second captured image") is transmitted to terminal device 10A of player U1 via network 20.

[0026] Then, the control unit 18A of the terminal device 10A generates a composite image M3 for display on the display unit 13A, which includes the players U1 and U2 and in which the gaze of the player U2 is directed toward the position of the player U1, based on the first captured image of the player U1, the second captured image of another player U2, and the display positions at which the first captured image and the second captured image are displayed on the display unit 13A.

[0027] This allows each player to have an experience of making eye contact with the other player playing next to him through a mirror, for example, through a composite image displayed on the display unit 13 of each terminal device 10, with the line of sight corrected so that the line of sight of the other player is directed to the player's position.

[0028] <Operation flow of terminal device 10> The operation of the terminal device 10 according to this embodiment will be described with reference to Fig. 3. The operation in Fig. 3 corresponds to the method according to this embodiment. This operation is started, for example, when the player using the terminal device 10 starts up the terminal device 10, but the conditions and / or timing for starting this operation are not limited to this example.

[0029] 2, a specific example of remote duet playing will be described in which player U1 sits on the bass side of instrument 100A (to the left as one faces the keyboard 101) and player U2 sits on the treble side of instrument 100B (to the right as one faces the keyboard 101), and they share the range of the instrument to play. As mentioned above, player U1 corresponds to the "player" in this embodiment, and player U2 corresponds to the "other player" in this embodiment.

[0030] Step S100: The control unit 18 of the terminal device 10 transmits a captured image of the player (hereinafter also referred to as a "first captured image") to another terminal device 10 via the communication unit 14.

[0031] Specifically, the control unit 18 of the terminal device 10 captures an image of the player using the camera of the imaging unit 11. In this embodiment, the captured image of the player obtained by capturing an image is a moving image made up of multiple frame images. In this embodiment, the control unit 18 captures not only an image of the player from the front, but also captures an image of the player's hands, thereby obtaining a moving image of the player's finger movements. In the example shown in FIG. 2, the control unit 18A of the terminal device 10A captures an image of the player U1 using the camera of the imaging unit 11A. Then, the control unit 18 transmits the captured image of the player as a first captured image to the communication unit 14 of another terminal device 10 via the network 20 and the communication unit 14.

[0032] Furthermore, in addition to the first captured image, the control unit 18 may convert audio data received from the musical instrument 100 played by the player via the audio input / output 16 into an electrical signal and acquire the audio signal. The audio signal may be performance information such as MIDI (Musical Instruments Digital Interface). The control unit 18 may collect the sound of the musical instrument 100 played by the player using a microphone in the audio input / output 16. The control unit 18 may also collect the sound produced by the player using the microphone in the audio input / output 16, convert the collected sound into an electrical signal, and acquire the audio signal. The control unit 18 may transmit information such as the audio signal or audio signal acquired in this manner to another terminal device 10 together with the first captured image.

[0033] Step S101: The control unit 18 transmits, via the communication unit 14 to another terminal device 10, position information indicating the position of the player (hereinafter also referred to as "first position information").

[0034] In this embodiment, the first position information indicates the position of the player using the terminal device 10 relative to the display unit 13. Specifically, the control unit 18 of the terminal device 10 derives the position of the player based on the position of the display unit 13 of the terminal device 10 by performing image analysis on the first captured image captured in step S100. In this example, the display unit 13 of the terminal device 10 is disposed on the top board of the musical instrument 100 with the display surface facing the player, as shown in FIG. 2. In the example shown in FIG. 2, the player U1 is seated on the left side facing the keyboard 101A. Therefore, as viewed from the display unit 13A, the player U1 is located directly in front of the right side of the display unit 13A. In this case, the control unit 18A of the terminal device 10A can derive the position of the player U1 directly in front of the right side of the display unit 13A from the first captured image. If the terminal device 10 is equipped with a sensor module capable of acquiring information such as the distance and positional relationship between the subject (player U1) and the display unit 13, the position may be derived using the information acquired by the sensor module. Then, the control unit 18 transmits the derived position information indicating the player's position as first position information to the communication unit 14 of another terminal device 10 via the network 20 and the communication unit 14.

[0035] Note that steps S100 and S101 may be executed in parallel or in reverse order.

[0036] Step S102: The control unit 18 receives a captured image of another player (hereinafter also referred to as a "second captured image") from another terminal device 10 via the communication unit 14.

[0037] Specifically, the control unit 18 of the other terminal device 10 captures an image of the other player using the camera of the imaging unit 11. As described above in step S100, the captured image is a moving image consisting of a plurality of frame images. The control unit 18 of the other terminal device 10 transmits the captured image of the other player as a second captured image to the communication unit 14 of the terminal device 10 via the network 20 and the communication unit 14. In the example shown in FIG. 2, the control unit 18B of the terminal device 10B captures an image of player U2 using the camera of the imaging unit 11B. The control unit 18B transmits the captured image of player U2 to the terminal device 10A as the second captured image. In this manner, the control unit 18 of the terminal device 10 receives the second captured image from the other terminal device 10 via the communication unit 14. Furthermore, as described above in step S100, the control unit 18 may receive information such as an acoustic signal or an audio signal together with the second captured image from the other terminal device 10.

[0038] Step S103: The control unit 18 receives, via the communication unit 14, from another terminal device 10, position information indicating the position of another player (hereinafter also referred to as "second position information").

[0039] In this embodiment, the second position information indicates the relative position of the other player and the display unit 13 of the other terminal device 10. Specifically, the control unit 18 of the other terminal device 10 derives the position of the other player based on the position of the display unit 13B of the other terminal device 10 by performing image analysis on the second captured image captured in step S1002. In this example, the display unit 13 of the other terminal device 10 is disposed in the same manner as the terminal device 10, i.e., on the top board of the other musical instrument 100, with the display surface facing the other player. In the example shown in FIG. 2, the player U2 is seated on the right side of the keyboard 101B. Therefore, as viewed from the display unit 13B, the player U2 is located directly in front of the left side of the display unit 13B. In this case, the control unit 18B of the terminal device 10B can derive the left side of the display unit 13B as the position of the player U2 from the second captured image. Note that the position may also be derived using a sensor module as described above in step S101. The control unit 18B transmits the derived position information indicating the position of the player U2 as second position information to the terminal device 10A. In this way, the control unit 18 of the terminal device 10 receives the second position information from another terminal device 10 via the communication unit 14.

[0040] Note that steps S102 and S103 may be executed in parallel or in reverse order. Also, although steps S100 to S103 are described as being executed sequentially, steps S100 to S103 can be executed in parallel or in any order.

[0041] Step S104: The control unit 18 determines the display positions of the first captured image and the second captured image on the display unit 13 based on the first position information and the second position information.

[0042] Specifically, the control unit 18 uses information on the relative position between the player and the display unit 13 of the terminal device 10, which is indicated by the first position information, to determine in which direction, as viewed from the player, the first captured image (here, the player's own image) should be displayed. The control unit 18 also uses information on the relative position between another player and the display unit 13 of another terminal device 10, which is indicated by the second position information, to determine in which direction, as viewed from the player, the second captured image (here, the image of another player) should be displayed. For example, when displaying an image of the player on the display unit 13, it is more natural for the image to look like a mirror. Therefore, in this embodiment, the position of the player is reversed (mirror-reflected) and displayed. The control unit 18 determines the display position of each captured image on the display unit 13 based on the determined directions. Any method can be used to determine the display position of each captured image, and for example, the display position may be specified by pixel coordinates on the display unit 13. As a result, in this example, the composite image displayed on the display unit 13 in step S106, which will be described later, displays the player's own figure (a first display model, which will be described later) in a predetermined area located in front of the player on the display surface of the display unit 13. Also, the figure of another player (a second display model, which will be described later) is displayed in another predetermined area located in front of the player on the right side of the display surface of the display unit 13. As a result, in the composite image, which will be described later, it becomes easier to improve the reproducibility of the positional relationship between the players when one player and another player play one instrument (the piano in this case) together.

[0043] Step S105: The control unit 18 generates a composite image based on the first captured image of the player, the second captured image of the other player, and the display position on the display unit 13, to be displayed on the display unit 13, the composite image including the player and the other player, with the other player's line of sight directed toward the player's position.

[0044] Specifically, the control unit 18 generates a composite image based on the relative position between the player and the display unit 13 of the terminal device 10 indicated by the first position information acquired in step S101, the relative position between another player and the display unit 13 of another terminal device 10 indicated by the second position information acquired in step S103, and the display positions on the display unit 13 of each captured image specified in step S104.

[0045] Any method can be used to generate the composite image. For example, the control unit 18 generates display models by horizontally inverting (i.e., mirror-reflecting) the first captured image and the second captured image. Hereinafter, the display model for the first captured image will be referred to as the "first display model," and the display model for the second captured image will be referred to as the "second display model." In the example shown in FIG. 2, a first display model M1, which is a mirror-reflected image of player U1, and a second display model M2, which is a mirror-reflected image of player U2, are generated. The control unit 18 performs gaze correction based on the relative positions of the players and the display unit 13 derived in step S103, so that the line of sight of the second display model (i.e., the mirror-reflected image of another player) faces the position of the player derived from the first captured image. In the example shown in FIG. 2, the control unit 18A of the terminal device 10A may perform gaze correction so that the line of sight of the second display model M2 faces the position of player U1 (here, the right front of the display unit 13A). A known method can be used for the gaze correction. That is, in this embodiment, the relative position between the player and the display unit 13 of the terminal device 10 is identified from the first position information, and the control unit 18 can determine, based on the identified position, in which direction the gaze of the mirror-inverted image (second display model) of another player generated from the second captured image and displayed at a display position specified based on the second position information should be directed.

[0046] Then, by combining the first display model and the second display model after the line of sight correction, the control unit 18 can generate a composite image that includes the player and another player, with the line of sight of the other player directed toward the position of the player. In the example shown in Fig. 2, a composite image M3 can be generated that includes the first display model M1 and the second display model M2 after the line of sight correction, with the line of sight of the mirror-inverted image of player U2 displayed on the second display model M2 directed toward the position of player U1.

[0047] Step S106: The control unit 18 causes the display unit 13 to display the composite image generated in step S105.

[0048] This allows a player to have the experience of seeing another player playing next to them through a mirror (i.e., the display surface of the display unit 13), as if they were watching the other player playing next to them. As a result, the sense of realism of remote piano duets is easily improved.

[0049] In addition, when displaying a composite image on the display unit 13, the control unit 18 of the terminal device 10 may present to the player a composite sound obtained by combining an audio signal (hereinafter also referred to as a "first audio signal") produced by the player's performance of the instrument with an audio signal (hereinafter also referred to as a "second audio signal") produced by another player's performance of the instrument. The "player's performance of the instrument" refers to the player's performance of the instrument 100. The "another player's performance of the instrument" refers to the performance of another instrument 100 by another player. In this case, the control unit 18 may transmit the first audio signal to another terminal device 10 via the communication unit 14, and receive the second audio signal from the other terminal device 10. The composite sound may be presented to the player by outputting it from a speaker of the audio input / output 16, for example. Note that if the instrument played by the player is an instrument that produces performance sounds (e.g., an acoustic piano), only the second audio signal may be output from the speaker without using the composite sound. This allows the player to listen to a performance as if another player were sitting next to them and playing a duet on the musical instrument 100. As a result, the sense of realism of remote duet playing can be further improved.

[0050] In addition, the control unit 18 of the terminal device 10 may operate the player's musical instrument 100 in conjunction with the second captured image and the second audio signal. Specifically, the control unit 18 may move a predetermined key of the keyboard 101 up and down in synchronization with the movement of the second display model generated from the second captured image based on the received performance information. This allows the player to visually recognize the manner in which the keyboard 101 of the musical instrument 100 at hand moves up and down in sync with the performance of the synthesized sound (e.g., dashed line 102 in FIG. 2 ) while viewing the second display model after gaze correction. As a result, the sense of realism of remote piano duet playing is further enhanced.

[0051] Additionally or alternatively, the terminal device 10 may project an image of the playing hands of another player onto the player's instrument 100 using the projection unit 12. In the example shown in Fig. 2, the terminal device 10A may project an image of the playing hands of player U2 onto the instrument 100A of player U1 (for example, at the position indicated by the dashed line 102) using the projection unit 12A. As a result, the sense of realism of playing duets can be further improved.

[0052] In the above description, the system 1 includes a pair of terminal devices 10, but this embodiment also includes a case where the system 1 includes three or more terminal devices 10, that is, a case where there are multiple other terminal devices 10. In addition, in the above description, the terminal devices 10 communicate via the network 20, but this embodiment also includes a case where the terminal devices 10 communicate peer-to-peer.

[0053] As described above, the terminal device 10 according to this embodiment includes an imaging unit 11 that captures an image of a musical instrument player, and a control unit 18 that generates an image to be displayed on the display unit 13. The terminal device 10 further includes a communication unit 14 that communicates with another terminal device 10 used by another player. The control unit 18 of the terminal device 10 transmits a first captured image of the player to the other terminal device 10 via the communication unit 14, and further generates a composite image that includes the player and the other player, with the gaze of the other player directed toward the position of the player, based on the first captured image of the player, the second captured image of the other player, and the display positions on the display unit 13, to be displayed on the display unit 13.

[0054] With this configuration, a composite image is generated that includes the player and another player, with the other player's gaze directed toward the player. For example, by displaying the composite image on the display unit 13 of the terminal device 10 while a player is playing, the player can experience eye contact with the other player playing next to them through the composite image, as if through a mirror. This makes it easier to enhance the sense of realism of a duet.

[0055] Although the present disclosure has been described based on the drawings and examples, it should be noted that those skilled in the art may make various modifications and alterations based on the present disclosure. Therefore, it should be noted that these modifications and alterations are included in the scope of the present disclosure. For example, the functions included in each component or step can be rearranged so as not to be logically inconsistent, and multiple components or steps can be combined or divided into one. [Explanation of symbols]

[0056] 1 System 10 Terminal Equipment 11 Imaging unit 12 Projection section 13 Display section 14 Communications Department 15 Storage section 16 Audio Inputs / Outputs 17 Detection unit 18 Control Unit 20 Network 100 musical instruments 101 Keyboard U1 Player M1 First display model M2 Second display model M3 composite image

Claims

1. an imaging unit that images a user; A terminal device having a control unit that generates an image to be displayed on a display unit, a communication unit that communicates with another terminal device used by another user, The control unit transmitting a first captured image of the user to the other terminal device via the communication unit; A terminal device that generates, based on the first captured image, a second captured image of the other user, location information of the user and the other user acquired by the terminal device, and the display position of the display unit, a composite image that includes a first display model that is a mirror-inverted image of the user and a second display model that is a mirror-inverted image of the other user, and that has been gaze-corrected so that the gaze of the second display model is directed toward the position of the user, for display on the display unit.

2. The terminal device according to claim 1, The user is a player of an instrument, and the other user is a different player. Terminal device.

3. The terminal device according to claim 2, The control unit transmitting a first sound signal generated by the musical instrument performance by the player to the other terminal device via the communication unit; generating a synthesized sound by synthesizing the first sound signal and a second sound signal generated by the musical instrument performance of the other player; a terminal device that presents the synthesized sound to the player when the synthesized image is displayed on the display unit;

4. The terminal device according to claim 3, The control unit operates the instrument played by the player in conjunction with the second captured image and the second sound signal.

5. The terminal device according to claim 2, Further comprising a projection unit, The control unit projects an image of the playing hands of the other player onto the instrument of the other player using the projection unit.

6. 3. The terminal device according to claim 2, wherein one of the musical instruments is played jointly by the player and the other player.

7. an imaging unit that images a user; a terminal device having a control unit that generates an image to be displayed on a display unit; another terminal device used by another user and communicating with the terminal device; A control method for an information processing system comprising: a step of transmitting the first captured image to the other terminal device via a communication unit by the control unit; and displaying, on the display unit, a composite image that includes a first display model that is a mirror-inverted image of the user and a second display model that is a mirror-inverted image of the other user, the composite image having been subjected to gaze correction so that the gaze of the second display model is directed toward the position of the user, based on the first captured image of the user, the second captured image of the other user, position information of the user and the other user acquired by the terminal device, and the display position of the display unit. Control method.

8. A program for causing a computer to execute the control method for an information processing system described in claim 7.

Images