Terminal equipment

Abstract

To improve the sense of unity and realism during remote sessions.SOLUTION: A terminal device includes an imaging unit that images an instrument player and the audience at the venue, and a control unit that generates images, and further includes a communication unit that communicates with another terminal device used by another player at another venue, and the control unit transmits the captured image of the player, the captured image of the audience at the venue, and information on the audience's focus area to the other terminal device via the communication unit, and generates an image of the other player and an image of the other audience member directing his or her gaze at the focus area on the basis of the captured image of the other player, the captured image of another audience member at the other venue, and information on the audience's focus area for display.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a terminal device. [Background technology]

[0002] Conventionally, there are known techniques related to remote music sessions. For example, Patent Document 1 discloses a technique for controlling the timing of imaging a performer and the timing of displaying an audience image to differ from each other. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2022 / 209130 Summary of the Invention [Problem to be solved by the invention]

[0004] In remote sessions, there is room to improve the sense of unity and realism of the session.

[0005] The objective of the present disclosure, made in consideration of the above circumstances, is to enable an improvement in the sense of unity and realism during remote sessions. [Means for solving the problem]

[0006] A terminal device according to one embodiment of the present disclosure has an imaging unit that images instrument players and spectators at a venue, and a control unit that generates images, and further includes a communication unit that communicates with another terminal device used by another instrument player at another venue, and the control unit transmits the captured image of the instrument player, the captured image of the spectators at the venue, and information about the spectators' focus points to the other terminal device via the communication unit, and generates, based on the captured image of the other instrument player, the captured image of another spectator at the other venue, and information about the spectators' focus points, an image of the other instrument player and an image of the other spectator directing his or her gaze at the focus point for display. Effect of the Invention

[0007] According to one embodiment of the present disclosure, it is possible to improve the sense of unity and realism in a remote session. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram illustrating an example configuration of a system according to an embodiment of the present disclosure. [Figure 2A] 13 is a flowchart showing an example of the operation of the terminal device. [Figure 2B] 13 is a flowchart showing an example of the operation of the terminal device. [Diagram 3] FIG. 1 illustrates an example of how the system is used. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0010] <Overview of the embodiment> With reference to FIG. 1, a configuration example of a system 1 according to an embodiment of the present disclosure will be described. The system 1 includes at least a pair of terminal devices 10 and a server 20. The terminal devices 10 are connected to each other so as to be able to communicate information with each other via the server 20, which is connected to a network 30 including, for example, the Internet and a mobile communication network. The server 20 belongs to, for example, a cloud computing system or other computing system, and implements various functions. The terminal devices 10 transmit and receive information regarding musical instrument performances, players, and audience members. A user of the terminal device 10 uses the terminal device 10 to perform a music session (hereinafter also referred to as a "remote session") via the network 30 at a venue such as a concert hall that accommodates an audience. In this embodiment, the system 1 includes two terminal devices 10, but may include any number of terminal devices 10 depending on the number of players or venues that perform the remote session.

[0011] The terminal device 10 of this embodiment has an imaging unit 11 that images the instrument players and the audience at the venue, and a control unit 18 that generates images. The terminal device 10 further includes a communication unit 14 that communicates with another terminal device 10 used by another player at another venue. The control unit 18 of the terminal device 10 transmits the captured images of the players, the captured images of the audience at the venue, and information on the audience's focus target to the other terminal device 10 via the communication unit 14. Based on the captured images of the other player, the captured images of the other audience at the other venue, and information on the audience's focus target, the control unit 18 generates for display an image of the other player and an image of the other audience directing their gaze towards the focus target.

[0012] In this way, according to this embodiment, by displaying each image during a performance, the performer can experience as if he or she were receiving the gaze of an audience member in another venue and making eye contact with them, which makes it easier to improve the sense of unity and realism in a remote session.

[0013] Next, each component of the system 1 will be described in detail.

[0014] <Terminal device configuration> As shown in FIG. 1, the terminal device 10 includes an imaging unit 11, a projection unit 12, a display unit 13, a communication unit 14, a storage unit 15, an audio input / output 16, a detection unit 17, and a control unit 18.

[0015] The imaging unit 11 includes a camera that captures an image of a subject using visible light, and a distance sensor that measures the distance to the subject to obtain a distance image. The camera captures an image of the subject at, for example, 30 frames per second to generate a moving image consisting of continuous captured images. The distance sensor includes a ToF (Time Of Flight) camera, a LiDAR (Light Detection And Ranging), and a stereo camera, and generates a distance image of the subject including distance information. The distance sensor can obtain information such as the position (spatial coordinates, etc.) of the subject captured by the imaging unit 11 and the relative position of the subject with respect to the display unit 13. The imaging unit 11 sends the captured image and the distance image to the control unit 18. In one embodiment, the imaging unit 11 includes a player imaging unit for capturing an image of a player, and an audience imaging unit for capturing an image of an audience. The terminal device 10 captures an image of a player of an instrument using the terminal device 10 and the audience as subjects using the player imaging unit and the audience imaging unit of the imaging unit 11, and obtains captured images of the player and the audience. For example, the control unit 18 can obtain information such as the pupil position, gaze point, and line of sight of the subject by analyzing the obtained captured image.

[0016] The projection unit 12 is a projector of any type, such as an LCD (liquid crystal display) projector, an LCOS (liquid crystal on silicon) projector, or a DLP (digital light processing) projector. The projector includes an image display element, a light source, and a projection lens. For example, a liquid crystal is used as the image display element in an LCD or LCOS projector, and a DMD (Digital Micromirror Device) is used in a DLP projector. The light source irradiates the image display element with projection light, and projects an image on a projection surface through the projection unit 12. The light source may irradiate illumination light in addition to the projection light. The irradiation range of the projection light and the irradiation range of the illumination light may be the same or different. The projection lens displays the image of the image display element on the projection surface by enlarging or reducing it at an arbitrary magnification. In one embodiment, the projection unit 12 includes a player projection unit for projecting a captured image of the player, and an audience projection unit for projecting a captured image of the audience. The terminal device 10 can use the player projection section or audience projection section of the projection unit 12 to project captured images of the instrument player or the audience onto any projection surface (for example, the screen of the display unit 13) at any magnification.

[0017] The display unit 13 is disposed at an arbitrary position where the user of the terminal device 10 can easily see. For example, when the musical instrument is a piano, the easily visible position may be the top of the piano or a seating position where a session partner is expected to sit for the player display unit described later, and a predetermined area of ​​the audience seats for the audience display unit described later. The display unit 13 is, for example, an LCD or an organic EL (Electro Luminescence) display, or a reflective or transmissive screen. The display unit 13 displays data obtained by the operation of the terminal device 10, for example, an image acquired by the imaging unit 11. The display unit 13 may be connected to the terminal device 10 as an external output device instead of being provided in the terminal device 10. For example, an interface compatible with a standard such as USB (Universal Serial Bus) or Bluetooth (registered trademark) may be used as the connection interface. In one embodiment, the display unit 13 includes a player display unit for displaying a captured image of the player and an audience display unit for displaying a captured image of the audience. The terminal device 10 causes the player display section or the audience display section of the display section 13 to display captured images of the instrument player or the audience, and images generated based on the captured images.

[0018] The communication unit 14 includes one or more communication interfaces. The communication interfaces include an interface compatible with a wired or wireless LAN standard, an interface compatible with a mobile communication standard such as LTE (Long Term Evolution), 4G (4th Generation), or 5G (5th Generation), etc. The communication unit 14 receives information used in the operation of the terminal device 10 and transmits information obtained by the operation of the terminal device 10. The terminal device 10 is connected to a network 30 via a nearby router device or a mobile communication base station by the communication unit 14, and communicates with an external device such as another terminal device 10 via the network 30. In one embodiment, the terminal device 10 communicates with another terminal device 10 used by another player and the server 20 using the communication unit 14.

[0019] The storage unit 15 includes, for example, one or more semiconductor memories, one or more magnetic memories, one or more optical memories, or a combination of at least two of them, functioning as a main storage device, an auxiliary storage device, or a cache memory. The semiconductor memory is, for example, a random access memory (RAM) or a read only memory (ROM). The RAM is, for example, a static RAM (SRAM) or a dynamic RAM (DRAM). The ROM is, for example, an electrically erasable programmable ROM (EEPROM). The storage unit 15 stores information used in the operation of the terminal device 10 and information obtained by the operation of the terminal device 10.

[0020] The audio input / output 16 includes a microphone for accepting acoustic and voice input, and a speaker. In one embodiment, the audio input / output 16 provides an audio interface between a musical instrument player and the terminal device 10. The audio input / output 16 receives audio data from a musical instrument played by the player, converts the audio data into electrical signals, and transmits the electrical signals to a speaker. The speaker converts the electrical signals into sound waves audible to humans. The audio input / output 16 may include a headset jack. The headset jack provides an interface between a removable audio input / output peripheral, such as a headset having headphones and / or a microphone.

[0021] The detection unit 17 has any sensor module capable of acquiring data on the movement of a person's eyes. The sensor module includes, for example, an eye tracking camera, a near-infrared LED (Infrared Emitting Diode), a laser sensor, a photoelectric sensor, an ultrasonic sensor, or a combination thereof. However, the detection unit 17 is not limited to these examples, and may have any other sensor module that detects the state or operation of each part of the terminal device 10, or an interface with each sensor module. The detection unit 17 sends information indicating the detection result by the sensor module to the control unit 18. In one embodiment, the detection unit 17 includes a player detection unit that acquires data on the movement of the player's eyes, and an audience detection unit that acquires data on the movement of the audience's eyes. In addition to or instead of analyzing the captured image acquired using the above-mentioned imaging unit 11, the control unit 18 processes the data information sent from the detection unit 17 to acquire information such as the position of the pupils, the gaze point, and the line of sight of each person of the player or the audience.

[0022] The control unit 18 includes one or more processors, one or more programmable circuits, one or more dedicated circuits, or a combination thereof. The processor is, for example, but not limited to, a general-purpose processor such as a central processing unit (CPU) or a graphics processing unit (GPU), or a dedicated processor specialized for a specific process. The programmable circuit is, for example, but not limited to, a field-programmable gate array (FPGA). The dedicated circuit is, for example, but not limited to, an application specific integrated circuit (ASIC). The control unit 18 controls the operation of the entire terminal device 10. In one embodiment, the terminal device 10 generates images of the performers and the audience using the control unit 18.

[0023] The functions of the terminal device 10 are realized by a processor included in the control unit 18 executing a control program. The control program is a program for causing a computer to function as the terminal device 10. Some or all of the functions of the terminal device 10 may be realized by a dedicated circuit included in the control unit 18. The control program may be stored in a non-transitory recording or storage medium readable by the terminal device 10, and read by the terminal device 10 from the medium.

[0024] <Server configuration> As shown in FIG. 1, the server 20 includes a control unit 21, a communication unit 22, and a storage unit .

[0025] The control unit 21 includes one or more processors, one or more programmable circuits, one or more dedicated circuits, or a combination of these. The control unit 21 controls the operation of the entire server 20.

[0026] The communication unit 22 includes one or more communication interfaces connected to the network 30. The communication interface corresponds to, for example, a mobile communication standard, a wired LAN (Local Area Network) standard, or a wireless LAN standard, but is not limited thereto, and may correspond to any communication standard. In this embodiment, the server 20 communicates with each terminal device 10 via the communication unit 22 and the network 30.

[0027] The storage unit 23 includes one or more memories. Each memory included in the storage unit 23 may function as, for example, a main storage device, an auxiliary storage device, or a cache memory. The storage unit 23 stores any information used in the operation of the server 20. For example, the storage unit 23 may store a system program, an application program, a database, map information, music concert information, and the like. The information stored in the storage unit 23 may be updatable with information acquired from the network 30 via the communication unit 22, for example.

[0028] The functions of the server 20 are realized by a processor included in the control unit 21 executing a control program. The control program is a program for causing a computer to function as the server 20. Also, some or all of the functions of the server 20 may be realized by a dedicated circuit included in the control unit 21. Also, the control program may be stored in a non-transitory recording or storage medium readable by the server 20, and the server 20 may read it from the medium.

[0029] <Operation of terminal device> The operation of the terminal device 10 according to the present embodiment will be described with reference to FIG. 2A, FIG. 2B, and FIG. 3. As shown in the flowcharts of FIG. 2A and FIG. 2B, each terminal device 10 mutually transmits and receives captured images of the players and the audience and information on the target of attention, thereby generating display models of the players and the audience, and generates images of the players and the audience directing their gaze to the target of attention from the display models. Each terminal device 10 acquires captured images of the players and the audience and information on the target of attention while displaying images of other players and the audience in other venues on the display unit 13. Such an operation is executed by each terminal device 10 when a remote session is performed between a player U1 in a venue D1 and a player U2 in another venue D2, as shown in FIG. 3. For example, a situation is shown in which a player U1 of a musical instrument 100A using a terminal device 10 in a venue D1 and a player U2 of a musical instrument 100C using another terminal device 10 in another venue D2 perform a remote session in front of the audience at each venue. In the following, in order to distinguish the terminal device 10 on the venue side from the terminal device 10 on the other venue side, they are also referred to as terminal device 10A and terminal device 10B, respectively, for convenience. Similarly, the components of each terminal device are also distinguished by adding a sub-number "A" or "B" to the reference number as necessary. Also, the musical instruments 100A-100D are also collectively referred to as "musical instruments 100." The musical instruments 100A-100D may be electronic musical instruments that generate electronic sounds. The musical instruments 100A-100D are pianos in the example shown in FIG. 3, but are not limited thereto and may be any other musical instruments such as other keyboard instruments (e.g., organs) or string instruments (e.g., violins), or a combination of these.

[0030] Fig. 2A shows a process for transmitting captured images of performers and spectators at one venue and information on a target object from one terminal device 10 to another terminal device 10. Fig. 2B shows a process for generating an image based on captured images of another performer and spectator at another venue and information on a target object received from another terminal device 10, and displaying or projecting the image on the display unit 13. Note that the processes of the respective steps in Figs. 2A and 2B are executed in parallel with each other. Here, an example will be described using terminal device 10A at venue D1 shown in Fig. 3.

[0031] The operation procedure of the terminal device 10A in FIG. 2A is as follows.

[0032] In step S100, the control unit 18A of the terminal device 10A transmits captured images of the player U1 and audience G1 at the venue D1 to another terminal device 10B via the communication unit 14A. Specifically, the control unit 18A of the terminal device 10A captures the figure of the player U1 playing the musical instrument 100A at the venue D1 and the audience G1 using the player imaging unit and audience imaging unit of the imaging unit 11A, respectively. The captured images of the player U1 and the audience G1 obtained by imaging are, for example, moving images consisting of a plurality of frames of images. The control unit 18A transmits the captured images of the player U1 and the audience G1 to the communication unit 14B of the other terminal device 10B via the communication unit 14A, the network 30, and the server 20.

[0033] Furthermore, the control unit 18A of the terminal device 10A may convert audio data received from the musical instrument 100A played by the player U1 via the audio input / output 16A into an electric signal and acquire it as an audio signal, in addition to the captured image of the player U1. The audio signal is, for example, performance information such as MIDI (Musical Instruments Digital Interface). The control unit 18A may collect the sound of the musical instrument 100A being played and / or the voice of the player U1 using a microphone of the audio input / output 16. Furthermore, the control unit 18A may collect the voice emitted by the audience G1 using a microphone of the audio input / output 16, in addition to the captured image of the audience G1, convert it into an electric signal, and acquire it as an audio signal. The control unit 18A may transmit information such as the audio signal or audio signal acquired in this manner to another terminal device 10B, together with the captured image of the player U1 and the captured image of the audience G1.

[0034] In step S101, the control unit 18A of the terminal device 10A transmits information on the focus of the player U1 and the audience G1 in the venue D1 to another terminal device 10B via the communication unit 14A. Specifically, the control unit 18A acquires information on the subjects in the venue D1, that is, the gaze point of the player U1 and the gaze point of the audience G1. For example, the control unit 18A derives or calculates the position of the player U1 relative to the display unit 13A (the player display unit 131 and the audience display unit 132), the position of the audience G1 relative to the display unit 13A (the player display unit 131 and the audience display unit 132), and the gaze points of the player U1 and the audience G1 by analyzing the captured images of the player U1 and the audience G1 in the venue D1. The position of each subject and the position of the gaze point are specified by the spatial coordinates of the respective reference points. The reference points are, for example, the centers of both eyes of each subject. The gaze point is, for example, a point (focus) where the lines of sight of both eyes of each person, the player U1 or the spectator G1, intersect. The control unit 18A recognizes each subject and both eyes by image processing such as pattern matching on the captured image, detects position information of at least one of the pupils and irises of the left and right eyes, and detects or calculates the gaze point based on the position information. The position information includes the convergence angle of both eyes. Furthermore, the control unit 18A derives, for example, the spatial coordinates of both eyes of each person, the player U1 and the spectator G1, relative to the position of the camera of the imaging unit 11A based on the distance image. The control unit 18A derives the spatial coordinates of the center based on the spatial coordinates of the both eyes, derives the lines of sight of both eyes based on the spatial coordinates of the both eyes and the convergence angle, and derives the spatial coordinates of the position where the lines of sight intersect as the gaze point.

[0035] In addition, information on the gaze points of each of the performers and spectators may be obtained by analyzing data information sent from the eye tracking cameras of the performer detection section and spectator detection section of the detection unit 17 in addition to or instead of image analysis.

[0036] The control unit 18A specifies the target of attention for each of the player U1 and the audience G1 based on the information on the gaze point. For example, when the length of time that the gaze point of each person of the player U1 or the audience G1 is fixed continuously or intermittently during a predetermined period is longer than a reference, the target on which the gaze point is fixed is specified as the target of attention. Furthermore, when specifying the target of attention, the control unit 18A may take into consideration the smallness of the change in the position of the gaze point or the largeness of the change in the pupil diameter, or both, in addition to the length of the gaze time during the predetermined period. The target of attention of the player U1 specified in this way is an image of another player U2 (display model M2 described later), an audience G1, or an image of another audience G2 (display model N2 described later) at another venue D2. Furthermore, the target of attention of the audience G1 is an image of the player U1 or an image of another player U2 (display model M2 described later). The control unit 18 transmits information specifying the target of attention of each of the other player U2 or the audience G2 as information on the target of attention to another terminal device 10B.

[0037] The operation procedure of the terminal device 10A in FIG. 2B is as follows.

[0038] In step S200, the control unit 18A of the terminal device 10A receives captured images of another player U2 and audience G2 in another venue D2 from another terminal device 10B via the communication unit 14A. Specifically, the same process as in step S100 is executed in the terminal device 10B. That is, the control unit 18B captures images of the player U2 and audience G2 playing the musical instrument 100C in the venue D2 using the player imaging unit and audience imaging unit of the imaging unit 11B, respectively, and transmits the captured images of the player U2 and audience G2 to the terminal device 10A in the venue D1 via the communication unit 14B. The control unit 18A receives the captured images of the player U2 and audience G2 in the venue D2 via the communication unit 14A.

[0039] Furthermore, as described in step S100, the terminal device 10A may receive information such as audio signals or voice signals of the player U2 and the audience G2 together with the captured images of the player U2 and the audience G2 from the terminal device 10B.

[0040] In step S201, the control unit 18A receives information on the attention target of the player U2 and the audience G2 from the terminal device 10B via the communication unit 14A. This procedure corresponds to the execution of a process similar to step S101 in another venue D2. That is, the control unit 18B of the terminal device 10B obtains information on the gaze point of the player U2 by analyzing the captured image of the player U2, and obtains information on the gaze point of the audience G2 by analyzing the captured image of the audience G2. The control unit 18B identifies a display model that is relatively more attention-grabbing for each of the player U2 and the audience G2, based on the information on the gaze points for the player display unit 133 and the audience display unit 134. The control unit 18B identifies the subject corresponding to the identified display model, i.e., the player U1 or the audience G1, as the attention target of each person of the player U2 or the audience G2. The control unit 18B transmits information identifying the player U1 or spectator G1 identified as the attention target to the terminal device 10A as information on the attention target of the player U2 or spectator G2. For example, when an image of the player U1 (display model M1 described below) displayed on the display unit 133 is identified as the attention target of each person in the spectator G2, the control unit 18A receives information identifying the player U1 from another terminal device 10B as information on the attention target of the spectator G2.

[0041] In step S202, the control unit 18A generates a display image in which another player U2 and another spectator G2 direct their gaze toward their respective targets of interest based on the captured image received in step S200 and the information about the target of interest received in step S201.

[0042] Specifically, the control unit 18A performs gaze correction on the captured image of the player U2 based on the information of the target of attention of the player U2, thereby generating a display model in which the player U2 directs his / her gaze to the target of attention. For example, when the player U2 shifts his / her gaze from the musical instrument 100C to the display model M1, and the target of attention of the player U2 changes from the musical instrument 100C to the display model M1, the control unit 18A of the terminal device 10A performs gaze correction on the captured image of the player U2, thereby generating a display model M2 in which the player U2 directs his / her gaze to the player U1. This allows the player U1 to experience as if he / she were making eye contact in response to receiving the gaze of the player U2 in another venue D2.

[0043] Furthermore, the control unit 18A performs gaze correction on the captured image of the spectator G2 based on the information on the target of the spectator G2, thereby generating a display model in which the spectator G2 directs his / her gaze to the target of the spectator G2. For example, when the spectator G2 shifts his / her gaze from the performer U2 to the display model M1, and the target of the spectator G2 changes from the performer U2 to the display model M1, the control unit 18A of the terminal device 10A performs gaze correction on the captured image of the spectator G2, thereby generating a display model N2 in which the spectator G2 directs his / her gaze to the performer U1. This allows the performer U1 to have the experience of making eye contact in response to receiving the gaze of the spectator G2 in another venue D2.

[0044] In this embodiment, the relative position between the player U1 and the player display section 131 or the spectator display section 132 of the display section 13A of the terminal device 10A is specified, so the control section 18A can determine in which direction the line of sight of the captured image of the player U2 or the spectator G2 displayed or projected on the player display section 131 or the spectator display section 132 of the display section 13A should be directed based on the specified position, and perform line of sight correction. A known method can be used as a method of line of sight correction. For example, the control section 18A can perform line of sight correction based on information on the relative position with respect to the player display section 131 or the spectator display section 132 acquired via a distance measuring sensor of the player imaging section or the spectator imaging section of the imaging section 11A.

[0045] A similar process is executed at another venue D2. That is, in the example shown in Fig. 3, a display model M1 is generated in which a performer U1 directs his / her gaze to the target of attention, and a display model N1 is generated in which a spectator G1 directs his / her gaze to the target of attention.

[0046] In the above explanation, the line of sight correction is performed on the display models of both the performer and the spectator, but the scope of this embodiment also includes the case where the line of sight correction is performed on the display model of the spectator only.

[0047] In step S203, the control unit 18A displays a display image including the display model generated in step S202. Specifically, the control unit 18A displays a display model M2 of the player U2 on the player display unit 131 of the display unit 13A. The display model M2 may be projected onto the screen of the player display unit 131 by the player projection unit of the projection unit 12A. The control unit 18A also displays a display model N2 of the audience G2 on the audience display unit 132 of the display unit 13A. The display model N2 may be projected onto the screen of the audience display unit 132 by the audience projection unit of the projection unit 12A. This allows the player U1 to experience as if he or she were receiving gazes from the player U2 or the audience G2 and making eye contact through the display models M2 and N2. Similar processing is also executed in another venue D2. In other words, the player U2 can experience as if he or she were receiving the gaze of the player U1 or the audience G1 and making eye contact with them through the display models M1 and N1.

[0048] As described above in steps S100 and S200, when information such as audio signals or voice signals of the players and audience at each venue is transmitted and received between the terminal devices 10, the audio signals or voice signals may be output from the speakers of the audio input / output 16. In the example shown in FIG. 3, audio signals of the sounds of the musical instruments 100A and 100C, audio signals of the voices of the players U1 and U2, or audio signals of the audience's voices collected from the microphones 161 and 163 of the audio input / output 16 are transmitted and received between the terminal devices 10A and 10B. In this case, the audio signals or voice signals are output from the speakers 162 and 164. This allows each player to experience as if they were having a session in the same space in front of an audience including the audience at another venue, while listening to the performance and singing of another player at another venue. This makes it easier to further improve the sense of unity and realism in a remote session.

[0049] In the above description, the system 1 includes two terminal devices 10, but the present embodiment also includes a case where the system 1 includes three or more terminal devices 10. In addition, in the above description, the terminal devices 10 communicate with each other via the network 30, but the present embodiment also includes a case where the terminal devices 10 communicate with each other in a peer-to-peer manner.

[0050] As described above, the terminal device 10 according to this embodiment has an imaging unit 11 that images the instrument player and the audience at the venue, and a control unit 18 that generates images. The terminal device 10 further includes a communication unit 14 that communicates with another terminal device 10 used by another player at another venue. The control unit 18 of the terminal device 10 transmits the captured image of the player, the captured image of the audience at the venue, and information on the audience's focus target to the other terminal device 10 via the communication unit 14. Based on the captured image of the other player, the captured image of another audience at the other venue, and information on the audience's focus target, the control unit 18 generates an image of the other player and an image of the other audience directing their gaze to the focus target for display.

[0051] With this configuration, by displaying each image during a performance, the performer can experience as if he or she were receiving the gaze of an audience member in another venue and making eye contact with them, which makes it easier to improve the sense of unity and realism in a remote session.

[0052] Although the present disclosure has been described based on the drawings and examples, it should be noted that those skilled in the art may make various modifications and alterations based on the present disclosure. Therefore, it should be noted that these modifications and alterations are included in the scope of the present disclosure. For example, the functions included in each component or step can be rearranged so as not to be logically inconsistent, and multiple components or steps can be combined or divided into one. [Explanation of symbols]

[0053] 1 System 10 Terminal Equipment 11 Imaging unit 12 Projection section 13 Display section 14 Communications Department 15 Storage section 16 Audio Inputs / Outputs 17 Detection unit 18 Control Unit 20 Servers 21 Control section 22 Communications Department 23 Memory section 30 Network 100A, 100B, 100C, 100D Musical Instruments 111,113 Player Imaging Section 112,114 Spectator Imaging Section 131,133 Player display section 132,134 Spectator Display Section 161,163 Microphones 162,164 speakers D1 Venue D2 Another venue U1 Player U2 Another Player G1 Spectators G2 Another audience M1 Player Display Model M2 Model for displaying different players N1 Spectator display model N2 Another model for viewing by spectators

Claims

[Claim 1] an imaging unit for imaging instrument players and audience members at a venue; A terminal device having a control unit for generating an image, a communication unit for communicating with another terminal device used by another player at another venue, The control unit is transmitting a captured image of the performer, a captured image of an audience at the venue, and information on an object of attention of the audience to the other terminal device via the communication unit; A terminal device generates, based on an image of the other performer, an image of another spectator at the other venue, and information on the object of attention of the other spectator, to display an image of the other performer and an image of the other spectator directing his or her gaze toward the object of attention.

Images