Image generation method and system

Abstract

To suppress deterioration of a communication speed while providing a real communication.SOLUTION: An image generation method includes the steps of: displaying a model image of a second user of a second terminal device by a first terminal device; and determining whether or not a visual line of a first user of the first terminal device is directed to the model image of a second user to be displayed. The image generation method contains a step of generating the model image of the first user on the basis of an imaging image of the first user received from the first terminal device by the second terminal device in the case where it is determined that the visual line of the first user is directed to the model image of the second user to be displayed by the first terminal device. The image generation method includes a step of generating the model image of the first user on the basis of data of the first user previously acquired by the second terminal device in the case where it is determined that the visual line of the first user is not directed to the model image of the second user to be displayed by the first terminal device.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to image generation methods and systems. [Background technology]

[0002] Conventionally, there is known a technique for displaying an avatar representing another user on a terminal device. Among such techniques, a technique for displaying a real face image of the other user by pasting it onto the avatar in order to provide realistic communication is known (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2001-16563 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the conventional technology, in order to constantly attach and display the real face images of other users to an avatar, it is necessary to continuously receive the real face image data of other users from other terminal devices. If the real face image data of other users is continuously received, the amount of communication between the terminal devices increases, and the communication speed decreases. If the communication speed decreases, users cannot smoothly communicate with each other in real life.

[0005] In consideration of the above, an object of the present disclosure is to prevent a decrease in communication speed while providing real communication. [Means for solving the problem]

[0006] An image generating method according to an embodiment of the present disclosure includes: An image generating method executed by a first terminal device and a second terminal device, displaying, by the first terminal device, a model image of the second user of the second terminal device, and determining whether or not a line of sight of the first user of the first terminal device is directed toward the displayed model image of the second user; generating, by the second terminal device, a model image of the first user based on a captured image of the first user received from the first terminal device, when the first terminal device determines that a line of sight of the first user is directed toward the displayed model image of the second user; When the first terminal device determines that the line of sight of the first user is not directed toward the displayed model image of the second user, generating a model image of the first user based on data of the first user previously acquired by the second terminal device.

[0007] In one embodiment of the present disclosure, the system includes: A first terminal device and a second terminal device are included, the first terminal device displays a model image of a second user of the second terminal device, and determines whether or not a line of sight of the first user of the first terminal device is directed toward the displayed model image of the second user; When the first terminal device determines that the line of sight of the first user is directed toward the displayed model image of the second user, the second terminal device generates a model image of the first user based on the captured image of the first user received from the first terminal device; If the first terminal device determines that the first user's gaze is not directed toward the displayed model image of the second user, the second terminal device generates a model image of the first user based on data of the first user that has been previously acquired. Effect of the Invention

[0008] According to an embodiment of the present disclosure, it is possible to suppress a decrease in communication speed while providing realistic communication. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating a schematic configuration of a system according to an embodiment of the present disclosure. [Diagram 2] FIG. 13 is a diagram for explaining an example of an event. [Diagram 3] FIG. 13 is a diagram for explaining an example of an event. [Figure 4] FIG. 2 is a block diagram of the system shown in FIG. [Diagram 5] 10 is a flowchart showing an example of the operation of the first terminal device. [Figure 6] 10 is a flowchart showing an example of the operation of the second terminal device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0011] As shown in Fig. 1, a system 1 according to this embodiment includes a server device 10 and terminal devices 20A, 20B, 20C, 20D, and 20E. Hereinafter, when the terminal devices 20A to 20E are not particularly distinguished from each other, they are referred to as "terminal devices 20". The system 1 includes five terminal devices 20. However, the system 1 only needs to include at least two terminal devices 20.

[0012] The system 1 is a system for providing an event in which a user can participate using a terminal device 20. The event is, for example, an online class.

[0013] The server device 10 and the terminal device 20 are capable of communicating with each other via a network 2. The network 2 may be any network including a mobile communication network, the Internet, and the like.

[0014] The server device 10 executes transmission and reception of information and information processing required for providing an event. For example, the server device 10 relays communications between multiple terminal devices 20 during the event.

[0015] The server device 10 is, for example, a dedicated computer configured to function as a server, a general-purpose personal computer, or a cloud computing system.

[0016] The terminal device 20 is, for example, a desktop personal computer (PC), a tablet PC, a notebook PC, a smartphone, or the like.

[0017] The terminal devices 20A, 20B, 20C, 20D, and 20E are used by the users 3A, 3B, 3C, 3D, and 3E, respectively. The users 3A to 3E use the terminal devices 20A to 20E, respectively, to participate in an event. For example, FIG. 2 shows a state in which the user 3B participates in an event using the terminal device 20B. The model images 4A, 4C, 4D, and 4E are displayed on the terminal device 20B. The model images 4A, 4C, 4D, and 4E are model images of the users 3A, 3C, 3D, and 3E, respectively. Here, in an event, one of the users 3A to 3E becomes a fixed user. A fixed user is a user whose model image display position is fixed. For example, a teacher or a student who speaks in an online class becomes a fixed user. In FIG. 2, the user 3A is a fixed user. Therefore, the display position of the model image 4A of the user 3A is fixed. The model image of the fixed user is fixed, for example, at the center position of the display unit 24. The model images of the other users are arranged so as to surround the model image of the fixed user. The model images of the other users are arranged so as to direct the gaze of the model image of the fixed user. The model image of the fixed user may be displayed larger than the model images of the other users. The model image of the user may be an image including at least the head of the user. By including the user's head in the model image of the user, multiple users can communicate while recognizing each other's facial expressions. The model image of the user may be an image including the user's neck and the upper body of the user in addition to the user's head.

[0018] Any two of the terminal devices 20A to 20E transmit and receive captured images of users in order to generate and display model images of the users. The two terminal devices 20 transmitting and receiving captured images of users are described as a "first terminal device" and a "second terminal device." The user of the first terminal device is described as a "first user." The user of the second terminal device is described as a "second user." The first terminal device generates a model image of the second user and displays the generated model image of the second user. The second terminal device generates a model image of the first user and displays the generated model image of the first user.

[0019] Here, when the first terminal device displays the model image of the second user, it judges whether the first user's gaze is directed to the displayed model image of the second user. For example, in FIG. 2, it is assumed that the terminal device 20B is the first terminal device, the user 3B is the first user, and the model image of the second user is the model image 4C of the user 3C. In this case, the terminal device 20B judges whether the gaze of the user 3B is directed to the model image 4C.

[0020] When the first terminal device determines that the line of sight of the first user is directed to the displayed model image of the second user, the first terminal device transmits data of the captured image of the first user to the second terminal device. For example, in FIG. 2, when the terminal device 20B, which is the first terminal device, determines that the line of sight of the user 3B, which is the first user, is directed to the model image 4C of the second user, the terminal device 20B transmits data of the captured image of the user 3B to the terminal device 20C, which is the second terminal device. In this case, the terminal device 20C receives data of the captured image of the user 3B. As shown in FIG. 3, the terminal device 20C, which is the second terminal device, generates and displays a model image 4B of the user 3B based on the received captured image of the user 3B. When the line of sight of the user 3B, which is the first user, is directed to the model image 4C in FIG. 2, as shown in FIG. 3, the model image 4B of the user 3B, which is the first user, is an image in which the displayed face is directed toward the user 3C, which is the second user. By displaying such a model image of the first user on the second terminal device, it is possible to provide realistic communication.

[0021] When the first terminal device determines that the line of sight of the first user is not directed to the displayed model image of the second user, the first terminal device does not transmit the captured image data of the first user to the second terminal device. For example, in FIG. 2, when the terminal device 20B, which is the first terminal device, determines that the line of sight of the user 3B is not directed to the model image 4C of the second user, the terminal device 20B does not transmit the captured image data of the user 3B to the terminal device 20C, which is the second terminal device. In this case, the terminal device 20C generates a model image of the user 3B, which is the first user, based on the data of the user 3B acquired in advance. The data of the user 3B acquired in advance is, for example, the data of the captured image of the user 3B received in advance. The terminal device 20C generates an image of the back of the user's head or the side of the face as the model image of the user 3B, as described later. Here, when the line of sight of the user 3B is not directed to the model image 4C, the user 3B and the user 3C are not communicating with each other. Therefore, even if an image of the back of the user 3B's head or the like is displayed, there is no impediment to communication between the user 3B and the user 3C. In addition, generally, an image of the back of the user's head is not as complicated as an image of the user's face. Therefore, an image of the back of the user's head can be easily created based on the data of the captured image of the user that has been received in advance. Furthermore, by not transmitting the captured image of the first user to the second terminal device, the amount of communication between the terminal devices 20 can be reduced.

[0022] (Server device configuration) As shown in FIG. 4, the server device 10 includes a communication unit 11, a storage unit 12, and a control unit 13.

[0023] The communication unit 11 includes at least one communication module that can be connected to the network 2. The communication module is, for example, a communication module that complies with standards such as a wired LAN (Local Area Network) or a wireless LAN. The communication unit 11 is connected to the network 2 via the wired LAN or wireless LAN by the communication module.

[0024] The storage unit 12 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of at least two of these. The semiconductor memory is, for example, a random access memory (RAM) or a read only memory (ROM). The RAM is, for example, a static random access memory (SRAM) or a dynamic random access memory (DRAM). The ROM is, for example, an electrically erasable programmable read only memory (EEPROM). The storage unit 12 may function as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 12 stores data used in the operation of the server device 10 and data obtained by the operation of the server device 10.

[0025] The control unit 13 includes at least one processor, at least one dedicated circuit, or a combination of these. The processor is a general-purpose processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), or a dedicated processor specialized for a specific process. The dedicated circuit is, for example, an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). The control unit 13 executes processes related to the operation of the server device 10 while controlling each part of the server device 10.

[0026] (Terminal Device Configuration) As shown in FIG. 4, the terminal device 20 includes a communication unit 21, an input unit 22, a speaker unit 23, a display unit 24, an imaging unit 25, a storage unit 26, and a control unit 27.

[0027] The communication unit 21 is configured to include at least one communication module connectable to the network 2. The communication module is, for example, a communication module compatible with standards such as a wired LAN (Local Area Network) or a wireless LAN, or a communication module compatible with mobile communication standards such as LTE (Long Term Evolution), 4G (4th Generation), or 5G (5th Generation).

[0028] The input unit 22 can receive input from a user. The input unit 22 includes at least one input interface capable of receiving input from a user. The input interface is, for example, a physical key, a capacitive key, a pointing device, a touch screen or a microphone that is integral with the display of the display unit 24, or the like.

[0029] The speaker unit 23 is capable of outputting sound. The speaker unit 23 may be configured to include any number of speakers.

[0030] The display unit 24 is capable of displaying data. The display unit 24 includes, for example, a display. The display is, for example, a Liquid Crystal Display (LCD) or an organic EL (Electro Luminescence) display.

[0031] The imaging unit 25 can capture an image of a subject and generate a captured image. The imaging unit 25 is, for example, a visible light camera. The imaging unit 25 continuously captures images of the subject at, for example, an arbitrary frame rate. The captured image is a color image (RGB image). However, the captured image may be a monochrome image.

[0032] The storage unit 26 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of at least two of these. The semiconductor memory is, for example, a RAM or a ROM. The RAM is, for example, an SRAM or a DRAM. The ROM is, for example, an EEPROM. The storage unit 26 may function as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 26 stores data used in the operation of the terminal device 20 and data obtained by the operation of the terminal device 20. For example, the storage unit 26 stores data of captured images of a user of at least one terminal device 20 that communicates with the terminal device itself.

[0033] The control unit 27 includes at least one processor, at least one dedicated circuit, or a combination of these. The processor is a general-purpose processor such as a CPU or a GPU, or a dedicated processor specialized for a specific process. The dedicated circuit is, for example, an FPGA or an ASIC. The control unit 27 executes processes related to the operation of the terminal device 20 while controlling each part of the terminal device 20.

[0034] (Operation of the first terminal device) 5 is a flowchart showing an example of the operation of the first terminal device. For example, when an event is held, the control unit 27 of the first terminal device starts the process of S1. The first terminal device may transmit and receive user voice data to and from the second terminal device in parallel with the following process.

[0035] The control unit 27 determines whether or not the fixed mode is selected (S1). If the first user is a fixed user, the fixed mode is selected in the first terminal device. If the control unit 27 determines that the fixed mode is selected (S1: YES), it ends the process shown in Fig. 5. On the other hand, if the control unit 27 determines that the fixed mode is not selected (S1: NO), it proceeds to the process of S2.

[0036] In the process of S2, the control unit 27 receives data of the captured image of the second user from the second terminal device via the network 2 and the server device 10 through the communication unit 21. The control unit 27 may store the received data of the captured image of the second user in the storage unit 26. The data of the captured image of the second user stored in the storage unit 26 may be used when the first terminal device executes the process of S13 described below as the second terminal device.

[0037] In the process of S3, the control unit 27 generates a model image of the second user based on the captured image data of the second user received in the process of S2. For example, the control unit 27 generates the model image of the second user by extracting an image of the head and neck of the second user from the captured image data of the second user.

[0038] In the process of S4, the control unit 27 causes the display unit 24 to display the model image of the second user generated in the process of S3.

[0039] In the process of S5, the control unit 27 determines whether or not the line of sight of the first user is directed toward the model image of the second user displayed on the display unit 24. For example, as described above, in Fig. 2, it is assumed that the terminal device 20B is the first terminal device, and the model image of the second user is the model image 4C of the user 3C. In this case, the control unit 27 of the terminal device 20B determines whether or not the line of sight of the user 3B is directed toward the model image 4C.

[0040] As an example of the process of S5, first, the control unit 27 generates a captured image of the first user facing the display unit 24 by the imaging unit 25. The control unit 27 analyzes the generated captured image of the first user and recognizes at least one of the positions of the right eye and the left eye of the first user and the rotation angle of the iris. The control unit 27 detects the position where the line of sight of the right eye and the line of sight of the left eye of the first user intersect by recognizing at least one of the positions of the right eye and the left eye of the first user and the rotation angle of the iris of the first user. The position where the line of sight of the right eye and the line of sight of the left eye of the first user intersect corresponds to the position of what the first user is looking at, that is, the position where the line of sight of the first user is fixed. The control unit 27 detects the position where the line of sight of the right eye and the line of sight of the left eye of the first user intersect as the position of the viewpoint of the first user. The control unit 27 determines whether the line of sight of the first user is directed to the model image of the second user displayed on the display unit 24 based on the detected position of the viewpoint of the first user. Here, the viewpoint of the first user is also called the focal point of the first user. In general, the depth of field is different depending on the user. Therefore, there may be a deviation amount of the focal point depending on the user. In the process of S5, such a deviation amount may be allowed.

[0041] When the control unit 27 determines that the first user's gaze is directed toward the model image of the second user displayed on the display unit 24 (S5: YES), the control unit 27 proceeds to the process of S6. When the control unit 27 determines that the first user's gaze is not directed toward the model image of the second user displayed on the display unit 24 (S5: NO), the control unit 27 proceeds to the process of S7.

[0042] In the process of S6, the control unit 27 generates a captured image of the first user facing the display unit 24 by the imaging unit 25. The control unit 27 transmits data of the generated captured image of the first user to the server device 10 via the network 2 by the communication unit 21. The data of the captured image of the first user is transmitted to the second terminal device via the server device 10.

[0043] In the process of S7, the control unit 27 generates coordinate data of the first user's viewpoint. The coordinates of the first user's viewpoint may be coordinates in a coordinate system based on the position of the display unit 24. The control unit 27 may generate the coordinate data of the first user's viewpoint by detecting the position of the first user's viewpoint as described above. The control unit 27 transmits the generated coordinate data of the first user's viewpoint to the server device 10 via the network 2 by the communication unit 21. The coordinate data of the first user's viewpoint is transmitted to the second terminal device via the server device 10.

[0044] In the processes of S6 and S7, in addition to the above data, the control unit 27 transmits data of the display image displayed on the display unit 24 of the first terminal device to the server device 10 via the network 2 by the communication unit 21. The data of the display image of the first terminal device is transmitted to the second terminal device via the server device 10. Here, the display image of the first terminal device is the image seen by the first user. For example, in FIG. 2, when the first user is user 3B, the display image of the first terminal device is the image displayed on the display unit 24 of the terminal device 20B. As described later, the second terminal device determines the positions of the model images of the multiple users on the display unit 24 by using the data of the display image of the first terminal device.

[0045] After the process of S6 or S7, the control unit 27 may set another terminal device 20 as the second terminal device and execute the process from S2. Alternatively, the control unit 27 may set its own device as the second terminal device and the user of its own device as the second user, and proceed to the process of S12 shown in Fig. 6 described later.

[0046] (Operation of the second terminal device) 6 is a flowchart showing an example of the operation of the second terminal device. For example, when an event is held, the control unit 27 of the second terminal device starts the process of S11. The second terminal device may transmit and receive user voice data to and from the first terminal device in parallel with the following process.

[0047] The control unit 27 determines whether or not the fixed mode is selected (S11). When the control unit 27 determines that the fixed mode is selected (S11: YES), the control unit 27 ends the process shown in Fig. 6. On the other hand, when the control unit 27 determines that the fixed mode is not selected (S11: NO), the control unit 27 proceeds to the process of S12.

[0048] In the process of S12, the control unit 27 determines whether data of the captured image of the first user or coordinate data of the viewpoint of the first user has been received from the first terminal device via the network 2 and the server device 10. When the process of S6 has been executed by the first terminal device, the control unit 27 determines that data of the captured image of the first user has been received from the first terminal device (S12: YES). When the process of S7 has been executed by the first terminal device, the control unit 27 determines that coordinate data of the viewpoint of the first user has been received from the first terminal device (S12: NO).

[0049] When the control unit 27 determines that the coordinate data of the viewpoint of the first user has been received from the first terminal device (S12: NO), the control unit 27 proceeds to processing of S13. On the other hand, when the control unit 27 determines that the captured image of the first user has been received from the first terminal device (S12: YES), the control unit 27 proceeds to processing of S14.

[0050] In the process of S12, in addition to the above-mentioned determination process, the control unit 27 receives data of a display image of the first terminal device from the first terminal device via the network 2 and the server device 10 through the communication unit .

[0051] In the process of S13, the control unit 27 generates a model image of the first user based on the data of the first user acquired in advance. The data of the first user acquired in advance is, for example, data of a captured image of a user (first user) received in advance in the process of S2 executed by the second terminal device as the first terminal device. The control unit 27 may generate an image including the head of the first user as a model image of the first user based on the coordinate data of the viewpoint of the first user received in the process of step S12 and the data of the captured image of the first user received in advance. As an example of this process, first, the control unit 27 specifies the direction in which the first user displayed on the display unit 24 is looking as seen by the second user facing the display unit 24 based on the coordinate data of the viewpoint of the first user. Furthermore, the control unit 27 performs image processing or the like on the data of the captured image of the first user received in advance, thereby generating an image of the head of the first user in which the face of the first user displayed on the display unit 24 is looking in the direction in which the first user is looking as seen by the second user facing the display unit 24. Here, if it is determined that the first user's line of sight is not directed toward the model image of the second user (S5: NO), the process of S13 is executed. Therefore, the generated image of the head of the first user in which the face of the first user is directed toward the direction the first user is looking is an image of the back of the head or the side of the face of the first user. For example, in FIG. 3, the second user facing the display unit 24 is user 3C, the first user is user 3D, and the process of S13 is executed. In this case, as shown in FIG. 3, the model image of the first user, user 3C, as viewed from the second user facing the display unit 24 is model image 4D including the back of the head. Here, generally, an image of the back of the head or the side of the face of a user is not complicated compared to an image of the user's face. Therefore, an image of the back of the head or the side of the face of a user can be easily generated from another image.

[0052] In the process of S14, the control unit 27 generates a model image of the first user based on the captured image data of the first user received in the process of S12. For example, the control unit 27 generates a model image of the first user by extracting an image of the head and neck of the first user from the captured image data of the first user. Here, if it is determined that the line of sight of the first user is directed toward the model image of the second user (S5: YES), the process of S14 is executed. Therefore, the model image of the first user generated from the captured image of the first user is an image in which the face of the first user displayed on the display unit 24 is directed toward the second user facing the display unit 24. For example, the model image of the first user is an image like the model image 4B shown in FIG. 3.

[0053] In the process of S15, the control unit 27 causes the display unit 24 to display the model image of the first user generated in the process of S13 or S14. The control unit 27 may cause the display unit 24 to display the model image of the first user by synthesizing the generated model image of the first user with the display image being displayed on the display unit 24. The control unit 27 may determine the positions, etc. of the model images of the multiple users on the display unit 24 based on the data of the display image of the first terminal device received in the process of S12. The control unit 27 may cause the display unit 24 to display the model image of the first user generated by synthesizing the display image of the first terminal device received in the process of S12.

[0054] In the process of S15, the control unit 27 may adjust the perspective of the model image of the first user based on data of the distance from the display unit 24 to the second user. As an example of adjusting the perspective, the control unit 27 may increase the size of the model image of the first user displayed on the display unit 24 when the distance from the display unit 24 to the second user is long, compared to when the distance from the display unit 24 to the second user is short. The control unit 27 may decrease the size of the model image of the first user displayed on the display unit 24 when the distance from the display unit 24 to the second user is short, compared to when the distance from the display unit 24 to the second user is long. The control unit 27 may measure the distance from the display unit 24 to the second user based on data of the captured image of the second user generated by the imaging unit 25, the position of the display unit 24, and the position of the imaging unit 25. The positions of the display unit 24 and the imaging unit 25 may be acquired in advance and stored in the storage unit 26.

[0055] After the process of S15, the control unit 27 may proceed to the process of S5 with the device itself as the first terminal device and the user of the device itself as the first user. In the process of S5, the control unit 27 may set the model image of the user displayed on the display unit 24 in the process of S15 as the model image of the second user.

[0056] As described above, in this embodiment, the first terminal device displays a model image of the second user on the display unit 24, and determines whether or not the first user's line of sight is directed toward the displayed model image of the second user.

[0057] When the first terminal device determines that the first user's gaze is directed toward the displayed model image of the second user, the second terminal device generates a model image of the first user based on the captured image of the first user received from the first terminal device. For example, as described above, in FIG. 3, the terminal device 20C, which is the second terminal device, generates a model image 4B of the user 3B, which is the first user, based on the captured image of the user 3B. With this configuration, it is possible to provide realistic communication, as described above.

[0058] When the first terminal device determines that the line of sight of the first user is not directed to the displayed model image of the second user, the second terminal device generates a model image of the first user based on the data of the first user acquired in advance. For example, as described above, in FIG. 3, the terminal device 20C, which is the second terminal device, generates a model image 4D of the user 3C, who is the first user. The model image 4D is an image of the back of the head. As described above, an image of the back of the head or the side of the face of the user can be easily generated from other images. Furthermore, when the line of sight of the first user is not directed to the displayed model image of the second user, the first user and the second user are not in communication with each other. Therefore, even if the image of the back of the head of the first user is displayed on the second terminal device, there is no problem in communication between the first user and the second user. In addition, since the first terminal device does not need to transmit the captured image of the first user to the second terminal device, the amount of communication between the terminal devices 20 can be reduced. By reducing the amount of communication between the terminal devices 20, a decrease in communication speed can be suppressed.

[0059] Therefore, according to this embodiment, it is possible to provide realistic communication while suppressing a decrease in communication speed.

[0060] Although the present disclosure has been described based on the drawings and examples, it should be noted that those skilled in the art may make various modifications and alterations based on the present disclosure. Therefore, it should be noted that these modifications and alterations are included in the scope of the present disclosure. For example, the functions included in each component or step can be rearranged so as not to be logically inconsistent, and multiple components or steps can be combined or divided into one.

[0061] For example, in the above-described embodiment, the process contents of S3 and S4 may be the same as or similar to the process contents of S14 and S15.

[0062] For example, in the above-described embodiment, the data of the first user acquired in advance in the process of S13 has been described as data of a captured image of a user (first user) received in advance in the process of S2 executed by the second terminal device as the first terminal device. However, the data of the captured image of the first user received in advance is not limited to the data of the captured image received in the process of S2. The data of the captured image of the first user received in advance may be data of a captured image received in a process other than S2.

[0063] For example, in the above-described embodiment, in the process of S15, the control unit 27 has been described as measuring the distance from the display unit 24 to the second user based on data of the captured image of the second user generated by the imaging unit 25, etc. However, the control unit 27 may measure the distance from the display unit 24 to the second user by other methods. For example, in the case where the second terminal device includes a distance measuring sensor, the control unit 27 may measure the distance from the display unit 24 to the second user by the distance measuring sensor.

[0064] Some of the embodiments of the present disclosure will be described below. However, it should be noted that the embodiments of the present disclosure are not limited to these. [Appendix 1] An image generating method executed by a first terminal device and a second terminal device, displaying, by the first terminal device, a model image of the second user of the second terminal device, and determining whether or not a line of sight of the first user of the first terminal device is directed toward the displayed model image of the second user; generating, by the second terminal device, a model image of the first user based on a captured image of the first user received from the first terminal device, when the first terminal device determines that a line of sight of the first user is directed toward the displayed model image of the second user; When the first terminal device determines that the line of sight of the first user is not directed toward the displayed model image of the second user, generating a model image of the first user based on data of the first user previously acquired by the second terminal device. [Appendix 2] The image generating method described in Appendix 1, further comprising displaying the generated model image of the first user by the second terminal device. [Appendix 3] The image generating method described in Appendix 1 or 2, further comprising, when the first terminal device determines that the line of sight of the first user is directed toward the displayed model image of the second user, transmitting a captured image of the first user to the second terminal device. [Appendix 4] An image generating method as described in any one of Appendices 1 to 3, wherein when the first terminal device determines that the gaze of the first user is directed toward the displayed model image of the second user, the model image of the first user generated by the second terminal device is an image in which the displayed face of the first user is directed toward the second user facing the display unit of the second terminal device. [Appendix 5] The image generating method according to any one of appendices 1 to 4, wherein generating a model image of the first user by the second terminal device based on data of the first user previously acquired is generating a model image of the first user based on an image of the first user previously received. [Appendix 6] The image generating method described in Appendix 5, further comprising, when the first terminal device determines that the first user's gaze is not directed toward the displayed model image of the second user, transmitting coordinate data of the first user's viewpoint to the second terminal device. [Appendix 7] The image generating method described in Appendix 6, further comprising generating, by the second terminal device, an image including the head of the first user as a model image of the first user based on the coordinate data of the viewpoint of the first user received from the first terminal device and the previously received captured image of the first user. [Appendix 8] The image generating method described in Appendix 7, wherein the image of the head of the first user is an image in which, when viewed from a second user facing the display unit of the second terminal device, the face of the first user is displayed facing in the direction the first user is looking. [Appendix 9] The image generating method of claim 8, wherein the image of the first user's head is an image of the back of the first user's head or the side of the face. [Appendix 10] The image generating method of any one of Appendix 1 to 9, further comprising, when displaying the model image of the first user by the second terminal device, adjusting the perspective of the model image of the first user based on the distance from a display unit of the second terminal device to the second user. [Appendix 11] A first terminal device and a second terminal device are included, the first terminal device displays a model image of a second user of the second terminal device, and determines whether or not a line of sight of the first user of the first terminal device is directed toward the displayed model image of the second user; When the first terminal device determines that the line of sight of the first user is directed toward the displayed model image of the second user, the second terminal device generates a model image of the first user based on the captured image of the first user received from the first terminal device; A system in which, when the first terminal device determines that the first user's gaze is not directed toward the displayed model image of the second user, the second terminal device generates a model image of the first user based on previously acquired data of the first user. [Appendix 12] The system described in Appendix 11, wherein the second terminal device displays the generated model image of the first user. [Appendix 13] The system described in Appendix 11 or 12, wherein the first terminal device transmits a captured image of the first user to the second terminal device when it determines that the first user's gaze is directed toward a displayed model image of the second user. [Appendix 14] The system described in any one of Appendices 11 to 13, wherein when the first terminal device determines that the first user's gaze is directed toward the displayed model image of the second user, the model image of the first user generated by the second terminal device is an image in which the displayed face of the first user is directed toward the second user facing the display unit of the second terminal device. [Appendix 15] The system described in any one of Appendix 11 to 14, wherein the second terminal device generates a model image of the first user based on a captured image of the first user that has been previously received as data of the first user that has been previously acquired. [Appendix 16] The system described in Appendix 15, wherein when the first terminal device determines that the first user's gaze is not directed toward the displayed model image of the second user, it transmits coordinate data of the first user's viewpoint to the second terminal device. [Appendix 17] The system described in Appendix 16, wherein the second terminal device generates an image including the head of the first user as a model image of the first user based on the coordinate data of the first user's viewpoint received from the first terminal device and the previously received captured image of the first user. [Appendix 18] The system described in Appendix 17, wherein the image of the first user's head is an image in which, when viewed from a second user facing the display unit of the second terminal device, the face of the first user is displayed facing in the direction the first user is looking. [Appendix 19] 19. The system of claim 18, wherein the image of the first user's head is an image of the back of the first user's head or the side of the face. [Appendix 20] The system described in any one of Appendix 11 to 19, wherein when the second terminal device displays the model image of the first user, the second terminal device adjusts the perspective of the model image of the first user based on the distance from a display unit of the second terminal device to the second user. [Explanation of symbols]

[0065] 1: system, 2: network, 3A, 3B, 3C, 3D, 3E: user, 4A, 4B, 4C, 4D: model image, 10: server device (11: communication unit, 12: memory unit, 13: control unit), 20, 20A, 20B, 20C, 20D, 20E: terminal device (21: communication unit, 22: input unit, 23: speaker unit, 24: display unit, 25: imaging unit, 26: memory unit, 27: control unit)

Claims

1. An image generation method executed by a first terminal device and a second terminal device, comprising: displaying, by the first terminal device, a model image of the second user of the second terminal device, and determining whether or not a line of sight of the first user of the first terminal device is directed toward the displayed model image of the second user; generating, by the second terminal device, a model image of the first user based on a captured image of the first user received from the first terminal device when the first terminal device determines that a line of sight of the first user is directed toward a displayed model image of the second user; When the first terminal device determines that the first user's gaze is not directed toward the displayed model image of the second user, generating a model image of the first user based on data of the first user previously acquired by the second terminal device.

2. The image generating method according to claim 1 , further comprising displaying the generated model image of the first user by the second terminal device.

3. The image generating method according to claim 1, further comprising: when the first terminal device determines that the first user's gaze is directed toward the displayed model image of the second user, transmitting a captured image of the first user to the second terminal device.

4. 4. The image generating method according to claim 3, wherein when the first terminal device determines that the gaze of the first user is directed toward the displayed model image of the second user, the model image of the first user generated by the second terminal device is an image in which the face of the first user displayed is directed toward the second user facing the display unit of the second terminal device.

5. The image generating method according to claim 1, wherein generating a model image of the first user by the second terminal device based on data of the first user previously acquired is generating a model image of the first user based on an image of the first user previously received.

6. The image generating method according to claim 5, further comprising: when the first terminal device determines that the first user's gaze is not directed toward the displayed model image of the second user, transmitting coordinate data of the first user's viewpoint to the second terminal device.

7. The image generating method according to claim 6, further comprising generating, by the second terminal device, an image including the head of the first user as a model image of the first user based on the coordinate data of the viewpoint of the first user received from the first terminal device and the previously received captured image of the first user.

8. The image generating method according to claim 7, wherein the image of the head of the first user is an image in which, when viewed from a second user facing the display unit of the second terminal device, the face of the first user is displayed facing in the direction the first user is looking.

9. The image generating method according to claim 8 , wherein the image of the first user's head is an image of the back of the head or a side of the face of the first user.

10. The image generating method according to claim 1, further comprising, when the second terminal device displays the model image of the first user, adjusting the perspective of the model image of the first user based on the distance from a display unit of the second terminal device to the second user.

11. A first terminal device and a second terminal device are included, the first terminal device displays a model image of a second user of the second terminal device, and determines whether or not a line of sight of the first user of the first terminal device is directed toward the displayed model image of the second user; When the first terminal device determines that the line of sight of the first user is directed toward the displayed model image of the second user, the second terminal device generates a model image of the first user based on the captured image of the first user received from the first terminal device, A system in which, when the first terminal device determines that the first user's gaze is not directed toward the displayed model image of the second user, the second terminal device generates a model image of the first user based on previously acquired data of the first user.

12. The system according to claim 11 , wherein the second terminal device displays the generated model image of the first user.

13. The system according to claim 11 , wherein the first terminal device transmits a captured image of the first user to the second terminal device when it determines that the first user's gaze is directed toward the displayed model image of the second user.

14. The system described in claim 13, wherein when the first terminal device determines that the first user's gaze is directed toward the displayed model image of the second user, the model image of the first user generated by the second terminal device is an image in which the displayed face of the first user is directed toward the second user facing the display unit of the second terminal device.

15. The system according to claim 11 , wherein the second terminal device generates a model image of the first user based on a captured image of the first user previously received as previously acquired data of the first user.

16. The system described in claim 15, wherein the first terminal device transmits coordinate data of the first user's viewpoint to the second terminal device when it determines that the first user's gaze is not directed toward the displayed model image of the second user.

17. The system described in claim 16, wherein the second terminal device generates an image including the head of the first user as a model image of the first user based on the coordinate data of the first user's viewpoint received from the first terminal device and the previously received captured image of the first user.

18. The system described in claim 17, wherein the image of the first user's head is an image in which the face of the first user displayed is facing in the direction the first user is looking, as viewed by a second user facing the display unit of the second terminal device.

19. The system of claim 18 , wherein the image of the first user's head is an image of the back of the first user's head or the side of the face.

20. The system according to claim 11, wherein the second terminal device, when displaying the model image of the first user, adjusts the perspective of the model image of the first user based on the distance from a display unit of the second terminal device to the second user.

Images