Image generation method and system

Abstract

The present disclosure relates to an image generation method and system that suppress a decrease in communication speed while providing real communication. The image generation method includes displaying, by a first terminal device, a model image of a second user of a second terminal device, and determining whether a line of sight of a first user of the first terminal device faces the displayed model image of the second user. The image generation method includes generating, by a second terminal device, a model image of a first user on the basis of a captured image of the first user received from the first terminal device when it is determined by the first terminal device that the line of sight of the first user faces the displayed model image of the second user. The image generation method includes generating, by a second terminal device, a model image of a first user based on data of the first user acquired in advance when it is determined by the first terminal device that the line of sight of the first user does not face the displayed model image of the second user.

Description

Technical Field

[0001] The present disclosure relates to an image generation method and system. Background Art

[0002] In the past, it is known that the avatar representing other users is displayed on the technology of the terminal device. In such technology, in order to provide real communication, it is known that the real face image of other users is pasted on the avatar to display the technology (patent document 1).

[0003] Patent Document 1: Japanese Patent Application Publication No. 2001-16563

[0004] In the prior art, in order to always paste the real face image of another user on the virtual image for display, it is necessary to continuously receive the data of the real face image of another user from another terminal device. If the data of the real face image of another user is continuously received, the communication volume between the terminal devices increases, resulting in a decrease in the communication speed. If the communication speed is reduced, users cannot smoothly communicate with each other. Summary of the invention

[0005] The present disclosure made in view of this point aims to provide true communication while suppressing a decrease in communication speed.

[0006] An image generation method according to an embodiment of the present disclosure is an image generation method executed by a first terminal device and a second terminal device, comprising:

[0007] displaying a model image of a second user of the second terminal device through the first terminal device, and determining whether the sight line of the first user of the first terminal device is directed toward the displayed model image of the second user;

[0008] When it is determined by the first terminal device that the line of sight of the first user is directed toward the displayed model image of the second user, generating, by the second terminal device, a model image of the first user based on a captured image of the first user received from the first terminal device; and

[0009] When the first terminal device determines that the line of sight of the first user is not directed toward the displayed model image of the second user, the second terminal device generates a model image of the first user based on data of the first user acquired in advance.

[0010] A system according to one embodiment of the present disclosure includes a first terminal device and a second terminal device.

[0011] The first terminal device displays a model image of a second user of the second terminal device, and determines whether the line of sight of the first user of the first terminal device is directed toward the displayed model image of the second user.

[0012] When the first terminal device determines that the line of sight of the first user is directed toward the displayed model image of the second user, the second terminal device generates a model image of the first user based on the captured image of the first user received from the first terminal device,

[0013] When the first terminal device determines that the line of sight of the first user is not directed toward the displayed model image of the second user, the second terminal device generates the model image of the first user based on data of the first user acquired in advance.

[0014] According to one embodiment of the present disclosure, it is possible to suppress a decrease in communication speed while providing real communication. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a diagram showing a schematic configuration of a system according to one embodiment of the present disclosure.

[0016] Figure 2 This is a diagram for explaining an example of how an event occurs.

[0017] Figure 3 This is a diagram for explaining an example of how an event occurs.

[0018] Figure 4 yes Figure 1 A block diagram of the system is shown.

[0019] Figure 5 This is a flowchart showing an example of the operation of the first terminal device.

[0020] Figure 6 This is a flowchart showing an example of the operation of the second terminal device. DETAILED DESCRIPTION

[0021] Hereinafter, embodiments according to the present disclosure will be described with reference to the drawings.

[0022] like Figure 1 As shown, the system 1 involved in this embodiment includes a server device 10 and terminal devices 20A, 20B, 20C, 20D, and 20E. Hereinafter, when the terminal devices 20A to 20E are not particularly distinguished, they are recorded as "terminal devices 20". The system 1 includes five terminal devices 20. However, the system 1 only needs to include at least two terminal devices 20.

[0023] The system 1 is a system for providing an event that can be attended using a terminal device 20. The event is, for example, online education.

[0024] The server device 10 and the terminal device 20 can communicate via the network 2. The network 2 may be any network including a mobile communication network and the Internet.

[0025] The server device 10 performs transmission and reception of information and information processing required for providing an event. For example, the server device 10 relays communication between a plurality of terminal devices 20 during the implementation of an event.

[0026] The server device 10 is, for example, a dedicated computer configured to function as a server, a general-purpose personal computer, or a cloud computing system.

[0027] The terminal device 20 is, for example, a desktop PC (Personal Computer), a tablet PC, a notebook PC, or a smartphone.

[0028] Terminal devices 20A, 20B, 20C, 20D, and 20E are used by users 3A, 3B, 3C, 3D, and 3E, respectively. Users 3A to 3E use terminal devices 20A to 20E, respectively, to participate in an event. For example, Figure 2 FIG. 2 shows a user 3B participating in an event using a terminal device 20B. The terminal device 20B displays model images 4A, 4C, 4D, and 4E. Model images 4A, 4C, 4D, and 4E are model images of users 3A, 3C, 3D, and 3E, respectively. Here, in the event, any one of users 3A to 3E becomes a fixed user. A fixed user is a user whose model image display position is fixed. For example, in online education, a teacher or a speaking student becomes a fixed user. Figure 2 , user 3A is a fixed user. Therefore, the display position of the model image 4A of user 3A is fixed. The model image of the fixed user is fixed, for example, at the center of the display unit 24. The model images of other users are configured to surround the model image of the fixed user. The model images of other users are configured to direct the line of sight toward the model image of the fixed user. The model image of the fixed user may be displayed larger than the model images of other users. The model image of the user may be an image including at least the head of the user. By including the head of the user in the model image of the user, multiple users can communicate while recognizing each other's expressions. The model image of the user may also be an image including the neck of the user and the upper body of the user in addition to the head of the user.

[0029] Any two of the terminal devices 20A to 20E transmit and receive the captured image of the user in order to generate and display a model image of the user. The two terminal devices 20 that transmit and receive the captured image of the user are recorded as "the first terminal device" and "the second terminal device". The user of the first terminal device is recorded as "the first user". The user of the second terminal device is recorded as "the second user". The first terminal device generates a model image of the second user and displays the generated model image of the second user. The second terminal device generates a model image of the first user and displays the generated model image of the first user.

[0030] Here, if the first terminal device displays the model image of the second user, it is determined whether the first user's line of sight is directed toward the displayed model image of the second user. Figure 2 In FIG. 4 , it is assumed that the terminal device 20B is the first terminal device, the user 3B is the first user, and the model image of the second user is the model image 4C of the user 3C. In this case, the terminal device 20B determines whether the line of sight of the user 3B is toward the model image 4C.

[0031] When the first terminal device determines that the first user's line of sight is directed toward the displayed model image of the second user, the first terminal device transmits the data of the captured image of the first user to the second terminal device. Figure 2 In the example, when the terminal device 20B as the first terminal device determines that the sight line of the user 3B as the first user is directed toward the model image 4C of the second user, the terminal device 20B sends the data of the captured image of the user 3B to the terminal device 20C as the second terminal device. In this case, the terminal device 20C receives the data of the captured image of the user 3B. Figure 3 As shown, the terminal device 20C as the second terminal device generates and displays the model image 4B of the user 3B based on the received captured image of the user 3B. Figure 2 In the case where the sight line of the user 3B as the first user is directed toward the model image 4C, as shown in FIG. Figure 3 As shown, the model image 4B of the user 3B as the first user is an image in which the face is displayed facing the user 3C as the second user. By displaying such a model image of the first user on the second terminal device, it is possible to provide realistic communication.

[0032] When the first terminal device determines that the first user's line of sight is not directed toward the displayed model image of the second user, the first terminal device does not transmit the data of the captured image of the first user to the second terminal device. Figure 2In the example, when the terminal device 20B as the first terminal device determines that the sight line of the user 3B is not directed toward the model image 4C of the second user, the terminal device 20B does not send the data of the captured image of the user 3B to the terminal device 20C as the second terminal device. In this case, the terminal device 20C generates the model image of the user 3B as the first user based on the data of the user 3B acquired in advance. The data of the user 3B acquired in advance is, for example, the data of the captured image of the user 3B received in advance. As described later, the terminal device 20C generates an image of the back of the user's head or the side of the face as the model image of the user 3B. Here, when the sight line of the user 3B is not directed toward the model image 4C, the user 3B and the user 3C are in a state where no communication is achieved. Therefore, even if the image of the back of the user 3B is displayed, there is no obstacle to the communication between the user 3B and the user 3C. In addition, the image of the back of the user's head is generally not complicated compared with the image of the user's face. Therefore, the image of the back of the user's head can be easily created based on the data of the captured image of the user received in advance. Furthermore, by not transmitting the captured image of the first user to the second terminal device, the communication volume between the terminal devices 20 can be reduced.

[0033] (Configuration of server device)

[0034] like Figure 4 As shown, the server device 10 includes a communication unit 11 , a storage unit 12 , and a control unit 13 .

[0035] The communication unit 11 includes at least one communication module that can connect to the network 2. The communication module is, for example, a communication module that complies with a standard such as a wired LAN (Local Area Network) or a wireless LAN. The communication unit 11 is connected to the network 2 via the wired LAN or the wireless LAN through the communication module.

[0036] The storage unit 12 is configured to include at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of at least two of them. The semiconductor memory is, for example, a RAM (Random Access Memory) or a ROM (Read Only Memory). The RAM is, for example, an SRAM (Static Random Access Memory) or a DRAM (Dynamic Random Access Memory). The ROM is, for example, an EEPROM (Electrically Erasable Programmable Read Only Memory). The storage unit 12 can function as a main storage device, an auxiliary storage device, or a buffer memory. The storage unit 12 stores data used in the operation of the server device 10 and data obtained through the operation of the server device 10.

[0037] The control unit 13 is configured to include at least one processor, at least one dedicated circuit, or a combination thereof. The processor is a general-purpose processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) or a dedicated processor customized for specific processing. The dedicated circuit is, for example, an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). The control unit 13 controls each unit of the server device 10 and executes processing related to the operation of the server device 10.

[0038] (Terminal Device Configuration)

[0039] like Figure 4 As shown, the terminal device 20 includes a communication unit 21 , an input unit 22 , a speaker unit 23 , a display unit 24 , a camera unit 25 , a storage unit 26 , and a control unit 27 .

[0040] The communication unit 21 is configured to include at least one communication module that can be connected to the network 2. The communication module is, for example, a communication module corresponding to a standard such as a wired LAN (Local Area Network) or a wireless LAN, or a communication module corresponding to a mobile communication standard such as LTE (Long Term Evolution), 4G (4th Generation) or 5G (5th Generation).

[0041] The input unit 22 can receive input from the user. The input unit 22 is configured to include at least one input interface that can receive input from the user. The input interface is, for example, a physical key, an electrostatic capacitance key, a pointing device, a touch screen or a microphone that is integrally provided with the display of the display unit 24.

[0042] The speaker unit 23 can output sound. The speaker unit 23 may be configured to include any number of speakers.

[0043] The display unit 24 can display data. The display unit 24 includes, for example, a display. The display is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display.

[0044] The imaging unit 25 can capture a subject and generate a captured image. The imaging unit 25 is, for example, a visible light camera. The imaging unit 25 captures the subject continuously at, for example, an arbitrary frame rate. The captured image is a color image (RGB image). However, the captured image may also be a black and white image.

[0045] The storage unit 26 is configured to include at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of at least two of them. The semiconductor memory is, for example, a RAM or a ROM. The RAM is, for example, an SRAM or a DRAM. The ROM is, for example, an EEPROM. The storage unit 26 can function as a main storage device, an auxiliary storage device, or a buffer memory. The storage unit 26 stores data used in the operation of the terminal device 20 and data obtained by the operation of the terminal device 20. For example, the storage unit 26 stores data of a captured image of a user of at least one terminal device 20 communicating with the present device.

[0046] The control unit 27 is configured to include at least one processor, at least one dedicated circuit, or a combination thereof. The processor is a general-purpose processor such as a CPU or a GPU, or a dedicated processor customized for a specific process. The dedicated circuit is, for example, an FPGA or an ASIC. The control unit 27 controls each part of the terminal device 20 and executes processing related to the operation of the terminal device 20.

[0047] (Operation of the first terminal device)

[0048] Figure 5 2 is a flowchart showing an example of the operation of the first terminal device. For example, when an event is held, the control unit 27 of the first terminal device starts the process of S1. The first terminal device can transmit and receive the user's voice data to and from the second terminal device in parallel with the following process.

[0049] The control unit 27 determines whether the fixed mode is selected (S1). If the first user is a fixed user, the fixed mode is selected in the first terminal device. If the control unit 27 determines that the fixed mode is selected (S1: Yes), the process ends. Figure 5 On the other hand, when the control unit 27 determines that the fixed mode is not selected (S1: No), it proceeds to the process of S2.

[0050] In the process of S2, the control unit 27 receives the data of the captured image of the second user from the second terminal device through the network 2 and the server device 10 through the communication unit 21. The control unit 27 can store the received data of the captured image of the second user in the storage unit 26. The data of the captured image of the second user stored in the storage unit 26 can be used when the first terminal device performs the process of S13 described later as the second terminal device.

[0051] In the process of S3, the control unit 27 generates a model image of the second user based on the data of the captured image of the second user received in the process of S2. For example, the control unit 27 generates the model image of the second user by extracting images of the head and neck of the second user from the data of the captured image of the second user.

[0052] In the process of S4 , the control unit 27 causes the display unit 24 to display the model image of the second user generated in the process of S3 .

[0053] In the process of S5, the control unit 27 determines whether the line of sight of the first user is directed toward the model image of the second user displayed on the display unit 24. For example, as described above, Figure 2 It is assumed that the terminal device 20B is the first terminal device and the model image of the second user is the model image 4C of the user 3C. In this case, the control unit 27 of the terminal device 20B determines whether the line of sight of the user 3B is toward the model image 4C.

[0054] As an example of the processing of S5, first, the control unit 27 generates a photographed image of the first user facing the display unit 24 through the photographing unit 25. The control unit 27 analyzes the generated photographed image of the first user and recognizes at least one of the positions of the right eye and the left eye and the rotation angle of the black pupil of the eye of the first user. The control unit 27 detects the position where the line of sight of the right eye and the line of sight of the left eye of the first user intersect by recognizing at least one of the positions of the right eye and the left eye of the first user and the rotation angle of the black pupil of the eye. The position where the line of sight of the right eye and the line of sight of the left eye of the first user intersect corresponds to the position of the object being observed by the first user, that is, the position where the line of sight of the first user is focused. The control unit 27 detects the position where the line of sight of the right eye and the line of sight of the left eye of the first user intersect as the position of the viewpoint of the first user. The control unit 27 determines whether the line of sight of the first user is directed to the model image of the second user displayed on the display unit 24 based on the detected position of the viewpoint of the first user. Here, the viewpoint of the first user is also referred to as the focus point of the first user. In general, the depth of field varies depending on the user. Therefore, there is a case where the focus point is shifted depending on the user. In the process of S5, such a shift may be allowed.

[0055] When the control unit 27 determines that the line of sight of the first user is directed toward the model image of the second user displayed on the display unit 24 (S5: Yes), the control unit 27 proceeds to the process of S6. When the control unit 27 determines that the line of sight of the first user is not directed toward the model image of the second user displayed on the display unit 24 (S5: No), the control unit 27 proceeds to the process of S7.

[0056] In the process of S6, the control unit 27 generates a captured image of the first user facing the display unit 24 through the imaging unit 25. The control unit 27 transmits the generated data of the captured image of the first user to the server device 10 through the network 2 through the communication unit 21. The data of the captured image of the first user is transmitted to the second terminal device through the server device 10.

[0057] In the process of S7, the control unit 27 generates the coordinate data of the viewpoint of the first user. The coordinates of the viewpoint of the first user may be coordinates of a coordinate system based on the position of the display unit 24. The control unit 27 may generate the coordinate data of the viewpoint of the first user by detecting the position of the viewpoint of the first user as described above. The control unit 27 transmits the generated coordinate data of the viewpoint of the first user to the server device 10 via the network 2 through the communication unit 21. The coordinate data of the viewpoint of the first user is transmitted to the second terminal device via the server device 10.

[0058] In the processing of S6 and S7, in addition to the above data, the control unit 27 also sends the data of the display image displayed by the display unit 24 of the first terminal device to the server device 10 via the network 2 through the communication unit 21. The data of the display image of the first terminal device is sent to the second terminal device via the server device 10. Here, the display image of the first terminal device becomes the image that the first user is observing. For example, Figure 2 In the case where the first user is user 3B, the display image of the first terminal device becomes the image displayed on the display unit 24 of the terminal device 20B. The second terminal device determines the positions of the model images of the plurality of users on the display unit 24, etc., by using the display image data of the first terminal device as described later.

[0059] After the processing of S6 or S7, the control unit 27 may execute the processing from S2 with another terminal device 20 as the second terminal device. Alternatively, the control unit 27 may enter the following described method with the present device as the second terminal device and the user of the present device as the second user. Figure 6 The process of S12 is shown.

[0060] (Operation of the Second Terminal Device)

[0061] Figure 6 2 is a flowchart showing an example of the operation of the second terminal device. For example, when an event is held, the control unit 27 of the second terminal device starts the process of S11. The second terminal device can transmit and receive the user's voice data to and from the first terminal device in parallel with the following process.

[0062] The control unit 27 determines whether the fixed mode is selected (S11). If the control unit 27 determines that the fixed mode is selected (S11: Yes), the process ends. Figure 6 On the other hand, when the control unit 27 determines that the fixed mode is not selected (S11: No), the control unit 27 proceeds to the process of S12.

[0063] In the process of S12, the control unit 27 determines whether the data of the captured image of the first user or the coordinate data of the viewpoint of the first user is received from the first terminal device via the network 2 and the server device 10. When the process of S6 is executed by the first terminal device, the control unit 27 determines that the data of the captured image of the first user is received from the first terminal device (S12: Yes). When the process of S7 is executed by the first terminal device, the control unit 27 determines that the coordinate data of the viewpoint of the first user is received from the first terminal device (S12: No).

[0064] When the control unit 27 determines that the coordinate data of the viewpoint of the first user is received from the first terminal device (S12: No), the process proceeds to S13. On the other hand, when the control unit 27 determines that the captured image of the first user is received from the first terminal device (S12: Yes), the process proceeds to S14.

[0065] In the process of S12 , the control unit 27 performs the above-mentioned determination process and receives the display image data of the first terminal device from the first terminal device via the network 2 and the server device 10 through the communication unit 21 .

[0066] In the process of S13, the control unit 27 generates a model image of the first user based on the data of the first user acquired in advance. The data of the first user acquired in advance is, for example, data of a captured image of a user (the first user) received in advance in the process of S2 executed by the second terminal device as the first terminal device. The control unit 27 can generate an image including the head of the first user as a model image of the first user based on the coordinate data of the viewpoint of the first user received in the process of step S12 and the data of the captured image of the first user received in advance. As an example of this process, first, the control unit 27 determines the direction in which the first user is viewed from the second user facing the display unit 24 and displayed on the display unit 24 based on the coordinate data of the viewpoint of the first user. And, the control unit 27 generates an image of the head of the first user, whose face is facing the direction in which the first user is viewed, as viewed from the second user facing the display unit 24 and displayed on the display unit 24, by performing image processing on the data of the captured image of the first user received in advance. Here, if it is determined that the first user's line of sight is not directed toward the model image of the second user (S5: No), the process of S13 is executed. Therefore, the image of the first user's head with the face facing the direction the first user is looking at becomes an image of the back of the first user's head or the side of the face. Figure 3 In the example, it is assumed that the second user facing the display unit 24 is the user 3C and the first user is the user 3D, and the process of S13 is executed. Figure 3 As shown, the model image of user 3D as the first user becomes model image 4D including the back of the head when viewed from user 3C, the second user, who is opposite to display unit 24. Here, the image of the back of the head or the side of the face of a general user is not complicated compared with the image of the face of the user. Therefore, the image of the back of the head or the side of the face of the user can be easily generated from other images.

[0067] In the process of S14, the control unit 27 generates a model image of the first user based on the data of the captured image of the first user received in the process of S12. For example, the control unit 27 generates the model image of the first user by extracting the image of the head and neck of the first user from the data of the captured image of the first user. Here, in a case where it is determined that the line of sight of the first user is directed toward the model image of the second user (S5: Yes), the process of S14 is executed. Therefore, the model image of the first user generated based on the captured image of the first user becomes an image of the first user displayed on the display unit 24, with the face of the first user directed toward the second user facing the display unit 24. For example, the model image of the first user becomes Figure 3 An image like the model image 4B shown.

[0068] In the process of S15, the control unit 27 causes the display unit 24 to display the model image of the first user generated in the process of S13 or S14. The control unit 27 can cause the display unit 24 to display the model image of the first user by synthesizing the generated model image of the first user with the display image currently displayed on the display unit 24. The control unit 27 can determine the positions of the model images of the plurality of users on the display unit 24 based on the data of the display image of the first terminal device received in the process of S12. The control unit 27 can cause the display unit 24 to display the model image of the first user by synthesizing the generated model image of the first user with the display image of the first terminal device received in the process of S12.

[0069] In the process of S15, the control unit 27 may adjust the perspective of the model image of the first user based on the data of the distance from the display unit 24 to the second user. As an example of adjusting the perspective, the control unit 27 may make the size of the model image of the first user displayed on the display unit 24 larger when the distance from the display unit 24 to the second user is long than when the distance from the display unit 24 to the second user is short. The control unit 27 may make the size of the model image of the first user displayed on the display unit 24 smaller when the distance from the display unit 24 to the second user is short than when the distance from the display unit 24 to the second user is long. The control unit 27 may measure the distance from the display unit 24 to the second user based on the data of the captured image of the second user generated by the imaging unit 25, the position of the display unit 24, and the position of the imaging unit 25. The position of the display unit 24 and the position of the imaging unit 25 may be acquired in advance and stored in the storage unit 26.

[0070] After the process of S15, the control unit 27 can enter the process of S5 by treating the device as the first terminal device and the user of the device as the first user. In the process of S5, the control unit 27 can use the model image of the user displayed on the display unit 24 in the process of S15 as the model image of the second user.

[0071] As described above, in the present embodiment, the first terminal device displays the model image of the second user on the display unit 24 and determines whether the line of sight of the first user is directed toward the displayed model image of the second user.

[0072] When the first terminal device determines that the first user's line of sight is directed toward the displayed model image of the second user, the second terminal device generates a model image of the first user based on the captured image of the first user received from the first terminal device. Figure 3 In the example, the terminal device 20C as the second terminal device generates the model image 4B of the user 3B based on the captured image of the user 3B as the first user. According to such a configuration, as described above, it is possible to provide a realistic communication.

[0073] When the first terminal device determines that the first user's line of sight is not directed toward the displayed model image of the second user, the second terminal device generates a model image of the first user based on the data of the first user acquired in advance. Figure 3 , the terminal device 20C as the second terminal device generates a model image 4D of the user 3C as the first user. The model image 4D is an image of the back of the head. As described above, the image of the back of the head or the side of the face of the user can be easily generated based on other images. In addition, when the line of sight of the first user is not directed toward the displayed model image of the second user, the first user and the second user are in a state of not communicating. Therefore, even if the image of the back of the head of the first user is displayed on the second terminal device, there is no obstacle to the communication between the first user and the second user. In addition, since the first terminal device does not need to send the captured image of the first user to the second terminal device, the communication volume between the terminal devices 20 can be reduced. By reducing the communication volume between the terminal devices 20, the reduction in communication speed can be suppressed.

[0074] Therefore, according to the present embodiment, it is possible to provide true communication while suppressing a decrease in communication speed.

[0075] The present disclosure is described based on the drawings and embodiments, but it should be noted that anyone skilled in the art can make various modifications and changes based on the present disclosure. Therefore, it should be noted that these modifications and changes are included in the scope of the present disclosure. For example, the functions included in each component or each step can be reconfigured in a logically non-contradictory manner, and multiple components or steps can be combined into one or divided.

[0076] For example, in the above-mentioned embodiment, the processing contents of S3 and S4 may be the same as or similar to the processing contents of S14 and S15.

[0077] For example, in the above-mentioned embodiment, in the process of S13, the data of the first user acquired in advance is assumed to be the data of the captured image of the user (first user) received in advance in the process of S2 executed by the second terminal device as the first terminal device. However, the data of the captured image of the first user received in advance is not limited to the data of the captured image received in the process of S2. The data of the captured image of the first user received in advance may also be the data of the captured image received in a process other than S2.

[0078] For example, in the above-mentioned embodiment, in the process of S15, it is assumed that the control unit 27 measures the distance from the display unit 24 to the second user based on the data of the captured image of the second user generated by the imaging unit 25. However, the control unit 27 may measure the distance from the display unit 24 to the second user by other methods. For example, when the second terminal device has a distance measuring sensor, the control unit 27 may measure the distance from the display unit 24 to the second user by the distance measuring sensor.

[0079] Hereinafter, some embodiments of the present disclosure are exemplified, but it should be noted that the embodiments of the present disclosure are not limited to these embodiments.

[0080] [Note 1]

[0081] An image generation method is performed by a first terminal device and a second terminal device, comprising:

[0082] displaying a model image of a second user of the second terminal device through the first terminal device, and determining whether the sight line of the first user of the first terminal device is directed toward the displayed model image of the second user;

[0083] When it is determined by the first terminal device that the line of sight of the first user is directed toward the displayed model image of the second user, generating, by the second terminal device, a model image of the first user based on a captured image of the first user received from the first terminal device; and

[0084] When the first terminal device determines that the line of sight of the first user is not directed toward the displayed model image of the second user, the second terminal device generates a model image of the first user based on data of the first user acquired in advance.

[0085] [Note 2]

[0086] The image generation method according to Note 1, further comprising:

[0087] The generated model image of the first user is displayed on the second terminal device.

[0088] [Note 3]

[0089] The image generation method according to note 1 or 2, further comprising:

[0090] When it is determined by the first terminal device that the line of sight of the first user is directed toward the displayed model image of the second user, the captured image of the first user is transmitted to the second terminal device.

[0091] [Note 4]

[0092] The image generating method according to any one of Notes 1 to 3, wherein:

[0093] In a case where it is determined by the first terminal device that the line of sight of the first user is directed toward the displayed model image of the second user, the model image of the first user generated by the second terminal device is an image in which the face of the first user is displayed directed toward the direction of the second user opposite to the display unit of the second terminal device.

[0094] [Note 5]

[0095] The image generating method according to any one of Notes 1 to 4, wherein:

[0096] Generating the model image of the first user based on the data of the first user acquired in advance by the second terminal device means generating the model image of the first user based on a captured image of the first user received in advance.

[0097] [Note 6]

[0098] The image generation method according to Note 5, further comprising:

[0099] When it is determined by the first terminal device that the line of sight of the first user is not directed toward the displayed model image of the second user, coordinate data of the viewpoint of the first user is transmitted to the second terminal device.

[0100] [Note 7]

[0101] The image generation method according to Note 6, further comprising:

[0102] The second terminal device generates an image including the head of the first user as a model image of the first user based on the coordinate data of the viewpoint of the first user received from the first terminal device and the previously received captured image of the first user.

[0103] [Note 8]

[0104] The image generation method according to note 7, wherein:

[0105] The image of the head of the first user is an image viewed from a second user facing the display unit of the second terminal device, in which the face of the first user is displayed facing a direction viewed by the first user.

[0106] [Note 9]

[0107] The image generation method according to note 8, wherein:

[0108] The image of the head of the first user is an image of the back of the head or the side of the face of the first user.

[0109] [Note 10]

[0110] The image generation method according to any one of Notes 1 to 9, further comprising:

[0111] When the model image of the first user is displayed by the second terminal device, the perspective of the model image of the first user is adjusted based on the distance from the display unit of the second terminal device to the second user.

[0112] [Note 11]

[0113] A system wherein:

[0114] comprising a first terminal device and a second terminal device,

[0115] The first terminal device displays a model image of a second user of the second terminal device, and determines whether the line of sight of the first user of the first terminal device is directed toward the displayed model image of the second user.

[0116] When the first terminal device determines that the line of sight of the first user is directed toward the displayed model image of the second user, the second terminal device generates a model image of the first user based on the captured image of the first user received from the first terminal device,

[0117] When the first terminal device determines that the line of sight of the first user is not directed toward the displayed model image of the second user, the second terminal device generates the model image of the first user based on data of the first user acquired in advance.

[0118] [Note 12]

[0119] A system as set forth in Note 11, wherein:

[0120] The second terminal device displays the generated model image of the first user.

[0121] [Note 13]

[0122] A system as set forth in note 11 or 12, wherein:

[0123] The first terminal device transmits the captured image of the first user to the second terminal device when it is determined that the line of sight of the first user is directed toward the displayed model image of the second user.

[0124] [Note 14]

[0125] A system as set forth in any of Notes 11 to 13, wherein:

[0126] When the first terminal device determines that the line of sight of the first user is directed toward the displayed model image of the second user, the model image of the first user generated by the second terminal device is an image in which the face of the first user is displayed directed toward the direction of the second user opposite to the display unit of the second terminal device.

[0127] [Note 15]

[0128] A system as described in any of Notes 11 to 14, wherein:

[0129] The second terminal device generates a model image of the first user based on a captured image of the first user received in advance as data of the first user acquired in advance.

[0130] [Note 16]

[0131] A system as set forth in Note 15, wherein:

[0132] The first terminal device transmits coordinate data of the viewpoint of the first user to the second terminal device when determining that the line of sight of the first user is not directed toward the displayed model image of the second user.

[0133] [Note 17]

[0134] A system as set forth in Note 16, wherein:

[0135] The second terminal device generates an image including a head of the first user as a model image of the first user based on the coordinate data of the viewpoint of the first user received from the first terminal device and the previously received captured image of the first user.

[0136] [Note 18]

[0137] A system as set forth in Note 17, wherein:

[0138] The image of the head of the first user is an image viewed from a second user facing the display unit of the second terminal device, in which the face of the first user is displayed facing a direction viewed by the first user.

[0139] [Note 19]

[0140] A system as set forth in Note 18, wherein:

[0141] The image of the head of the first user is an image of the back of the head or the side of the face of the first user.

[0142] [Note 20]

[0143] A system as described in any of Notes 11 to 19, wherein:

[0144] When displaying the model image of the first user, the second terminal device adjusts the perspective of the model image of the first user based on the distance from the display unit of the second terminal device to the second user.

[0145] Description of reference numerals:

[0146] 1…system; 2…network; 3A, 3B, 3C, 3D, 3E…user; 4A, 4B, 4C, 4D…model image; 10…server device (11…communication unit, 12…storage unit, 13…control unit); 20, 20A, 20B, 20C, 20D, 20E…terminal device (21…communication unit, 22…input unit, 23…speaker unit, 24…display unit, 25…shooting unit, 26…storage unit, 27…control unit).

Claims

1. An image generation method, performed by a first terminal device and a second terminal device, wherein: include: displaying a model image of a second user of the second terminal device through the first terminal device, and determining whether a line of sight of the first user of the first terminal device is directed toward the displayed model image of the second user; When it is determined by the first terminal device that the line of sight of the first user is directed toward the displayed model image of the second user, generating, by the second terminal device, a model image of the first user based on a captured image of the first user received from the first terminal device; as well as When the first terminal device determines that the line of sight of the first user is not directed toward the displayed model image of the second user, the second terminal device generates a model image of the first user based on data of the first user acquired in advance.

2. The image generation method according to claim 1, wherein: Also includes: The generated model image of the first user is displayed on the second terminal device.

3. The image generation method according to claim 1, wherein: Also includes: When it is determined by the first terminal device that the line of sight of the first user is directed toward the displayed model image of the second user, a captured image of the first user is transmitted to the second terminal device.

4. The image generation method according to claim 3, wherein: When it is determined by the first terminal device that the line of sight of the first user is directed toward the displayed model image of the second user, the model image of the first user generated by the second terminal device is an image in which the face of the first user is displayed directed toward the direction of the second user opposite to the display unit of the second terminal device.

5. The image generation method according to claim 1, wherein: Generating the model image of the first user based on the data of the first user acquired in advance by the second terminal device means generating the model image of the first user based on a captured image of the first user received in advance.

6. The image generation method according to claim 5, wherein: Also includes: When it is determined by the first terminal device that the line of sight of the first user is not directed toward the displayed model image of the second user, coordinate data of the viewpoint of the first user is transmitted to the second terminal device.

7. The image generation method according to claim 6, wherein: Also includes: The second terminal device generates an image including the head of the first user as a model image of the first user based on the coordinate data of the viewpoint of the first user received from the first terminal device and the previously received captured image of the first user.

8. The image generation method according to claim 7, wherein: The image of the head of the first user is an image viewed from a second user facing the display unit of the second terminal device, in which the face of the first user is displayed facing a direction viewed by the first user.

9. The image generation method according to claim 8, wherein: The image of the first user's head is an image of the back of the first user's head or a side view of the face.

10. The image generation method according to claim 1, wherein: Also includes: When the model image of the first user is displayed by the second terminal device, the perspective of the model image of the first user is adjusted based on the distance from the display unit of the second terminal device to the second user.

11. A system, wherein: comprising a first terminal device and a second terminal device, The first terminal device displays a model image of a second user of the second terminal device, and determines whether the line of sight of the first user of the first terminal device is directed toward the displayed model image of the second user. When the first terminal device determines that the line of sight of the first user is directed toward the displayed model image of the second user, the second terminal device generates a model image of the first user based on the captured image of the first user received from the first terminal device, When the first terminal device determines that the line of sight of the first user is not directed toward the displayed model image of the second user, the second terminal device generates the model image of the first user based on the data of the first user acquired in advance.

12. The system according to claim 11, wherein: The second terminal device displays the generated model image of the first user.

13. The system according to claim 11, wherein: The first terminal device transmits the captured image of the first user to the second terminal device when determining that the line of sight of the first user is directed toward the displayed model image of the second user.

14. The system according to claim 13, wherein: When the first terminal device determines that the first user's line of sight is directed toward the displayed model image of the second user, the model image of the first user generated by the second terminal device is an image in which the face of the first user is displayed facing the direction of the second user opposite to the display unit of the second terminal device.

15. The system according to claim 11, wherein: The second terminal device generates a model image of the first user based on a captured image of the first user received in advance as data of the first user acquired in advance.

16. The system of claim 15, wherein: The first terminal device transmits coordinate data of the viewpoint of the first user to the second terminal device when determining that the line of sight of the first user is not directed toward the displayed model image of the second user.

17. The system of claim 16, wherein: The second terminal device generates an image including a head of the first user as a model image of the first user based on the coordinate data of the viewpoint of the first user received from the first terminal device and the previously received captured image of the first user.

18. The system of claim 17, wherein: The image of the head of the first user is an image viewed from a second user facing the display unit of the second terminal device, in which the face of the first user is displayed facing a direction viewed by the first user.

19. The system of claim 18, wherein: The image of the first user's head is an image of the back of the first user's head or a side view of the face.

20. The system of claim 11, wherein: When displaying the model image of the first user, the second terminal device adjusts the perspective of the model image of the first user based on the distance from the display unit of the second terminal device to the second user.

Images