Control device

Abstract

To provide an improved technique to create a video obtained by superimposing a plurality of images.SOLUTION: A controller 27 comprises a control unit 29. When displaying two or more images of an image 4A of a first user 3A, an image 4B of a second user 3B facing a display 24, and an image 5 of a predetermined object, superimposed on the display 24, the control unit 29 determines an image to be preferentially superimposed on the other(s) based on the situation of the first user 3A or the situation of the second user 3B.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a control device. [Background technology]

[0002] Conventionally, there is known a technique for generating a video in which a plurality of images are superimposed. For example, Patent Document 1 describes generating a superimposed video in which a first virtual object and an adjusted second virtual object are superimposed. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2021 / 230101 Summary of the Invention [Problem to be solved by the invention]

[0004] There is room for improvement in the conventional technology. For example, the image that should be displayed on top among a plurality of images may change depending on the situation of the user viewing the video.

[0005] In view of the above, an object of the present disclosure is to provide an improved technique for generating a video in which multiple images are superimposed. [Means for solving the problem]

[0006] A control device according to one embodiment of the present disclosure includes a control unit that, when two or more images of an image of a first user, an image of a second user facing a display unit, and an image of a specified object are superimposed and displayed on the display unit, determines the image to be preferentially superimposed on top based on the status of the first user or the status of the second user. Effect of the Invention

[0007] According to an embodiment of the present disclosure, a technique for generating improved multi-image overlay videos may be provided. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram of a system according to an embodiment of the present disclosure. [Diagram 2] 2 is a flowchart showing an operation procedure of the terminal device shown in FIG. [Diagram 3] 2 is a flowchart showing an operation procedure of the terminal device shown in FIG. [Figure 4] 2 is a diagram showing an example of a two-dimensional image displayed on a display unit of the terminal device shown in FIG. 1. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0010] As shown in Fig. 1, the system 1 includes at least one server device 10, a terminal device 20A, and a terminal device 20B. Hereinafter, when there is no particular distinction between the terminal device 20A and the terminal device 20B, they are also referred to as "terminal device 20". The system 1 includes two terminal devices 20. However, the system 1 may include two or more terminal devices 20.

[0011] The server device 10 and the terminal device 20 are capable of communicating with each other via a network 2. The network 2 may be any network including a mobile communication network, the Internet, and the like.

[0012] The system 1 is a system for providing a virtual event. The virtual event is provided by utilizing a virtual space. The system 1 according to the present embodiment provides calligraphy by providing the virtual event.

[0013] The server device 10 is, for example, a server computer that belongs to a cloud computing system or other computing system and functions as a server that implements various functions.

[0014] The server device 10 executes processes necessary for providing a virtual event. For example, the server device 10 transmits information necessary for providing the virtual event to the terminal device 20 via the network 2. In addition, the server device 10 mediates the transmission and reception of information between the terminal device 20A and the terminal device 20B during the virtual event.

[0015] The terminal device 20A is used by a first user 3A. The first user 3A participates in a virtual event using the terminal device 20A. The first user 3A is a calligraphy teacher. The first user 3A is, for example, an adult. The first user 3A faces the display unit 24 of the terminal device 20A. The first user 3A teaches calligraphy to a second user 3B.

[0016] The terminal device 20B is used by the second user 3B. The second user 3B participates in a virtual event using the terminal device 20B. The second user 3B is a calligraphy student. The second user 3B is, for example, a child. The second user 3B faces the display unit 24 of the terminal device 20B. The second user 3B is taught calligraphy by the first user 3A.

[0017] The terminal device 20 is, for example, a desktop personal computer (PC), a tablet PC, a notebook PC, a smartphone, or the like.

[0018] (Server device configuration) As shown in FIG. 1, the server device 10 includes a communication unit 11, a storage unit 12, and a control unit 13.

[0019] The communication unit 11 includes at least one communication module that can be connected to the network 2. The communication module is, for example, a communication module that complies with standards such as a wired LAN (Local Area Network) or a wireless LAN. The communication unit 11 is connected to the network 2 via the wired LAN or wireless LAN by the communication module.

[0020] The storage unit 12 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of at least two of these. The storage unit 12 may function as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 12 stores data used in the operation of the server device 10 and data obtained by the operation of the server device 10.

[0021] The control unit 13 includes at least one processor, at least one dedicated circuit, or a combination of these. The processor is, for example, a general-purpose processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), or a dedicated processor specialized for a specific process. The control unit 13 executes processes related to the operation of the server device 10 while controlling each part of the server device 10.

[0022] (Terminal Device Configuration) As shown in FIG. 1, the terminal device 20 includes a communication unit 21, an input unit 22, an output unit 23, a display unit 24, a camera 25, a distance measurement sensor 26, and a control device 27.

[0023] The communication unit 21 is configured to include at least one communication module connectable to the network 2. The communication module is, for example, a communication module compatible with standards such as wired LAN or wireless LAN, or a communication module compatible with mobile communication standards such as LTE (Long Term Evolution), 4G (4th Generation), or 5G (5th Generation).

[0024] The input unit 22 can receive input from a user. The input unit 22 includes at least one input interface capable of receiving input from a user. The input interface is, for example, a physical key, a capacitive key, a pointing device, a touch screen or a microphone that is integral with the display of the display unit 24, or the like.

[0025] The output unit 23 is capable of outputting data. The output unit 23 includes at least one output interface capable of outputting data. The output interface includes a projector, a speaker, and the like.

[0026] The display unit 24 is capable of displaying data. The display unit 24 is, for example, a display. The display is, for example, a Liquid Crystal Display (LCD) or an organic EL (Electro Luminescence) display.

[0027] The camera 25 can capture an image of a subject and generate a captured image. The camera 25 is, for example, a visible light camera. The camera 25 continuously captures an image of the subject at, for example, an arbitrary frame rate. A plurality of cameras 25 may be arranged around the user.

[0028] The distance measurement sensor 26 can generate a distance image of the subject by measuring the distance from the display of the display unit 24 to the subject. The distance image is an image in which the pixel value of each pixel corresponds to the distance. The distance measurement sensor 26 includes, for example, a ToF (Time Of Flight) camera, a LiDAR (Light Detection And Ranging) camera, or a stereo camera.

[0029] The control device 27 includes a storage unit 28 and a control unit 29 .

[0030] The storage unit 28 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of at least two of these. The storage unit 28 may function as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 28 stores data used in the operation of the terminal device 20 and data obtained by the operation of the terminal device 20.

[0031] The control unit 29 includes at least one processor, at least one dedicated circuit, or a combination of these. The processor is, for example, a general-purpose processor such as a CPU or a GPU, or a dedicated processor specialized for a specific process. The control unit 29 executes processes related to the operation of the terminal device 20 while controlling each unit of the terminal device 20.

[0032] (Terminal device operation) Fig. 2 is a flowchart showing the operation procedure of the terminal device 20 shown in Fig. 1. The operation procedure shown in Fig. 2 is common to the terminal devices 20A and 20B. In the following, the explanation will be given assuming that the terminal device 20A executes the operation procedure shown in Fig. 2. When the virtual event starts, the control unit 29 starts the process of step S1.

[0033] In the process of step S1, the control unit 29 acquires data of the first user 3A. The data of the first user 3A includes data of a distance image of the first user 3A, data of a captured image of the first user 3A, data of a peripheral image of the first user 3A, voice data of the first user 3A, and data of the size of a predetermined body part of the first user 3A. The peripheral image of the first user 3A includes a captured image of the character "eternal" written on a piece of paper as a model by the first user 3A, who is a calligraphy teacher. The predetermined body part may be set according to a virtual event, or may be determined by the first user 3A, who is a teacher. In this embodiment, the predetermined body part is the right arm.

[0034] In the processing of step S1, control unit 29 acquires distance image data of first user 3A by having distance measuring sensor 26 generate it. Control unit 29 acquires captured image data of first user 3A by having camera 25 generate it. Control unit 29 acquires peripheral image data of first user 3A by having camera 25 generate it. Control unit 29 acquires voice data of first user 3A by collecting the voice of first user 3A with the microphone of input unit 22. Control unit 29 acquires the size of a predetermined body part of first user 3A by analyzing the captured image of first user 3A.

[0035] In the process of step S2, control unit 29 transmits the data of first user 3A acquired in the process of step S1 to server device 10 via network 2 by communication unit 21. The data of first user 3A is transmitted via server device 10 to terminal device 20B.

[0036] In the process of step S3, the control unit 29 determines whether or not an input to interrupt imaging or the like or an input to exit from the virtual event has been received by the input unit 22. When the control unit 29 determines that an input to interrupt imaging or the like or an input to exit from the virtual event has been received (step S3: YES), the control unit 29 ends the operation procedure as shown in Fig. 2. When the control unit 29 does not determine that an input to interrupt imaging or the like or an input to exit from the virtual event has been received (step S3: NO), the control unit 29 returns to the process of step S1.

[0037] Fig. 3 is a flowchart showing the operation procedure of the terminal device 20 shown in Fig. 1. The operation procedure shown in Fig. 3 is common to the terminal devices 20A and 20B. The operation procedure shown in Fig. 3 is an example of a display method according to the present embodiment. The following description will be given assuming that the terminal device 20B executes the operation procedure shown in Fig. 3.

[0038] In the process of step S11, control unit 29 receives data of first user 3A from terminal device 20A via network 2 and server device 10 through communication unit .

[0039] In the process of step S12, control unit 29 generates a three-dimensional model of first user 3A using the data of first user 3A received in the process of step S11. For example, control unit 29 generates a polygon model using distance image data of first user 3A. Furthermore, control unit 29 generates the three-dimensional model of first user 3A by applying texture mapping to the polygon model using captured image data of first user 3A.

[0040] In the processing of step S13, control unit 29 generates a three-dimensional model of second user 3B. For example, control unit 29 acquires distance image data of second user 3B by having distance measuring sensor 26 generate it. Control unit 29 acquires captured image data of second user 3B by having camera 25 generate it. Control unit 29 uses these data to generate a three-dimensional model of second user 3B in the same or similar manner as in the processing of step S12.

[0041] In the process of step S14, the control unit 29 generates data of a predetermined object. In this embodiment, the predetermined object is a piece of Japanese calligraphy paper as shown in FIG. 4 described later. However, the predetermined object may be set according to a virtual event. In the Japanese calligraphy paper as the predetermined object, the character "Ei" drawn by the first user 3A, who is a calligraphy teacher, is superimposed on the character "Ei" written by the second user 3B, who is a calligraphy student. First, the control unit 29 acquires data of a captured image of the Japanese calligraphy paper in front of the second user 3B by having the camera 25 generate the data. The control unit 29 acquires data of a captured image of the character "Ei" drawn by the first user 3A as a model from the data of the peripheral image of the first user 3A received in the process of step S11. The control unit 29 generates data of the predetermined object by combining data of the captured image of the Japanese calligraphy paper in front of the second user 3B and data of the captured image of the character "Ei" drawn by the first user 3A as a model.

[0042] In the process of step S15, the control unit 29 estimates the status of the first user 3A or the status of the second user 3B. The control unit 29 may estimate the status of the first user 3A from the voice data of the first user 3A or the captured image data of the first user 3A received in the process of step S11. The control unit 29 may estimate the status of the second user 3B from the voice data of the second user or the captured image data of the second user 3B. The control unit 29 acquires the captured image of the second user 3B by the camera 25 as described above. The control unit 29 acquires the voice data of the second user 3B by collecting the voice of the second user 3B by the microphone of the input unit 22. The control unit 29 may estimate the status of the first user 3A or the second user 3B from a predetermined keyword included in the voice data of the first user 3A or the second user 3B.

[0043] For example, by analyzing the voice data of first user 3A, control unit 29 estimates that the situation of first user 3A is a situation that describes a posture. For example, control unit 29 estimates that the situation of first user 3A is a situation that describes a posture when writing characters.

[0044] For example, by analyzing the voice data of first user 3A, control unit 29 estimates that the situation of first user 3A is being explained by comparing the posture of first user 3A with the posture of second user 3B.

[0045] For example, when the control unit 29 analyzes the voice data of the first user 3A and detects the keyword "Please look at the writing paper" as a specified keyword from the voice data of the first user 3A, it infers that the situation of the first user 3A is one in which the first user 3A is describing a specified object.

[0046] For example, the control unit 29 analyzes the data of the captured image of the second user 3B and detects the line of sight of the second user 3B, thereby inferring that the situation of the second user 3B is one in which the second user's attention is directed to a specified object.

[0047] In the process of step S16, the control unit 29 adjusts the size of the three-dimensional model of the first user 3A generated in the process of step S12. For example, the control unit 29 adjusts the three-dimensional model of the first user 3A based on a result of comparing the size of a predetermined body part of the first user 3A with the size of a predetermined body part of the second user 3B. As described above, in this embodiment, the predetermined body part is the right arm. The control unit 29 acquires data on the size of the right arm of the first user 3A from the data of the first user 3A received in the process of step S11. The control unit 29 acquires data on the size of the right arm of the second user 3B by analyzing the captured image of the second user 3B acquired by the camera 25 as described above. For example, it is assumed that the size of the right arm of the first user 3A is 30 [cm] and the size of the right arm of the second user 3B is 20 [cm]. In this case, the control unit 29 adjusts the size of the three-dimensional model of the first user 3A to be 3 / 2 times the size of the three-dimensional model of the second user 3B.

[0048] In the process of step S17, control unit 29 places, within the virtual space, the three-dimensional model of first user 3A after the size has been adjusted in the process of step S16, the three-dimensional model of second user 3B, and a predetermined object.

[0049] In the process of step S17, the control unit 29 arranges the three-dimensional model of the first user 3A and the three-dimensional model of the second user 3B based on a reference point. The reference point may be set according to a virtual event or may be determined by the first user 3A, who is the teacher. In this embodiment, the reference point is the right shoulder. The control unit 29 arranges the three-dimensional models of the first user 3A and the second user 3B so that the right shoulder of the three-dimensional model of the first user 3A is located on the right shoulder of the three-dimensional model of the second user 3B. The control unit 29 may move the three-dimensional model of the first user 3A so that the right shoulder of the three-dimensional model of the first user 3A is located on the right shoulder of the three-dimensional model of the second user 3B.

[0050] In the process of step S18, the control unit 29 generates a two-dimensional image by rendering the three-dimensional model and the predetermined object arranged in the virtual space, photographed from a virtual viewpoint. The virtual viewpoint may be set based on the viewpoint of the second user 3B. For example, the control unit 29 sets the virtual line of sight so that the second user 3B and the image 4B of the second user 3B displayed on the display unit 24 face each other, as shown in FIG. 4 described later. With this configuration, the second user 3B can feel as if he or she is looking at himself or herself through a mirror.

[0051] The two-dimensional image generated by the process of step S18 includes an image 4A of the first user 3A obtained by two-dimensionalizing a three-dimensional model of the first user 3A, as shown in Fig. 4 described later. This two-dimensional image also includes an image 4B of the second user 3B obtained by two-dimensionalizing a two-dimensional model of the second user 3B, as shown in Fig. 4 described later. This two-dimensional image also includes an image 5 of a predetermined object obtained by two-dimensionalizing a predetermined object, as shown in Fig. 4 described later.

[0052] In the process of step S19, when two or more of image 4A of first user 3A, image 4B of second user 3B, and image 5 of a predetermined object overlap, control unit 29 determines the image to be preferentially overlaid on top. In this embodiment, the higher the image is overlaid, the closer it is to be displayed on display unit 24. Control unit 29 determines the image to be preferentially overlaid on top based on the situation of first user 3A or the situation of second user 3B estimated in the process of step S15.

[0053] For example, it is assumed that the image 4A of the first user 3A is included in the two or more overlapping images. In this case, when the control unit 29 estimates in the process of step S15 that the situation of the first user 3A is a situation in which the posture is being explained, the control unit 29 decides to overlay the image 4A of the first user 3A on the other images that overlap with the image 4A. The control unit 29 may decide to overlay the image 4A of the first user 3A on the top of the two or more overlapping images. By deciding to overlay the image 4A of the first user 3A on the other images, the image 4A of the first user 3A is displayed on the front side of the display unit 24 of the terminal device 20B when the calligraphy teacher who is the first user 3A is explaining the posture. By displaying the image 4A of the first user 3A on the front side of the display unit 24, the possibility that the image 4A of the first user 3A will be hidden by the other images is reduced. This allows the second user 3B to observe the posture of the first user 3A well.

[0054] For example, it is assumed that the image 4B of the second user 3B is included in the two or more overlapping images. In this case, when the control unit 29 estimates in the process of step S15 that the situation of the first user 3A is a situation in which the posture of the first user 3A and the posture of the second user 3B are being compared and explained, the control unit 29 determines to superimpose the image 4B of the second user 3B on another image overlapping the image 4B. With this configuration, when the teacher who is the first user 3A is explaining by comparing his / her posture with that of the second user 3B, a two-dimensional model of the second user 3B is displayed on the front side of the display unit 24 of the terminal device 20B. By displaying the image 4B of the second user 3B on the front side of the display unit 24, the possibility that the image 4B of the second user 3B will be hidden by another image is reduced. As a result, the student who is the second user 3B can listen to the explanation by the first user 3A in which the posture of the first user 3A is compared with that of the second user 3B while closely observing his / her own posture. Note that the two or more overlapping images may include the image 4A of the first user 3A and the image 5 of a predetermined object in addition to the image 4B of the second user 3B. In this case, the control unit 29 may determine to overlay the image 5 of the predetermined object below the image 4B of the second user 3B, and the image 4A of the first user 3A below the image 5 of the predetermined object. Alternatively, the control unit 29 may determine to overlay the image 4B of the second user 3B on top, the image 5 of the predetermined object on top second, and the image 4A of the first user 3A on top third.

[0055] For example, it is assumed that the image 5 of the predetermined object is included in two or more overlapping images. In this case, when the control unit 29 estimates in the process of step S15 that the situation of the first user 3A is a situation in which the predetermined object is being explained, the control unit 29 decides to overlay the image 5 of the predetermined object on the other images that overlap with the image 5. The control unit 29 may decide to overlay the image 5 of the predetermined object on the top of the two or more overlapping images. By deciding to overlay the image 5 of the predetermined object on the other images, when the first user 3A is explaining the characters on the half-sheet of paper that is the predetermined object, the image 5 of the predetermined object is displayed on the front side of the display unit 24 of the terminal device 20B. By displaying the predetermined object on the front side of the display unit 24, the possibility that the predetermined object will be hidden by the other images is reduced. This allows the second user 3B to closely observe the predetermined object while listening to the explanation of the first user 3A.

[0056] For example, it is assumed that the image 5 of the predetermined object is included in two or more overlapping images. In this case, when the control unit 29 estimates in the process of step S15 that the situation of the second user 3B is a situation in which the attention of the second user 3B is directed to the predetermined object, the control unit 29 decides to overlay the image 5 of the predetermined object on the other images overlapping with the image 5. The control unit 29 may decide to overlay the image 5 of the predetermined object on the top of the two or more overlapping images. By deciding to overlay the image 5 of the predetermined object on the other images, when the second user 3B is directing his / her attention to the character on the writing paper, which is the predetermined object, the predetermined object is displayed on the front side of the display unit 24 of the terminal device 20B. By displaying the image 5 of the predetermined object on the front side of the display unit 24, the possibility that the predetermined object will be hidden by the other images is reduced. This allows the second user 3B to observe the predetermined object well.

[0057] In the process of step S20, the control unit 29 corrects the data of the two-dimensional image generated in the process of step S18 based on the process result of step S19. That is, the control unit 29 corrects the data of the two-dimensional image so that the image determined to be preferentially overlaid in the process of step S19 is overlaid on another image.

[0058] In the process of step S21, control unit 29 causes display unit 24 to display the two-dimensional image corrected in the process of step 20. Control unit 29 also causes the speaker of output unit 23 to output the voice data of first user 3A as voice.

[0059] After executing the process of step S21, control unit 29 returns to the process of step S11. Control unit 29 repeatedly executes the processes of steps S11 to S21, for example, until data of first user 3A is no longer transmitted from terminal device 20A to terminal device 20B or the virtual event ends.

[0060] By executing the processes of steps S11 to S21, the display unit 24 of the terminal device 20B displays, for example, an image 4A of the first user 3A, an image 4B of the second user 3B, and an image 5 of a predetermined object as shown in FIG. 4. In FIG. 4, the image 4A of the first user 3A and the image 4B of the second user 3B are included in the two or more overlapping images in the process of step S19. In the process of step S19, the control unit 29 estimates that the situation of the first user 3A is the situation explained by comparing the posture of the first user 3A with the posture of the second user 3B in the process of step S15, and determines to superimpose the image 4B of the second user 3B on the image 4B. By such a process, in FIG. 4, the image 4B of the second user 3B is superimposed on the image 4A of the first user 3A. As shown in FIG. 4, the display unit 24 may display an image of an object such as a desk in addition to the image 4A.

[0061] Here, the two-dimensional image generated in the process of step S18 may include a two-dimensional image generated by the terminal device 20A and a two-dimensional image generated by the terminal device 20B. In this case, the control unit 29 of the terminal device 20A may execute the process of step S19 on the two-dimensional image generated by the terminal device 20A and the two-dimensional image generated by the terminal device 20B. For example, when two or more of the two-dimensional image generated by the terminal device 20A and the two-dimensional image generated by the terminal device 20B overlap, the control unit 29 of the terminal device 20A may determine an image to be preferentially overlaid on top.

[0062] In this manner, in terminal device 20B, when two or more images of image 4A of first user 3A, image 4B of second user 3B, and image 5 of a predetermined object are displayed on display unit 24 in an overlapping manner, control unit 29 determines which image to preferentially overlay. Control unit 29 determines which image to preferentially overlay based on the status of first user 3A or the status of second user 3B. By determining which image to preferentially overlay based on the status of first user 3A or the status of second user 3B, it is possible to improve user convenience.

[0063] Here, when the three-dimensional model of the first user 3A and the three-dimensional model of the second user 3B are arranged in the virtual space in the process of step S17, the arrangement position of the three-dimensional model of the first user 3A and the arrangement position of the three-dimensional model of the second user 3B may become close to each other. For example, as shown in FIG. 4, in a virtual event of calligraphy, when the first user 3A writes a model behind the second user 3B, the arrangement position of the three-dimensional model of the first user 3A and the arrangement position of the three-dimensional model of the second user 3B become close to each other. In this case, in the two-dimensional image generated in the process of step S18, a part of the image 4A of the first user 3A may be hidden by the image 4B of the second user 3B, or a part of the image 4B of the second user 3B may be hidden by the image 4A of the first user 3A. In addition, a part of the writing paper may be hidden by the hand of the second user 3B, so that a part of the image 5 of the predetermined object may be hidden by the image 4B of the second user 3B in the two-dimensional image generated in the process of step S18. Even if a portion of the images 4A, 4B, 5 is hidden in this way, the possibility that a portion of the images 4A, 4B, 5 will be hidden on the display unit 24 is reduced by determining the image to be preferentially overlaid in the process of step S19.

[0064] Thus, the present embodiment provides an improved technique for generating a video in which multiple images are superimposed.

[0065] Also, for example, when the first user 3A is an adult and the second user 3B is a child, the body size of the first user 3A who is a teacher may differ greatly from the body size of the second user 3B who is a student. Also, the position of the first user 3A with respect to the display unit 24 of the terminal device 20A may differ from the position of the second user 3B with respect to the display unit 24 of the terminal device 20B. In this case, if the three-dimensional models of the first user 3A and the second user 3B are simply arranged in the virtual space, the second user 3B may not be able to look at the image displayed on the display unit 24 and use the posture of the first user 3A as a model. In this embodiment, the process of step S16 makes the size ratio between the three-dimensional model of the first user 3A and the three-dimensional model of the second user 3B close to the size ratio between the body of the first user 3A and the body of the second user 3B. Also, in the process of step S17, the control unit 29 arranges the three-dimensional model of the first user 3A and the three-dimensional model of the second user 3B based on the reference point. With this configuration, second user 3B can see the two-dimensional image displayed on display unit 24 of terminal device 20B by the process of step S21 and correctly compare his / her own posture with that of first user 3A. In other words, second user 3B can use the posture of first user 3A as a model.

[0066] A modification of this embodiment will now be described.

[0067] In the process of step S21 described above, control unit 29 may cause a projector in output unit 23 to project the character "eternal" sampled by first user 3A onto an actual piece of writing paper as shown in FIG.

[0068] When the processes of steps S11 to S21 described above are repeatedly executed, the control unit 29 may execute the process of step S16 when a predetermined trigger is detected. The predetermined trigger is, for example, an input of the second user 3B to the input unit 22 of the terminal device 20B, a predetermined action of the second user, or a predetermined keyword such as "Pay attention to your posture." The predetermined keyword may be set by the first user 3A. The predetermined action of the second user is, for example, holding a brush, or the second user 3B's line of sight meeting the line of sight of the first user 3A displayed as the image 4A on the display unit 24 of the terminal device 20B. By executing step S16 when the predetermined trigger is detected, the size of the three-dimensional model of the first user 3A is adjusted at a timing when the second user 3B wants to see the posture of the first user 3A as a model.

[0069] Although the present disclosure has been described based on the drawings and examples, it should be noted that those skilled in the art may make various modifications and alterations based on the present disclosure. Therefore, it should be noted that these modifications and alterations are included in the scope of the present disclosure. For example, the functions included in each component or step can be rearranged so as not to be logically inconsistent, and multiple components or steps can be combined or divided into one.

[0070] For example, in the above-described embodiment, the terminal device 20A and the terminal device 20B have been described as implementing a virtual event via the server device 10. However, the terminal device 20A and the terminal device 20B may implement a virtual event without the intervention of the server device 10. As an example, the terminal device 20A and the terminal device 20B may implement a virtual event in a state where they are connected via a P2P (Peer to Peer) type architecture.

[0071] For example, in the above-described embodiment, a virtual event is provided to provide calligraphy lessons. However, a virtual event may be provided to provide other lessons such as flower arranging or pottery. In the case of flower arranging, the predetermined object may be a flower. In the case of pottery, the predetermined object may be ceramics or the like.

[0072] Some of the embodiments of the present disclosure will be described below. However, it should be noted that the embodiments of the present disclosure are not limited to these. [Appendix 1] A control device comprising: a control unit that, when two or more images of an image of a first user, an image of a second user facing a display unit, and an image of a specified object are superimposed and displayed on the display unit, determines which image to preferentially superimpose on top based on the status of the first user or the status of the second user. [Appendix 2] The control device described in Appendix 1, wherein when the two or more images include an image of the first user, the control unit determines that the image of the first user is overlaid on another image that overlaps with the image of the first user when it estimates that the situation of the first user is a situation that describes a posture. [Appendix 3] The control device described in Appendix 1 or 2, wherein when the two or more images include an image of the second user, the control unit decides to overlay the image of the second user on another image that overlaps with the image of the second user when it estimates that the situation of the first user is a situation that is explained by comparing the posture of the first user with the posture of the second user. [Appendix 4] The control device according to any one of appendices 1 to 3, wherein when the two or more images include an image of the first user and an image of the specified object in addition to the image of the second user, the control unit decides to overlay the image of the specified object under the image of the second user and to overlay the image of the first user under the image of the specified object. [Appendix 5] The control device according to any one of appendices 1 to 4, wherein when the two or more images include an image of the specified object, the control unit determines to overlay the image of the specified object on another image that overlaps with the image of the specified object when it estimates that the situation of the first user is a situation that describes the specified object. [Appendix 6] The control device of any one of appendices 1 to 5, wherein when the two or more images include an image of the specified object, the control unit determines to overlay the image of the specified object on another image that overlaps with the image of the specified object when it estimates that the second user's situation is one in which the second user's attention is directed to the specified object. [Appendix 7] The control device according to any one of claims 1 to 6, wherein the control unit estimates the situation of the first user or the situation of the second user based on voice data of the first user or the voice data of the second user. [Appendix 8] The control device according to any one of appendices 1 to 7, wherein the control unit estimates the situation of the first user or the situation of the second user based on a predetermined keyword contained in the voice data of the first user or the voice data of the second user. [Appendix 9] A display unit; a control unit that, when displaying two or more images among an image of a first user, an image of a second user facing a display unit, and an image of a predetermined object in an overlapping manner on the display unit, determines an image to be preferentially overlapped on top based on a state of the first user or a state of the second user; A terminal device comprising: [Appendix 10] A display method including, when two or more images of an image of a first user, an image of a second user facing a display unit, and an image of a specified object are displayed in an overlapping manner on the display unit, determining which image to preferentially overlay based on the status of the first user or the status of the second user. [Explanation of symbols]

[0073] 1: system, 2: network, 3A: first user, 3B: second user, 4A, 4B, 5: image, 10: server device, 11: communication unit, 12: memory unit, 13: control unit, 20, 20A, 20B: terminal device, 21: communication unit, 22: input unit, 23: output unit, 24: display unit, 25: camera, 26: distance measurement sensor, 27: control device, 28: memory unit, 29: control unit

Claims

1. a control unit that, when displaying two or more images among an image of a first user, an image of a second user facing a display unit, and an image of a predetermined object in an overlapping manner on the display unit, determines an image to be preferentially overlapped on top based on a status of the first user or a status of the second user; A control device, wherein when the two or more images include an image of the first user, the control unit determines that the situation of the first user is a situation that describes a posture, and then decides to overlay the image of the first user on another image that overlaps with the image of the first user.

2. When two or more images selected from an image of a first user, an image of a second user facing a display unit, and an image of a predetermined object are superimposed and displayed on the display unit, a control unit is provided which determines which image is to be preferentially superimposed on top based on the status of the first user or the status of the second user, A control device, wherein when the two or more images include an image of the second user, the control unit determines that the situation of the first user is a situation that is explained by comparing the posture of the first user with the posture of the second user, and then decides to overlay the image of the second user on another image that overlaps with the image of the second user.

3. The control device according to claim 2, wherein when the two or more images include an image of the first user and an image of the specified object in addition to the image of the second user, the control unit decides to overlay the image of the specified object under the image of the second user and to overlay the image of the first user under the image of the specified object.

4. A control unit is provided which, when two or more images selected from an image of a first user, an image of a second user facing a display unit, and an image of a predetermined object are superimposed and displayed on the display unit, determines which image is to be preferentially superimposed on top based on the status of the first user or the status of the second user, A control device, wherein when the two or more images include an image of the specified object, the control unit determines, when it estimates that the situation of the first user is a situation that describes the specified object, to overlay the image of the specified object on another image that overlaps with the image of the specified object.

Images