Terminal device, terminal device operation method and program

Abstract

To provide a technique for appropriately displaying characters to a user.SOLUTION: A terminal device 20 includes a control unit 28. The control unit 28 is configured to invert characters included in an illustration drawn by a first user 3A on a virtual screen 33 in a virtual space 30 such that the characters are readable when viewed from a virtual camera 31.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to a terminal device, an operation method for a terminal device, and a program. [Background technology]

[0002] Conventionally, there is known a technology for multiple users to communicate using a virtual space. For example, Patent Document 1 describes defining a virtual space including a first avatar associated with a first user and a second avatar associated with a second user. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-64399 Summary of the Invention [Problem to be solved by the invention]

[0004] In play or creative activities, users may wish to share illustrations or the like they have drawn in order to improve communication between them. When an illustration includes text, it is required that the text be displayed appropriately to the user.

[0005] In view of the above, an object of the present disclosure is to provide a technique for appropriately displaying characters to a user. [Means for solving the problem]

[0006] A terminal device according to an embodiment of the present disclosure includes: The device includes a control unit that inverts characters included in the illustration of the first user drawn on the virtual screen in the virtual space so that the characters are readable when viewed from the virtual camera.

[0007] A method for operating a terminal device according to an embodiment of the present disclosure includes: This includes inverting characters included in the first user's illustration drawn on a virtual screen in the virtual space so that they are readable when viewed from the virtual camera.

[0008] A program according to an embodiment of the present disclosure includes: On the computer, Inverting characters included in a first user's illustration drawn on a virtual screen in a virtual space so that the characters are readable when viewed from a virtual camera. The method executes an operation including: Effect of the Invention

[0009] According to an embodiment of the present disclosure, a technique for appropriately displaying characters to a user can be provided. [Brief description of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating a schematic configuration of a provision system according to an embodiment of the present disclosure. [Diagram 2] FIG. 2 is a block diagram of the provision system shown in FIG. [Diagram 3] 2 is a sequence diagram showing an operation procedure of the providing system shown in FIG. 1. [Figure 4] 3 is a flowchart showing an operation procedure of the terminal device shown in FIG. 2. [Diagram 5] 3 is a flowchart showing an operation procedure of the terminal device shown in FIG. 2. [Figure 6] FIG. 1 is a diagram for explaining a virtual space. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0012] As shown in FIG. 1, the provision system 1 includes at least one server device 10, a terminal device 20A, and a terminal device 20B.

[0013] Hereinafter, when there is no particular distinction between the terminal device 20A and the terminal device 20B, they are also referred to as "terminal device 20." The provision system 1 includes two terminal devices 20. However, the provision system 1 may include two or more terminal devices 20.

[0014] The server device 10 and the terminal device 20 are capable of communicating with each other via a network 2. The network 2 may be any network including a mobile communication network, the Internet, and the like.

[0015] The provision system 1 is a system for providing a virtual event. The virtual event is provided by utilizing a virtual space. A user can participate in the virtual event as a participant using a terminal device 20. In the virtual event according to this embodiment, the user can share illustrations. The virtual event according to this embodiment may be implemented in a place for play, creative activity, or the like.

[0016] The server device 10 is, for example, a server computer that belongs to a cloud computing system or other computing system and functions as a server that implements various functions. The server device 10 may be configured of two or more server computers that are communicatively connected and operate in cooperation with each other.

[0017] The server device 10 executes processes necessary for providing a virtual event. For example, the server device 10 transmits information necessary for providing the virtual event to the terminal device 20 via the network 2. In addition, the server device 10 mediates the transmission and reception of information between the terminal device 20A and the terminal device 20B during the virtual event.

[0018] The terminal device 20 is, for example, a desktop personal computer (PC), a tablet PC, a notebook PC, a smartphone, or the like.

[0019] The terminal device 20 includes a display 24 as a display device. However, the terminal device 20 may include a display device other than the display 24. For example, the terminal device 20 may include a projector as a display device.

[0020] The display 24 is, for example, a liquid crystal display (LCD) or an organic electroluminescence (EL) display, etc. The display 24 is configured as, for example, a touch panel display or a touch screen display together with a touch screen of the input unit 22 described later.

[0021] The terminal device 20A is used by a first user 3A. The first user 3A participates in a virtual event using the terminal device 20A.

[0022] The terminal device 20B is used by a second user 3B. The second user 3B participates in a virtual event using the terminal device 20B.

[0023] The first user 3A and the second user 3B can communicate with each other by sharing illustrations and the like at the virtual event.

[0024] For example, the first user 3A draws an illustration on the display 24 by touching the display 24 with a finger or a stylus pen. The terminal device 20A transmits the illustration data of the first user 3A to the terminal device 20B via the network 2 and the server device 10. The terminal device 20B receives the data of the illustration of the first user 3A from the terminal device 20A and displays the illustration of the first user 3A on the display 24. The second user 3B can see the illustration of the first user 3A by displaying the illustration of the first user 3A on the display 24 of the terminal device 20B. Here, the illustration of the first user 3A includes the character "hellо". The terminal device 20B inverts the character "hellо" included in the data of the illustration of the first user 3A so that it is readable when viewed from the second user 3B. With this configuration, the character can be appropriately displayed to the second user 3B.

[0025] (Server device configuration) As shown in FIG. 2, the server device 10 includes a communication unit 11, a storage unit 12, and a control unit 13.

[0026] The communication unit 11 includes at least one communication module that can be connected to the network 2. The communication module is, for example, a communication module that is compatible with standards such as a wired LAN (Local Area Network) or a wireless LAN. However, the communication module is not limited to this. The communication module may be compatible with any communication standard. The communication unit 11 is connected to the network 2 via the wired LAN or wireless LAN by the communication module.

[0027] The storage unit 12 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of at least two of these. The semiconductor memory is, for example, a random access memory (RAM) or a read only memory (ROM). The RAM is, for example, a static random access memory (SRAM) or a dynamic random access memory (DRAM). The ROM is, for example, an electrically erasable programmable read only memory (EEPROM). The storage unit 12 may function as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 12 stores data used in the operation of the server device 10 and data obtained by the operation of the server device 10.

[0028] The control unit 13 includes at least one processor, at least one dedicated circuit, or a combination of these. The processor is, for example, a general-purpose processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), or a dedicated processor specialized for a specific process. The dedicated circuit is, for example, an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). The control unit 13 executes processes related to the operation of the server device 10 while controlling each part of the server device 10.

[0029] The functions of the server device 10 are realized by executing a processing program according to this embodiment in a processor corresponding to the control unit 13. That is, the functions of the server device 10 are realized by software. The processing program causes a computer to execute the operations of the server device 10, thereby causing the computer to function as the server device 10. That is, the computer functions as the server device 10 by executing the operations of the server device 10 in accordance with the processing program.

[0030] A part or all of the functions of the server device 10 may be realized by a dedicated circuit corresponding to the control unit 13. In other words, a part or all of the functions of the server device 10 may be realized by hardware.

[0031] (Terminal Device Configuration) 2, the terminal device 20 includes a communication unit 21, an input unit 22, an output unit 23, a camera 25, a distance measurement sensor 26, a storage unit 27, and a control unit 28. The terminal device 20 may include a plurality of cameras 25.

[0032] The communication unit 21 is configured to include at least one communication module connectable to the network 2. The communication module is, for example, a communication module compatible with standards such as wired LAN or wireless LAN, or a communication module compatible with mobile communication standards such as LTE (Long Term Evolution), 4G (4th Generation), or 5G (5th Generation). However, the communication module is not limited to this. The communication module may be compatible with any communication standard.

[0033] The input unit 22 is capable of accepting input from a user. The input unit 22 is configured to include at least one input interface capable of accepting input from a user. The input interface includes a touch screen that is integral with the display 24. Furthermore, the input interface may include physical keys, capacitive keys, a pointing device, a microphone, or the like. However, the input interface is not limited to these.

[0034] The output unit 23 is capable of outputting data. The output unit 23 is configured to include at least one output interface capable of outputting data. The output interface includes a display 24 and a speaker, etc. The output interface may include, for example, a plurality of speakers provided around the display 24. However, the output interface is not limited to this.

[0035] The camera 25 is capable of capturing an image of a subject and generating a captured image. The camera 25 is, for example, a visible light camera. The camera 25 continuously captures images of the subject at a frame rate of, for example, 15 to 30 fps.

[0036] The camera 25 is disposed at a position where the camera 25 can capture an image of the user facing the display 24 as a subject. The camera 25 may be disposed at a position where the camera 25 can capture an image of the entire body of the user. FIG. 1 shows the positions of the multiple cameras 25 provided in the terminal device 20A as an example. For example, some of the multiple cameras 25 provided in the terminal device 20 are embedded in front of the display 24. The camera 25 embedded in front of the display 24 can capture a front image of the user. In addition, other parts of the multiple cameras 25 provided in the terminal device 20 are disposed around the user so as to capture an image of the user from multiple directions. However, the arrangement of the cameras 25 is not limited to this. As another example, the camera 25 may be disposed at a position where the camera 25 can capture an image of the user from diagonally above.

[0037] The distance measurement sensor 26 can measure the distance to the subject. The distance measurement sensor 26 is disposed at a position where the distance from the display 24 to the subject can be measured. The distance measurement sensor 26 is disposed at a position where the distance to the user as the subject can be measured. The distance measurement sensor 26 generates a distance image. The distance image is an image in which the pixel value of each pixel corresponds to a distance. The distance measurement sensor 26 includes, for example, a ToF (Time Of Flight) camera, a LiDAR (Light Detection And Ranging) or a stereo camera.

[0038] The storage unit 27 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of at least two of these. The semiconductor memory is, for example, a RAM or a ROM. The RAM is, for example, an SRAM or a DRAM. The ROM is, for example, an EEPROM. The storage unit 27 may function as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 27 stores data used in the operation of the terminal device 20 and data obtained by the operation of the terminal device 20.

[0039] The control unit 28 includes at least one processor, at least one dedicated circuit, or a combination of these. The processor is, for example, a general-purpose processor such as a CPU or a GPU, or a dedicated processor specialized for a specific process. The dedicated circuit is, for example, an FPGA or an ASIC. The control unit 28 executes processes related to the operation of the terminal device 20 while controlling each part of the terminal device 20.

[0040] The functions of the terminal device 20 are realized by executing a terminal program according to this embodiment on a processor corresponding to the control unit 28. That is, the functions of the terminal device 20 are realized by software. The terminal program causes a computer to execute the operations of the terminal device 20, thereby causing the computer to function as the terminal device 20. That is, the computer functions as the terminal device 20 by executing the operations of the terminal device 20 in accordance with the terminal program.

[0041] A part or all of the functions of the terminal device 20 may be realized by a dedicated circuit equivalent to the control unit 28. In other words, a part or all of the functions of the terminal device 20 may be realized by hardware.

[0042] (Operation of the provided system) Fig. 3 is a sequence diagram showing the operation procedure of the provision system 1 shown in Fig. 1. Hereinafter, it is assumed that the first user 3A sets a virtual event as a virtual event manager. In addition, it is assumed that the first user 3A and the second user 3B participate in the virtual event as participants.

[0043] In the process of step S1, in the terminal device 20A, the control unit 28 accepts an input of setting information from the first user 3A through the input unit 22. The setting information is information for setting a virtual event. The setting information includes, for example, a schedule of the virtual event and a participant list. The participant list includes the name of the participant and the email address of the participant. Here, the participant list includes the name of the second user 3B who is a participant and the email address of the second user 3B. For example, the control unit 28 accesses a site for setting a virtual event provided by the server device 10 through the communication unit 21, and acquires data of an input screen for inputting setting information. The control unit 28 displays the input screen on the display 24 and presents it to the first user 3A. The first user 3A looks at the input screen and inputs setting information through the input unit 22.

[0044] In the process of step S2, in the terminal device 20A, the control unit 28 transmits the setting information received by the input unit 22 to the server device 10 via the network 2 through the communication unit 21.

[0045] In the process of step S3, in the server device 10, the control unit 13 receives the setting information from the terminal device 20A via the network 2 through the communication unit 11.

[0046] In the process of step S4, in the server device 10, the control unit 13 sets the virtual event based on the setting information received in the process of step S3. For example, the control unit 13 generates authentication information. The authentication information is information for authenticating the second user 3B who participates in the virtual event using the terminal device 20B. The authentication information includes a participant ID and a passcode, etc. The participant ID is identification information used when the second user 3B participates in the virtual event as a participant.

[0047] In the process of step S5, in the server device 10, the control unit 13 transmits the generated authentication information to the terminal device 20B via the network 2 by the communication unit 11. For example, the control unit 13 transmits the authentication information to the terminal device 20B by attaching it to an e-mail.

[0048] In the process of step S6, in the terminal device 20B, the control unit 28 receives the authentication information from the server device 10 via the network 2 by the communication unit 21. The control unit 28 receives the authentication information attached to the e-mail.

[0049] In the processing of step S7, in terminal device 20B, control unit 28 accepts input of authentication information and participation application information from second user 3B through input unit 22. For example, control unit 28 accesses a site for setting up a virtual event provided by server device 10 through communication unit 21, and acquires data of an input screen for inputting authentication information and participation application information. Control unit 28 causes display 24 to display the input screen, and presents it to second user 3B. Second user 3B looks at the input screen, and inputs the authentication information attached to the e-mail and the participation application information through input unit 22.

[0050] In the process of step S8, in the terminal device 20B, the control unit 28 transmits the authentication information and participation application information received by the input unit 22 to the server device 10 via the network 2 through the communication unit 21.

[0051] In the processing of step S9, in server device 10, control unit 13 receives authentication information and participation application information from terminal device 20B via network 2 through communication unit 11. By receiving the authentication information and participation application information, control unit 13 completes acceptance of participation of second user 3B (step S10).

[0052] In the process of step S11, in the server device 10, the control unit 13 transmits, via the network 2, a notification of the start of the event to each of the terminal devices 20A and 20B through the communication unit 11.

[0053] In the processing of step S12, in terminal device 20A, control unit 28 receives an event start notification from server device 10 via network 2 through communication unit 21. Upon receiving the event start notification, control unit 28 starts capturing an image of first user 3A.

[0054] In the processing of step S13, in terminal device 20B, control unit 28 receives an event start notification from server device 10 via network 2 through communication unit 21. Upon receiving the event start notification, control unit 28 starts imaging second user 3B.

[0055] In the process of step S14, the terminal device 20A and the terminal device 20B carry out a virtual event via the server device 10.

[0056] (Terminal device operation) Fig. 4 is a flowchart showing the operation procedure of the terminal device 20 shown in Fig. 2. The operation procedure shown in Fig. 4 is common to the terminal devices 20A and 20B. The operation procedure shown in Fig. 4 is an example of an operation method of the terminal device 20 according to this embodiment. The operation procedure shown in Fig. 4 is executed in the processing of step S14 shown in Fig. 3. The following description will be given assuming that the terminal device 20A executes the operation procedure shown in Fig. 4.

[0057] In the process of step S21, control unit 28 acquires motion data of first user 3A. In this embodiment, control unit 28 acquires distance image data of first user 3A as motion data of first user 3A using distance measurement sensor 26. When first user 3A moves, the distance from display 24 to first user 3A changes. Therefore, the distance image data of first user 3A can be used as motion data of first user 3A.

[0058] In the process of step S21, control unit 28 acquires, by camera 25, data of a captured image of first user 3A.

[0059] In the process of step S21, control unit 28 acquires data of an illustration drawn on display 24 by first user 3A in response to a touch operation on the touch screen of input unit 22.

[0060] In the process of step S21, control unit 28 collects sounds such as speech uttered by first user 3A with the microphone of input unit 22, thereby acquiring sound data.

[0061] Here, in the process of step S21, control unit 28 acquires motion data of first user 3A, captured image data of first user 3A, illustration data of first user 3A, and sound data of first user 3A at the same timing. That is, the captured image data of first user 3A, illustration data of first user 3A, and sound data of first user 3A are synchronized with the motion data of first user 3A.

[0062] In the process of step S22, the control unit 28 generates encoded data by encoding the data acquired in the process of step S21. In the encoding, the control unit 28 may perform any processing (e.g., resolution change, trimming, etc.) on the captured image data, etc.

[0063] In the process of step S23, the control unit 28 transmits the encoded data as a packet to the server device 10 via the network 2 by the communication unit 21. The encoded data is then transmitted via the server device 10 to the terminal device 20B.

[0064] In the process of step S24, the control unit 28 determines whether or not an input to interrupt imaging or the like or an input to exit from the virtual event has been received by the input unit 22. If the control unit 28 determines that an input to interrupt imaging or the like or an input to exit from the virtual event has been received (step S24: YES), the control unit 28 ends the operation procedure as shown in Fig. 4. If the control unit 28 does not determine that an input to interrupt imaging or the like or an input to exit from the virtual event has been received (step S24: NO), the control unit 28 returns to the process of step S21.

[0065] Fig. 5 is a flowchart showing the operation procedure of the terminal device 20 shown in Fig. 2. The operation procedure shown in Fig. 5 is common to the terminal devices 20A and 20B. The operation procedure shown in Fig. 5 is an example of an operation method of the terminal device 20 according to this embodiment. The operation procedure shown in Fig. 5 is executed in the processing of step S14 shown in Fig. 3. The following description will be given assuming that the terminal device 20B executes the operation procedure shown in Fig. 5.

[0066] In the process of step S31, the control unit 28 receives the encoded data from the terminal device 20A via the network 2 and the server device 10 through the communication unit 21.

[0067] In the process of step S32, control unit 28 decodes the received encoded data. By decoding the encoded data, control unit 28 obtains motion data of first user 3A, captured image data of first user 3A, illustration data of first user 3A, and sound data of first user 3A. The captured image data of first user 3A, illustration data of first user 3A, and sound data of first user 3A are synchronized with the motion data of first user 3A by the process of step S21 described above.

[0068] In the process of step S33, control unit 28 generates avatar 32 of first user 3A as shown in Fig. 6 using the data acquired in the process of step S32. For example, control unit 28 generates a polygon model using motion data of first user 3A, i.e., distance image data. Furthermore, control unit 28 generates the appearance of avatar 32 from the captured image data of first user 3A by applying texture mapping to the polygon model using the captured image data of first user 3A. However, control unit 28 may employ any method to generate avatar 32 of first user 3A.

[0069] Here, in the process of step S33, the control unit 28 may generate the front appearance of the avatar 32 from the captured image data of the front of the first user 3A, which is included in the captured image data of the first user 3A acquired in the process of step S32. The captured image of the front of the first user 3A is captured, for example, by the camera 25 embedded in the front of the display 24 of the terminal device 20A. If the captured image data acquired in the process of step S32 does not include the captured image data of the front of the first user 3A, the control unit 28 may generate the front appearance of the avatar 32 from the data of other captured images of the first user 3A. For example, the control unit 28 generates the front appearance of the avatar 32 by interpolating or correcting the captured image data generated by capturing the first user 3A from multiple directions, which is included in the captured image data acquired in the process of step S32. Alternatively, the control unit 28 may generate the front appearance of the avatar 32 by duplicating the captured image data of the front of the first user 3A captured in the past.

[0070] In the process of step S34, control unit 28 places virtual camera 31, avatar 32, and virtual screen 33 in virtual space 30, as shown in Fig. 6. Control unit 28 places virtual screen 33 between virtual camera 31 and avatar 32. Virtual screen 33 includes surface 33A on the avatar 32 side and surface 33B on the virtual camera 31 side. Surface 33A corresponds to the screen of display 24 of terminal device 20A. Surface 33B corresponds to the screen of display 24 of terminal device 20B.

[0071] In the process of step S34, the control unit 28 may place the avatar 32 in the virtual space 30 so that the avatar 32 is drawing an illustration on the virtual screen 33. For example, when the first user 3A touches the display 24 with a finger or a stylus pen to draw an illustration, the control unit 28 places the avatar 32 in the virtual space 30 so that the finger or stylus pen of the avatar 32 is touching the surface 33A of the virtual screen 33.

[0072] In the process of step S34, the control unit 28 may determine or adjust the position, orientation, and field of view of the virtual camera 31 based on the line of sight of the second user 3B facing the display 24 of the terminal device 20B as shown in FIG. 1. For example, the control unit 28 determines the position of the virtual camera 31 based on the height of the line of sight of the second user 3B. The control unit 28 also determines the orientation of the virtual camera 31 based on the direction of the line of sight of the second user 3B. The control unit 28 may detect the line of sight data of the second user 3B by analyzing the data of the captured image of the second user 3B captured by the camera 25 of the terminal device 20B. However, the control unit 28 may determine or adjust the position, orientation, and field of view of the virtual camera 31 based on the operation input of the second user 2B received from the input unit 22.

[0073] In the process of step S35, control unit 28 draws an illustration of first user 3A on virtual screen 33 based on the data of the illustration of first user 3A acquired in the process of step S32, as shown in Fig. 6. For example, control unit 28 draws an illustration on surface 33A of virtual screen 33.

[0074] In the process of step S36, the control unit 28 determines whether or not the illustration drawn on the virtual screen 33 includes text by any method such as machine learning. When the control unit 28 determines that the illustration drawn on the virtual screen 33 includes text (step S36: YES), the process proceeds to step S37. In the configuration shown in FIG. 6, the control unit 28 determines that the illustration includes the text "hellо". On the other hand, when the control unit 28 determines that the illustration drawn on the virtual screen 33 does not include text (step S36: NO), the process proceeds to step S38.

[0075] In the process of step S37, the control unit 28 inverts the characters on the virtual screen 33 so that they are readable from the virtual camera 31. As an example, the control unit 28 may specify the direction in which the characters on the virtual screen 33 are read from the avatar 32 by any method such as machine learning. Furthermore, the control unit 28 may invert the characters on the virtual screen 33 so that the characters on the virtual screen 33 are readable from the virtual camera 31 in the specified reading direction. In the configuration shown in FIG. 6, the control unit 28 specifies that the direction in which the characters "hellо" on the virtual screen 33 are read is from the left to the right as seen from the avatar 32. The control unit 28 inverts the characters "hellо" on the virtual screen 33 so that the characters "hellо" are readable from the left to the right as seen from the virtual camera 31. As another example, the control unit 28 may specify that the direction in which the characters on the virtual screen 33 are read from the avatar 32 is from the top to the bottom. In this case, control unit 28 inverts the characters on virtual screen 33 from left to right so that the characters are readable when viewed from virtual camera 31. Control unit 28 may set an axis at the center of the characters to be inverted, and invert the characters around the set axis.

[0076] In the process of step S37, the control unit 28 may invert the characters included in the illustration drawn on the virtual screen 33 on a word-by-word or sentence-by-sentence basis so that they are readable as viewed from the virtual camera 31. For example, when the control unit 28 determines that the characters included in the illustration drawn on the virtual screen 33 constitute a word but do not constitute a sentence, the control unit 28 inverts the characters on a word-by-sentence basis so that they are readable as viewed from the virtual camera 31. When the control unit 28 determines that the characters included in the illustration drawn on the virtual screen 33 constitute a sentence, the control unit 28 inverts the characters on a sentence-by-sentence basis so that they are readable as viewed from the virtual camera 31. When the control unit 28 determines that the characters included in the illustration drawn on the virtual screen 33 do not constitute a word or a sentence, the control unit 28 may invert the characters on a sentence-by-sentence basis so that they are readable as viewed from the virtual camera 31. The control unit 28 may determine whether the characters on the virtual screen 33 constitute a word or a sentence, etc., by any method such as machine learning. By automatically inverting the characters on the virtual screen 33 on a word-by-word or sentence-by-sentence basis, the second user 3B can easily read the characters. This improves convenience for second user 3B.

[0077] In the process of step S38, the control unit 28 generates, by rendering, a two-dimensional image of the virtual space 30 projected onto the virtual screen 33 as seen from the virtual camera 31 as shown in FIG.

[0078] In the process of step S39, control unit 28 causes display 24 to display the two-dimensional image generated in the process of step 38. Control unit 28 also causes the speaker of output unit 23 to output the sound acquired in the process of step S32. Control unit 28 may appropriately select a speaker that outputs the sound from among the multiple speakers of output unit 23. For example, an avatar 32 is drawn in the two-dimensional image displayed on display 24. Control unit 28 selects a speaker near the drawn avatar 32 from among the multiple speakers provided around display 24 as the speaker that outputs the sound, so that sound is output from the vicinity of the drawn avatar 32.

[0079] After executing the process of step S39, the control unit 28 returns to the process of step S31. The control unit 28 repeatedly executes the processes of steps S31 to S39 until, for example, the encoded data is no longer transmitted from the terminal device 20A or the virtual event ends.

[0080] In the processing of steps S34 and S35 from the second time onwards, the control unit 28 moves the avatar 32 as shown in Fig. 6 based on the motion data of the first user 3A and the data of the illustration of the first user 3A synchronized with the motion data. The control unit 28 draws the illustration on the virtual screen 33 while moving the avatar 32 so that the avatar 32 draws an illustration of the first user 3A on the virtual screen 33. The control unit 28 may draw the illustration on the virtual screen 33 in real time while moving the avatar 32 so that the avatar 32 draws an illustration of the first user 3A on the virtual screen 33. By moving the avatar 32 in real time, the first user 3A and the second user 3B can communicate with each other in real time.

[0081] In this way, by the process of step S39, a two-dimensional image of the virtual space 30 as shown in FIG. 6 is displayed on the display 24 of the terminal device 20B. With this configuration, the second user 3B sees the first user 3A drawing an illustration on the display 24 of the terminal device 20B while facing the second user 3B. Therefore, the second user 3B can feel as if he is looking at the first user 3A drawing an illustration and the illustration through glass. Therefore, the first user 3A and the second user 3B can share the illustration and the figure of the first user 3A drawing the illustration.

[0082] Furthermore, by generating the appearance of avatar 32 from the captured image data of first user 3A in the process of step S33, second user 3B can feel as if first user 3A himself is drawing the illustration. With this configuration, communication between users can be improved.

[0083] Furthermore, the processing of step S37 inverts the characters included in the illustration drawn on the virtual screen 33 so that they are readable when viewed from the virtual camera 31. With this configuration, when the two-dimensional image of the virtual space 30 is displayed on the display 24 of the terminal device 20B, the characters become readable when viewed from the second user 3B. This improves the convenience for the second user 3B. This makes it possible to provide a technology for appropriately displaying characters to a user.

[0084] A modification of this embodiment will now be described.

[0085] In the process of step S33 described above, the control unit 28 may generate the appearance of the avatar 32 from the character image data. In this case, the control unit 28 generates the appearance of the avatar 32 from the character image data by applying texture mapping using the character image data to the polygon model. By generating the appearance of the avatar 32 from the character image data, it is possible to enhance playability. Therefore, it is possible to improve communication between users.

[0086] In the process of step S34 described above, the control unit 28 may place a virtual object 34 as shown in FIG. 6 in the virtual space instead of the avatar 32. In this case, the control unit 28 places a virtual screen 33 between the virtual camera 31 and the virtual object 34. The control unit 28 may generate the virtual object 34 by an object captured in the captured image of the first user 3A. Alternatively, the control unit 28 may use a preset object as the virtual object 34. Even if the virtual object 34 is used instead of the avatar 32, the second user 3B can feel the depth in the two-dimensional image of the virtual space 30 displayed on the display 24 of the terminal device 20B. By feeling the depth in the two-dimensional image displayed on the display 24, the second user 3B can feel as if he or she is looking at the illustration of the first user 3A displayed on the display 24 through glass.

[0087] In the process of step S37 described above, control unit 28 may invert characters included in the illustration drawn on virtual screen 33 so as to be readable when viewed from virtual camera 31, based on the operation input of second user 3B received by input unit 22. As an example, control unit 28 may receive a touch operation by second user 3B on the character portion of display 24, using a touch screen of input unit 22 provided integrally with display 24. Upon receiving this touch operation, control unit 28 may invert characters included in the illustration drawn on virtual screen 33 so as to be readable when viewed from virtual camera 31. With this configuration, second user 3B can invert characters displayed on display 24 at a desired timing.

[0088] Although the present disclosure has been described based on the drawings and examples, it should be noted that those skilled in the art may make various modifications and alterations based on the present disclosure. Therefore, it should be noted that these modifications and alterations are included in the scope of the present disclosure. For example, the functions included in each component or step can be rearranged so as not to be logically inconsistent, and multiple components or steps can be combined or divided into one.

[0089] For example, in the above-described embodiment, the display device has been described as a display. However, the display device is not limited to a display. As another example, the display device may be a projector. When the display device is a projector, control unit 28 may acquire both the motion data and illustration data of first user 3A by distance measurement sensor 26. As first user 3A moves when drawing an illustration, data of the illustration of first user 3A can be estimated and acquired from the motion data of first user 3A.

[0090] For example, the control unit 28 of the terminal device 20 may not execute the processes of steps S36 and S37 as shown in FIG. 5. In this case, for example, in the processes of steps S34 and S35 that are repeatedly executed, the control unit 28 of the terminal device 20B may move the avatar 32 based on the motion data of the first user 3A. The control unit 28 may draw an illustration of the first user 3A on the virtual screen 33 while moving the avatar 32 so that the avatar 32 draws an illustration of the first user 3A on the virtual screen 33. With this configuration, as described above, the second user 3B can feel as if he or she is looking at the figure of the first user 3A drawing the illustration and the illustration through glass. Therefore, the first user 3A and the second user 3B can share the illustration and the figure of drawing the illustration.

[0091] For example, in the above-described embodiment, the terminal device 20A and the terminal device 20B have been described as implementing a virtual event via the server device 10. However, the terminal device 20A and the terminal device 20B may implement a virtual event without the intervention of the server device 10. As an example, the terminal device 20A and the terminal device 20B may implement a virtual event in a state where they are connected via a P2P (Peer to Peer) type architecture.

[0092] For example, an embodiment is also possible in which a general-purpose computer functions as the terminal device 20 according to the above-described embodiment. Specifically, a program describing the processing contents for realizing each function of the terminal device 20 according to the above-described embodiment is stored in the memory of the general-purpose computer, and the program is read and executed by a processor. Therefore, the present disclosure can also be realized as a program executable by a processor, or a non-transitory computer-readable medium that stores the program. [Explanation of symbols]

[0093] 1. Provision system 2 Network 10. Server equipment 11 Communications Department 12 Storage section 13 Control section 20, 20A, 20B Terminal equipment 21 Communications Department 22 Input section 23 Output section 24 Display 25 Camera 26 Distance Sensor 27 Memory section 28 Control Unit 30 Virtual Space 31 Virtual Camera 32 Avatar 33 Virtual Screen 33A,33B side 34 Virtual Objects 3A First User 3B Second User

Claims

1. A terminal device including a control unit, a communication unit, and a display device, The control unit is receiving illustration data of a first user of another terminal device from the other terminal device by the communication unit; A virtual screen is disposed between a virtual camera and an avatar of the first user in the virtual space; determining a position, orientation and field of view of the virtual camera based on a line of sight of a second user of the terminal device; Inverting characters included in the illustration of the first user drawn on the avatar side of the virtual screen so as to be readable when viewed from the virtual camera; When it is determined that the characters included in the illustration form a word but do not form a sentence, the characters included in the illustration are inverted on a word-by-word basis so as to be readable when viewed from the virtual camera; When it is determined that the characters included in the illustration constitute a sentence, the characters included in the illustration are inverted on a sentence-by-sentence basis so as to be readable when viewed from a virtual camera; generating, by rendering, a two-dimensional image of the virtual space projected onto the virtual screen as viewed from the virtual camera; A terminal device that displays the generated two-dimensional image on the display device.

2. Further comprising an input unit, The terminal device according to claim 1 , wherein the control unit inverts characters included in the illustration so as to be readable when viewed from the virtual camera, based on an operation input of the second user received by the input unit.

3. The control unit is receiving, by the communication unit, motion data of the first user and data of the illustration synchronized with the motion data; The terminal device according to claim 1 or 2, further comprising: a display unit configured to display a display of the first user on the virtual screen based on the received motion data; a display unit configured to display the display of the first user on the virtual screen based on the received motion data;

4. The terminal device according to claim 1 , wherein the control unit is configured to place a virtual object in place of the avatar in the virtual space.

5. The control unit is receiving data of a captured image of the first user by the communication unit; The terminal device according to claim 1 , wherein the appearance of the avatar is generated from data of a captured image of the first user.

6. The terminal device according to claim 1 , wherein the control unit generates the appearance of the avatar from character image data.

7. The terminal device according to claim 1 , wherein the control unit draws the illustration on the virtual screen in real time while moving the avatar so that the avatar draws the illustration on the virtual screen.

8. The display device is a display, The motion data of the first user is acquired by a distance measuring sensor of the other terminal device; The terminal device according to claim 1 , wherein the data of the illustration of the first user is acquired in response to a touch operation on a touch screen of a display of the other terminal device.

9. A method for operating a terminal device, comprising: receiving, from another terminal device, data of an illustration of a first user of the other terminal device; disposing a virtual screen between a virtual camera and an avatar of the first user in a virtual space; determining a position, orientation and field of view of the virtual camera based on a line of sight of a second user of the terminal device; Inverting characters included in the illustration of the first user drawn on the avatar side of the virtual screen so as to be readable when viewed from the virtual camera, When it is determined that the characters included in the illustration form a word but do not form a sentence, the characters included in the illustration are inverted on a word-by-word basis so as to be readable when viewed from the virtual camera; When it is determined that the characters included in the illustration constitute a sentence, the characters included in the illustration are inverted on a sentence-by-sentence basis so as to be readable when viewed from a virtual camera; generating, by rendering, a two-dimensional image of the virtual space projected onto the virtual screen as viewed from the virtual camera; and displaying the generated two-dimensional image on a display device.

10. A terminal device comprising: receiving, from another terminal device, data of an illustration of a first user of the other terminal device; disposing a virtual screen between a virtual camera and an avatar of the first user in a virtual space; determining a position, orientation and field of view of the virtual camera based on a line of sight of a second user of the terminal device; Inverting characters included in the illustration of the first user drawn on the avatar side of the virtual screen so as to be readable when viewed from the virtual camera, When it is determined that the characters included in the illustration form a word but do not form a sentence, the characters included in the illustration are inverted on a word-by-word basis so as to be readable when viewed from the virtual camera; When it is determined that the characters included in the illustration constitute a sentence, the characters included in the illustration are inverted on a sentence-by-sentence basis so as to be readable when viewed from a virtual camera; generating, by rendering, a two-dimensional image of the virtual space projected onto the virtual screen as viewed from the virtual camera; Displaying the generated two-dimensional image on a display device; A program that causes an operation including

Images