terminal device

Abstract

To enhance reality in conversation and user experience.SOLUTION: A terminal device includes a communication unit, and a control unit for implementing communication by the communication unit. The terminal device further includes: an input unit which accepts input of an image of a user who makes drawing while facing a reference direction, and a drawn image by the user; and an output unit which displays the images. The control unit causes the output unit to display, on the basis of information for displaying another user and displaying a drawn image by the other user, received from another terminal device, a composite image in which both the user and the other user face the reference direction while the other drawn image is displayed close to the drawn image with both the reference directions coincide with each other.SELECTED DRAWING: Figure 3B

Description

[Technical Field]

[0001] The present disclosure relates to a terminal device. [Background technology]

[0002] There is a known technology that allows users at multiple locations to communicate with each other by sending and receiving images and voices via a network using computers. Patent Document 1 discloses a technology in which an instructor system and a receiver system communicate with each other, and the receiver performs work by receiving instructions from the receiver through a projected image of the instructor and output sound. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 4323446 Summary of the Invention [Problem to be solved by the invention]

[0004] There is room for further improvement in the realism of calls on the network and the user experience.

[0005] The present disclosure provides a terminal device and the like that contributes to improving the realism of calls and the user experience. [Means for solving the problem]

[0006] The terminal device of the present disclosure is a terminal device having a communication unit and a control unit that communicates via the communication unit, and further has an input unit that accepts input of an image of a user drawing while facing a reference direction and a drawing image by the user, and an output unit that displays an image, and the control unit, based on information received from another terminal device for displaying the other user and another drawing image by the other user, causes the output unit to display a composite image in which the user and the other user are both facing the reference direction, and displays the other drawing images near the drawing image so that their respective reference directions are aligned. [Effects of the Invention]

[0007] According to the terminal device etc. of the present disclosure, it is possible to improve the realism of calls and the user experience. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a call system. [Figure 2] 1 is a diagram showing an example of how a terminal device 12 is used. [Figure 3A] FIG. 10 is a flowchart illustrating an example of the operation of the terminal device. [Figure 3B] FIG. 10 is a flowchart illustrating an example of the operation of the terminal device. DETAILED DESCRIPTION OF THE INVENTION

[0009] The following describes the embodiments.

[0010] [Example of call system configuration] 1 is a diagram showing an example of the configuration of a call system according to an embodiment. The call system 1 includes a server device 10 and multiple terminal devices 12 that are connected to each other via a network 11 so that they can communicate information with each other. The call system 1 is a system that supports calls, including the sending and receiving of images, between the terminal devices 12 used by multiple users.

[0011] The server device 10 is, for example, a server computer that belongs to a cloud computing system or other computing system and functions as a server that implements various functions. The server device 10 may be configured with two or more server computers that are communicatively connected and operate in cooperation with each other. The server device 10 has a communication unit 101, a storage unit 102, and a control unit 103 as components for transmitting and receiving images, audio information, etc. necessary for calls and for performing information processing. Details of each unit will be described later.

[0012] The network 11 is, for example, the Internet, but may also include an ad-hoc network, a local area network (LAN), a metropolitan area network (MAN), or other networks, or any combination thereof.

[0013] The terminal device 12 is a device equipped with a communication function and an information processing function, and is used by each user who makes a call. The terminal device 12 may be configured to include, for example, an information processing terminal such as a smartphone or a tablet terminal, or an information processing device such as a personal computer. Hereinafter, when it is necessary to distinguish between the terminal devices 12 that communicate with each other, they will be referred to as terminal devices 12-1 and 12-2.

[0014] Terminal device 12-1 has a communication unit 121 and a control unit 123 that communicates via communication unit 121, and further has an input unit 125 that accepts input of an image of a user drawing while facing a reference direction and an image drawn by the user, and an output unit 126 that displays the image. Based on information received from other terminal device 12-2 for displaying the other user and another image drawn by the other user, control unit 123 causes output unit 126 to display a composite image in which the user and the other user both face the reference direction, and causes the other image drawn near the drawn image to match their respective reference directions.

[0015] 2 shows a specific example of how the terminal devices 12-1 and 12-2 are used. Here, the users 20-1 and 20-2 use the terminal devices 12-1 and 12-2, respectively, to communicate with each other by sending and receiving their own drawn images, drawn images, and voices. The drawn images may be characters, symbols, pictures, etc.

[0016] User 20-1 draws a drawing image 26 in the drawing area 25 while facing a reference direction 28. The drawing area 25 is, for example, a piece of writing paper or drawing paper. The reference direction 28 is a direction in front of user 20-1 or from the front side to the back. User 20-1 and drawing image 26 are captured by camera 22. Camera 22 is included in the input unit 125 of terminal device 12-1. Camera 22 is provided at any position where it can capture an image of user 20-1 and drawing image 26. Alternatively, the drawing area 25 may be configured with a touch panel included in the input unit 125, and the drawing image 26 may be received by the input unit 125 via the touch panel. Voices uttered by user 20-1 are also picked up by a microphone included in the input unit 125. Information on the captured image and voice of user 20-1 and drawing image 26 is sent to terminal device 12-2 via network 11 and server device 10. Similarly, the terminal device 12-2 captures an image of the user 20-2 and the drawing, and sends the captured image and audio information to the terminal device 12-1.

[0017] In the terminal device 12-1, the display 23 displays the composite image 24. The display 23 is included in the output unit 126. The composite image 24 includes images of both the user 20-1 and the user 20-2 facing the reference direction 28. The images of the user 20-1 and the user 20-2 in the composite image 24 may be 2D models or 3D models. When the images of the user 20-1 and the user 20-2 are configured as 3D models, the display may be a light field display capable of stereoscopically representing the 3D models. By disposing the display 23 on the reference direction 28 side, the user 20-1 can view the composite image 24 while drawing while facing the reference direction. Furthermore, the drawing image 27 by the user 20-2 is projected onto the drawing area 25 by the projector 21. The projector 21 is included in the output unit 126. Furthermore, the voice of the user 20-2 is output by a speaker included in the output unit 126. Similarly, the terminal device 12-2 displays the composite image and the drawn image 26 to the user 20-2, and outputs the voice of the user 20-1.

[0018] By viewing the composite image 24, user 20-1 experiences the same experience as if he or she were talking to user 20-2 beside him or her through a mirror placed in front of him or her. User 20-1 can also see user 20-2's drawing image 27 displayed near his or her own drawing image 26. User 20-2 also experiences the same experience. For example, if user 20-1 is a student of calligraphy, painting, or the like, and user 20-2 is an instructor, the student can draw while looking at the instructor's example and have the instructor correct the drawing. In this case, the student can experience the same experience as if the instructor were right next to him or her. This makes it possible to improve the realism of the call and the user experience.

[0019] [Configuration example of server device 10] The server device 10 includes a communication unit 101, a storage unit 102, and a control unit 103. When the server device 10 is configured with two or more server computers, these components are appropriately arranged in the two or more computers.

[0020] The communication unit 101 includes one or more communication interfaces. The communication interface is, for example, a LAN interface. The communication unit 101 receives information used in the operation of the server device 10 and transmits information obtained by the operation of the server device 10. The server device 10 is connected to a network 11 by the communication unit 101 and communicates information with a terminal device 12 via the network 11.

[0021] The storage unit 102 includes, for example, one or more semiconductor memories, one or more magnetic memories, one or more optical memories, or a combination of at least two of them, that function as a main storage device, an auxiliary storage device, or a cache memory. The semiconductor memory is, for example, a random access memory (RAM) or a read only memory (ROM). The RAM is, for example, a static RAM (SRAM) or a dynamic RAM (DRAM). The ROM is, for example, an electrically erasable programmable read only memory (EEPROM). The storage unit 102 stores information used in the operation of the server device 10 and information obtained by the operation of the server device 10.

[0022] The control unit 103 includes one or more processors, one or more dedicated circuits, or a combination thereof. The processor is, for example, a general-purpose processor such as a CPU (Central Processing Unit), or a dedicated processor such as a GPU (Graphics Processing Unit) specialized for a specific process. The dedicated circuit is, for example, an FPGA (Field-Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), etc. The control unit 103 executes information processing related to the operation of the server device 10 while controlling each unit of the server device 10.

[0023] The functions of the server device 10 are realized by a processor included in the control unit 103 executing a control program. The control program is a program for causing a computer to function as the server device 10. Alternatively, some or all of the functions of the server device 10 may be realized by a dedicated circuit included in the control unit 103. Alternatively, the control program may be stored in a non-transitory recording / storage medium readable by the server device 10, and read by the server device 10 from the medium.

[0024] [Configuration example of terminal device 12] The terminal device 12 includes a communication unit 121 , a storage unit 122 , a control unit 123 , an input unit 125 , and an output unit 126 .

[0025] The communication unit 121 includes a communication module compatible with wired or wireless LAN standards, a module compatible with mobile communication standards such as LTE, 4G, 5G, etc. The terminal device 12 is connected to the network 11 by the communication unit 121 via a nearby router device or a mobile communication base station, and performs information communication with the server device 10, etc. via the network 11.

[0026] The storage unit 122 includes one or more semiconductor memories, one or more magnetic memories, one or more optical memories, or a combination of at least two of these. The semiconductor memories are, for example, RAM or ROM. The RAM is, for example, SRAM or DRAM. The ROM is, for example, EEPROM. The storage unit 122 functions as, for example, a main storage device, an auxiliary storage device, or a cache memory. The storage unit 122 stores information used in the operation of the control unit 123 and information obtained by the operation of the control unit 123.

[0027] The control unit 123 has, for example, one or more general-purpose processors such as a CPU, an MPU (Micro Processing Unit), etc., or one or more dedicated processors such as a GPU specialized for a specific process. Alternatively, the control unit 123 may have one or more dedicated circuits such as an FPGA, an ASIC, etc. The control unit 123 performs overall control of the operation of the terminal device 12 by operating according to a control / processing program or operating according to an operating procedure implemented as a circuit. The control unit 123 then transmits and receives various information to and from the server device 10, etc. via the communication unit 121, and performs the operation according to this embodiment.

[0028] The input unit 125 includes one or more input interfaces. Examples of the input interfaces include physical keys, capacitive keys, a pointing device, a touch screen integrated with a display, a microphone for accepting voice input, and a camera for capturing captured images. The camera captures images of a subject using visible light. The camera captures images of the subject at, for example, 15 to 30 frames per second to generate a video consisting of successive captured images. The camera may also include a ranging sensor that measures the distance to the subject and acquires a distance image. The ranging sensor includes a ToF (Time Of Flight) camera, a LiDAR (Light Detection And Ranging), or a stereo camera, and generates a distance image of the subject including distance information. The input unit 125 sends input or acquired information, images, etc. to the control unit 123.

[0029] The output unit 126 includes one or more output interfaces. The output interfaces include, for example, a display, a projector, and a speaker. The display is, for example, an LCD or an organic EL display. The display may include a light field display capable of displaying a 3D model. The output unit 126 functions as a "display unit" in this embodiment. The output unit 126 outputs information obtained by the operation of the control unit 123.

[0030] The functions of the control unit 123 are realized by a processor included in the control unit 123 executing a control program. The control program is a program for causing the processor to function as the control unit 123. Alternatively, some or all of the functions of the control unit 123 may be realized by a dedicated circuit included in the control unit 123. Alternatively, the control program may be stored in a non-transitory recording / storage medium readable by the terminal device 12, and read by the terminal device 12 from the medium.

[0031] [Example of operation of terminal device 12] 3A and 3B are flow charts showing an example of the operation procedure of the terminal device 12. FIG.

[0032] FIG. 3A shows the operation procedure of the control unit 123 in each terminal device 12 when sending a drawing or the like of a user using the terminal device 12.

[0033] In step S301, the control unit 123 acquires a captured image, a drawn image, and audio information of the user. The control unit 123 acquires the captured image by causing the camera of the input unit 125 to capture the user and the drawn image at an arbitrarily set frame rate. Alternatively, the control unit 123 may acquire the drawn image from a touch panel of the input unit 125. Furthermore, the control unit 123 acquires audio information by causing the microphone of the input unit 125 to collect the user's voice.

[0034] In step S302, the control unit 123 derives information about the reference direction and the size of the drawing area. The control unit 123 recognizes an image of the user and an image of the work area in the captured image using any image processing. The control unit 123 derives the reference direction using the position of the work area relative to the user. The position of the work area is detected, for example, as coordinate information of the vertices of a rectangle in the captured image. The control unit 123 also derives the size of the work area using any algorithm. The size of the work area is specified, for example, by the length of the sides of the rectangular work area. For example, the control unit 123 derives the distance from the camera to the work area using a distance image, and derives the length of the sides of the work area based on that distance. Alternatively, information about the length of the sides of the touch panel that constitutes the work area may be pre-stored in the storage unit 122, and the control unit 123 may read that information from the storage unit 122. Information indicating the reference direction is added to the drawing image. The information about the reference direction is, for example, vector information in spatial coordinates.

[0035] In step S303, the control unit 123 encodes the user's image, drawing image, size information, and audio information to generate encoded information. When encoding, the control unit 123 may perform any processing (e.g., changing the resolution, trimming, etc.) on the captured image, etc.

[0036] In step S304, the control unit 123 causes the communication unit 121 to packetize the encoded information and send it to the server device 10 for other terminal devices 12.

[0037] When the control unit 123 acquires information corresponding to a user operation to interrupt imaging and sound collection or to end a call (Yes in S305), it terminates the processing procedure of Figure 3A, and while it does not acquire information corresponding to an operation to interrupt or end (No in S305), it executes steps S301 to S304 and sends information for displaying the user and drawn images and information for outputting audio to other terminal devices 12.

[0038] Fig. 3B relates to the operation procedure of the control unit 123 when each terminal device 12 displays a drawing image or the like sent from another terminal device 12. When the control unit 123 receives, via the server device 10, a packet sent by another terminal device 12 executing the procedure of Fig. 3A, the control unit 123 executes the procedure of Fig. 3B.

[0039] In step S311, the control unit 123 decodes the coded information contained in the packet received from the other terminal device 12 to obtain the image, the drawn image, the size information, and the audio information of the other user.

[0040] In step S312, the control unit 123 generates a composite image and a drawing image for display. The control unit 123 generates a composite image in which the user and the other user both face the reference direction by using an arbitrary image processing method.

[0041] For example, the control unit 123 positions the other user diagonally behind the user and combines the image of the user with the image of the other user so that the user and the other user face forward or from the front to the back of the user. The control unit 123 may generate the images of the user and the other user as 3D models. For example, the control unit 123 can generate polygon models using distance images of the user and the other user, and generate the 3D models by applying texture mapping to the polygon models using the captured images of each user.

[0042] Furthermore, the control unit 123 uses the size information to generate a drawing image for display, for example, a drawing image for displaying a drawing image by another user, the drawing image having a size corresponding to the user's drawing image and facing a common reference direction. For example, the control unit 123 derives the size of the drawing image drawn by the other user based on the size of the drawing area in which the other user draws. The size of the drawing image is determined by the size of the sides of the rectangular area that the drawing image occupies in the drawing area. The control unit 123 generates a drawing image for display using an arbitrary algorithm that takes into account the known characteristics of the projector or display, such that the drawing image has the derived size when displayed and the reference direction of the drawing image coincides with the reference direction for the user when displayed.

[0043] In step S313, the control unit 123 displays the composite image and the drawing image for display by the output unit 126, and outputs audio. For example, the control unit 123 causes the composite image to be displayed on a display, and causes the drawing image for display to be projected by a projector. Alternatively, the control unit 123 causes the drawing image for display to be displayed on a touch panel display. At the same time, the control unit 123 causes the audio of other users to be output to a speaker.

[0044] By operating each terminal device 12 according to the above procedure, each user can experience a conversation as if they were talking to another user nearby through a mirror placed in front of them. Each user can also view the other user's drawing image displayed near their own drawing image. Since the other user's drawing image is displayed along a common reference direction at a size corresponding to the size of the user's own drawing image, the convenience of viewing the other user's drawing image is improved. This makes it possible to improve the realism of the conversation and the user experience.

[0045] 3A and 3B is executed by the server device 10. For example, the server device 10 may execute processes such as deriving the size of a drawing area for a drawing image sent from another terminal device 12, determining a reference direction, and generating a composite image.

[0046] In the above-described embodiment, the processing / control program that defines the operation of the control unit 123 of the terminal device 12 may be stored in the memory unit of the server device 10 or another server device and downloaded to each terminal device 12 via the network 11, or may be stored in a recording / storage medium readable by each terminal device 12 and read from the medium by each terminal device 12.

[0047] Although the embodiments have been described above based on the drawings and examples, it should be noted that those skilled in the art can easily make various modifications and alterations based on the present disclosure. Therefore, it should be noted that these modifications and alterations are included in the scope of the present disclosure. For example, the functions included in each means, step, etc. can be rearranged so as not to be logically inconsistent, and multiple means, steps, etc. can be combined or divided into one. [Explanation of symbols]

[0048] 1. Call System 10 Server device 11 Network 12 Terminal Equipment 20-1, 20-2 users 21 Projector 22 Camera 24 Composite Images 25 Drawing area 26, 27 Drawing images 28 Reference direction 101, 111 Communications Department 102, 112 Storage section 103, 113 Control section 115 Input section 116 Output section

Claims

1. A terminal device having a communication unit and a control unit that performs communication via the communication unit, an input unit that receives an image of a user drawing while facing a reference direction and an input of the drawing image by the user, and an output unit that displays the image; the control unit derives the reference direction from the image of the user, and based on information received from another terminal device for displaying another user and displaying another drawing image by the other user, causes the output unit to display a composite image in which the user and the other user located behind the user both face the reference direction, and causes the other drawing image to be displayed near the drawing image at a size corresponding to the size of the drawing image so that the reference directions of the respective images coincide with each other; Terminal device.

2. In claim 1, and wherein either or both of the image of the user and the image of the other user are 3D models. Terminal device.

3. In claim 1, the output unit has a display arranged to display the composite image toward the user on a side of the reference direction of the user. Terminal device.

4. In claim 1, the output unit has a projector that projects the other drawing image near the drawing image; Terminal device.

Images