Terminal device
The terminal device enhances network call realism and user experience by aligning user images and drawn images to a common reference direction, improving interaction realism and experience.
Patent Information
- Application Number
- JP2023076254
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-02
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2043-05-02
AI Technical Summary
Existing network communication technologies lack realism and user experience in phone calls, particularly in interactions involving image and sound communication.
A terminal device that includes a communication unit, input and output units, and a control unit to align images of users facing a common reference direction, displaying composite images with drawn images coinciding in orientation, enhancing the realism and user experience.
Improves the realism and user experience of phone calls by allowing users to interact as if they are facing each other, facilitating better learning and instruction scenarios.
Smart Images

Figure 2024160837000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to a terminal device. [Background technology]
[0002] There is a known technology that allows users at multiple locations to communicate with each other by sending and receiving images and sounds via a network using a computer. Patent Document 1 discloses a technology in which an instructor system and a receiver system communicate with each other, and the receiver performs a task by receiving instructions from the receiver based on a projected instructor image and output sound. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] Patent No. 4323446 Summary of the Invention [Problem to be solved by the invention]
[0004] There is room to further improve the realism of calls on the network and the user experience.
[0005] The present disclosure provides a terminal device etc. that contributes to improving the realism of calls and the user experience. [Means for solving the problem]
[0006] The terminal device of the present disclosure is a terminal device having a communication unit and a control unit that communicates via the communication unit, and further has an input unit that accepts input of an image of a user drawing while facing a reference direction and a drawing image by the user, and an output unit that displays an image, and based on information received from another terminal device for displaying the other user and displaying another drawing image by the other user, the control unit causes the output unit to display a composite image in which the user and the other user are both facing the reference direction, and displays the other drawing image near the drawing image so that their respective reference directions are aligned. Effect of the Invention
[0007] According to the terminal device etc. of the present disclosure, it is possible to improve the realism of calls and the user experience. [Brief description of the drawings]
[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a telephone system. [Diagram 2] 1 is a diagram showing an example of how a terminal device 12 is used; [Figure 3A] FIG. 11 is a flowchart illustrating an example of the operation of the terminal device. [Figure 3B] FIG. 11 is a flowchart illustrating an example of the operation of the terminal device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] The following describes the embodiments.
[0010] [Example of call system configuration] 1 is a diagram showing an example of the configuration of a call system in one embodiment. The call system 1 has a server device 10 and multiple terminal devices 12 that are connected to each other so that they can communicate information with each other via a network 11. The call system 1 is a system that supports calls, including sending and receiving images between the terminal devices 12 used by multiple users.
[0011] The server device 10 is, for example, a server computer that belongs to a cloud computing system or other computing system and functions as a server that implements various functions. The server device 10 may be composed of two or more server computers that are communicably connected and operate in cooperation with each other. The server device 10 has a communication unit 101, a storage unit 102, and a control unit 103 as components for transmitting and receiving images, audio information, etc. necessary for a call and for performing information processing. The details of each unit will be described later.
[0012] Network 11 may be, for example, the Internet, but may also include an ad-hoc network, a Local Area Network (LAN), a Metropolitan Area Network (MAN), or other network, or any combination thereof.
[0013] The terminal device 12 is a device equipped with a communication function and an information processing function, and is used by each user who makes a call. The terminal device 12 may be configured to include, for example, an information processing terminal such as a smartphone or a tablet terminal, or an information processing device such as a personal computer. In the following, when the terminal devices 12 that communicate with each other are to be distinguished from each other, they are referred to as terminal devices 12-1 and 12-2.
[0014] Terminal device 12-1 has a communication unit 121 and a control unit 123 that communicates via communication unit 121, and further has an input unit 125 that accepts input of an image of a user drawing while facing a reference direction and a drawing image by the user, and an output unit 126 that displays an image. Based on information received from other terminal device 12-2 for displaying the other user and displaying another drawing image by the other user, control unit 123 causes output unit 126 to display a composite image in which the user and the other user both face the reference direction, and causes the other drawing image to be displayed near the drawing image so that their respective reference directions match.
[0015] 2 shows a specific example of how the terminal devices 12-1 and 12-2 are used. Here, the users 20-1 and 20-2 use the terminal devices 12-1 and 12-2 to communicate with each other by sending and receiving their own drawn images, drawn images, and voices. The drawn images are characters, symbols, pictures, etc.
[0016] The user 20-1 draws the drawing image 26 in the drawing area 25 while facing the reference direction 28. The drawing area 25 is, for example, a piece of writing paper or drawing paper. The reference direction 28 is a direction from the front side of the user 20-1 to the back side. The user 20-1 and the drawing image 26 are captured by the camera 22. The camera 22 is included in the input unit 125 of the terminal device 12-1. The camera 22 is provided at an arbitrary position where the user 20-1 and the drawing image 26 can be captured. Alternatively, the drawing area 25 may be configured with a touch panel included in the input unit 125, and the drawing image 26 may be received by the input unit 125 through the touch panel. The voice of the user 20-1 is also picked up by a microphone included in the input unit 125. The captured image and voice information of the user 20-1 and the drawing image 26 are sent to the terminal device 12-2 via the network 11 and the server device 10. Similarly, the terminal device 12-2 captures an image of the user 20-2 and the drawing, and transmits the captured image and voice information to the terminal device 12-1.
[0017] In the terminal device 12-1, the display 23 displays the composite image 24. The display 23 is included in the output unit 126. The composite image 24 includes images of the user 20-1 and the user 20-2 both facing the reference direction 28. The images of the user 20-1 and the user 20-2 in the composite image 24 may be 2D models or 3D models. When the images of the user 20-1 and the user 20-2 are configured as 3D models, the display may be a light field display capable of stereoscopically expressing the 3D models. The display 23 is disposed on the reference direction 28 side, so that the user 20-1 can view the composite image 24 while drawing while facing the reference direction. Moreover, the drawing image 27 by the user 20-2 is projected by the projector 21 onto the drawing area 25. The projector 21 is included in the output unit 126. Moreover, the voice of the user 20-2 is output by a speaker included in the output unit 126. Similarly, the terminal device 12-2 displays the composite image and the drawn image 26 to the user 20-2, and outputs the voice of the user 20-1.
[0018] By looking at the composite image 24, the user 20-1 can experience a conversation with the user 20-2 beside him / her through a mirror placed in front of him / her. The user 20-1 can also see the drawing image 27 of the user 20-2 displayed near his / her own drawing image 26. The user 20-2 can also experience the same. For example, if the user 20-1 is a student of calligraphy, painting, or the like, and the user 20-2 is an instructor, the student can draw a picture while looking at the instructor's example and have the drawing corrected by the instructor. At that time, the student can experience a conversation as if the instructor was beside him / her. This makes it possible to improve the reality of the call and the user experience.
[0019] [Example of configuration of server device 10] The server device 10 includes a communication unit 101, a storage unit 102, and a control unit 103. When the server device 10 is configured with two or more server computers, these components are appropriately arranged in the two or more computers.
[0020] The communication unit 101 includes one or more communication interfaces. The communication interface is, for example, a LAN interface. The communication unit 101 receives information used in the operation of the server device 10, and transmits information obtained by the operation of the server device 10. The server device 10 is connected to a network 11 by the communication unit 101, and communicates information with a terminal device 12 via the network 11.
[0021] The storage unit 102 includes, for example, one or more semiconductor memories, one or more magnetic memories, one or more optical memories, or a combination of at least two of them, functioning as a main storage device, an auxiliary storage device, or a cache memory. The semiconductor memory is, for example, a random access memory (RAM) or a read only memory (ROM). The RAM is, for example, a static RAM (SRAM) or a dynamic RAM (DRAM). The ROM is, for example, an electrically erasable programmable ROM (EEPROM). The storage unit 102 stores information used in the operation of the server device 10 and information obtained by the operation of the server device 10.
[0022] The control unit 103 includes one or more processors, one or more dedicated circuits, or a combination of these. The processor is, for example, a general-purpose processor such as a CPU (Central Processing Unit), or a dedicated processor such as a GPU (Graphics Processing Unit) specialized for a specific process. The dedicated circuit is, for example, an FPGA (Field-Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), etc. The control unit 103 executes information processing related to the operation of the server device 10 while controlling each unit of the server device 10.
[0023] The functions of the server device 10 are realized by a processor included in the control unit 103 executing a control program. The control program is a program for causing a computer to function as the server device 10. Also, some or all of the functions of the server device 10 may be realized by a dedicated circuit included in the control unit 103. Also, the control program may be stored in a non-transitory recording / storage medium readable by the server device 10, and read by the server device 10 from the medium.
[0024] [Example of the configuration of the terminal device 12] The terminal device 12 includes a communication unit 121 , a storage unit 122 , a control unit 123 , an input unit 125 , and an output unit 126 .
[0025] The communication unit 121 has a communication module compatible with a wired or wireless LAN standard, a module compatible with a mobile communication standard such as LTE, 4G, 5G, etc. The terminal device 12 is connected to the network 11 by the communication unit 121 via a nearby router device or a mobile communication base station, and performs information communication with the server device 10, etc. via the network 11.
[0026] The memory unit 122 includes one or more semiconductor memories, one or more magnetic memories, one or more optical memories, or a combination of at least two of these. The semiconductor memories are, for example, RAM or ROM. The RAM is, for example, SRAM or DRAM. The ROM is, for example, EEPROM. The memory unit 122 functions as, for example, a main memory device, an auxiliary memory device, or a cache memory. The memory unit 122 stores information used in the operation of the control unit 123 and information obtained by the operation of the control unit 123.
[0027] The control unit 123 has, for example, one or more general-purpose processors such as a CPU, an MPU (Micro Processing Unit), or one or more dedicated processors such as a GPU specialized for a specific process. Alternatively, the control unit 123 may have one or more dedicated circuits such as an FPGA, an ASIC, or the like. The control unit 123 generally controls the operation of the terminal device 12 by operating according to a control / processing program, or operating according to an operating procedure implemented as a circuit. The control unit 123 transmits and receives various information to and from the server device 10, etc. via the communication unit 121, and executes the operation according to this embodiment.
[0028] The input unit 125 includes one or more input interfaces. The input interfaces include, for example, a physical key, a capacitive key, a pointing device, a touch screen integrally provided with a display, a microphone for receiving voice input, and a camera for capturing captured images. The camera is a camera that captures an image of a subject using visible light. The camera captures an image of a subject at, for example, 15 to 30 frames per second to generate a moving image consisting of continuous captured images. The camera may also include a distance measurement sensor that measures the distance to the subject and obtains a distance image. The distance measurement sensor includes a ToF (Time Of Flight) camera, a LiDAR (Light Detection And Ranging), and a stereo camera, and generates a distance image of the subject including distance information. The input unit 125 sends information, images, etc. that are input or obtained to the control unit 123.
[0029] The output unit 126 includes one or more output interfaces. The output interfaces include, for example, a display, a projector, and a speaker. The display is, for example, an LCD or an organic EL display. The display may include a light field display capable of expressing a 3D model. The output unit 126 has a function as a "display unit" in this embodiment. The output unit 126 outputs information obtained by the operation of the control unit 123.
[0030] The functions of the control unit 123 are realized by a processor included in the control unit 123 executing a control program. The control program is a program for causing the processor to function as the control unit 123. A part or all of the functions of the control unit 123 may be realized by a dedicated circuit included in the control unit 123. The control program may be stored in a non-transitory recording / storage medium readable by the terminal device 12, and may be read by the terminal device 12 from the medium.
[0031] [Example of operation of terminal device 12] 3A and 3B are flow charts showing an example of the operation procedure of the terminal device 12. In FIG.
[0032] FIG. 3A relates to an operation procedure of the control unit 123 in each terminal device 12 when sending a drawn image or the like of a user using that terminal device 12.
[0033] In step S301, the control unit 123 acquires a captured image, a drawn image, and audio information of the user. The control unit 123 acquires the captured image by having the camera of the input unit 125 capture the user and the drawn image at an arbitrarily set frame rate. Alternatively, the control unit 123 may acquire the drawn image from the touch panel of the input unit 125. Furthermore, the control unit 123 acquires audio information by having the microphone of the input unit 125 collect the user's voice.
[0034] In step S302, the control unit 123 derives the reference direction and the size information of the drawing area. The control unit 123 recognizes the image of the user and the image of the working area in the captured image by any image processing. The control unit 123 derives the reference direction using the position of the working area relative to the user. The position of the working area is detected as, for example, coordinate information of the vertices of a rectangle in the captured image. The control unit 123 also derives the size of the working area by any algorithm. The size of the working area is specified, for example, by the length of the sides of the rectangular working area. For example, the control unit 123 derives the distance from the camera to the working area by using a distance image, and derives the length of the sides of the working area based on the distance. Alternatively, information on the length of the sides of the touch panel that constitutes the working area may be stored in advance in the storage unit 122, and the control unit 123 may read out the information from the storage unit 122. Information indicating the reference direction is added to the drawing image. The information on the reference direction is, for example, information on a vector in spatial coordinates.
[0035] In step S303, the control unit 123 encodes the user's image, drawing image, size information, and voice information to generate encoded information. When encoding, the control unit 123 may perform any processing (e.g., resolution change, trimming, etc.) on the captured image, etc.
[0036] In step S304, the control unit 123 packetizes the encoded information using the communication unit 121, and transmits the packetized information to the server device 10 and other terminal devices 12.
[0037] When the control unit 123 acquires information corresponding to a user operation to interrupt imaging / audio collection or to end a call (Yes in S305), it terminates the processing procedure of FIG. 3A. If the control unit 123 does not acquire information corresponding to an operation to interrupt or end the call (No in S305), it executes steps S301 to S304 and sends information for displaying user and drawn images and information for outputting audio to other terminal devices 12.
[0038] Fig. 3B relates to an operation procedure of the control unit 123 when each terminal device 12 displays a drawn image or the like sent from another terminal device 12. When the control unit 123 receives a packet sent by another terminal device 12 executing the procedure of Fig. 3A via the server device 10, the control unit 123 executes the procedure of Fig. 3B.
[0039] In step S311, the control unit 123 decodes the encoded information contained in the packet received from the other terminal device 12 to obtain the image, the drawn image, the size information, and the audio information of the other user.
[0040] In step S312, the control unit 123 generates a composite image and a drawing image for display. The control unit 123 generates a composite image in which the user and the other user both face the reference direction by using an arbitrary image processing method.
[0041] For example, the control unit 123 positions the other users diagonally behind the user, and synthesizes an image of the user with an image of the other users so that the user and the other users face in front of the user or from the front side to the back. The control unit 123 may generate images of the user and the other users as 3D models. For example, the control unit 113 can generate a polygon model using distance images of the user and the other users, and generate the 3D model by applying texture mapping using the captured images of each to the polygon model.
[0042] Furthermore, the control unit 123 uses the size information to generate a drawing image for display, for example, a drawing image by another user, which has a size corresponding to the user's drawing image and faces a common reference direction. For example, the control unit 123 derives the size of the drawing image drawn by the other user based on the size of the drawing area in which the other user draws. The size of the drawing image is determined by the size of the sides of the rectangular area that the drawing image occupies in the drawing area. The control unit 123 generates a drawing image for display, using an arbitrary algorithm that takes into account the known characteristics of the projector or display, such that the drawing image has the derived size when displayed and the reference direction of the drawing image coincides with the reference direction for the user when displayed.
[0043] In step S313, the control unit 123 displays the composite image and the drawing image for display by the output unit 126 and outputs the sound. For example, the control unit 123 causes the composite image to be displayed on a display and causes the drawing image for display to be projected by a projector. Alternatively, the control unit 123 causes the drawing image for display to be displayed on a touch panel display. At the same time, the control unit 123 causes the sound of other users to be output to a speaker.
[0044] By operating each terminal device 12 according to the above procedure, each user can experience a conversation as if he / she were talking to another user beside him / her through a mirror installed in front of him / her. Also, each user can see the drawing images of other users displayed near his / her own drawing image. At that time, the drawing images of other users are displayed along a common reference direction at a size corresponding to the size of the drawing image of the user, improving the convenience of referring to the drawing images of other users. This makes it possible to improve the reality of the conversation and the user experience.
[0045] 3A and 3B are also included in the present embodiment. For example, the server device 10 may perform processes such as deriving the size of a drawing area for a drawing image sent from another terminal device 12, determining a reference direction, and generating a composite image.
[0046] In the above-described embodiment, the processing / control program that specifies the operation of the control unit 123 of the terminal device 12 may be stored in a memory unit of the server device 10 or another server device and downloaded to each terminal device 12 via the network 11, or may be stored in a recording / storage medium readable by each terminal device 12 and read from the medium by each terminal device 12.
[0047] In the above, the embodiment has been described based on the drawings and examples, but it should be noted that those skilled in the art can easily make various modifications and corrections based on the present disclosure. Therefore, it should be noted that these modifications and corrections are included in the scope of the present disclosure. For example, the functions included in each means, each step, etc. can be rearranged so as not to be logically inconsistent, and multiple means, steps, etc. can be combined into one or divided into one. [Explanation of symbols]
[0048] 1. Call System 10. Server equipment 11 Network 12 Terminal Equipment 20-1, 20-2 User 21 Projector 22 Camera 24 Composite Image 25 Drawing area 26, 27 Drawing images 28 Reference direction 101, 111 Communications Department 102, 112 Storage section 103, 113 Control section 115 Input section 116 Output section
Claims
1. A terminal device having a communication unit and a control unit that performs communication by the communication unit, The device further includes an input unit that receives an image of a user drawing while facing a reference direction and an input of a drawing image by the user, and an output unit that displays the image, the control unit, based on information received from another terminal device for displaying another user and displaying another drawing image by the other user, causes the output unit to display a composite image in which the user and the other user are both facing the reference direction, and causes the output unit to display the other drawing image near the drawing image so that the reference directions of the two drawings are aligned. Terminal device.
2. In claim 1, The control unit causes a size of the other drawn image to correspond to a size of the drawn image. Terminal device.
3. In claim 1, Either or both of the image of the user and the image of the other user are 3D models. Terminal device.
4. In claim 1, The output unit has a display that displays the composite image on a side of the user in the reference direction. Terminal device.
5. In claim 1, the output unit has a projector that projects the other drawing image near the drawing image, Terminal device.
Citation Information
Patent Citations
Communication system, information processor, and program
JP2015097070A
Work order system and its program
JP4323446B2
